Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/130/ for more details.

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/132/ for more details.

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/133/ for more details.

@douardda Thanks for helping me out to improve commit messages.
Although I was wondering before landing the diff we usually squash all the commits to one single one, so what is the need to follow strict guidelines for commit messages in the process of improving the diff. I mean at the end they all are going to be squashed to one single commit. :)

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/137/ for more details.

In D1492#33485, @nahimilega wrote:

@douardda Thanks for helping me out to improve commit messages.
Although I was wondering before landing the diff we usually squash all the commits to one single one, so what is the need to follow strict guidelines for commit messages in the process of improving the diff. I mean at the end they all are going to be squashed to one single commit. :)

No, commits MUST remains as you wrote them. Do NOT use arcanist to push your commits, use 'git push' directly, as described in https://wiki.softwareheritage.org/wiki/Code_review_in_Phabricator#Landing_your_change_onto_master

Although I was wondering before landing the diff we usually squash all the commits to one single one,

You can but you are not forced to.
You could keep logical modifications that belong together into multiple commits.
I'd say the only requisite is that those commits have their unit tests green.

so what is the need to follow strict guidelines for commit messages in the process of improving the diff.

It's not necessarily only to improve the diff.

That can also help you clarify what you wanted to do in the first place.

I think it's always an helpful exercise and everyone can benefit from it.

Cheers,

Thanks @ardumont @douardda, I will remember your advice.

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/139/ for more details.

You got your wish by inlining the R call inside of Python.

That is still if somewhat superfluous gain as it will make it harder to add any R processing, should you need or want it, later.
For starters, it is somewhat suboptimal to use the N x 65 returns (where N == number of CRAN packages, currently at 14000 and still growing by several thousand each year) when you only need package and version. So maybe

R> obj1 <- jsonlite::toJSON( tools::CRAN_package_db() )
R> obj2 <- jsonlite::toJSON( tools::CRAN_package_db()[, c("Package", "Version") ] )
R> dang::lsos()  # little helper to post-process object.size()                                                                                                                                                     
     Type     Size Rows Columns
obj1 json 17174872    1      NA
obj2 json   579312    1      NA
R>

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/140/ for more details.

@eddelbuettel As you recommended, I did some preprocessing only to get the useful data.
Also, I changed from a separate file to a single line R script because it is difficult to determine the location of the R script to execute it when the module will be shipped as a package or run in production.

Also, I changed from a separate file to a single line R script because it is difficult to determine the location of the R script to execute it when the module will be shipped as a package or run in production.

Let's just agree to disgree here. Some of us know how to do this with packages, and have done for years if not decades. But my role here is not to impose or suggest policy (but rather to help with R details as needed) so I will leave this to others as they see fit.

Let's just agree to disgree here. Some of us know how to do this with packages, and have done for years if not decades.

@eddelbuettel I didn't had an opportunity to work on a project with two languages and combine them, it is my first time. I would love to learn the way it is done and made ready for production.

In D1492#33863, @nahimilega wrote:

Let's just agree to disgree here. Some of us know how to do this with packages, and have done for years if not decades.

@eddelbuettel I didn't had an opportunity to work on a project with two languages and combine them, it is my first time. I would love to learn the way it is done and made ready for production.

Note that I did not mean you have to embed the R code as a string in your python code when I made my comment on this part of your code. There are ways to handle such a non-python file properly in a python package. Please read https://packaging.python.org/ and https://setuptools.readthedocs.io/en/latest/setuptools.html (and especially https://setuptools.readthedocs.io/en/latest/setuptools.html#including-data-files ).

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/147/ for more details.

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/156/ for more details.

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/163/ for more details.

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/175/ for more details.

I ran the lister in docker to test it, it ran without any problem
(the content of tcran.py is same 4 lines that are present in README of lister

(swh) archit@work-pc:~/swh-environment/swh-lister$ python tcran.py 
DEBUG:swh.lister.core.lister_base:Loading config from lister_cran
INFO:swh.core.config:Loading config file /home/archit/.config/swh/lister_cran.yml
DEBUG:swh.lister.core.lister_base:<swh.lister.cran.lister.CRANLister object at 0x7f03ea55c518> CONFIG={'content_size_limit': 104857600, 'log_db': 'dbname=softwareheritage-log', 'storage': {'cls': 'remote', 'args': {'url': 'http://localhost:5002/'}}, 'scheduler': {'cls': 'remote', 'args': {'url': 'http://localhost:5008/'}}, 'lister': {'cls': 'local', 'args': {'db': 'postgresql:///lister-cran'}}, 'credentials': [], 'cache_responses': True, 'cache_dir': '/home/archit/.cache/swh/lister/cran/'}
DEBUG:root:models: 10000
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:5002
DEBUG:urllib3.connectionpool:http://localhost:5002 "POST /origin/add_multi HTTP/1.1" 200 1
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:5008
DEBUG:urllib3.connectionpool:http://localhost:5008 "POST /create_tasks HTTP/1.1" 200 1
DEBUG:root:models: 4372
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:5002 "POST /origin/add_multi HTTP/1.1" 200 1
DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
DEBUG:urllib3.connectionpool:http://localhost:5008 "POST /create_tasks HTTP/1.1" 200 1

Here is one of the task that was created by the lister
https://forge.softwareheritage.org/P424

@douardda Can you please take a look at it and give me confirmation to merge it.

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/178/ for more details.