Page MenuHomeSoftware Heritage

franckbret (Franck Bret)
User

Projects

User does not belong to any projects.

User Details

User Since
Mar 17 2022, 11:24 AM (9 w, 17 h)

Recent Activity

Wed, May 18

franckbret updated subscribers of D7812: [WIP] Add arch lister module..

I've made several experiments in order to find a better way to list arch linux package.

Wed, May 18, 11:26 AM

Wed, May 11

franckbret updated the diff for D7812: [WIP] Add arch lister module..

Updating D7812: [WIP] Add arch lister module.

Wed, May 11, 3:22 PM
franckbret requested review of D7812: [WIP] Add arch lister module..
Wed, May 11, 3:06 PM

Wed, May 4

franckbret closed D7713: crates: rework to take advantage of data returned by the crates lister.
Wed, May 4, 10:31 AM
franckbret committed rDLDBASEa097a946c2f2: crates: rework to take advantage of data returned by the crates lister (authored by franckbret).
crates: rework to take advantage of data returned by the crates lister
Wed, May 4, 10:31 AM
franckbret updated the diff for D7713: crates: rework to take advantage of data returned by the crates lister.

Updating D7713: crates: rework to take advantage of data returned by the crates lister

Wed, May 4, 10:07 AM

Fri, Apr 29

franckbret added inline comments to D7713: crates: rework to take advantage of data returned by the crates lister.
Fri, Apr 29, 3:56 PM
franckbret updated the diff for D7713: crates: rework to take advantage of data returned by the crates lister.

Update back the diff

Fri, Apr 29, 10:42 AM
franckbret updated the diff for D7713: crates: rework to take advantage of data returned by the crates lister.

Add 'Franck Bret' to contributors

Fri, Apr 29, 10:38 AM
franckbret requested review of D7713: crates: rework to take advantage of data returned by the crates lister.
Fri, Apr 29, 10:17 AM
franckbret added a revision to T4104: Ingest crates.io (Rust): D7713: crates: rework to take advantage of data returned by the crates lister.
Fri, Apr 29, 10:14 AM · Crates loader, Crates lister, Archive coverage

Thu, Apr 28

franckbret closed D7654: crates: create one origin per package instead of per version.
Thu, Apr 28, 4:11 PM
franckbret committed rDLS985b71e80c66: crates: Create one origin per package instead of per version (authored by franckbret).
crates: Create one origin per package instead of per version
Thu, Apr 28, 4:11 PM
franckbret updated the diff for D7654: crates: create one origin per package instead of per version.

prepare to push

Thu, Apr 28, 4:11 PM
franckbret closed D7501: Rust lang, Crates loader.
Thu, Apr 28, 4:09 PM
franckbret committed rDLDBASE2e27f7c7a697: Rust lang, Crates loader (authored by franckbret).
Rust lang, Crates loader
Thu, Apr 28, 4:09 PM
franckbret updated the diff for D7501: Rust lang, Crates loader.

Updating D7501: Rust lang, Crates loader

Thu, Apr 28, 4:09 PM
franckbret retitled D7654: crates: create one origin per package instead of per version from Crates.io lister, create one origin per package instead of per version to crates: create one origin per package instead of per version.
Thu, Apr 28, 9:06 AM
franckbret updated the diff for D7654: crates: create one origin per package instead of per version.

Updating D7654: crates, create one origin per package instead of per version

Thu, Apr 28, 9:06 AM

Wed, Apr 27

franckbret updated the diff for D7654: crates: create one origin per package instead of per version.

Add missing "v1" in http api url

Wed, Apr 27, 5:48 PM
franckbret updated the diff for D7654: crates: create one origin per package instead of per version.

Added 'version' key to 'artifacts' dict.

Wed, Apr 27, 5:27 PM
franckbret added a comment to D7501: Rust lang, Crates loader.

Proposed evolution on crate lister to be consistent with the loader here :

https://forge.softwareheritage.org/D7654

Nice, and now the loader needs some rework to match that diff's adaptations ;)

Wed, Apr 27, 4:10 PM
franckbret added a comment to D7654: crates: create one origin per package instead of per version.

I've run the lister in the docker environment, it looks good.

Wed, Apr 27, 3:39 PM
franckbret retitled D7654: crates: create one origin per package instead of per version from Refactor crates.io lister to Crates.io lister, create one origin per package instead of per version.
Wed, Apr 27, 2:34 PM
franckbret updated the diff for D7654: crates: create one origin per package instead of per version.

Updating D7654: Crates.io lister, create one origin per package instead of per version

Wed, Apr 27, 2:34 PM
franckbret retitled D7654: crates: create one origin per package instead of per version from Change 'extra_loarder_arguments' arg 'name' to 'package_name', and rename 'visit_type' from 'rust-crate' to 'crates' in order to be consistent with what have been decided for the crate loader, see https://forge.softwareheritage.org/D7501. to Refactor crates.io lister.
Wed, Apr 27, 12:52 PM
franckbret updated the diff for D7654: crates: create one origin per package instead of per version.

Refactor crates.io lister

Wed, Apr 27, 12:51 PM

Tue, Apr 26

franckbret added a comment to D7654: crates: create one origin per package instead of per version.

Ok, but what will be the usage of versioned artifacts as extra_loader_arguments?

We don't want to create one origin per version of a package.
We want all versions of a packages seen under the same origin (well, at the time of the ingestion).

See this thread where we made the mistake for the maven ingestion [1]

[1] https://sympa.inria.fr/sympa/arc/swh-devel/2022-04/msg00043.html

Is that possible at all for the rust ingestion?

Cheers,

Tue, Apr 26, 6:23 PM
franckbret added a comment to D7654: crates: create one origin per package instead of per version.

wait... we are creating one origin for each version of the crate?

Yes it is.

I guess Instead of returning an url for a version (i.e: https://static.crates.io/crates/{package_name}/{package_name}-{version}.crate ) lister should return an http api url (i.e : https://crates.io/api/v1/crates/{package_name} ) ?

Yes, this one.

And i'd say, return the list of associated versioned artifacts as extra_loader_arguments.

@vlorentz thoughts?

Tue, Apr 26, 5:56 PM
franckbret added a comment to D7654: crates: create one origin per package instead of per version.

Once @vlorentz's comment is adressed (we should list one origin, the package i guess, and
gather all crate versions under that same origin).

Make sure your commit message (one to one with the diff description) is one short
summary (in imperative form) [1]. For some extra description, use multiple lines.

[1] https://docs.softwareheritage.org/devel/contributing/git-style-guide.html

TIA

Tue, Apr 26, 4:43 PM
franckbret added a comment to D7654: crates: create one origin per package instead of per version.

wait... we are creating one origin for each version of the crate?

Tue, Apr 26, 4:41 PM
franckbret requested review of D7654: crates: create one origin per package instead of per version.
Tue, Apr 26, 10:57 AM
franckbret added a comment to D7501: Rust lang, Crates loader.

Proposed evolution on crate lister to be consistent with the loader here :

Tue, Apr 26, 10:54 AM
franckbret added a comment to D7501: Rust lang, Crates loader.

yes, rename to package_name, the other change would make for a much wider impact (than you probably want to deal with ;).

It would not. The get_loader function is not used anywhere outside of this cli.py

right, nvm then.

I was thinking of something else (iirc, there is something similar in the lister or something)

i'd prefer that renaming nonetheless though, it's more explicit.

Tue, Apr 26, 10:33 AM
franckbret added inline comments to D7501: Rust lang, Crates loader.
Tue, Apr 26, 10:32 AM
franckbret updated the diff for D7501: Rust lang, Crates loader.

Refactor according to @vlorentz & @ardumont comments.

Tue, Apr 26, 10:31 AM

Mon, Apr 25

franckbret updated the diff for D7501: Rust lang, Crates loader.

fix missing whitespace in loader specification documentation

Mon, Apr 25, 4:26 PM
franckbret updated the diff for D7501: Rust lang, Crates loader.

Add crates package loader specification

Mon, Apr 25, 4:04 PM

Sat, Apr 23

franckbret added a comment to D7501: Rust lang, Crates loader.

Not sure you have seen this comment before, and because its an important point to go further, I repost :

Sat, Apr 23, 9:20 AM
franckbret updated the diff for D7501: Rust lang, Crates loader.

Trying another alternative to make mypy happy (don't get why got no error in dev env running tox -e mypy or mypy -c mypy.ini but fail on CI..)

Sat, Apr 23, 9:09 AM
franckbret updated the diff for D7501: Rust lang, Crates loader.

mypy multiline ignore fix

Sat, Apr 23, 8:24 AM

Fri, Apr 22

franckbret updated the diff for D7501: Rust lang, Crates loader.

Rebase upstream

Fri, Apr 22, 5:08 PM
franckbret updated the diff for D7501: Rust lang, Crates loader.

Fixes after ardumont and vlorentz review

Fri, Apr 22, 3:35 PM
franckbret added a comment to D7501: Rust lang, Crates loader.

Some fixes after @Alphare review. Mainly add 'name' and 'version' to loader args as those are given through the 'extra_loader_arguments' from lister.

One important thing to note after adding 'name' and 'version' args is that it does not work with the CLI in the docker environment..

franck@debian-franck:~/workspace/swh-environment/docker$ docker-compose exec swh-loader swh loader run crates "https://static.crates.io/crates/micro-timer/micro-timer-0.4.0.crate" version="0.4.0" name="micro-timer"

Traceback (most recent call last):
...
  File "/src/swh-loader-core/swh/loader/cli.py", line 104, in run
    loader = get_loader(type, url=url, storage=conf["storage"], **kw)
TypeError: get_loader() got multiple values for argument 'name'

What do you think about that ?
Better renaming 'name' to 'package_name' in the crate lister and loader or should the cli arg evolve to something like 'loader_name'?

Fri, Apr 22, 3:35 PM

Apr 14 2022

franckbret added a comment to D7501: Rust lang, Crates loader.

Some fixes after @Alphare review. Mainly add 'name' and 'version' to loader args as those are given through the 'extra_loader_arguments' from lister.

Apr 14 2022, 4:40 PM
franckbret retitled D7501: Rust lang, Crates loader from WIP: Rust lang, Crates loader to Rust lang, Crates loader.
Apr 14 2022, 4:32 PM
franckbret added inline comments to D7501: Rust lang, Crates loader.
Apr 14 2022, 4:32 PM
franckbret updated the diff for D7501: Rust lang, Crates loader.

Some fixes after @Alphare review. Mainly add 'name' and 'version' to loader args as those are given through the 'extra_loader_arguments' from lister.

Apr 14 2022, 4:29 PM

Apr 11 2022

franckbret published D7501: Rust lang, Crates loader for review.

Here is a first crates.io loader.

Apr 11 2022, 5:30 PM

Mar 28 2022

franckbret closed D7367: Start rust crates lister.
Mar 28 2022, 8:51 AM
franckbret committed rDLSfea6fc04aab5: lister: Add new rust crates lister (authored by franckbret).
lister: Add new rust crates lister
Mar 28 2022, 8:51 AM
franckbret updated the diff for D7367: Start rust crates lister.

Rebase

Mar 28 2022, 8:45 AM

Mar 24 2022

franckbret retitled D7367: Start rust crates lister from WIP: Start rust crates lister to Start rust crates lister.
Mar 24 2022, 9:14 AM
franckbret updated the diff for D7367: Start rust crates lister.

lister: Add new rust crates lister

Mar 24 2022, 8:36 AM
franckbret planned changes to D7367: Start rust crates lister.

Hi, thanks you all for the review.

Mar 24 2022, 8:06 AM

Mar 22 2022

franckbret updated the diff for D7367: Start rust crates lister.

Fixes after first code review

Mar 22 2022, 4:15 PM

Mar 21 2022

franckbret updated the diff for D7367: Start rust crates lister.

Docstring

Mar 21 2022, 12:31 PM

Mar 18 2022

franckbret planned changes to D7367: Start rust crates lister.

Pulling this as draft

Mar 18 2022, 10:46 AM
franckbret requested review of D7367: Start rust crates lister.
Mar 18 2022, 10:45 AM
franckbret abandoned D7376: Add tasks for crates lister.

Cancelling to move those changes in D7367

Mar 18 2022, 10:26 AM

Mar 17 2022

franckbret requested review of D7376: Add tasks for crates lister.
Mar 17 2022, 7:42 PM