Page MenuHomeSoftware Heritage
Feed Advanced Search

Oct 25 2022

anlambert added a comment to T4548: Add a public API endpoint and documentation to trigger Save Code Now from webhook.

I used ngrok to forward webhook requests to my local machine.

Oct 25 2022, 5:21 PM · Web app
anlambert closed D8767: replay: Ensure proper removal of external paths when deleting directory.
Oct 25 2022, 4:51 PM
anlambert committed rDLDSVN8c709079ce28: replay: Ensure proper removal of external paths when deleting directory (authored by anlambert).
replay: Ensure proper removal of external paths when deleting directory
Oct 25 2022, 4:51 PM
anlambert added a comment to T4548: Add a public API endpoint and documentation to trigger Save Code Now from webhook.

I started looking how to implement that task.

Oct 25 2022, 4:22 PM · Web app
anlambert created P1510 (An Untitled Masterwork).
Oct 25 2022, 3:32 PM
anlambert requested review of D8767: replay: Ensure proper removal of external paths when deleting directory.
Oct 25 2022, 3:25 PM
anlambert accepted D8766: Puppet: Artifacts as lists.

Looks good to me, thanks !

Oct 25 2022, 3:00 PM
anlambert added inline comments to D8763: nixguix: Allow lister to ignore specific extensions.
Oct 25 2022, 12:05 PM
anlambert accepted D8758: swh.core.tarball: Wire support to existing tarball.
Oct 25 2022, 11:04 AM
anlambert accepted D8761: nixguix/test: Add all supported tarball extensions to test manifest.
Oct 25 2022, 11:02 AM
anlambert accepted D8762: Puppet: Switch artifacts from dict to list.

Looks good to me, I guess you need to update the loader too.

Oct 25 2022, 11:01 AM
anlambert closed D8747: conda: Yield listed origins after all artifacts in a page are processed.
Oct 25 2022, 10:58 AM
anlambert committed rDLS4f6b3f3f09b8: conda: Yield listed origins after all artifacts in a page are processed (authored by anlambert).
conda: Yield listed origins after all artifacts in a page are processed
Oct 25 2022, 10:58 AM
anlambert accepted D8763: nixguix: Allow lister to ignore specific extensions.

LGTM, one thought posted as inline comment.

Oct 25 2022, 10:58 AM
anlambert updated the diff for D8747: conda: Yield listed origins after all artifacts in a page are processed.

Rebase

Oct 25 2022, 10:52 AM

Oct 24 2022

anlambert accepted D8755: Reset Sentry tags when leaving an object's context.
Oct 24 2022, 4:59 PM
anlambert added inline comments to D8757: Add support for more tarball/zip extensions.
Oct 24 2022, 4:47 PM
anlambert added inline comments to D8757: Add support for more tarball/zip extensions.
Oct 24 2022, 4:34 PM
anlambert accepted D8757: Add support for more tarball/zip extensions.
Oct 24 2022, 4:16 PM
anlambert updated the diff for D8747: conda: Yield listed origins after all artifacts in a page are processed.

Rebase and address @vlorentz comments

Oct 24 2022, 1:40 PM
anlambert added inline comments to D8747: conda: Yield listed origins after all artifacts in a page are processed.
Oct 24 2022, 1:36 PM
anlambert added inline comments to D8747: conda: Yield listed origins after all artifacts in a page are processed.
Oct 24 2022, 1:33 PM
anlambert added inline comments to D8379: Hackage: Loads Hackage Listed origins.
Oct 24 2022, 1:29 PM

Oct 21 2022

anlambert accepted D8752: gogs/lister: Allow public gogs instance listing.

LGTM

Oct 21 2022, 6:27 PM
anlambert added inline comments to D8752: gogs/lister: Allow public gogs instance listing.
Oct 21 2022, 6:18 PM
anlambert accepted D8751: Make the replayer not crash on kafka messages that fail to be converted as model objects.

Looks good to me , just noticed some remaining typos: one in code and another in commit message (s/sctructure/structure).

Oct 21 2022, 3:42 PM
anlambert closed D8749: utils/highlightjs: Do not report pygments exception to sentry.
Oct 21 2022, 12:29 PM
anlambert committed rDWAPPSa2526ef5bb45: utils/highlightjs: Do not report pygments exception to sentry (authored by anlambert).
utils/highlightjs: Do not report pygments exception to sentry
Oct 21 2022, 12:29 PM
anlambert added a project to T4648: Introduce RPM Loader: RPM loader.
Oct 21 2022, 11:38 AM · RPM loader
anlambert edited projects for T4448: Implementation of Fedora Lister, added: RPM lister; removed Lister.
Oct 21 2022, 11:38 AM · RPM lister, Archive coverage
anlambert created RPM loader.
Oct 21 2022, 11:38 AM
anlambert created RPM lister.
Oct 21 2022, 11:37 AM
anlambert accepted D8566: Conda: Anaconda packages archive loader.

Looks good to me, thanks !

Oct 21 2022, 11:35 AM
anlambert closed T4646: CVSProtocolError: Error from CVS server: b"E cvs checkout: Skipping `$Log$' keyword due to excessive comment leader... as Resolved by committing rDLDCVSd00badc39fa3: cvsclient: Do not abort checkout when skipping $Log$ keyword expansion.
Oct 21 2022, 10:45 AM · CVS loader
anlambert closed D8750: cvsclient: Do not abort checkout when skipping $Log$ keyword expansion.
Oct 21 2022, 10:45 AM
anlambert committed rDLDCVSd00badc39fa3: cvsclient: Do not abort checkout when skipping $Log$ keyword expansion (authored by anlambert).
cvsclient: Do not abort checkout when skipping $Log$ keyword expansion
Oct 21 2022, 10:45 AM

Oct 20 2022

anlambert updated the diff for D8747: conda: Yield listed origins after all artifacts in a page are processed.

Add missing test

Oct 20 2022, 3:40 PM
anlambert retitled D8747: conda: Yield listed origins after all artifacts in a page are processed from conda: Yield listed origins after all artifacts in a page processed to conda: Yield listed origins after all artifacts in a page are processed.
Oct 20 2022, 3:39 PM
anlambert requested changes to D8569: Add rubygems loader.

@Alphare, fyi I improved the rubygems lister in that commit in order to gather all artifacts related to a gem and send these info to the loader as extra arguments.
Below is an example of the lister output for a gem:

ListedOrigin(
    url="https://rubygems.org/gems/haar_joke",
    visit_type="rubygems",
    last_update=iso8601.parse_date("2016-11-05T00:00:00+00:00"),
    extra_loader_arguments={
        "artifacts": [
            {
                "url": "https://rubygems.org/downloads/haar_joke-0.0.2.gem",
                "length": 8704,
                "version": "0.0.2",
                "filename": "haar_joke-0.0.2.gem",
                "checksums": {
                    "sha256": "85a8cf5f41890e9605265eeebfe9e99aa0350a01a3c799f9f55a0615a31a2f5f"
                },
            },
            {
                "url": "https://rubygems.org/downloads/haar_joke-0.0.1.gem",
                "length": 8704,
                "version": "0.0.1",
                "filename": "haar_joke-0.0.1.gem",
                "checksums": {
                    "sha256": "a2ee7052fb8ffcfc4ec0fdb77fae9a36e473f859af196a36870a0f386b5ab55e"
                },
            },
        ],
        "rubygem_metadata": [
            {
                "date": "2016-11-05T00:00:00+00:00",
                "authors": "Gemma Gotch",
                "version": "0.0.2",
                "extrinsic_metadata_url": "https://rubygems.org/api/v2/rubygems/haar_joke/versions/0.0.2.json",
            },
            {
                "date": "2016-07-23T00:00:00+00:00",
                "authors": "Gemma Gotch",
                "version": "0.0.1",
                "extrinsic_metadata_url": "https://rubygems.org/api/v2/rubygems/haar_joke/versions/0.0.1.json",
            },
        ],
    },
}

It enables to improve the scheduling of loading tasks for Ruby gems (by providing last_update value to ListedOrigin)
and it will save you of couple of calls to RubyGems Web API in the loader to fetch the list of versions for a gem.
So loader implementation must be adapted to use these new arguments.

Oct 20 2022, 3:25 PM
anlambert requested review of D8750: cvsclient: Do not abort checkout when skipping $Log$ keyword expansion.
Oct 20 2022, 3:22 PM
anlambert added a revision to T4646: CVSProtocolError: Error from CVS server: b"E cvs checkout: Skipping `$Log$' keyword due to excessive comment leader...: D8750: cvsclient: Do not abort checkout when skipping $Log$ keyword expansion.
Oct 20 2022, 3:20 PM · CVS loader
anlambert triaged T4646: CVSProtocolError: Error from CVS server: b"E cvs checkout: Skipping `$Log$' keyword due to excessive comment leader... as Normal priority.
Oct 20 2022, 3:18 PM · CVS loader

Oct 19 2022

anlambert requested changes to D8379: Hackage: Loads Hackage Listed origins.

Apart the md5 decoding issue, loader looks pretty stable when testing it in docker.

Oct 19 2022, 5:51 PM
anlambert requested review of D8749: utils/highlightjs: Do not report pygments exception to sentry.
Oct 19 2022, 4:51 PM
anlambert requested review of D8747: conda: Yield listed origins after all artifacts in a page are processed.
Oct 19 2022, 4:13 PM
anlambert added inline comments to D8566: Conda: Anaconda packages archive loader.
Oct 19 2022, 4:11 PM
anlambert added inline comments to D8566: Conda: Anaconda packages archive loader.
Oct 19 2022, 3:54 PM
anlambert requested changes to D8566: Conda: Anaconda packages archive loader.

Looks good to me, just intrinsic metadata parsing code to improve for errors handling and one test file to update after recent changes in swh-loader-core, see inline comments.

Oct 19 2022, 3:51 PM
anlambert committed rDVAUd6de3a4d1e7f: git_bare: Fix typo detected by codespell (authored by anlambert).
git_bare: Fix typo detected by codespell
Oct 19 2022, 3:10 PM
anlambert committed rDSTO784f730e3aa7: Fix typos detected by codespell (authored by anlambert).
Fix typos detected by codespell
Oct 19 2022, 3:08 PM
anlambert committed rDSCRUB17cf8b668739: sql: Fix typos detected by codespell (authored by anlambert).
sql: Fix typos detected by codespell
Oct 19 2022, 3:00 PM
anlambert committed rDTSCN0f4f9d4f8eb8: cli: Fix import ordering reported by isort (authored by anlambert).
cli: Fix import ordering reported by isort
Oct 19 2022, 2:58 PM
anlambert committed rDDEP9f8535874de7: docs/api/metadata: Fix typo detected by codespell (authored by anlambert).
docs/api/metadata: Fix typo detected by codespell
Oct 19 2022, 2:57 PM
anlambert closed D8746: Exclude atom tests data from pre-commit processing and fix typo.
Oct 19 2022, 2:57 PM
anlambert committed rDDEP85015daece77: pre-commit: exclude atom tests data (authored by anlambert).
pre-commit: exclude atom tests data
Oct 19 2022, 2:57 PM
anlambert committed rDOBJS3718273e1e24: Fix typos detected by codespell and ignore inflight db column name (authored by anlambert).
Fix typos detected by codespell and ignore inflight db column name
Oct 19 2022, 2:55 PM
anlambert committed rDLDHG0efa949b4dd9: identify: Fix typo detected by codespell (authored by anlambert).
identify: Fix typo detected by codespell
Oct 19 2022, 2:51 PM
anlambert committed rDLDG7324b60ed187: docs/attic/api-backend-protocol: Fix typo detected by codespell (authored by anlambert).
docs/attic/api-backend-protocol: Fix typo detected by codespell
Oct 19 2022, 2:49 PM
anlambert committed rDLS0baaf68cff95: nixguix: Fix typo detected by codespell (authored by anlambert).
nixguix: Fix typo detected by codespell
Oct 19 2022, 2:47 PM
anlambert committed rDGQL7522a848faf2: app: Fix typo detected by codespell (authored by anlambert).
app: Fix typo detected by codespell
Oct 19 2022, 2:46 PM
anlambert requested review of D8746: Exclude atom tests data from pre-commit processing and fix typo.
Oct 19 2022, 2:44 PM
anlambert closed D8745: test_retry: Use proper way to mock sleep of retryable storage methods.
Oct 19 2022, 2:39 PM
anlambert committed rDSTO3c08d9f04f93: test_retry: Use proper way to mock sleep of retryable storage methods (authored by anlambert).
test_retry: Use proper way to mock sleep of retryable storage methods
Oct 19 2022, 2:39 PM
anlambert requested review of D8745: test_retry: Use proper way to mock sleep of retryable storage methods.
Oct 19 2022, 2:30 PM
anlambert committed rDWAPPSef821584d5a7: add_forge_now/api_views: Fix typo detected by codespell (authored by anlambert).
add_forge_now/api_views: Fix typo detected by codespell
Oct 19 2022, 1:51 PM
anlambert closed D8742: cypress: Explicitely set support file path in config.
Oct 19 2022, 1:51 PM
anlambert committed rDWAPPSb0481ba8ca23: cypress: Explicitely set support file path in config (authored by anlambert).
cypress: Explicitely set support file path in config
Oct 19 2022, 1:51 PM
anlambert accepted D8744: Fix format issues in the schema.
Oct 19 2022, 1:49 PM
anlambert closed D8743: debian/gbp-buildpackage: Use mirror of cassandra debian repository.
Oct 19 2022, 1:29 PM
anlambert committed rCJSWHb8f7539eada2: debian/gbp-buildpackage: Use mirror of cassandra debian repository (authored by anlambert).
debian/gbp-buildpackage: Use mirror of cassandra debian repository
Oct 19 2022, 1:29 PM
anlambert committed rDLDSVN790fb183f1fb: docs: Fix typo detected by codespell (authored by anlambert).
docs: Fix typo detected by codespell
Oct 19 2022, 12:07 PM
anlambert closed D8739: utils: Report svnadmin load error in init_svn_repo_from_dump.
Oct 19 2022, 12:07 PM
anlambert committed rDLDSVNfcf2addb17e9: utils: Report svnadmin load error in init_svn_repo_from_dump (authored by anlambert).
utils: Report svnadmin load error in init_svn_repo_from_dump
Oct 19 2022, 12:07 PM
anlambert requested review of D8743: debian/gbp-buildpackage: Use mirror of cassandra debian repository.
Oct 19 2022, 11:30 AM
anlambert closed T4641: Jenkins jobs for swh-graph time out while cloning repository, a subtask of T4637: Document/showcase examples gRPC clients of the swh-graph , as Resolved.
Oct 19 2022, 11:09 AM · Documentation, Compressed graph service
anlambert closed T4641: Jenkins jobs for swh-graph time out while cloning repository as Resolved.

Jenkins jobs for swh packages have been updated to only fetch the master branch of repositories, closing this.

Oct 19 2022, 11:09 AM · Compressed graph service, System administration, Continuous Integration
anlambert updated the diff for D8742: cypress: Explicitely set support file path in config.

Rebase and improve commit message

Oct 19 2022, 11:04 AM
anlambert updated the diff for D8739: utils: Report svnadmin load error in init_svn_repo_from_dump.

Rebase

Oct 19 2022, 11:01 AM
anlambert closed D8707: templates/swh-pipeline: Fetch master branch only when cloning repo.
Oct 19 2022, 10:59 AM
anlambert committed rCJSWHcabe23bcecf6: templates/swh-pipeline: Fetch master branch only when cloning repo (authored by anlambert).
templates/swh-pipeline: Fetch master branch only when cloning repo
Oct 19 2022, 10:59 AM
anlambert updated the diff for D8707: templates/swh-pipeline: Fetch master branch only when cloning repo.

Rebase

Oct 19 2022, 10:58 AM
anlambert closed D8733: jobs/swh-environment: Only fetch master branches of swh repositories.
Oct 19 2022, 10:58 AM
anlambert committed rCJSWH29744ebc975f: jobs/swh-environment: Only fetch master branches of swh repositories (authored by anlambert).
jobs/swh-environment: Only fetch master branches of swh repositories
Oct 19 2022, 10:58 AM

Oct 18 2022

anlambert requested review of D8742: cypress: Explicitely set support file path in config.
Oct 18 2022, 8:59 PM
anlambert updated the summary of D8739: utils: Report svnadmin load error in init_svn_repo_from_dump.
Oct 18 2022, 8:14 PM
anlambert requested review of D8739: utils: Report svnadmin load error in init_svn_repo_from_dump.
Oct 18 2022, 8:11 PM
anlambert requested review of D8733: jobs/swh-environment: Only fetch master branches of swh repositories.
Oct 18 2022, 7:18 PM
anlambert added a revision to T4641: Jenkins jobs for swh-graph time out while cloning repository: D8733: jobs/swh-environment: Only fetch master branches of swh repositories.
Oct 18 2022, 7:18 PM · Compressed graph service, System administration, Continuous Integration
anlambert requested review of D8707: templates/swh-pipeline: Fetch master branch only when cloning repo.
Oct 18 2022, 7:12 PM
anlambert added a revision to T4641: Jenkins jobs for swh-graph time out while cloning repository: D8707: templates/swh-pipeline: Fetch master branch only when cloning repo.
Oct 18 2022, 7:12 PM · Compressed graph service, System administration, Continuous Integration
anlambert accepted D8691: docs/grpc-api.rst: Add Python examples.

Looks good to me.

Oct 18 2022, 4:08 PM
anlambert closed D8702: tests: Simplify mercurial loading task creation tests implementation.
Oct 18 2022, 3:46 PM
anlambert committed rDLDHGfd757254dd0a: tests: Simplify mercurial loading task creation tests implementation (authored by anlambert).
tests: Simplify mercurial loading task creation tests implementation
Oct 18 2022, 3:46 PM
anlambert closed D8701: tests: Simplify bzr loading task creation test implementation.
Oct 18 2022, 3:46 PM
anlambert committed rDLDBZRba7ebcba871b: tests: Simplify bzr loading task creation test implementation (authored by anlambert).
tests: Simplify bzr loading task creation test implementation
Oct 18 2022, 3:46 PM
anlambert closed D8700: tests: Simplify cvs loading task creation test implementation.
Oct 18 2022, 3:45 PM
anlambert committed rDLDCVSde594fcab27e: tests: Simplify cvs loading task creation test implementation (authored by anlambert).
tests: Simplify cvs loading task creation test implementation
Oct 18 2022, 3:45 PM
anlambert closed D8699: tests: Simplify svn loading tasks creation tests implementation.
Oct 18 2022, 3:45 PM
anlambert committed rDLDSVN5d3433e9cee1: tests: Simplify svn loading tasks creation tests implementation (authored by anlambert).
tests: Simplify svn loading tasks creation tests implementation
Oct 18 2022, 3:45 PM
anlambert closed D8698: tests: Simplify git loading tasks creation tests implementation.
Oct 18 2022, 3:45 PM