Projects Origin-Pypi

Origin-PypiTag
ActivePublic
Watch Project

Members

This project does not have any members.
View All

Watchers

This project does not have any watchers.
View All

Recent Activity
View All

Jan 8 2023

gitlab-migration closed T1246: pypi loader: Analyze existing errors as Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 10:21 PM · Archive coverage, Origin-Pypi

gitlab-migration changed the status of T2098: Deploy package loaders from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 10:00 PM · Origin-Debian, Origin-CRAN, Origin-GNU, Origin-npm, Origin-Pypi, Archive coverage

gitlab-migration changed the status of T2023: Re-implement gnu loader with package loader, a subtask of T1389: Implement a base "package" loader for package managers, from Resolved to Migrated.

Jan 8 2023, 9:59 PM · Origin-Debian, Origin-CRAN, Origin-GNU, Origin-npm, Origin-Pypi, Archive coverage

gitlab-migration changed the status of T2022: Re-implement npm loader with base loader, a subtask of T1389: Implement a base "package" loader for package managers, from Resolved to Migrated.

Jan 8 2023, 9:59 PM · Origin-Debian, Origin-CRAN, Origin-GNU, Origin-npm, Origin-Pypi, Archive coverage

gitlab-migration changed the status of T2021: Re-implement pypi loader with package loader from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 9:59 PM · Origin-Pypi, Archive coverage

gitlab-migration changed the status of T2021: Re-implement pypi loader with package loader, a subtask of T1389: Implement a base "package" loader for package managers, from Resolved to Migrated.

Jan 8 2023, 9:59 PM · Origin-Debian, Origin-CRAN, Origin-GNU, Origin-npm, Origin-Pypi, Archive coverage

gitlab-migration changed the status of T1389: Implement a base "package" loader for package managers from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 9:58 PM · Origin-Debian, Origin-CRAN, Origin-GNU, Origin-npm, Origin-Pypi, Archive coverage

gitlab-migration changed the status of T1181: pypi: Schedule ingestion from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 9:58 PM · Archive coverage, Origin-Pypi

gitlab-migration changed the status of T1181: pypi: Schedule ingestion, a subtask of T419: ingest PyPI into the Software Heritage archive (meta task), from Resolved to Migrated.

Jan 8 2023, 9:58 PM · Archive coverage, Origin-Pypi

gitlab-migration changed the status of T420: mirror PyPI, a subtask of T419: ingest PyPI into the Software Heritage archive (meta task), from Wontfix to Migrated.

Jan 8 2023, 9:56 PM · Archive coverage, Origin-Pypi

gitlab-migration changed the status of T420: mirror PyPI from Wontfix to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 9:56 PM · Origin-Pypi

gitlab-migration changed the status of T422: PyPI lister, a subtask of T419: ingest PyPI into the Software Heritage archive (meta task), from Resolved to Migrated.

Jan 8 2023, 9:56 PM · Archive coverage, Origin-Pypi

gitlab-migration changed the status of T422: PyPI lister from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 9:56 PM · Developers, Origin-Pypi

gitlab-migration changed the status of T419: ingest PyPI into the Software Heritage archive (meta task) from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 9:56 PM · Archive coverage, Origin-Pypi

gitlab-migration changed the status of T3399: Improve PyPI lister to pull last update information when running incrementally from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 4:34 PM · Origin-Pypi, Lister

gitlab-migration changed the status of T2025: Re-implement debian loader with package loader mechanism, a subtask of T1389: Implement a base "package" loader for package managers, from Resolved to Migrated.

Jan 8 2023, 4:28 PM · Origin-Debian, Origin-CRAN, Origin-GNU, Origin-npm, Origin-Pypi, Archive coverage

gitlab-migration changed the status of T2024: Re-implement deposit loader with package loader mechanism, a subtask of T1389: Implement a base "package" loader for package managers, from Resolved to Migrated.

Jan 8 2023, 4:28 PM · Origin-Debian, Origin-CRAN, Origin-GNU, Origin-npm, Origin-Pypi, Archive coverage

gitlab-migration changed the status of T2017: package loader: Discuss revision metadata normalization, a subtask of T1389: Implement a base "package" loader for package managers, from Resolved to Migrated.

Jan 8 2023, 4:28 PM · Origin-Debian, Origin-CRAN, Origin-GNU, Origin-npm, Origin-Pypi, Archive coverage

gitlab-migration changed the status of T1890: pypi lister: Add tests from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 4:27 PM · Origin-Pypi, Lister

gitlab-migration changed the status of T421: PyPI loader from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 4:19 PM · PyPI loader, Origin-Pypi

gitlab-migration changed the status of T421: PyPI loader, a subtask of T419: ingest PyPI into the Software Heritage archive (meta task), from Resolved to Migrated.

Jan 8 2023, 4:19 PM · Archive coverage, Origin-Pypi

Jul 9 2021

ardumont closed T3399: Improve PyPI lister to pull last update information when running incrementally as Resolved.

Deployed and running so closing.

Jul 9 2021, 2:52 PM · Origin-Pypi, Lister

ardumont added a comment to T3399: Improve PyPI lister to pull last update information when running incrementally.

Deployed in production as well and triggered a run:

Jul 9 2021, 2:51 PM · Origin-Pypi, Lister

ardumont added a comment to T3399: Improve PyPI lister to pull last update information when running incrementally.

From a quick test, it looks like the "Project and release activity details" feed can go back multiple years without any issue, allowing us to backfill the data for all known origins, before adding the incremental behavior to the lister.

Jul 9 2021, 2:09 PM · Origin-Pypi, Lister

ardumont added a comment to T3399: Improve PyPI lister to pull last update information when running incrementally.

Deployed in staging and triggered a run:

Jul 9 2021, 2:03 PM · Origin-Pypi, Lister

ardumont closed D5977: Make PyPI lister incremental and complete in regards to last_update.

Jul 9 2021, 12:59 PM · Scheduling utilities, Origin-Pypi

swh-public-ci added a comment to D5977: Make PyPI lister incremental and complete in regards to last_update.

Build is green

Jul 9 2021, 12:56 PM · Scheduling utilities, Origin-Pypi

ardumont updated the diff for D5977: Make PyPI lister incremental and complete in regards to last_update.

Adapt according to last suggestion about stub
and from the irc discussion, adding a sleep call to avoid the first throttling due to the initialization of the xmlrpc client [1]

Jul 9 2021, 12:53 PM · Scheduling utilities, Origin-Pypi

swh-public-ci added a comment to D5977: Make PyPI lister incremental and complete in regards to last_update.

Build is green

Jul 9 2021, 12:21 PM · Scheduling utilities, Origin-Pypi

ardumont added inline comments to D5977: Make PyPI lister incremental and complete in regards to last_update.

Jul 9 2021, 12:19 PM · Scheduling utilities, Origin-Pypi

ardumont added inline comments to D5977: Make PyPI lister incremental and complete in regards to last_update.

Jul 9 2021, 12:18 PM · Scheduling utilities, Origin-Pypi

ardumont updated the diff for D5977: Make PyPI lister incremental and complete in regards to last_update.

Attend to the major part of the review (thx)

Jul 9 2021, 12:18 PM · Scheduling utilities, Origin-Pypi

olasd added a comment to D5977: Make PyPI lister incremental and complete in regards to last_update.

In D5977#154053, @ardumont wrote:

Jul 9 2021, 12:11 PM · Scheduling utilities, Origin-Pypi

ardumont added a comment to D5977: Make PyPI lister incremental and complete in regards to last_update.

Looks good, thanks a lot!

Jul 9 2021, 12:06 PM · Scheduling utilities, Origin-Pypi

olasd accepted D5977: Make PyPI lister incremental and complete in regards to last_update.

Looks good, thanks a lot!

Jul 9 2021, 11:57 AM · Scheduling utilities, Origin-Pypi

ardumont updated the test plan for D5977: Make PyPI lister incremental and complete in regards to last_update.

Jul 9 2021, 11:47 AM · Scheduling utilities, Origin-Pypi

ardumont updated the test plan for D5977: Make PyPI lister incremental and complete in regards to last_update.

Jul 9 2021, 11:45 AM · Scheduling utilities, Origin-Pypi

ardumont updated the test plan for D5977: Make PyPI lister incremental and complete in regards to last_update.

Jul 9 2021, 11:41 AM · Scheduling utilities, Origin-Pypi

ardumont updated the test plan for D5977: Make PyPI lister incremental and complete in regards to last_update.

Jul 9 2021, 11:40 AM · Scheduling utilities, Origin-Pypi

ardumont updated the test plan for D5977: Make PyPI lister incremental and complete in regards to last_update.

Jul 9 2021, 11:15 AM · Scheduling utilities, Origin-Pypi

swh-public-ci added a comment to D5977: Make PyPI lister incremental and complete in regards to last_update.

Build is green

Jul 9 2021, 11:15 AM · Scheduling utilities, Origin-Pypi

ardumont updated the diff for D5977: Make PyPI lister incremental and complete in regards to last_update.

Handle throttling appropriately (adapt throttling decorators [1])
Rework logic to still deal with pages of results so the write in dbs occur incrementally

Jul 9 2021, 11:12 AM · Scheduling utilities, Origin-Pypi

ardumont updated the test plan for D5977: Make PyPI lister incremental and complete in regards to last_update.

Jul 9 2021, 11:10 AM · Scheduling utilities, Origin-Pypi

ardumont planned changes to D5977: Make PyPI lister incremental and complete in regards to last_update.

Running through docker, i actually need to change a few things:

throttling needs to change (D5983)
actual logic of sending all origins in one go won't be ok for the first run (no flush prior to actually finish the run...) so i'll need to rework this (in-progress) [1]

Jul 9 2021, 10:59 AM · Scheduling utilities, Origin-Pypi

ardumont added a revision to T3399: Improve PyPI lister to pull last update information when running incrementally: D5983: pypi.lister: Handle xml-rpc throttling properly.

Jul 9 2021, 10:50 AM · Origin-Pypi, Lister

swh-public-ci added a comment to D5977: Make PyPI lister incremental and complete in regards to last_update.

Build is green

Jul 9 2021, 10:18 AM · Scheduling utilities, Origin-Pypi

ardumont updated the diff for D5977: Make PyPI lister incremental and complete in regards to last_update.

Rebase

Jul 9 2021, 10:15 AM · Scheduling utilities, Origin-Pypi

swh-public-ci added a comment to D5977: Make PyPI lister incremental and complete in regards to last_update.

Build is green

Jul 9 2021, 10:11 AM · Scheduling utilities, Origin-Pypi

ardumont updated the diff for D5977: Make PyPI lister incremental and complete in regards to last_update.

Rebase
Refactor

Jul 9 2021, 10:08 AM · Scheduling utilities, Origin-Pypi

swh-public-ci added a comment to D5977: Make PyPI lister incremental and complete in regards to last_update.

Build is green

Jul 9 2021, 10:03 AM · Scheduling utilities, Origin-Pypi