Page MenuHomeSoftware Heritage
Differential D5293

Add a non-incremental sourceforge lister
ClosedPublic
Actions

Authored by Alphare on Mar 19 2021, 6:08 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDLSf7b27c693022: Add a non-incremental sourceforge lister
Summary

Following @zack's work on T735, this change introduces an actual SWH lister for
SourceForge.

SourceForge provides a main sitemap that lists sharded sitemaps, which
themselves list pages. Each page belongs to a project (or sub-project,
though those are rare), information about which can be found by querying
a REST API, which gives us the list of any and all VCS used for said
project. Both sitemaps and pages have a "last modified" timestamp that
will be used in a future patch to implement incremental listing.

More precise information can be found as inline comments or docstrings.

Diff Detail

Repository
rDLS Listers
Branch
sourceforge-lister
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 20025
Build 31086: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 31085: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
35 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.sourceforge.tests.test_lister::test_sourceforge_lister_full
swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7f68361ca5c0> requests_mock = <requests_mock.mocker.Mocker object at 0x7f68361ca198> mocker = <pytest_mock.plugin.MockerFixture object at 0x7f6836c84390>
52 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.bitbucket.tests.test_lister::test_bitbucket_full_lister
1,171 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.bitbucket.tests.test_lister::test_bitbucket_incremental_lister
43 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.bitbucket.tests.test_lister::test_bitbucket_lister_rate_limit_hit
1,003 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.bitbucket.tests.test_tasks::test_full_listing
View Full Test Results (1 Failed · 155 Passed)

Event Timeline

Alphare created this revision.Mar 19 2021, 6:08 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptMar 19 2021, 6:08 PM
swh-public-ci added a comment.Mar 19 2021, 6:10 PM

Build has FAILED

Patch application report for D5293 (id=18945)

Rebasing onto df73073a67...

First, rewinding head to replay your work on top of it...
Applying: Add a non-incremental sourceforge lister
Changes applied before test
commit 5c77c477538af6e1a7b67c620a578618d08a5774
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Wed Mar 17 17:39:41 2021 +0100

    Add a non-incremental sourceforge lister
    
    Following zack's work on T735, this change introduces an actual SWH lister for
    SourceForge.
    
    SourceForge provides a main sitemap that lists sharded sitemaps, which
    themselves list pages. Each page belongs to a project (or sub-project,
    though those are rare), information about which can be found by querying
    a REST API, which gives us the list of any and all VCS used for said
    project. Both sitemaps and pages have a "last modified" timestamp that
    will be used in a future patch to implement incremental listing.
    
    More precise information can be found as inline comments or docstrings.

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/258/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/258/console

Harbormaster returned this revision to the author for changes because remote builds failed.Mar 19 2021, 6:10 PM
Harbormaster failed remote builds in B20025: Diff 18945!
zack edited the summary of this revision. (Show Details)Mar 19 2021, 7:16 PM
zack added a subscriber: zack.
Alphare updated this revision to Diff 18953.Mar 19 2021, 10:49 PM

Add python 3.7 compat

swh-public-ci added a comment.Mar 19 2021, 10:51 PM

Build is green

Patch application report for D5293 (id=18953)

Rebasing onto df73073a67...

First, rewinding head to replay your work on top of it...
Applying: Add a non-incremental sourceforge lister
Changes applied before test
commit a085f108debd45ac7c7dc8b2777869bfe3aca822
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Wed Mar 17 17:39:41 2021 +0100

    Add a non-incremental sourceforge lister
    
    Following zack's work on T735, this change introduces an actual SWH lister for
    SourceForge.
    
    SourceForge provides a main sitemap that lists sharded sitemaps, which
    themselves list pages. Each page belongs to a project (or sub-project,
    though those are rare), information about which can be found by querying
    a REST API, which gives us the list of any and all VCS used for said
    project. Both sitemaps and pages have a "last modified" timestamp that
    will be used in a future patch to implement incremental listing.
    
    More precise information can be found as inline comments or docstrings.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/259/ for more details.

Harbormaster completed remote builds in B20033: Diff 18953.Mar 19 2021, 10:51 PM
Alphare requested review of this revision.Mar 19 2021, 10:51 PM
vlorentz added a subscriber: vlorentz.Mar 20 2021, 9:41 AM

Very nice! My only suggestion would be to add some tests for error conditions (eg. to make sure it retries).

swh/lister/sourceforge/lister.py
149

-> None

Alphare updated this revision to Diff 18994.Mar 23 2021, 10:16 AM

Update typing and add retry tests + check other fatal errors

swh-public-ci added a comment.Mar 23 2021, 10:18 AM

Build is green

Patch application report for D5293 (id=18994)

Rebasing onto 879170a57d...

First, rewinding head to replay your work on top of it...
Applying: Add a non-incremental sourceforge lister
Changes applied before test
commit b6bc9bd0d0b00ec731773617871ff6d3c1a53a1d
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Wed Mar 17 17:39:41 2021 +0100

    Add a non-incremental sourceforge lister
    
    Following zack's work on T735, this change introduces an actual SWH lister for
    SourceForge.
    
    SourceForge provides a main sitemap that lists sharded sitemaps, which
    themselves list pages. Each page belongs to a project (or sub-project,
    though those are rare), information about which can be found by querying
    a REST API, which gives us the list of any and all VCS used for said
    project. Both sitemaps and pages have a "last modified" timestamp that
    will be used in a future patch to implement incremental listing.
    
    More precise information can be found as inline comments or docstrings.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/260/ for more details.

Harbormaster completed remote builds in B20071: Diff 18994.Mar 23 2021, 10:18 AM
Alphare updated this revision to Diff 19015.Mar 23 2021, 12:03 PM
Alphare marked an inline comment as not done.

Record lastmod for origins

swh-public-ci added a comment.Mar 23 2021, 12:05 PM

Build is green

Patch application report for D5293 (id=19015)

Rebasing onto 879170a57d...

First, rewinding head to replay your work on top of it...
Applying: Add a non-incremental sourceforge lister
Changes applied before test
commit 247a8a25fd7e44b37766e2d14094227ecf56d32b
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Wed Mar 17 17:39:41 2021 +0100

    Add a non-incremental sourceforge lister
    
    Following zack's work on T735, this change introduces an actual SWH lister for
    SourceForge.
    
    SourceForge provides a main sitemap that lists sharded sitemaps, which
    themselves list pages. Each page belongs to a project (or sub-project,
    though those are rare), information about which can be found by querying
    a REST API, which gives us the list of any and all VCS used for said
    project. Both sitemaps and pages have a "last modified" timestamp that
    will be used in a future patch to implement incremental listing.
    
    More precise information can be found as inline comments or docstrings.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/261/ for more details.

Harbormaster completed remote builds in B20091: Diff 19015.Mar 23 2021, 12:05 PM
Alphare updated this revision to Diff 19016.Mar 23 2021, 12:08 PM

Simplify date tests (sorry for the update spam)

swh-public-ci added a comment.Mar 23 2021, 12:10 PM

Build is green

Patch application report for D5293 (id=19016)

Rebasing onto 879170a57d...

First, rewinding head to replay your work on top of it...
Applying: Add a non-incremental sourceforge lister
Changes applied before test
commit 5903a523aaa8108a007261d6923004cb6d400934
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Wed Mar 17 17:39:41 2021 +0100

    Add a non-incremental sourceforge lister
    
    Following zack's work on T735, this change introduces an actual SWH lister for
    SourceForge.
    
    SourceForge provides a main sitemap that lists sharded sitemaps, which
    themselves list pages. Each page belongs to a project (or sub-project,
    though those are rare), information about which can be found by querying
    a REST API, which gives us the list of any and all VCS used for said
    project. Both sitemaps and pages have a "last modified" timestamp that
    will be used in a future patch to implement incremental listing.
    
    More precise information can be found as inline comments or docstrings.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/262/ for more details.

Harbormaster completed remote builds in B20092: Diff 19016.Mar 23 2021, 12:10 PM
vlorentz accepted this revision.Mar 23 2021, 12:28 PM
This revision is now accepted and ready to land.Mar 23 2021, 12:28 PM
Alphare updated this revision to Diff 19067.Mar 23 2021, 6:41 PM

Rebase

This revision was landed with ongoing or failed builds.Mar 23 2021, 6:41 PM
Closed by commit rDLSf7b27c693022: Add a non-incremental sourceforge lister (authored by Alphare). · Explain Why
This revision was automatically updated to reflect the committed changes.
Alphare added a commit: rDLSf7b27c693022: Add a non-incremental sourceforge lister.
swh-public-ci added a comment.Mar 23 2021, 6:43 PM

Build is green

Patch application report for D5293 (id=19067)

Rebasing onto 879170a57d...

Current branch diff-target is up to date.
Changes applied before test
commit f7b27c6930220b4ed3e1141167d61e67416852c9
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Wed Mar 17 17:39:41 2021 +0100

    Add a non-incremental sourceforge lister
    
    Following zack's work on T735, this change introduces an actual SWH lister for
    SourceForge.
    
    SourceForge provides a main sitemap that lists sharded sitemaps, which
    themselves list pages. Each page belongs to a project (or sub-project,
    though those are rare), information about which can be found by querying
    a REST API, which gives us the list of any and all VCS used for said
    project. Both sitemaps and pages have a "last modified" timestamp that
    will be used in a future patch to implement incremental listing.
    
    More precise information can be found as inline comments or docstrings.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/263/ for more details.

Harbormaster completed remote builds in B20159: Diff 19067.Mar 23 2021, 6:43 PM

Revision Contents
Changeset List

PathSize
1 line
swh/
lister/
sourceforge/
12 lines
214 lines
20 lines
tests/
data/
1 line
1 line
1 line
10 lines
1 line
1 line
1 line
58 lines
38 lines
120 lines
34 lines

Diff 18945

View Options

setup.py

Loading...
View Options

swh/lister/sourceforge/__init__.py

Loading...
View Options

swh/lister/sourceforge/lister.py

Loading...
View Options

swh/lister/sourceforge/tasks.py

Loading...
View Options

swh/lister/sourceforge/tests/__init__.py

Loading...
View Options

swh/lister/sourceforge/tests/data/adobexmp.json

Loading...
View Options

swh/lister/sourceforge/tests/data/backapps-website.json

Loading...
View Options

swh/lister/sourceforge/tests/data/backapps.json

Loading...
View Options

swh/lister/sourceforge/tests/data/main-sitemap.xml

Loading...
View Options

swh/lister/sourceforge/tests/data/mojunk.json

Loading...
View Options

swh/lister/sourceforge/tests/data/mramm.json

Loading...
View Options

swh/lister/sourceforge/tests/data/os3dmodels.json

Loading...
View Options

swh/lister/sourceforge/tests/data/subsitemap-0.xml

Loading...
View Options

swh/lister/sourceforge/tests/data/subsitemap-1.xml

Loading...
View Options

swh/lister/sourceforge/tests/test_lister.py

Loading...
View Options

swh/lister/sourceforge/tests/test_tasks.py

Loading...
About · Developer information · Legal