Details

Reviewers

anlambert

Group Reviewers

Reviewers

Maniphest Tasks

T1524: save code now: also add new origins for unknown repos

Commits

rDWAPPS01ffb31ff72a: Schedule save code now as recurring origins to ingest when successful

Summary

To allow users to request only once their save code now origins, once the first
ingestion is successfully ingested, we also mark it as recurrent origin to crawl.

Implementation wise, the scheduling routine in charge of updating the save code now
statuses reported in the save code now ui is in charge of this.

Related to T1524

Test Plan

tox fails for now

Diff Detail

Repository

rDWAPPS Web applications

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 21998
Build 34223: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 34222: arc lint + arc unit

Event Timeline

ardumont created this revision.Jun 11 2021, 4:06 PM

anlambert added inline comments.Jun 11 2021, 4:09 PM

swh/web/common/origin_save.py
514–520 ↗	(On Diff #20965)	not sure if we need to reschedule those kind of visit types as artifacts list will not change here

WIP as tests need updating. Also opened to discuss where that part should go. I've
chosen immediately when creating the oneshot task but this could also be put after the
first actually ingestion occur and is successful (and not a plain not_found and failed
status for example)...

I would go for the second option, we are sure that the origin is valid in that case.

Build has FAILED

Patch application report for D5858 (id=20965)

Rebasing onto da39599f34...

Current branch diff-target is up to date.

Changes applied before test

commit 681af839d2708759d30d9ad424a7818a2c013e25
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Jun 11 16:01:36 2021 +0200

    Schedule save code now as recurring origins to ingest as well
    
    To avoid for the user to come back and schedule again the same origin later, this also
    manage origins as if the save code now was a lister and record the origin submitted by
    the user.
    
    Related to T1524

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/863/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/863/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 11 2021, 4:14 PM

Harbormaster failed remote builds in B21944: Diff 20965!

In D5858#149508, @anlambert wrote:

WIP as tests need updating. Also opened to discuss where that part should go. I've
chosen immediately when creating the oneshot task but this could also be put after the
first actually ingestion occur and is successful (and not a plain not_found and failed
status for example)...

I would go for the second option, we are sure that the origin is valid in that case.

If that's not too much work (with the current status polling, I guess it's not), I agree with that

swh/web/common/origin_save.py
512 ↗	(On Diff #20965)	Should we make this instance name the name of the frontend that sent the request? (e.g. archive.softwareheritage.org)
514–520 ↗	(On Diff #20965)	Yeah, I'd agree with that.

Thanks for the inputs both ;)

swh/web/common/origin_save.py
512 ↗	(On Diff #20965)	yes, sounds reasonable
514–520 ↗	(On Diff #20965)	oh yeah, good point ;)

Adapt according to suggestions (still no tests)

ardumont edited the summary of this revision. (Show Details)Jun 11 2021, 5:12 PM

Build has FAILED

Patch application report for D5858 (id=20968)

Rebasing onto da39599f34...

Current branch diff-target is up to date.

Changes applied before test

commit 3c5199803577dfe934c47f3ea1e667769a1be006
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Jun 11 16:01:36 2021 +0200

    Schedule save code now as recurring origins to ingest as well
    
    To avoid for the user to come back and schedule again the same origin later, this also
    manage origins as if the save code now was a lister and record the origin submitted by
    the user.
    
    Related to T1524

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/865/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/865/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 11 2021, 5:16 PM

Harbormaster failed remote builds in B21948: Diff 20968!

ardumont added inline comments.Jun 14 2021, 5:04 PM

swh/web/common/management/commands/refresh_savecodenow_statuses.py
28	^ fixme: find the archive's instance name instead of host

Update tests

Build has FAILED

Patch application report for D5858 (id=21015)

Rebasing onto a0db251b32...

Current branch diff-target is up to date.

Changes applied before test

commit b12529f49d83e173502d470317699a93424eb204
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Jun 11 16:01:36 2021 +0200

    Schedule save code now as recurring origins to ingest as well
    
    To avoid for the user to come back and schedule again the same origin later, this also
    manage origins as if the save code now was a lister and record the origin submitted by
    the user.
    
    Related to T1524

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/871/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/871/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 15 2021, 11:39 AM

Harbormaster failed remote builds in B21994: Diff 21015!

ardumont retitled this revision from wip: Schedule save code now as recurring origins to ingest as well to Schedule save code now as recurring origins to ingest as well.Jun 15 2021, 11:40 AM

ardumont edited the summary of this revision. (Show Details)

rework commit message
make mypy happier

ardumont retitled this revision from Schedule save code now as recurring origins to ingest as well to Schedule save code now as recurring origins to ingest when successful.Jun 15 2021, 11:47 AM

ardumont edited the summary of this revision. (Show Details)

Build was aborted

Patch application report for D5858 (id=21017)

Rebasing onto a0db251b32...

Current branch diff-target is up to date.

Changes applied before test

commit c2723ffa1bd5409e69a9a23c708dd66f9326fbcf
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Jun 11 16:01:36 2021 +0200

    Schedule save code now as recurring origins to ingest as well
    
    To allow users to request only once their save code now origins. We also schedule the
    origin when the ingestion is successfully ingested in the archive.
    
    Implementation wise, this is the scheduling routine in charge of updating the save code
    now status which is in charge of this.
    
    Related to T1524

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/872/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/872/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 15 2021, 11:53 AM

Harbormaster failed remote builds in B21995: Diff 21017!

rework commit message

Build is green

Patch application report for D5858 (id=21020)

Rebasing onto a0db251b32...

Current branch diff-target is up to date.

Changes applied before test

commit 3b16bc9051b34b1afdfbdabfbdbe7ad3d746311c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Jun 11 16:01:36 2021 +0200

    Schedule save code now as recurring origins to ingest when successful
    
    To allow users to request only once their save code now origins, once the first
    ingestion is successfully ingested, we also mark it as recurrent origin to crawl
    
    Implementation wise, this is the scheduling routine in charge of updating the save code
    now status which is in charge of this.
    
    Related to T1524

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/874/ for more details.

Harbormaster completed remote builds in B21998: Diff 21020.Jun 15 2021, 12:04 PM

ardumont requested review of this revision.Jun 15 2021, 12:04 PM

Possible duplicated origins should be handled but looks good otherwise.

swh/web/common/management/commands/refresh_savecodenow_statuses.py
32–43	You can have duplicated origins in the final list as multiple save code now requests can be submitted for a same origin URL. You should avoid that duplication as it will result in errors when trying to insert the data in scheduler database. Also maybe the `last_update` field of `ListedOrigin` could be set to the visit date ?
swh/web/tests/common/test_django_command.py
31–39	I would have split that fixture in two separate ones but not a big deal.

This revision now requires changes to proceed.Jun 15 2021, 2:06 PM

ardumont added inline comments.Jun 15 2021, 2:21 PM

swh/web/common/management/commands/refresh_savecodenow_statuses.py
32–43	definitely a good idea for the first part. The second part sounds good as well but I'm not sure what that entails.
swh/web/tests/common/test_django_command.py
31–39	I agree. I did not spend much time in trying to refactor the tests after making it green ;)

anlambert added inline comments.Jun 15 2021, 2:26 PM

swh/web/common/management/commands/refresh_savecodenow_statuses.py
32–43	The second part sounds good as well but I'm not sure what that entails. Thinking it back, this should be set only if the visit was eventful as we know for sure we ingest some new code here. If I recall correctly, last update date can be of interest in some new scheduling strategies but also not sure if it makes sense for the save code now.

ardumont added inline comments.Jun 15 2021, 2:27 PM

swh/web/common/management/commands/refresh_savecodenow_statuses.py
32–43	The second part sounds good as well but I'm not sure what that entails. I played the bonus card and asked a friend ;) 14:24 <+olasd> ardumont: if you set last_update, then it won't get visited again until that field is updated So no.

anlambert added inline comments.Jun 15 2021, 2:29 PM

swh/web/common/management/commands/refresh_savecodenow_statuses.py
32–43	yep, using that field seems only relevant for a real lister exploiting forge API.

Rebase
adapt according to review (split fixtures, filter out duplicates prior to record listed origins)
Open a configuration key instance_name for naming lister (webapp1/moma needs the same configuration as they are 2 instances of the same webapp)

ardumont marked 4 inline comments as done.Jun 15 2021, 3:39 PM

Looks good to me !

This revision is now accepted and ready to land.Jun 15 2021, 3:42 PM

Build is green

Patch application report for D5858 (id=21040)

Rebasing onto 8b4e77cae6...

Current branch diff-target is up to date.

Changes applied before test

commit 01ffb31ff72a9dbb1ea3eabad32ecd7fa1e231cc
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Jun 11 16:01:36 2021 +0200

    Schedule save code now as recurring origins to ingest when successful
    
    To allow users to request only once their save code now origins, once the first
    ingestion is successfully ingested, we also mark it as recurrent origin to crawl.
    
    Implementation wise, the scheduling routine in charge of updating the save code now
    statuses reported in the save code now ui is in charge of this.
    
    Related to T1524

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/877/ for more details.

Harbormaster completed remote builds in B22033: Diff 21040.Jun 15 2021, 3:51 PM

Closed by commit rDWAPPS01ffb31ff72a: Schedule save code now as recurring origins to ingest when successful (authored by ardumont). · Explain WhyJun 15 2021, 3:55 PM

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rDWAPPS01ffb31ff72a: Schedule save code now as recurring origins to ingest when successful.

Schedule save code now as recurring origins to ingest when successful
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Patch application report for D5858 (id=20965)

Changes applied before test

Patch application report for D5858 (id=20968)

Changes applied before test

Patch application report for D5858 (id=21015)

Changes applied before test

Patch application report for D5858 (id=21017)

Changes applied before test

Patch application report for D5858 (id=21020)

Changes applied before test

Patch application report for D5858 (id=21040)

Changes applied before test

Revision Contents
Changeset List

Diff 21020

conftest.py

requirements-test.txt

swh/web/common/management/commands/refresh_savecodenow_statuses.py

swh/web/tests/common/test_django_command.py

Schedule save code now as recurring origins to ingest when successfulClosedPublicActions

Details

Diff Detail

Event Timeline

Patch application report for D5858 (id=20965)

Changes applied before test

Patch application report for D5858 (id=20968)

Changes applied before test

Patch application report for D5858 (id=21015)

Changes applied before test

Patch application report for D5858 (id=21017)

Changes applied before test

Patch application report for D5858 (id=21020)

Changes applied before test

Patch application report for D5858 (id=21040)

Changes applied before test

Revision ContentsChangeset List

Diff 21020

conftest.py

requirements-test.txt

swh/web/common/management/commands/refresh_savecodenow_statuses.py

swh/web/tests/common/test_django_command.py

Schedule save code now as recurring origins to ingest when successful
ClosedPublic
Actions

Revision Contents
Changeset List