Page MenuHomeSoftware Heritage

Allow to specify the visit grab parameters per visit type and policy
ClosedPublic

Authored by vsellier on Wed, Jan 5, 9:19 AM.

Diff Detail

Repository
rDSCH Scheduling utilities
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Update a remaining test value

Build has FAILED

Patch application report for D6876 (id=24933)

Rebasing onto 559f3451c6...

Current branch diff-target is up to date.
Changes applied before test
commit 604f74c2677aedd16f1e59c5970a6558f8df1f4a
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Wed Jan 5 00:08:50 2022 +0100

    Allow to specify the visit grab parameters per visit type and policy
    
    Related to T3827

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/506/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/506/console

Build has FAILED

Patch application report for D6876 (id=24934)

Rebasing onto 559f3451c6...

Current branch diff-target is up to date.
Changes applied before test
commit 801d7a71da14994f9aee0d2d17b8d0baec1e31b8
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Wed Jan 5 00:08:50 2022 +0100

    Allow to specify the visit grab parameters per visit type and policy
    
    Related to T3827

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/507/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/507/console

Harbormaster returned this revision to the author for changes because remote builds failed.Wed, Jan 5, 9:24 AM
Harbormaster failed remote builds in B25805: Diff 24934!
ardumont added inline comments.
swh/scheduler/celery_backend/recurrent_visits.py
112

This one exists so might as well access it directly.
(Might even make the following assertion useless?)

lgtm

remains to unstuck the build.
one suggestion inline as well.

adapt according the feedback.
good catch, the assertion can be removed.

Build has FAILED

Patch application report for D6876 (id=24935)

Rebasing onto 559f3451c6...

Current branch diff-target is up to date.
Changes applied before test
commit 41cf0d40aeaba790520ebf7b97165ecfc80b2682
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Wed Jan 5 00:08:50 2022 +0100

    Allow to specify the visit grab parameters per visit type and policy
    
    Related to T3827

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/508/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/508/console

Harbormaster returned this revision to the author for changes because remote builds failed.Wed, Jan 5, 10:08 AM
Harbormaster failed remote builds in B25806: Diff 24935!

fix the tests for python 3.7

Build is green

Patch application report for D6876 (id=24936)

Rebasing onto 559f3451c6...

Current branch diff-target is up to date.
Changes applied before test
commit 1b6a69a783ad45bdaca6fe42408127b74ed23720
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Wed Jan 5 00:08:50 2022 +0100

    Allow to specify the visit grab parameters per visit type and policy
    
    Related to T3827

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/509/ for more details.

This revision is now accepted and ready to land.Wed, Jan 5, 11:25 AM

The shape of POLICY_ADDITIONAL_PARAMETERS is a bit clunky, with the dict key being POLICY_ADDITIONAL_PARAMETERS[visit_type][kwarg][policy]; having the nested dict "key" be POLICY_ADDITIONAL_PARAMETERS[visit_type][policy] = kwargs would be simpler?

If you make

POLICY_ADDITIONAL_PARAMETERS = {
    "git": {
        "already_visited_order_by_lag": {"tablesample": 0.1},
        ...
    }
}

Then grab_next_visits can just use

grab_next_visits(
    ...,
    **POLICY_ADDITIONAL_PARAMETERS.get(visit_type, {}).get(policy, {}),
)

upgrade accordingly the olasd's feedback

Thanks it's simpler this way.

Build is green

Patch application report for D6876 (id=24956)

Rebasing onto 559f3451c6...

Current branch diff-target is up to date.
Changes applied before test
commit 5c836d64a5fc460df5437e1167bf3b10613ba28b
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Wed Jan 5 00:08:50 2022 +0100

    Allow to specify the visit grab parameters per visit type and policy
    
    Related to T3827

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/510/ for more details.

This revision is now accepted and ready to land.Wed, Jan 5, 11:07 PM