Differential D6145

Add table sampling option to grab_next_visits
ClosedPublic
Actions

Authored by olasd on Aug 26 2021, 4:48 PM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Commits

rDSCH5e8007fdbfeb: Add table sampling option to grab_next_visits

Summary

Running common operations on all git origins is pretty intense. Using
table sampling gives us the opportunity to at least schedule some jobs
in (decently small) time.

Test Plan

smoke test added; not easy to test the *exact* table sampling behavior reliably

Diff Detail

Repository

rDSCH Scheduling utilities

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 23203
Build 36202: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 36201: arc lint + arc unit

Event Timeline

olasd created this revision.Aug 26 2021, 4:48 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptAug 26 2021, 4:48 PM

olasd added a child revision: D6146: Add a swh scheduler origin send-to-celery subcommand.Aug 26 2021, 4:49 PM

ardumont mentioned this in D5809: Direct scheduling of origin visits in celery.Aug 26 2021, 4:50 PM

Build is green

Patch application report for D6145 (id=22244)

Rebasing onto cc76a573e7...

Current branch diff-target is up to date.

Changes applied before test

commit 5e8007fdbfeb612ea394f97eeba25a1c4e529b7e
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Jun 1 15:48:05 2021 +0200

    Add table sampling option to grab_next_visits
    
    Running common operations on all git origins is pretty intense. Using
    table sampling gives us the opportunity to at least schedule some jobs
    in (decently small) time.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/446/ for more details.

Harbormaster completed remote builds in B23203: Diff 22244.Aug 26 2021, 4:51 PM

olasd requested review of this revision.Aug 26 2021, 4:51 PM

ardumont accepted this revision.Aug 26 2021, 4:51 PM

This revision is now accepted and ready to land.Aug 26 2021, 4:51 PM

Closed by commit rDSCH5e8007fdbfeb: Add table sampling option to grab_next_visits (authored by olasd). · Explain WhyAug 27 2021, 12:37 PM

This revision was automatically updated to reflect the committed changes.

olasd added a commit: rDSCH5e8007fdbfeb: Add table sampling option to grab_next_visits.