Page MenuHomeSoftware Heritage

Start journal client to populate the origin_visit_stats model
AbandonedPublic

Authored by ardumont on Jan 13 2021, 1:08 PM.

Details

Summary

This is only a function not plugged to anything yet.
Remains the cli part to actually be able to use it.

Questions inline in the code as comment.

Related to T2967

Test Plan

tox
(failing until swh.model > 0.9 [1] is released)

[1] D4848

Diff Detail

Repository
rDSCH Scheduling utilities
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 18323
Build 28298: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 28297: arc lint + arc unit

Event Timeline

Build has FAILED

Patch application report for D4852 (id=17178)

Rebasing onto a62003397d...

Current branch diff-target is up to date.
Changes applied before test
commit 52031853850394482c697fe2f9f634562cace31b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jan 13 13:03:28 2021 +0100

    Start journal client to populate the origin_visit_stats model
    
    Related to T2967

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/100/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/100/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jan 13 2021, 1:10 PM
Harbormaster failed remote builds in B18322: Diff 17178!

This needs more than introspecting the visit status information to determine the date
field to update in origin_visit_stats.

Build has FAILED

Patch application report for D4852 (id=17179)

Rebasing onto a62003397d...

Current branch diff-target is up to date.
Changes applied before test
commit ab9871d3f14ccc4510ef0912af45716259630878
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jan 13 13:03:28 2021 +0100

    Populate origin_visit_stats table out of the origin_visit_status topic
    
    Related to T2967

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/101/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/101/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jan 13 2021, 3:07 PM
Harbormaster failed remote builds in B18323: Diff 17179!

You should make origin_visit_stats_upsert take a list as arguments so that process_journal_objects can send them in batches.

Right now, it's doing one query + one commit for each stats, and it's the bottleneck of the simulator

Build is green

Patch application report for D4852 (id=17179)

Rebasing onto ca45d40f2a...

First, rewinding head to replay your work on top of it...
Applying: Populate origin_visit_stats table out of the origin_visit_status topic
Changes applied before test
commit a2a046e2bc67b455707888bca01c99af7cbdfc13
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jan 13 13:03:28 2021 +0100

    Populate origin_visit_stats table out of the origin_visit_status topic
    
    Related to T2967

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/122/ for more details.

This revision now requires changes to proceed.Jan 14 2021, 5:09 PM