Paths

Table of Contentst

Differential D4852

Start journal client to populate the origin_visit_stats model
AbandonedPublic
Actions

Authored by ardumont on Jan 13 2021, 1:08 PM.

Details

Reviewers

vlorentz

Group Reviewers

Reviewers

Maniphest Tasks

T2967: Write journal client subcribed to origin_visit_status topics

Summary

This is only a function not plugged to anything yet.
Remains the cli part to actually be able to use it.

Questions inline in the code as comment.

Related to T2967

Test Plan

tox
(failing until swh.model > 0.9 [1] is released)

[1] D4848

Diff Detail

Repository

rDSCH Scheduling utilities

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 18323
Build 28298: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 28297: arc lint + arc unit

Event Timeline

ardumont created this revision.Jan 13 2021, 1:08 PM

Build has FAILED

Patch application report for D4852 (id=17178)

Rebasing onto a62003397d...

Current branch diff-target is up to date.

Changes applied before test

commit 52031853850394482c697fe2f9f634562cace31b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jan 13 13:03:28 2021 +0100

    Start journal client to populate the origin_visit_stats model
    
    Related to T2967

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/100/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/100/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jan 13 2021, 1:10 PM

Harbormaster failed remote builds in B18322: Diff 17178!

This needs more than introspecting the visit status information to determine the date
field to update in origin_visit_stats.

Build has FAILED

Patch application report for D4852 (id=17179)

Rebasing onto a62003397d...

Current branch diff-target is up to date.

Changes applied before test

commit ab9871d3f14ccc4510ef0912af45716259630878
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jan 13 13:03:28 2021 +0100

    Populate origin_visit_stats table out of the origin_visit_status topic
    
    Related to T2967

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/101/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/101/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jan 13 2021, 3:07 PM

Harbormaster failed remote builds in B18323: Diff 17179!

ardumont mentioned this in D4853: visit_stats: Keep a reference to the last snapshot.Jan 13 2021, 3:37 PM

ardumont edited the test plan for this revision. (Show Details)Jan 13 2021, 6:43 PM

ardumont added a child revision: D4853: visit_stats: Keep a reference to the last snapshot.Jan 14 2021, 11:17 AM

You should make origin_visit_stats_upsert take a list as arguments so that process_journal_objects can send them in batches.

Right now, it's doing one query + one commit for each stats, and it's the bottleneck of the simulator

Build is green

Patch application report for D4852 (id=17179)

Rebasing onto ca45d40f2a...

First, rewinding head to replay your work on top of it...
Applying: Populate origin_visit_stats table out of the origin_visit_status topic

Changes applied before test

commit a2a046e2bc67b455707888bca01c99af7cbdfc13
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jan 13 13:03:28 2021 +0100

    Populate origin_visit_stats table out of the origin_visit_status topic
    
    Related to T2967

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/122/ for more details.

Harbormaster completed remote builds in B18323: Diff 17179.Jan 14 2021, 5:08 PM

ardumont requested review of this revision.Jan 14 2021, 5:08 PM

vlorentz requested changes to this revision.Jan 14 2021, 5:09 PM

This revision now requires changes to proceed.Jan 14 2021, 5:09 PM

superseded by D4866