This is only a function not plugged to anything yet.
Remains the cli part to actually be able to use it.
Questions inline in the code as comment.
Related to T2967
Details
Details
- Reviewers
vlorentz - Group Reviewers
Reviewers - Maniphest Tasks
- T2967: Write journal client subcribed to origin_visit_status topics
tox
(failing until swh.model > 0.9 [1] is released)
[1] D4848
Diff Detail
Diff Detail
- Repository
- rDSCH Scheduling utilities
- Branch
- master
- Lint
No Linters Available - Unit
No Unit Test Coverage - Build Status
Buildable 18322 Build 28296: Phabricator diff pipeline on jenkins Jenkins console · Jenkins Build 28295: arc lint + arc unit
| Time | Test | |
|---|---|---|
| 28 ms | Jenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.test_journal_client::test_journal_client_origin_visit_status_from_journal swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7f80a1ac62e8>
def test_journal_client_origin_visit_status_from_journal(swh_scheduler):
| |
| 6 ms | Jenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.cli.task::swh.scheduler.cli.task.pretty_print_task | |
| 4 ms | Jenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.cli.utils::swh.scheduler.cli.utils.parse_options | |
| 26 ms | Jenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.es.test_backend_es::test_backend_setup_basic | |
| 25 ms | Jenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.es.test_backend_es::test_backend_setup_index | |
| View Full Test Results (1 Failed · 142 Passed · 1 Skipped) |
Event Timeline
Comment Actions
Build has FAILED
Patch application report for D4852 (id=17178)
Rebasing onto a62003397d...
Current branch diff-target is up to date.
Changes applied before test
commit 52031853850394482c697fe2f9f634562cace31b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Wed Jan 13 13:03:28 2021 +0100
Start journal client to populate the origin_visit_stats model
Related to T2967Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/100/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/100/console
Comment Actions
This needs more than introspecting the visit status information to determine the date
field to update in origin_visit_stats.
Comment Actions
Build has FAILED
Patch application report for D4852 (id=17179)
Rebasing onto a62003397d...
Current branch diff-target is up to date.
Changes applied before test
commit ab9871d3f14ccc4510ef0912af45716259630878
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Wed Jan 13 13:03:28 2021 +0100
Populate origin_visit_stats table out of the origin_visit_status topic
Related to T2967Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/101/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/101/console
Comment Actions
You should make origin_visit_stats_upsert take a list as arguments so that process_journal_objects can send them in batches.
Right now, it's doing one query + one commit for each stats, and it's the bottleneck of the simulator
Comment Actions
Build is green
Patch application report for D4852 (id=17179)
Rebasing onto ca45d40f2a...
First, rewinding head to replay your work on top of it... Applying: Populate origin_visit_stats table out of the origin_visit_status topic
Changes applied before test
commit a2a046e2bc67b455707888bca01c99af7cbdfc13
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Wed Jan 13 13:03:28 2021 +0100
Populate origin_visit_stats table out of the origin_visit_status topic
Related to T2967See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/122/ for more details.