Page MenuHomeSoftware Heritage
Differential D5047

journal_client: Fix date computations for (un)eventful visits
ClosedPublic
Actions

Authored by vsellier on Feb 9 2021, 3:04 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T3000: Investigate scheduler journal client discrepancies
Commits
rDSCHcf32e3764920: journal_client: Fix date computations for (un)eventful visits
Summary

Fix a wrong computation when several messages (>=3) for the same
snapshot are received in the wrong order
For example, before the fix, the following occurs:

| date | snapshot |     | last_ev  | last_unev | Snap |
| ---- | -------- | --- | -------- | --------- | ---- |
| 2022 | S2       |     | 2022     |           | S2   |
| 2020 | S2       |     | 2020     | 2022      | S2   |
| 2021 | S2       |     | **2021** | **2020**  | S2   |

as it should be:

| date | snapshot |     | last_ev  | last_unev | Snap |
| ---- | -------- | --- | -------- | --------- | ---- |
| 2022 | S2       |     | 2022     |           | S2   |
| 2020 | S2       |     | 2020     | 2022      | S2   |
| 2021 | S2       |     | **2020** | **2022**  | S2   |

P945 for more scenario samples.

Related to T3000

Diff Detail

Repository
rDSCH Scheduling utilities
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 19117
Build 29644: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 29643: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
45 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.test_cli::test_list_tasks
swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7ff6caf88cc0> def test_list_tasks(swh_scheduler):
41 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.test_cli::test_list_tasks_limit
swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7ff6cb37b898> def test_list_tasks_limit(swh_scheduler):
7 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.cli.task::swh.scheduler.cli.task.pretty_print_task
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.cli.utils::swh.scheduler.cli.utils.parse_options
28 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.es.test_backend_es::test_backend_setup_basic
View Full Test Results (2 Failed · 262 Passed · 1 Skipped)

Event Timeline

vsellier created this revision.Feb 9 2021, 3:04 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 9 2021, 3:04 PM
swh-public-ci added a comment.Feb 9 2021, 3:07 PM

Build has FAILED

Patch application report for D5047 (id=17996)

Rebasing onto 14feab9523...

Current branch diff-target is up to date.
Changes applied before test
commit 6999b3be99996457e7387f54cd487faa633e9d25
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Feb 9 14:56:56 2021 +0100

    journal_client: Fix date computations for (un)eventful visits
    
    Fix a wrong computation when several messages (>=3) for the same
    snapshot are received in the wrong order
    For example, before the fix, the following occurs:
| date | snapshot |     | last_ev  | last_unev | Snap |
| ---- | -------- | --- | -------- | --------- | ---- |
| 2022 | S2       |     | 2022     |           | S2   |
| 2020 | S2       |     | 2020     | 2022      | S2   |
| 2021 | S2       |     | **2021** | **2020**  | S2   |
```

as it should be:
```
| date | snapshot |     | last_ev  | last_unev | Snap |
| ---- | -------- | --- | -------- | --------- | ---- |
| 2022 | S2       |     | 2022     |           | S2   |
| 2020 | S2       |     | 2020     | 2022      | S2   |
| 2021 | S2       |     | **2020** | **2022**  | S2   |
```

Related to T3000
Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/299/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/299/console
Harbormaster returned this revision to the author for changes because remote builds failed.Feb 9 2021, 3:07 PM
Harbormaster failed remote builds in B19117: Diff 17996!
vsellier updated this revision to Diff 18016.Feb 9 2021, 6:17 PM

rebase

swh-public-ci added a comment.Feb 9 2021, 6:19 PM

Build is green

Patch application report for D5047 (id=18016)

Rebasing onto aa507ac51c...

Current branch diff-target is up to date.
Changes applied before test
commit cf32e37649207cabac4e960fc8d45ba5af480a24
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Feb 9 14:56:56 2021 +0100

    journal_client: Fix date computations for (un)eventful visits
    
    Fix a wrong computation when several messages (>=3) for the same
    snapshot are received in the wrong order
    For example, before the fix, the following occurs:
| date | snapshot |     | last_ev  | last_unev | Snap |
| ---- | -------- | --- | -------- | --------- | ---- |
| 2022 | S2       |     | 2022     |           | S2   |
| 2020 | S2       |     | 2020     | 2022      | S2   |
| 2021 | S2       |     | **2021** | **2020**  | S2   |
```

as it should be:
```
| date | snapshot |     | last_ev  | last_unev | Snap |
| ---- | -------- | --- | -------- | --------- | ---- |
| 2022 | S2       |     | 2022     |           | S2   |
| 2020 | S2       |     | 2020     | 2022      | S2   |
| 2021 | S2       |     | **2020** | **2022**  | S2   |
```

Related to T3000
See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/300/ for more details.
Harbormaster completed remote builds in B19136: Diff 18016.Feb 9 2021, 6:19 PM
vsellier requested review of this revision.Feb 9 2021, 6:19 PM
vlorentz accepted this revision.Feb 10 2021, 9:50 AM
vlorentz added a subscriber: vlorentz.

This code hurts my brain, but it looks correct

This revision is now accepted and ready to land.Feb 10 2021, 9:50 AM
ardumont edited the summary of this revision. (Show Details)Feb 10 2021, 10:12 AM
Closed by commit rDSCHcf32e3764920: journal_client: Fix date computations for (un)eventful visits (authored by vsellier). · Explain WhyFeb 10 2021, 11:17 AM
This revision was automatically updated to reflect the committed changes.
vsellier added a commit: rDSCHcf32e3764920: journal_client: Fix date computations for (un)eventful visits.

Revision Contents
Changeset List

PathSize
swh/
scheduler/
14 lines
tests/
60 lines

Diff 17996

View Options

swh/scheduler/journal_client.py

Loading...
View Options

swh/scheduler/tests/test_journal_client.py

Loading...
About · Developer information · Legal