Page MenuHomeSoftware Heritage
Differential D4921

Make plotting optional in simulator cli command
ClosedPublic
Actions

Authored by douardda on Jan 22 2021, 10:51 AM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDSCHaaf7dd6f1d82: Make plottings optional in simulator cli output

Diff Detail

Event Timeline

douardda created this revision.Jan 22 2021, 10:51 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptJan 22 2021, 10:51 AM
douardda added a child revision: D4922: Simulation: log at infol level recorded metrics.Jan 22 2021, 10:51 AM
swh-public-ci added a comment.Jan 22 2021, 10:57 AM

Build is green

Patch application report for D4921 (id=17506)

Could not rebase; Attempt merge onto 03460207a1...

Updating 0346020..6698e51
Fast-forward
 swh/scheduler/cli/simulator.py      | 20 ++++++++++----
 swh/scheduler/simulator/__init__.py | 53 ++++++++++++++++++++++---------------
 swh/scheduler/simulator/common.py   | 34 +++++++++++++-----------
 3 files changed, 64 insertions(+), 43 deletions(-)
Changes applied before test
commit 6698e51903bfdfcc00cfc55be3056c25c8cb270f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 22 09:07:22 2021 +0100

    Make plotting histograms optional in simulator cli command

commit 1c069ca34add6b26d060588abb7958a089cb0735
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Jan 21 11:33:19 2021 +0100

    Randomize last_upadte in generated ListedOrigins in fill_test_data
    
    also insert objects by batches of 10k to make it nicer with ram usage.

commit 8a9aaf3942d5585e6af038ebced3cde6faf27c7e
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Jan 21 11:30:21 2021 +0100

    Add a --num-origins option to the fill-test-data cli command

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/246/ for more details.

Harbormaster completed remote builds in B18638: Diff 17506.Jan 22 2021, 10:57 AM
douardda requested review of this revision.Jan 22 2021, 10:57 AM
vlorentz requested changes to this revision.Jan 22 2021, 11:16 AM
vlorentz added a subscriber: vlorentz.

This will conflict with various other diffs, as we add other plots (that aren't histogram). Could you rebase on D4916?

This revision now requires changes to proceed.Jan 22 2021, 11:16 AM
douardda removed a child revision: D4922: Simulation: log at infol level recorded metrics.Jan 22 2021, 11:20 AM
douardda edited the summary of this revision. (Show Details)Jan 22 2021, 11:26 AM
douardda removed a parent revision: D4920: Randomize last_update in generated ListedOrigins in fill_test_data.
douardda updated this revision to Diff 17513.Jan 22 2021, 11:27 AM

kill unnedded dependency on D4920

swh-public-ci added a comment.Jan 22 2021, 11:33 AM

Build is green

Patch application report for D4921 (id=17513)

Rebasing onto 03460207a1...

Current branch diff-target is up to date.
Changes applied before test
commit 011cc3ddff7e5683f1f00a8c578d21776a923ae7
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 22 09:07:22 2021 +0100

    Make plotting histograms optional in simulator cli command

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/253/ for more details.

Harbormaster completed remote builds in B18645: Diff 17513.Jan 22 2021, 11:34 AM
douardda updated this revision to Diff 17521.Jan 22 2021, 12:20 PM

rebase on D4916

douardda retitled this revision from Make plotting histograms optional in simulator cli command to Make plotting optional in simulator cli command.Jan 22 2021, 12:21 PM
douardda added a parent revision: D4916: Run simulator tests on all known scheduling policies.
douardda added a child revision: D4923: Simulation: allow to export results in a csv file.
swh-public-ci added a comment.Jan 22 2021, 12:23 PM

Build is green

Patch application report for D4921 (id=17521)

Could not rebase; Attempt merge onto b93aa5be2c...

Updating b93aa5b..bd0941c
Fast-forward
 swh/scheduler/backend.py                    |  61 +++++++++--
 swh/scheduler/cli/simulator.py              |  12 ++-
 swh/scheduler/interface.py                  |  15 ++-
 swh/scheduler/simulator/__init__.py         |  27 ++---
 swh/scheduler/simulator/common.py           |  75 ++++++++-----
 swh/scheduler/simulator/origin_scheduler.py |   2 +-
 swh/scheduler/simulator/origins.py          | 162 +++++++++++++++++++++-------
 swh/scheduler/tests/test_simulator.py       |   9 +-
 8 files changed, 273 insertions(+), 90 deletions(-)
Changes applied before test
commit bd0941c722dae7ac14385b45048eee7f9565f735
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 22 12:15:47 2021 +0100

    Make plottings optional in simulator cli output

commit f878c6036ba7400dc08fc33dc8d3858cc234b4c9
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jan 21 17:55:16 2021 +0100

    Run simulator tests on all known scheduling policies

commit bdbc3a86f84772ec166764ca5169ec597cf89e14
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jan 21 17:48:38 2021 +0100

    simulator: record visit metrics alongside scheduler metrics
    
    This allows us to check the behavior of the archive over time in terms
    of number of visits.

commit 7afb0a498432d1e2641abf3a9de859354699c5c4
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jan 21 17:45:23 2021 +0100

    simulator: stop using the database as a cache for origin data
    
    This was a significant bottleneck of the simulator. To work around this,
    we:
    
     - Generate snapshot ids consistently in the OriginModel
     - Cache the origin data locally in the simulator, to compute the
       eventfulness of visits
     - Cache the last visit time for all origins to compute the estimated
       run time of visit tasks.

commit 8e7377d8af45ef8e8234b57dc6a16be75dd74ac5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 17:38:41 2021 +0100

    simulator: add a trivial heartbeat process to show progress
    
    For now, this process only writes a log every simulated day.

commit ba303f946ecd3e15e58de0072ce71b50aa423d59
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jan 21 17:31:43 2021 +0100

    grab_next_visits: don't re-schedule visits too fast
    
    The earlier implementation would just schedule new visits for origins
    forever, regardless of whether they were already scheduled or not.

commit 808ae6851faee9b633e773f9150d360cdb927146
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jan 21 17:29:45 2021 +0100

    Allow overriding the timestamp of grab_next_visits
    
    This makes the simulator behavior more consistent with reality.

commit 9943195d31c51a44325cba09d07fb6e904d45a00
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jan 21 17:27:40 2021 +0100

    Construct grab_next_visits query arguments incrementally

commit 72070b7bf628788b6872e90a3f8ac8f0c01b70d9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 14:57:42 2021 +0100

    simulator: add simple lister simulation

commit 1f1aad459c4b0740ecbe96e9809e4b31f66bf999
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 14:54:53 2021 +0100

    Factor out ListedOrigin generation to use the OriginModel
    
    This generates consistent last_update values according to the model and
    simulated time.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/258/ for more details.

Harbormaster completed remote builds in B18653: Diff 17521.Jan 22 2021, 12:23 PM
vlorentz added inline comments.Jan 22 2021, 12:45 PM
swh/scheduler/cli/simulator.py
50
douardda added inline comments.Jan 22 2021, 4:19 PM
swh/scheduler/cli/simulator.py
50

:-)

douardda updated this revision to Diff 17535.Jan 22 2021, 4:20 PM

s/-H/-P/

swh-public-ci added a comment.Jan 22 2021, 4:22 PM

Build is green

Patch application report for D4921 (id=17535)

Could not rebase; Attempt merge onto 86b255544c...

Auto-merging swh/scheduler/simulator/__init__.py
Auto-merging swh/scheduler/cli/simulator.py
Merge made by the 'recursive' strategy.
 swh/scheduler/backend.py                    |  61 +++++++++--
 swh/scheduler/cli/simulator.py              |  12 ++-
 swh/scheduler/interface.py                  |  15 ++-
 swh/scheduler/simulator/__init__.py         |  27 ++---
 swh/scheduler/simulator/common.py           |  75 ++++++++-----
 swh/scheduler/simulator/origin_scheduler.py |   2 +-
 swh/scheduler/simulator/origins.py          | 162 +++++++++++++++++++++-------
 swh/scheduler/tests/test_simulator.py       |   9 +-
 8 files changed, 273 insertions(+), 90 deletions(-)
Changes applied before test
commit 964ffa2a705904c6f93e489fe8a30d668ab8ab09
Merge: 86b2555 6a1b2e0
Author: Jenkins user <jenkins@localhost>
Date:   Fri Jan 22 15:20:16 2021 +0000

    Merge branch 'diff-target' into HEAD

commit 6a1b2e037498f5d1ad28effc5fb0a79f520ef46a
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 22 12:15:47 2021 +0100

    Make plottings optional in simulator cli output

commit f878c6036ba7400dc08fc33dc8d3858cc234b4c9
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jan 21 17:55:16 2021 +0100

    Run simulator tests on all known scheduling policies

commit bdbc3a86f84772ec166764ca5169ec597cf89e14
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jan 21 17:48:38 2021 +0100

    simulator: record visit metrics alongside scheduler metrics
    
    This allows us to check the behavior of the archive over time in terms
    of number of visits.

commit 7afb0a498432d1e2641abf3a9de859354699c5c4
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jan 21 17:45:23 2021 +0100

    simulator: stop using the database as a cache for origin data
    
    This was a significant bottleneck of the simulator. To work around this,
    we:
    
     - Generate snapshot ids consistently in the OriginModel
     - Cache the origin data locally in the simulator, to compute the
       eventfulness of visits
     - Cache the last visit time for all origins to compute the estimated
       run time of visit tasks.

commit 8e7377d8af45ef8e8234b57dc6a16be75dd74ac5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 17:38:41 2021 +0100

    simulator: add a trivial heartbeat process to show progress
    
    For now, this process only writes a log every simulated day.

commit ba303f946ecd3e15e58de0072ce71b50aa423d59
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jan 21 17:31:43 2021 +0100

    grab_next_visits: don't re-schedule visits too fast
    
    The earlier implementation would just schedule new visits for origins
    forever, regardless of whether they were already scheduled or not.

commit 808ae6851faee9b633e773f9150d360cdb927146
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jan 21 17:29:45 2021 +0100

    Allow overriding the timestamp of grab_next_visits
    
    This makes the simulator behavior more consistent with reality.

commit 9943195d31c51a44325cba09d07fb6e904d45a00
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jan 21 17:27:40 2021 +0100

    Construct grab_next_visits query arguments incrementally

commit 72070b7bf628788b6872e90a3f8ac8f0c01b70d9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 14:57:42 2021 +0100

    simulator: add simple lister simulation

commit 1f1aad459c4b0740ecbe96e9809e4b31f66bf999
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 14:54:53 2021 +0100

    Factor out ListedOrigin generation to use the OriginModel
    
    This generates consistent last_update values according to the model and
    simulated time.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/267/ for more details.

Harbormaster completed remote builds in B18667: Diff 17535.Jan 22 2021, 4:22 PM
vlorentz accepted this revision.Jan 25 2021, 10:45 AM
This revision is now accepted and ready to land.Jan 25 2021, 10:45 AM
douardda updated this revision to Diff 17758.Jan 29 2021, 5:23 PM

rebase

swh-public-ci added a comment.Jan 29 2021, 5:25 PM

Build is green

Patch application report for D4921 (id=17758)

Rebasing onto cf0583b079...

Current branch diff-target is up to date.
Changes applied before test
commit aaf7dd6f1d820012b588e780178aaefdc64e2685
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 22 12:15:47 2021 +0100

    Make plottings optional in simulator cli output

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/289/ for more details.

Harbormaster completed remote builds in B18899: Diff 17758.Jan 29 2021, 5:25 PM
Closed by commit rDSCHaaf7dd6f1d82: Make plottings optional in simulator cli output (authored by douardda). · Explain WhyFeb 1 2021, 3:11 PM
This revision was automatically updated to reflect the committed changes.
douardda added a commit: rDSCHaaf7dd6f1d82: Make plottings optional in simulator cli output.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
scheduler/
cli/
12 lines52%
Loading...
simulator/
2 lines92%
Loading...
34 lines70%
Loading...

Diff 17521

View Options

swh/scheduler/cli/simulator.py

Loading...
View Options

swh/scheduler/simulator/__init__.py

Loading...
View Options

swh/scheduler/simulator/common.py

Loading...
About · Developer information · Legal