Depends on D7868
Will be used to measure the effectiveness of T4219#85416
Details
Details
- Reviewers
anlambert - Group Reviewers
Reviewers - Maniphest Tasks
- T4219: Investigate why GitHub fork detection did not bring a speed-up
- Commits
- rDLDGf45ca1c2c0fa: Add metrics in store_data on ratios of objects already stored
Diff Detail
Diff Detail
- Repository
- rDLDG Git loader
- Branch
- storage-metrics
- Lint
No Linters Available - Unit
No Unit Test Coverage - Build Status
Buildable 29506 Build 46104: Phabricator diff pipeline on jenkins Jenkins console · Jenkins Build 46103: arc lint + arc unit
Event Timeline
Comment Actions
Build has FAILED
Patch application report for D7871 (id=28414)
Could not rebase; Attempt merge onto 85a4794094...
Updating 85a4794..78f51af Fast-forward swh/loader/git/base.py | 123 ++++++++++++++++++++++++++++++++++++ swh/loader/git/from_disk.py | 4 +- swh/loader/git/loader.py | 4 +- swh/loader/git/tests/test_loader.py | 79 ++++++++++++++++++++++- 4 files changed, 204 insertions(+), 6 deletions(-) create mode 100644 swh/loader/git/base.py
Changes applied before test
commit 78f51af62ad049e1883adca19b91ca100d63e06d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Fri May 20 13:46:58 2022 +0200
Add metrics in store_date on ratios of objects already stored
commit 083e1aa18e24cd3311162e259b15c1867b313060
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Fri May 20 12:16:23 2022 +0200
Move store_data from DVCSLoader to a new BaseGitLoader
In preparation for the removal of DVCSLoader from swh.loader.core,
as the git loader is the only one to use it anymore.Link to build: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/216/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/216/console
Comment Actions
Build has FAILED
Patch application report for D7871 (id=28415)
Could not rebase; Attempt merge onto 85a4794094...
Updating 85a4794..c145fcb Fast-forward requirements-swh.txt | 2 +- swh/loader/git/base.py | 123 ++++++++++++++++++++++++++++++++++++ swh/loader/git/from_disk.py | 4 +- swh/loader/git/loader.py | 4 +- swh/loader/git/tests/test_loader.py | 79 ++++++++++++++++++++++- 5 files changed, 205 insertions(+), 7 deletions(-) create mode 100644 swh/loader/git/base.py
Changes applied before test
commit c145fcb345af490f8dbc4067921b06b839eb4548
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Fri May 20 13:46:58 2022 +0200
Add metrics in store_date on ratios of objects already stored
commit 083e1aa18e24cd3311162e259b15c1867b313060
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Fri May 20 12:16:23 2022 +0200
Move store_data from DVCSLoader to a new BaseGitLoader
In preparation for the removal of DVCSLoader from swh.loader.core,
as the git loader is the only one to use it anymore.Link to build: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/217/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/217/console
Comment Actions
Looks good to me. There is a typo in commit message though, s/store_date/store_data/.
Comment Actions
Build has FAILED
Patch application report for D7871 (id=28418)
Could not rebase; Attempt merge onto 85a4794094...
Updating 85a4794..f45ca1c Fast-forward requirements-swh.txt | 2 +- swh/loader/git/base.py | 123 ++++++++++++++++++++++++++++++++++++ swh/loader/git/from_disk.py | 4 +- swh/loader/git/loader.py | 4 +- swh/loader/git/tests/test_loader.py | 79 ++++++++++++++++++++++- 5 files changed, 205 insertions(+), 7 deletions(-) create mode 100644 swh/loader/git/base.py
Changes applied before test
commit f45ca1c2c0fac2f57cd1b43af984251078c89169
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Fri May 20 13:46:58 2022 +0200
Add metrics in store_data on ratios of objects already stored
commit 083e1aa18e24cd3311162e259b15c1867b313060
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Fri May 20 12:16:23 2022 +0200
Move store_data from DVCSLoader to a new BaseGitLoader
In preparation for the removal of DVCSLoader from swh.loader.core,
as the git loader is the only one to use it anymore.Link to build: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/218/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/218/console
Comment Actions
Build is green
Patch application report for D7871 (id=28418)
Could not rebase; Attempt merge onto 85a4794094...
Updating 85a4794..f45ca1c Fast-forward requirements-swh.txt | 2 +- swh/loader/git/base.py | 123 ++++++++++++++++++++++++++++++++++++ swh/loader/git/from_disk.py | 4 +- swh/loader/git/loader.py | 4 +- swh/loader/git/tests/test_loader.py | 79 ++++++++++++++++++++++- 5 files changed, 205 insertions(+), 7 deletions(-) create mode 100644 swh/loader/git/base.py
Changes applied before test
commit f45ca1c2c0fac2f57cd1b43af984251078c89169
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Fri May 20 13:46:58 2022 +0200
Add metrics in store_data on ratios of objects already stored
commit 083e1aa18e24cd3311162e259b15c1867b313060
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Fri May 20 12:16:23 2022 +0200
Move store_data from DVCSLoader to a new BaseGitLoader
In preparation for the removal of DVCSLoader from swh.loader.core,
as the git loader is the only one to use it anymore.See https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/219/ for more details.
Comment Actions
Thanks for working on this, I think getting these insights will be useful.
Don't take this as a blocking suggestion but rather as an opportunity for generalization: do you think it would be possible to implement these "filtered objects" metrics inside the swh.storage filter proxy, rather than hardcode it only in the git loader? This way all loaders would be able to leverage them.
Even if the statsd poking only happens in the loader (to be able to use the cumulated tags on that statsd instance), I think we could push the collection of the cumulative counts down one layer.