Page MenuHomeSoftware Heritage

misc/coverage: Revamp archive coverage widget (WIP)
Needs ReviewPublic

Authored by anlambert on Fri, Jul 16, 6:21 PM.

Details

Reviewers
None
Group Reviewers
Reviewers
Summary

In order to implement T3127, revamp the archive coverage widget by:

  • adding new origins types that we now archived
  • splitting origins in multiple categories: listed, legacy, deposited and miscellaneous
  • adding global origins counts per origin type
  • adding detailed origins counts for each listed forges
  • removing dead code

The default view presents a high level overview of the archived origins.

In order to get more details about listed forges, collapsible tables have been added to the widget.
By clicking on any total origin counts, all those tables will be expanded to display detailed listers
metrics and links to search associated origins.

Below is how it looks so far:


Diff Detail

Repository
rDWAPPS Web applications
Branch
coverage-widget-upgrade
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 22740
Build 35463: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 35462: arc lint + arc unit

Event Timeline

anlambert edited the summary of this revision. (Show Details)

Build has FAILED

Patch application report for D6004 (id=21677)

Rebasing onto 4a9cf1d618...

Current branch diff-target is up to date.
Changes applied before test
commit 905265b5bc67f9655853e4814f33cbbc007bcd44
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Jul 16 18:09:30 2021 +0200

    misc/coverage: Revamp archive coverage widget (WIP)

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/967/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/967/console

Harbormaster returned this revision to the author for changes because remote builds failed.Fri, Jul 16, 6:39 PM
Harbormaster failed remote builds in B22645: Diff 21677!

Update: Add relevant deposit origins counts

Build is green

Patch application report for D6004 (id=21693)

Could not rebase; Attempt merge onto f667a87aa9...

Updating f667a87a..eb434d0c
Fast-forward
 assets/src/bundles/webapp/webapp.css |   1 +
 static/img/logos/cgit.png            | Bin 0 -> 1366 bytes
 static/img/logos/elife.png           | Bin 0 -> 36296 bytes
 static/img/logos/gitea.png           | Bin 0 -> 26874 bytes
 static/img/logos/launchpad.png       | Bin 0 -> 6346 bytes
 static/img/logos/sourceforge.png     | Bin 0 -> 8542 bytes
 swh/web/admin/deposit.py             |  32 +---
 swh/web/common/utils.py              |  30 ++-
 swh/web/config.py                    |   5 +-
 swh/web/misc/coverage.py             | 352 ++++++++++++++++++++---------------
 swh/web/templates/homepage.html      |   4 -
 swh/web/templates/misc/coverage.html |  57 +++---
 swh/web/tests/common/test_utils.py   |  59 +++++-
 13 files changed, 324 insertions(+), 216 deletions(-)
 create mode 100644 static/img/logos/cgit.png
 create mode 100644 static/img/logos/elife.png
 create mode 100644 static/img/logos/gitea.png
 create mode 100644 static/img/logos/launchpad.png
 create mode 100644 static/img/logos/sourceforge.png
Changes applied before test
commit eb434d0c9c219a528b70f32d020a461f3186f53a
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Jul 16 18:09:30 2021 +0200

    misc/coverage: Revamp archive coverage widget (WIP)

commit c5d441b1a8ec425b995ac72ab2c76c66a496ca1b
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Jul 19 17:18:31 2021 +0200

    common/utils: Wrap deposits list retrieval in a function
    
    It will enable to exploit those data in the archive coverage widget
    implementation to get deposit counts by origins type.
    
    Related to T3127

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/972/ for more details.

ardumont added inline comments.
swh/web/misc/coverage.py
230
240

how weird!?

250

That might pose an issue because long term, other deposit clients could also reference their origin through the doi site...
I don't know how deep that breaks the current assumpion on the netloc...

The ipol specific doi is https://doi.org/10.5201/ so might be change the netloc entry to what the changes suggest?

Or maybe it's yagni, let's consider this when the time comes?

At least now you know ^ ;)

swh/web/misc/coverage.py
230

Originally that view was also integrated in an iframe on softwareheritage.org but this is no longer the case.
I kept that particular view implementation in case we want to integrate it again.

240

Yes, do not know what is wrong here but this what we currently have in scheduler_metrics table:

softwareheritage-scheduler=> select name, instance_name, last_update, origins_known, origins_enabled, origins_never_visited, origins_with_pending_changes from listers inner join scheduler_metrics on id = lister_id order by name;
     name      |        instance_name         |          last_update          | origins_known | origins_enabled | origins_never_visited | origins_with_pending_changes 
---------------+------------------------------+-------------------------------+---------------+-----------------+-----------------------+------------------------------
 CRAN          | cran                         | 2021-07-13 12:14:07.919027+00 |         18292 |           18292 |                 18292 |                            0
250

I think the best way is to add a special processing for doi.org netloc and add the first path to it.

Update:

  • Add paragraphs detailing each origins category
  • Add missing Phabricator origins in listed ones
  • Address @ardumont comment about doi.org netlocs for deposits
  • Use only png logos to simplify their integration
  • Resize some large logos
  • Remove no longer used logos

Build has FAILED

Patch application report for D6004 (id=21698)

Rebasing onto c5d441b1a8...

Current branch diff-target is up to date.
Changes applied before test
commit fde0a120045513ff20bbac53eb73c42c1a24d340
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Jul 16 18:09:30 2021 +0200

    misc/coverage: Revamp archive coverage widget (WIP)

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/973/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/973/console

Build is green

Patch application report for D6004 (id=21698)

Rebasing onto c5d441b1a8...

Current branch diff-target is up to date.
Changes applied before test
commit fde0a120045513ff20bbac53eb73c42c1a24d340
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Jul 16 18:09:30 2021 +0200

    misc/coverage: Revamp archive coverage widget (WIP)

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/974/ for more details.

Build is green

Patch application report for D6004 (id=21708)

Rebasing onto 914c19e314...

Current branch diff-target is up to date.
Changes applied before test
commit 9322b749a9e9949a993a4eb266306b47cca511ed
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Jul 16 18:09:30 2021 +0200

    misc/coverage: Revamp archive coverage widget (WIP)

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/979/ for more details.

swh/web/misc/coverage.py
92–94

(proposal)

anlambert edited the summary of this revision. (Show Details)

Update:

  • Improve origin category texts
  • Fix code retrieving lister metrics as some of them were dicarded
  • Add display of distribution of origins per origins type, hidden by default, can be displayed by clicking on any collapsible elements (total origin counts)

Screenshots have been updated in task description, still work in progress.

Build is green

Patch application report for D6004 (id=21749)

Rebasing onto 481647f863...

Current branch diff-target is up to date.
Changes applied before test
commit 467ad35e36beb1304746bbb2e727aebb87f04226
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Jul 16 18:09:30 2021 +0200

    misc/coverage: Revamp archive coverage widget (WIP)

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/982/ for more details.

Update: Activate origins search links in collapsible elements.

Build is green

Patch application report for D6004 (id=21773)

Rebasing onto 481647f863...

Current branch diff-target is up to date.
Changes applied before test
commit bb36bd2fe48d3e69db1b97c7e25da73311bdbb44
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Jul 16 18:09:30 2021 +0200

    misc/coverage: Revamp archive coverage widget (WIP)

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/983/ for more details.

Below is how it looks so far:

It looks great, I love it!

swh/web/misc/coverage.py
204–205

There should be a fallback when this fails (eg. because the scheduler isn't available, or not configured in the testing env)

270

crashes with KeyError: 'instances' in an empty docker instance.

swh/web/templates/misc/coverage.html
65