Rebase

Test failures are not related to these changes, need to investigate.

In D7073#184077, @swh-public-ci wrote:

Build has FAILED

Patch application report for D7073 (id=25665)

Rebasing onto f990d789aa...

Current branch diff-target is up to date.

Changes applied before test

commit ca9ef5d15a350f9f609859e22e335798a4383d96
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Feb 3 13:47:43 2022 +0100

    auth: Add UserMailmap model and edition endpoints
    
    Add a new database table to store mailmap settings that will be
    used to add display names for an authors that can be found in the
    archive.
    
    Add endpoints to add or update these mailmap settings restricted
    to staff users or users with special permission.
    They will be used in the upcoming Web UI to manage mailmaps.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1314/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1314/console

Remove endpoints access to staff users.

Only the deposit clients mapping is missing.

The existing swh.web.admin.list_deposits role seems to correspond to the feature described in the task.

I found it back not on the swh-deposit role but on the swh-web role... that's why i got lost.

TTBOMK, it was instrospecting the connected user's keycloak role... but now i'm confused
when i read the code for that part. [1]

[1] https://forge.softwareheritage.org/source/swh-web/browse/master/swh/web/admin/deposit.py$19-30

[2] both staging and prod swh-web db

Looks good to me.

In T3870#78042, @vlorentz wrote:

seems to still happen on 1.1.1 https://sentry.softwareheritage.org/share/issue/d6330b7c048249d99342e4c9d56dce19/

Fixed and deployed.

Fixed and deployed.

Below are the results of the loading of the subversion origins listed above, I used a filtered version of the webapp database on staging.

13:36 $ sqlite3 swh/web/settings/db.sqlite3 
SQLite version 3.34.1 2021-01-20 14:10:07
Enter ".help" for usage hints.
sqlite> select visit_status, count(*) from save_origin_request group by visit_status;
failed|18
full|76
partial|83

So 76 origins that could not be fully loaded with swh-loader-svn v1.0.0 are now correctly processed with v1.1.0.

Rebase

Rebase

Build passed on Jenkins, great ! Abandoning this, will land it later.

swh-loader-svn v1.1.0 has been released, new tests have been scheduled on staging (T3875#77706).

Subversion origins that failed to load or with a partial visit status when using swh-loader-svn v1.0.0 (T3870#77580)
have been resubmitted on staging to check how many can now be fully loaded using swh-loader-svn v1.1.0.

In T3870#77435, @swh-sentry-integration wrote:

Sentry issue: SWH-LOADER-SVN-4Z

Slightly rephrase comment

Update: Fix recursive external detection on incremental loading.

All loading tasks submitted on staging last week got executed. Below are the results after updating the staging webapp database in my local environment:

(swh) ✔ ~/swh/swh-environment/swh-web [master|⚑ 106] 
11:10 $ sqlite3 swh/web/settings/db.sqlite3 
SQLite version 3.34.1 2021-01-20 14:10:07
Enter ".help" for usage hints.
sqlite> select visit_status, count(*) from save_origin_request group by visit_status;
failed|26
full|711
not_found|3
partial|153

In T3870#77439, @swh-sentry-integration wrote:

Sentry issue: SWH-LOADER-SVN-55

In T3870#77435, @swh-sentry-integration wrote:

Sentry issue: SWH-LOADER-SVN-4Z

In T3870#77434, @swh-sentry-integration wrote:

Sentry issue: SWH-LOADER-SVN-57

In T3870#77482, @swh-sentry-integration wrote:

Sentry issue: SWH-LOADER-SVN-2Y

In T3870#77432, @swh-sentry-integration wrote:

Sentry issue: SWH-LOADER-SVN-53

In T3870#77433, @swh-sentry-integration wrote:

Sentry issue: SWH-LOADER-SVN-4Y

Simplify regexp.

In T3870#77431, @swh-sentry-integration wrote:

Sentry issue: SWH-LOADER-SVN-50