In T3870#77430, @swh-sentry-integration wrote:

Sentry issue: SWH-LOADER-SVN-58

I think it will be better to add a new visit status, like restricted or unauthorized, what do you think ?

This task has been complete since a while now, closing it.

Is there a reason not to close this task?

Time to switch production database to a new sanitized one.

@ardumont deployed swh-loader-svn v1.0.0 on staging and restarted the loader service.

This is what I have done to sanitize the production database and plug it into our testbed for testing.

So turns out the customizer issue was due to the recent upgrade of the wp-extra-file-types plugin.

In D6962#181131, @vlorentz wrote:

Indeed! I have no idea what "ra" means :)

Rebase

Rebase

Fix wrong rebase

Update: Add missing root path check in DirEditor.add_directory

Currently the subversion loader is the only one that needs that directories diff feature so I think we can keep the implementation as it is at the
moment but I will create a task to implement the directories diff features in swh.model.from_disk.

Rebase

Rebase

Remove not needed pass instruction

lgtm but I'm not sure i understood everything (code wise).

Rebase

Rebase

Rebase

Rebase

Rebase

Update: Preserve symlinks when copying an external tree.

• Improve test when all externals got removed from a path

Update: Handle a couple of edge cases found when testing with real world repositories.

I modified the concerned line in the /srv/data/etc/cron/anacrontab file the following way:

It looks like it is not critical if the both tables above are not present in the WordPress database, publications list is still correctly displayed
while list of authors and tags will appear empty in the teachPress admin dashboard.

There is two tables with the index too large issue, wp_teachpress_authors and wp_teachpress_tags.

So it turns out that we cannot perform any operations on tables whose Index column size too large, sighs ...

Using the --single-transaction=TRUE option of mysqldump helped to get the name of the table with Index column size too large.

Add comment

In D6950#180642, @olasd wrote:

This looks like an impressive speedup, kudos.

Rather than add this logic on the svn loader only, we could consider either making swh.model.from_disk support incremental computations by keeping track of the ctime / mtime of on-disk data, and by collecting objects for the new loader. This would make this logic reusable by all loaders.

Without changing the swh.model.from_disk logic, we could also just diff the sets of objects between iterations (and rely on the OS cache for the new computation to be vaguely efficient), and only send new ones to the storage.

Rebase

Rebase

Also filter out empty lines and commented ones when parsing external
definitions after a checkout.

Rebase

Rebase

• Use context manager to create temporary directory for checkout
• Update years in test_loader.py license header

Rebase

Optimize subversion export operation when dealing with externals: use the
origin URL as export parameter only if we know that some externals are
defined as relative to the repository URL and targets a path outside the
repository.

Add a boolean in the tuple returned by the parse function indicating
if the external URL was defined as relative to the repository one
bu targets a path outside the repository.

Great, thanks !

This is because the vault modal does not contain a real HTML form so there is no submit event sent when pressing Enter.

Anti-DoS policy has been implemented and deployed. The max_edges thresholds can be easily changed by configuration.

Fixed and deployed.

Rebase and ensure path of external root directory is stored in the external paths set.

Rebase

Rebase

Use urlunparse.

How do the tests check integrity of the results? (ie. there are the right revisions, the right files, etc.)

Does the incremental loader update the remote repo if the svn:externals property was not touched?

It should be faster indeed, looks good to me.

Looks good to me.