Page MenuHomeSoftware Heritage
Feed Advanced Search

Jan 21 2022

ardumont closed T923: Mount the asf svn repository mirror as Wontfix.

That probably won't have to happen anymore.
svn monorepository needs to be ingested in a specific way.
A new task will get created for this.

Jan 21 2022, 6:23 PM · SVN Loader
anlambert added a comment to T3870: Analyze svn externals repository ingestion with loader.svn v1.0.

Sentry issue: SWH-LOADER-SVN-55

Jan 21 2022, 6:21 PM · SVN Loader
ardumont added a comment to T466: Test - Ingest XXL svn repository.

Related to T611 (asf repo is a monorepo and it's containing svn:externals properties)
Related to T3839 (we are dealing with large repositories here)

Jan 21 2022, 6:16 PM · SVN Loader
anlambert added a comment to T3870: Analyze svn externals repository ingestion with loader.svn v1.0.

Sentry issue: SWH-LOADER-SVN-4Z

Jan 21 2022, 6:01 PM · SVN Loader
anlambert added a comment to T3870: Analyze svn externals repository ingestion with loader.svn v1.0.

Sentry issue: SWH-LOADER-SVN-57

Jan 21 2022, 5:53 PM · SVN Loader
anlambert added a comment to T3870: Analyze svn externals repository ingestion with loader.svn v1.0.

Sentry issue: SWH-LOADER-SVN-2Y

Jan 21 2022, 5:47 PM · SVN Loader
anlambert added a revision to T3870: Analyze svn externals repository ingestion with loader.svn v1.0: D7019: replay: Fix detection of external paths to remove.
Jan 21 2022, 5:44 PM · SVN Loader
swh-sentry-integration added a comment to T3870: Analyze svn externals repository ingestion with loader.svn v1.0.

Sentry issue: SWH-LOADER-SVN-2Y

Jan 21 2022, 5:23 PM · SVN Loader
anlambert added a comment to T3870: Analyze svn externals repository ingestion with loader.svn v1.0.

Sentry issue: SWH-LOADER-SVN-53

Jan 21 2022, 5:06 PM · SVN Loader
anlambert added a comment to T3870: Analyze svn externals repository ingestion with loader.svn v1.0.

Sentry issue: SWH-LOADER-SVN-4Y

Jan 21 2022, 4:44 PM · SVN Loader
anlambert added a revision to T3870: Analyze svn externals repository ingestion with loader.svn v1.0: D7015: replay: Use peg_rev parameter of client.export instead of rev.
Jan 21 2022, 4:42 PM · SVN Loader
anlambert added a comment to T3870: Analyze svn externals repository ingestion with loader.svn v1.0.

Sentry issue: SWH-LOADER-SVN-50

Jan 21 2022, 3:41 PM · SVN Loader
anlambert added a comment to T3870: Analyze svn externals repository ingestion with loader.svn v1.0.

Sentry issue: SWH-LOADER-SVN-58

Jan 21 2022, 3:40 PM · SVN Loader
anlambert added a revision to T3870: Analyze svn externals repository ingestion with loader.svn v1.0: D7013: utils: Handle new edge cases in parse_external_definition.
Jan 21 2022, 3:38 PM · SVN Loader
swh-sentry-integration added a comment to T3870: Analyze svn externals repository ingestion with loader.svn v1.0.

Sentry issue: SWH-LOADER-SVN-55

Jan 21 2022, 3:09 PM · SVN Loader
swh-sentry-integration added a comment to T3870: Analyze svn externals repository ingestion with loader.svn v1.0.

Sentry issue: SWH-LOADER-SVN-4Z

Jan 21 2022, 2:32 PM · SVN Loader
swh-sentry-integration added a comment to T3870: Analyze svn externals repository ingestion with loader.svn v1.0.

Sentry issue: SWH-LOADER-SVN-57

Jan 21 2022, 2:30 PM · SVN Loader
swh-sentry-integration added a comment to T3870: Analyze svn externals repository ingestion with loader.svn v1.0.

Sentry issue: SWH-LOADER-SVN-4Y

Jan 21 2022, 2:30 PM · SVN Loader
swh-sentry-integration added a comment to T3870: Analyze svn externals repository ingestion with loader.svn v1.0.

Sentry issue: SWH-LOADER-SVN-53

Jan 21 2022, 2:29 PM · SVN Loader
swh-sentry-integration added a comment to T3870: Analyze svn externals repository ingestion with loader.svn v1.0.

Sentry issue: SWH-LOADER-SVN-50

Jan 21 2022, 2:29 PM · SVN Loader
swh-sentry-integration added a comment to T3870: Analyze svn externals repository ingestion with loader.svn v1.0.

Sentry issue: SWH-LOADER-SVN-58

Jan 21 2022, 2:28 PM · SVN Loader
anlambert updated the task description for T3870: Analyze svn externals repository ingestion with loader.svn v1.0.
Jan 21 2022, 2:21 PM · SVN Loader

Jan 20 2022

ardumont updated the task description for T3864: staging: Deploy swh.loader.svn v1.0.0.
Jan 20 2022, 3:04 PM · System administration, SVN Loader
ardumont updated the task description for T3864: staging: Deploy swh.loader.svn v1.0.0.
Jan 20 2022, 3:02 PM · System administration, SVN Loader
ardumont removed a project from T3870: Analyze svn externals repository ingestion with loader.svn v1.0: System administration.
Jan 20 2022, 11:52 AM · SVN Loader
ardumont triaged T3870: Analyze svn externals repository ingestion with loader.svn v1.0 as Normal priority.
Jan 20 2022, 11:48 AM · SVN Loader

Jan 19 2022

ardumont updated the task description for T3864: staging: Deploy swh.loader.svn v1.0.0.
Jan 19 2022, 5:45 PM · System administration, SVN Loader
ardumont moved T3864: staging: Deploy swh.loader.svn v1.0.0 from in-progress to deployed/landed/monitoring on the System administration board.
Jan 19 2022, 5:32 PM · System administration, SVN Loader
ardumont changed the status of T3864: staging: Deploy swh.loader.svn v1.0.0 from Open to Work in Progress.
Jan 19 2022, 5:32 PM · System administration, SVN Loader
ardumont updated the task description for T3864: staging: Deploy swh.loader.svn v1.0.0.
Jan 19 2022, 5:31 PM · System administration, SVN Loader
anlambert added a comment to T3864: staging: Deploy swh.loader.svn v1.0.0.

@ardumont deployed swh-loader-svn v1.0.0 on staging and restarted the loader service.

Jan 19 2022, 5:30 PM · System administration, SVN Loader
ardumont renamed T3864: staging: Deploy swh.loader.svn v1.0.0 from Deploy swh.loader.svn v1.0.0 to staging: Deploy swh.loader.svn v1.0.0.
Jan 19 2022, 2:49 PM · System administration, SVN Loader
ardumont triaged T3864: staging: Deploy swh.loader.svn v1.0.0 as Normal priority.
Jan 19 2022, 2:47 PM · System administration, SVN Loader
anlambert added a revision to T611: support for external definitions in the svn/subversion loader: D6977: replay: Prevent removal of external paths overlapping versioned ones.
Jan 19 2022, 2:38 PM · SVN Loader

Jan 14 2022

anlambert added a revision to T3839: Optimize SVN loader performance and memory consumption on large repositories: D6950: ra: Send modified objects only to storage after replaying a revision.
Jan 14 2022, 12:17 PM · SVN Loader

Jan 13 2022

swh-sentry-integration added a comment to T611: support for external definitions in the svn/subversion loader.

Sentry issue: SWH-LOADER-SVN-1Q

Jan 13 2022, 5:13 PM · SVN Loader

Jan 12 2022

anlambert added a revision to T611: support for external definitions in the svn/subversion loader: D6925: ra: Put externals in cache to avoid exporting them again.
Jan 12 2022, 2:57 PM · SVN Loader

Jan 10 2022

anlambert added a revision to T611: support for external definitions in the svn/subversion loader: D6895: ra: Add support for subversion external definitions.
Jan 10 2022, 2:19 PM · SVN Loader
anlambert updated the task description for T3839: Optimize SVN loader performance and memory consumption on large repositories.
Jan 10 2022, 1:05 PM · SVN Loader
anlambert triaged T3839: Optimize SVN loader performance and memory consumption on large repositories as Normal priority.
Jan 10 2022, 12:52 PM · SVN Loader

Dec 16 2021

stsp added a comment to T3692: Stabilize svn loader implementation.

This PDF contains my SVN Loader review report.

Dec 16 2021, 3:58 PM · SVN Loader, Archive coverage

Dec 14 2021

anlambert added a revision to T611: support for external definitions in the svn/subversion loader: D6839: utils: Add a function to parse a subversion external definition.
Dec 14 2021, 5:55 PM · SVN Loader
anlambert added a revision to T611: support for external definitions in the svn/subversion loader: D6838: Small code improvements for the svn loader.
Dec 14 2021, 5:53 PM · SVN Loader
anlambert added a comment to T611: support for external definitions in the svn/subversion loader.

My recent work on fixing the subversion loader issues in production based on sentry reports made me think again about how we could resolve that task.

Dec 14 2021, 3:51 PM · SVN Loader

Dec 1 2021

anlambert added a revision to T3695: Investigate revision reconstruction discrepancy with subversion export: D6722: ra: Rework export of non link file with svn:special property set.
Dec 1 2021, 7:58 PM · SVN Loader

Nov 29 2021

anlambert added a revision to T3695: Investigate revision reconstruction discrepancy with subversion export: D6704: ra: Fix parsing of svn link with multiple lines.
Nov 29 2021, 1:27 PM · SVN Loader
anlambert added a revision to T3695: Investigate revision reconstruction discrepancy with subversion export: D6702: ra: Persist link flag in FileState when replaying revisions.
Nov 29 2021, 11:43 AM · SVN Loader

Nov 26 2021

anlambert added a revision to T3695: Investigate revision reconstruction discrepancy with subversion export: D6698: ra: Persist path executable flag in FileState when replaying revisions.
Nov 26 2021, 5:08 PM · SVN Loader
ardumont reassigned T3747: svn: FileExistsError when creating directory from ardumont to anlambert.
Nov 26 2021, 10:01 AM · SVN Loader
ardumont closed T3747: svn: FileExistsError when creating directory as Resolved.
Nov 26 2021, 10:01 AM · SVN Loader
ardumont added a comment to T3747: svn: FileExistsError when creating directory.

Diff deployed through the v0.10.1

Nov 26 2021, 9:55 AM · SVN Loader

Nov 25 2021

anlambert added a revision to T3747: svn: FileExistsError when creating directory: D6690: svn: Use a temporary directory when checking if history got altered.
Nov 25 2021, 2:33 PM · SVN Loader

Nov 22 2021

vlorentz triaged T3747: svn: FileExistsError when creating directory as Normal priority.
Nov 22 2021, 2:16 PM · SVN Loader
ardumont added a comment to T3695: Investigate revision reconstruction discrepancy with subversion export.

If i'm not mistaken, there has been some more work about this.
This got deployed with the v0.10 package btw.

Nov 22 2021, 10:50 AM · SVN Loader

Nov 18 2021

anlambert added a revision to T3719: loader svn fails to clean up properly disk when it crashes: D6658: loader: Optimize SvnLoaderFromRemoteDump use on stale repository.
Nov 18 2021, 5:29 PM · SVN Loader

Nov 17 2021

ardumont added a comment to T3719: loader svn fails to clean up properly disk when it crashes.

Deployed ^ as part of the v0.9 package.
Let's see if that defiintely fixes that wrong pattern.

Nov 17 2021, 3:42 PM · SVN Loader
anlambert added a revision to T3719: loader svn fails to clean up properly disk when it crashes: D6648: loader: Create temp folder with pattern in SvnLoaderFromRemoteDump.
Nov 17 2021, 2:32 PM · SVN Loader

Nov 15 2021

anlambert added a comment to T3695: Investigate revision reconstruction discrepancy with subversion export.

Since D6608 got landed and deployed to production, revision reconstruction discrepancy is detected earlier and there is already a couple reported on sentry,
so keeping this opened.

Nov 15 2021, 10:55 AM · SVN Loader

Nov 10 2021

ardumont added a revision to T3719: loader svn fails to clean up properly disk when it crashes: D6636: Add types to the svn module.
Nov 10 2021, 3:49 PM · SVN Loader
ardumont added a revision to T3719: loader svn fails to clean up properly disk when it crashes: D6633: tests: Ensure loaders actually cleanup after themselves.
Nov 10 2021, 12:12 PM · SVN Loader
ardumont added a revision to T3719: loader svn fails to clean up properly disk when it crashes: D6632: test: Make sure loader runs use tmp_path fixture.
Nov 10 2021, 10:36 AM · SVN Loader
ardumont added a comment to P1215 temporary/working directories mess with the loader svn somewhere .

so:

/tmp/tmp0tm0182g/swh.loader.svn.2zdkkl_5-1740204/tmpi0q7t29s: mounted svn repository (from a dump)
/tmp/swh.loader.svn.mhx2lwbz-1740204/tmpi0q7t29s: svn loader's repository disk copy growing along the ingestion
Nov 10 2021, 9:54 AM · SVN Loader
ardumont updated the task description for T3719: loader svn fails to clean up properly disk when it crashes.
Nov 10 2021, 9:43 AM · SVN Loader
ardumont created P1215 temporary/working directories mess with the loader svn somewhere .
Nov 10 2021, 9:42 AM · SVN Loader

Nov 9 2021

ardumont added a comment to T3695: Investigate revision reconstruction discrepancy with subversion export.

I've triggered runs on those origins with the patched loader btw.

Nov 9 2021, 7:17 PM · SVN Loader
ardumont added a revision to T3719: loader svn fails to clean up properly disk when it crashes: D6630: Clean up loader, add types and update docstrings.
Nov 9 2021, 6:20 PM · SVN Loader
ardumont added a revision to T3719: loader svn fails to clean up properly disk when it crashes: D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.
Nov 9 2021, 10:40 AM · SVN Loader
ardumont updated the task description for T3719: loader svn fails to clean up properly disk when it crashes.
Nov 9 2021, 10:38 AM · SVN Loader
ardumont renamed T3701: loading some svn origins while ignoring history sometimes raises from loading some svn origin while ignoring history sometimes raises to loading some svn origins while ignoring history sometimes raises.
Nov 9 2021, 10:21 AM · SVN Loader
ardumont closed T3701: loading some svn origins while ignoring history sometimes raises as Resolved.

Diffs landed, packaged and deployed.
Closing.

Nov 9 2021, 10:21 AM · SVN Loader
ardumont closed T3701: loading some svn origins while ignoring history sometimes raises, a subtask of T3694: Investigate svn loading failure, as Resolved.
Nov 9 2021, 10:21 AM · SVN Loader
ardumont closed T3694: Investigate svn loading failure as Resolved.
Nov 9 2021, 9:48 AM · SVN Loader
ardumont added a comment to T3694: Investigate svn loading failure.

@posenato Hello, i was gonna check this morning and ping you to let you know.

Nov 9 2021, 9:47 AM · SVN Loader

Nov 8 2021

posenato added a comment to T3694: Investigate svn loading failure.

It has just finished saving the repository.
It worked perfectly.
Thank you a lot for your support.

Nov 8 2021, 7:03 PM · SVN Loader
ardumont triaged T3719: loader svn fails to clean up properly disk when it crashes as Normal priority.
Nov 8 2021, 6:49 PM · SVN Loader
ardumont added a comment to T3694: Investigate svn loading failure.

All fixes have currently landed, got packaged in v0.7.3.
I've deployed it and triggered a run [1] for the origin (ongoing).

Nov 8 2021, 6:48 PM · SVN Loader
ardumont added a revision to T3701: loading some svn origins while ignoring history sometimes raises: D6621: svn: Modify default loader to use the same as production task.
Nov 8 2021, 6:17 PM · SVN Loader

Nov 4 2021

anlambert added a revision to T3694: Investigate svn loading failure: D6611: loader: Perform a complete reloading when detecting altered history.
Nov 4 2021, 4:48 PM · SVN Loader
anlambert added a revision to T3695: Investigate revision reconstruction discrepancy with subversion export: D6608: loader: Implement post_load to check for revision divergence.
Nov 4 2021, 4:23 PM · SVN Loader

Nov 3 2021

anlambert added a revision to T3701: loading some svn origins while ignoring history sometimes raises: D6602: loader: Fix error when first revision number is not one.
Nov 3 2021, 6:07 PM · SVN Loader
anlambert added a revision to T3695: Investigate revision reconstruction discrepancy with subversion export: D6601: ra: Fix export of non link file with svn:special property set.
Nov 3 2021, 5:32 PM · SVN Loader
anlambert added a comment to T3701: loading some svn origins while ignoring history sometimes raises.

The issue is related to the difference of fetched revision data whether we use the swh.loader.svn.SvnLoader class (fetching revisions
one at a time through a ping pong with the svn server) or the swh.loader.svn.SvnLoaderFromRemoteDump class (fetching all revisions
to a dump file in one operation).

Nov 3 2021, 5:12 PM · SVN Loader
anlambert renamed T3695: Investigate revision reconstruction discrepancy with subversion export from loading some svn origin while ignoring history sometimes raises to Investigate revision reconstruction discrepancy with subversion export.
Nov 3 2021, 4:31 PM · SVN Loader
anlambert added a parent task for T3701: loading some svn origins while ignoring history sometimes raises: T3694: Investigate svn loading failure.
Nov 3 2021, 4:22 PM · SVN Loader
anlambert added a subtask for T3694: Investigate svn loading failure: T3701: loading some svn origins while ignoring history sometimes raises.
Nov 3 2021, 4:22 PM · SVN Loader
anlambert added a comment to T3701: loading some svn origins while ignoring history sometimes raises.

Original comment posted in T3695 by @ardumont

Nov 3 2021, 4:20 PM · SVN Loader
anlambert triaged T3701: loading some svn origins while ignoring history sometimes raises as Normal priority.
Nov 3 2021, 4:20 PM · SVN Loader

Nov 2 2021

anlambert added a revision to T3695: Investigate revision reconstruction discrepancy with subversion export: D6596: ra: Handle unsetting svn:special property on svn links.
Nov 2 2021, 6:15 PM · SVN Loader
anlambert added a revision to T3695: Investigate revision reconstruction discrepancy with subversion export: D6595: ra: Do not process end of lines for svn links.
Nov 2 2021, 6:14 PM · SVN Loader
anlambert added a comment to T3695: Investigate revision reconstruction discrepancy with subversion export.

The loading issue for the repository in T3694 description has been identified: a commit message has been modified
server side between two save code now requests leading to a discrepancy in computed hashes for revisions.

Nov 2 2021, 1:50 PM · SVN Loader

Oct 29 2021

anlambert added a revision to T3695: Investigate revision reconstruction discrepancy with subversion export: D6589: ra: Fix edge case in svn:eol-style property handling.
Oct 29 2021, 8:27 PM · SVN Loader
posenato added a comment to T3694: Investigate svn loading failure.

@anlambert, thank you for excellent analysis!
Yes, I made an adjustment of commit logs because there was an error during the release and I discover it very soon, so I preferred to make the release again adjusting the comments instead of to make a new release!

Oct 29 2021, 4:53 PM · SVN Loader
anlambert added a comment to T3694: Investigate svn loading failure.

@posenato, I did some deeper analysis on why your svn repository could not be loaded anymore into the archive
and it turned out @ardumont is right, the commit log of your repository got altered between two save code now requests.

Oct 29 2021, 3:41 PM · SVN Loader
posenato added a comment to T3694: Investigate svn loading failure.

Hi,
if you need to have an SVN log from the SVN server, I can ask our sysadmins to have an extract of the log.

Oct 29 2021, 10:26 AM · SVN Loader

Oct 28 2021

ardumont added a revision to T3695: Investigate revision reconstruction discrepancy with subversion export: D6576: loader: Rename start_from_scratch parameter to incremental.
Oct 28 2021, 1:06 PM · SVN Loader
ardumont renamed T3695: Investigate revision reconstruction discrepancy with subversion export from loading svn origin while ignoring history raises to loading some svn origin while ignoring history sometimes raises.
Oct 28 2021, 12:49 PM · SVN Loader
ardumont added a comment to T3695: Investigate revision reconstruction discrepancy with subversion export.

Ah, it's not for all origins though...
I tried with other origins which demonstrates the same issue [1] and they did not fail...

Oct 28 2021, 12:48 PM · SVN Loader
ardumont renamed T3695: Investigate revision reconstruction discrepancy with subversion export from loading svn origins from scratch raise to loading svn origin while ignoring history raises.
Oct 28 2021, 10:49 AM · SVN Loader
ardumont added a comment to T3694: Investigate svn loading failure.

For this one, i had a thought about it recently and it should be enough to trigger a
run from scratch (loader has the flag for it which is tested but got never used).

Oct 28 2021, 10:13 AM · SVN Loader
ardumont triaged T3695: Investigate revision reconstruction discrepancy with subversion export as Normal priority.
Oct 28 2021, 10:12 AM · SVN Loader