Page MenuHomeSoftware Heritage

Mercurial loaderFolder
ActivePublic

Members

  • This project does not have any members.

Watchers

  • This project does not have any watchers.

Recent Activity

Nov 1 2018

ardumont added a comment to T998: Mercurial loader sometimes creates snapshots that point to revisions that haven't been loaded.

Might be related to T1156.

Nov 1 2018, 10:06 AM · Mercurial loader

Oct 20 2018

ardumont edited P320 loader errors per loader type: ~/.config/swh/kibana/group-by.yml.
Oct 20 2018, 12:54 PM · Git loader, Mercurial loader, PyPI loader

Oct 19 2018

ardumont edited P320 loader errors per loader type: ~/.config/swh/kibana/group-by.yml.
Oct 19 2018, 10:20 AM · Git loader, Mercurial loader, PyPI loader
ardumont added projects to P320 loader errors per loader type: ~/.config/swh/kibana/group-by.yml: PyPI loader, Mercurial loader, Git loader.
Oct 19 2018, 9:54 AM · Git loader, Mercurial loader, PyPI loader

Oct 15 2018

ardumont created P315 querying mercurial loader errors.
Oct 15 2018, 2:33 PM · Mercurial loader

Oct 10 2018

ardumont added a comment to T954: Add tests to loader-mercurial.

(There is a verifier module which i don't use)

Oct 10 2018, 6:18 PM · Mercurial loader
ardumont closed T954: Add tests to loader-mercurial as Resolved.

And rDLDHG38b6ec659eee

Oct 10 2018, 2:45 PM · Mercurial loader

Oct 4 2018

zack removed a parent task for T329: hg / mercurial loader: T807: dogfooding: ingest the Software Heritage forge into the archive (via the canonical URLs).
Oct 4 2018, 11:51 AM · Mercurial loader

Oct 2 2018

ardumont closed T1189: Use mercurial tags as named pointer (referenced in the snapshot) as Invalid.

As per D409#8432 conclusion

Oct 2 2018, 5:54 PM · Mercurial loader

Oct 1 2018

zack raised the priority of T954: Add tests to loader-mercurial from Normal to High.

cc: @douardda , just because we discussed this today :)

Oct 1 2018, 7:39 PM · Mercurial loader

Sep 12 2018

ardumont triaged T1189: Use mercurial tags as named pointer (referenced in the snapshot) as Normal priority.
Sep 12 2018, 10:02 AM · Mercurial loader

Sep 11 2018

ardumont added a revision to T954: Add tests to loader-mercurial: D409: Use mercurial tags as named pointer (referenced in the snapshot).
Sep 11 2018, 6:24 PM · Mercurial loader

Aug 3 2018

ardumont added a comment to T682: Inject Google Code Mercurial repositories.

First pass have been done complete a while back.

Aug 3 2018, 3:05 PM · Archive coverage, Mercurial loader
ardumont added a subtask for T682: Inject Google Code Mercurial repositories: T1156: Fix release targets of already loaded mercurial type origins.
Aug 3 2018, 3:03 PM · Archive coverage, Mercurial loader
ardumont closed T329: hg / mercurial loader as Resolved.
Aug 3 2018, 3:03 PM · Mercurial loader
ardumont closed T329: hg / mercurial loader, a subtask of T682: Inject Google Code Mercurial repositories, as Resolved.
Aug 3 2018, 3:03 PM · Archive coverage, Mercurial loader

Jul 26 2018

ardumont added a parent task for T1155: Mercurial loader: release target is invalid: T1156: Fix release targets of already loaded mercurial type origins.
Jul 26 2018, 3:17 PM · Mercurial loader

Jul 24 2018

anlambert closed T1155: Mercurial loader: release target is invalid as Resolved.

I forgot to reference the task in commit rDLDHGdb2803207a2934da4665379c12224f9eb90e8995 fixing the issue.

Jul 24 2018, 11:14 AM · Mercurial loader

Jul 19 2018

ardumont added a comment to T1155: Mercurial loader: release target is invalid.
to correct the revisions...
Jul 19 2018, 5:39 PM · Mercurial loader
anlambert added a comment to T1155: Mercurial loader: release target is invalid.

T1156 created for loading the hg origins once again

Jul 19 2018, 4:16 PM · Mercurial loader
zack added a comment to T1155: Mercurial loader: release target is invalid.

Thanks for spotting. We also need a separate task to correct the
revisions that were already loaded in the archive. Can you please file
it? (tag "archive content")

Jul 19 2018, 4:09 PM · Mercurial loader
anlambert triaged T1155: Mercurial loader: release target is invalid as Normal priority.
Jul 19 2018, 2:35 PM · Mercurial loader

Jun 19 2018

zack edited projects for T682: Inject Google Code Mercurial repositories, added: Archive coverage; removed Archive content.
Jun 19 2018, 3:30 PM · Archive coverage, Mercurial loader

Mar 21 2018

ardumont added a comment to T682: Inject Google Code Mercurial repositories.
$ cat ~/.config/swh/kibana/query.yml
indexes:
  - swh_workers-2018.03.*
Mar 21 2018, 3:07 PM · Archive coverage, Mercurial loader
ardumont added a comment to P239 latest mercurial loader issue.

Grunt, we are missing information again.
It was supposed to be fixed.

Mar 21 2018, 2:50 PM · Mercurial loader
ardumont created P239 latest mercurial loader issue.
Mar 21 2018, 2:45 PM · Mercurial loader
ardumont added a comment to T682: Inject Google Code Mercurial repositories.

why no errors reported at all in logs (or logs for that matters..., removing all filters, this seems to stop around the 7th of march 2018)

Mar 21 2018, 2:29 PM · Archive coverage, Mercurial loader
ardumont added a comment to T682: Inject Google Code Mercurial repositories.

Current status, the queue is empty.

Mar 21 2018, 12:31 PM · Archive coverage, Mercurial loader

Mar 16 2018

olasd placed T998: Mercurial loader sometimes creates snapshots that point to revisions that haven't been loaded up for grabs.
Mar 16 2018, 6:03 PM · Mercurial loader
olasd triaged T998: Mercurial loader sometimes creates snapshots that point to revisions that haven't been loaded as High priority.
Mar 16 2018, 6:03 PM · Mercurial loader

Mar 14 2018

ardumont added a comment to T682: Inject Google Code Mercurial repositories.

Finally, rescheduled using swh-scheduler.
Heading towards T986.

Mar 14 2018, 3:25 PM · Archive coverage, Mercurial loader
ardumont added a comment to T682: Inject Google Code Mercurial repositories.

As in https://forge.softwareheritage.org/T879#16396, a limit of 2Gib on dump size was used to separate origins.
The current lists are stored at:

Mar 14 2018, 2:17 PM · Archive coverage, Mercurial loader

Mar 9 2018

ardumont closed T964: 2018-02-16 worker disk full postmortem as Resolved.

Wrapping up:

  • Loaders (swh-worker@swh_loader_{something}.service) now are part of a systemd slice to limit their memory usage (up to 90%). [1]
  • Loaders can now use a /tmp dedicated to their systemd service. That permits, when restarting the service to automatically clean that /tmp. This is activated for svn, mercurial and deposit loaders. [2]
  • Sibling typed loader can clean up amongst themselves (if some are killed and did not have time to finish their job). [3]
  • Relatedly, loaders are now dealing properly with the prepare phase exploding (it did not clean up properly nor update the visit status). [4]
Mar 9 2018, 2:00 PM · Mercurial loader
ardumont closed T964: 2018-02-16 worker disk full postmortem, a subtask of T329: hg / mercurial loader, as Resolved.
Mar 9 2018, 2:00 PM · Mercurial loader
ardumont closed T982: failing worker consumes remaining tasks without processing them, a subtask of T964: 2018-02-16 worker disk full postmortem, as Resolved.
Mar 9 2018, 11:17 AM · Mercurial loader

Mar 7 2018

ardumont closed T985: loader*: Make prepare method resilient to error and origin visit status compliant as Resolved.
Mar 7 2018, 12:51 PM · Mercurial loader
ardumont closed T985: loader*: Make prepare method resilient to error and origin visit status compliant, a subtask of T964: 2018-02-16 worker disk full postmortem, as Resolved.
Mar 7 2018, 12:51 PM · Mercurial loader
ardumont added a comment to T985: loader*: Make prepare method resilient to error and origin visit status compliant.

The gist of this is:

  • separate a prepare_origin_visit method from prepare method
  • prepare_origin_visit is an adapter method to setup origin/visit data (loader dependent because we don't have the same parameter structure...). This could fail (prod issues) but in extreme cases.
  • prepare is a state dependent on the loader's logic, but independent from the origin preparation (this can fail and that's what this issue is all about).
Mar 7 2018, 12:51 PM · Mercurial loader
ardumont triaged T985: loader*: Make prepare method resilient to error and origin visit status compliant as Normal priority.
Mar 7 2018, 12:45 PM · Mercurial loader

Feb 24 2018

ardumont closed T976: google import: Clean up wrong revisions as Resolved.
Feb 24 2018, 5:32 PM · Archive content, Mercurial loader
ardumont closed T976: google import: Clean up wrong revisions, a subtask of T682: Inject Google Code Mercurial repositories, as Resolved.
Feb 24 2018, 5:32 PM · Archive coverage, Mercurial loader

Feb 23 2018

ardumont changed the status of T976: google import: Clean up wrong revisions from Open to Work in Progress.

Status:

  • [DONE] backup
  • [IN-PROGRESS] Clean up in progress
Feb 23 2018, 10:30 AM · Archive content, Mercurial loader
ardumont changed the status of T976: google import: Clean up wrong revisions, a subtask of T682: Inject Google Code Mercurial repositories, from Open to Work in Progress.
Feb 23 2018, 10:30 AM · Archive coverage, Mercurial loader

Feb 21 2018

ardumont added a comment to T976: google import: Clean up wrong revisions.

Thanks for the heads up.

Feb 21 2018, 6:46 PM · Archive content, Mercurial loader
olasd added a comment to T976: google import: Clean up wrong revisions.

FWIW the backup has now completed.

Feb 21 2018, 4:57 PM · Archive content, Mercurial loader
ardumont added a comment to T329: hg / mercurial loader.
It seems like the biggest problem is/was
Feb 21 2018, 3:52 PM · Mercurial loader
fiendish added a comment to T329: hg / mercurial loader.

Can we associate the name of the temporary storage directory for a load with that loader's pid, and then make every new loader instance compare existing temp storage dirs during init? If a storage directory exists for a process that does not exist (because the process was killed) then it can be deleted.

Feb 21 2018, 3:42 PM · Mercurial loader
fiendish added a comment to T329: hg / mercurial loader.

I worry that RAM is way more constrained than disk space is. It seems like the biggest problem is/was

Feb 21 2018, 3:27 PM · Mercurial loader
ardumont updated subscribers of T329: hg / mercurial loader.

I think 6e12c90b160ad3277a1edea27a05f9adea1bc92f may be a bad idea. Have you tested how much RAM it takes to hold the whole dirs dict in memory on a very large repo like mozilla-unified?

Feb 21 2018, 11:40 AM · Mercurial loader
ardumont added a comment to T964: 2018-02-16 worker disk full postmortem.

If cache files are sticking around, then of course the code should make sure that they go away when done

Feb 21 2018, 11:04 AM · Mercurial loader