Page MenuHomeSoftware Heritage
Feed Advanced Search

Advanced Search
Use Results
Hide Query

Jul 10 2020

ardumont added revisions to T2483: tests: Make check-snapshot utility test function recursively check targetted object exists: D3502: check_snapshot: Check existence down to the first level directories, D3503: check_snapshot: Check existence down to contents.
Jul 10 2020, 7:01 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont removed revisions from T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3503: check_snapshot: Check existence down to contents, D3502: check_snapshot: Check existence down to the first level directories.
Jul 10 2020, 7:01 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3503: check_snapshot: Check existence down to contents.
Jul 10 2020, 6:14 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3502: check_snapshot: Check existence down to the first level directories.
Jul 10 2020, 5:40 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont reopened T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin, a subtask of T2481: Migrate dvcs loader tests code to pytest, as Open.
Jul 10 2020, 3:44 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont reopened T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin as "Open".

Reopening as i'm still refactoring/cleaning up more modules.

Jul 10 2020, 3:44 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3495: Expose a pytest_plugin module so other loaders can reuse for tests.
Jul 10 2020, 3:42 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3492: setup: Do no expose the pytest-plugin, let modules require it.
Jul 10 2020, 12:11 PM · SVN Loader, Mercurial loader, Git loader, Core Loader

Jul 9 2020

ardumont added a revision to T2483: tests: Make check-snapshot utility test function recursively check targetted object exists: D3478: loader.core: Make check_snapshot ensure snapshot object references exist.
Jul 9 2020, 4:17 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3475: setup: Do no expose the pytest-plugin any longer.
Jul 9 2020, 10:22 AM · SVN Loader, Mercurial loader, Git loader, Core Loader

Jul 8 2020

ardumont added a revision to T2483: tests: Make check-snapshot utility test function recursively check targetted object exists: D3473: loader.core: check_snapshot: Allow to check using both Snapshot and dict objects.
Jul 8 2020, 6:40 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2481: Migrate dvcs loader tests code to pytest: D3468: test_init: Use swh_storage fixture.
Jul 8 2020, 4:52 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont closed T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin, a subtask of T2481: Migrate dvcs loader tests code to pytest, as Resolved.
Jul 8 2020, 4:51 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont closed T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin as Resolved.
Jul 8 2020, 4:51 PM · SVN Loader, Mercurial loader, Git loader, Core Loader

Jul 7 2020

ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3453: tests: Use dedicated storage and scheduler fixtures.
Jul 7 2020, 6:05 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3452: Move sharable fixtures out of conftest into a dedicated pytest plugin.
Jul 7 2020, 5:55 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2481: Migrate dvcs loader tests code to pytest: D3449: tests: Rename resources directory to data.
Jul 7 2020, 4:44 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont closed T2487: Migrate hg loader tests code to pytest as Resolved.
Jul 7 2020, 4:38 PM · Mercurial loader
ardumont triaged T2488: Drop loader.core BaseLoaderTest and BaseLoaderStorageTest as Normal priority.
Jul 7 2020, 4:21 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2487: Migrate hg loader tests code to pytest: D3448: mercurial.loader: Refactor remaining tests to use pytest.
Jul 7 2020, 3:58 PM · Mercurial loader
ardumont closed T2486: Add missing coverage to the check-snapshot utility check function (and drop dead code), a subtask of T2481: Migrate dvcs loader tests code to pytest, as Resolved.
Jul 7 2020, 2:04 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont closed T2486: Add missing coverage to the check-snapshot utility check function (and drop dead code) as Resolved.
Jul 7 2020, 2:04 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont renamed T2486: Add missing coverage to the check-snapshot utility check function (and drop dead code) from Add missing coverage to the check-snapshot utility check function (or drop dead code) to Add missing coverage to the check-snapshot utility check function (and drop dead code).
Jul 7 2020, 2:04 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2486: Add missing coverage to the check-snapshot utility check function (and drop dead code): D3444: loaders.tests: Simplify and add coverage to check-snapshot.
Jul 7 2020, 1:16 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont renamed T2486: Add missing coverage to the check-snapshot utility check function (and drop dead code) from Add missing coverage to the check-snapshot utility check function to Add missing coverage to the check-snapshot utility check function (or drop dead code).
Jul 7 2020, 1:09 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a comment to T2486: Add missing coverage to the check-snapshot utility check function (and drop dead code).

In the end, it's more dead code since it's only code we pass into when the storage used is an in-memory instance.
This is no longer the case, tests are now using pg-storage instance.

Jul 7 2020, 1:09 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont removed a revision from T2481: Migrate dvcs loader tests code to pytest: D3440: loader.hg: Initialize and refactor deprecated tests to use pytest fixture.
Jul 7 2020, 1:08 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont closed T2482: Migrate git loader tests code to pytest, a subtask of T2481: Migrate dvcs loader tests code to pytest, as Resolved.
Jul 7 2020, 1:08 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2487: Migrate hg loader tests code to pytest: D3440: loader.hg: Initialize and refactor deprecated tests to use pytest fixture.
Jul 7 2020, 1:07 PM · Mercurial loader
ardumont triaged T2487: Migrate hg loader tests code to pytest as Normal priority.
Jul 7 2020, 1:07 PM · Mercurial loader
ardumont triaged T2486: Add missing coverage to the check-snapshot utility check function (and drop dead code) as Normal priority.
Jul 7 2020, 9:46 AM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2481: Migrate dvcs loader tests code to pytest: D3440: loader.hg: Initialize and refactor deprecated tests to use pytest fixture.
Jul 7 2020, 9:32 AM · SVN Loader, Mercurial loader, Git loader, Core Loader

Jul 6 2020

ardumont triaged T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin as Normal priority.
Jul 6 2020, 3:57 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont triaged T2483: tests: Make check-snapshot utility test function recursively check targetted object exists as Normal priority.
Jul 6 2020, 3:50 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2481: Migrate dvcs loader tests code to pytest: D3433: Move remaining common test utility functions to top-level arborescence.
Jul 6 2020, 3:34 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2481: Migrate dvcs loader tests code to pytest: D3432: Move common test utility function to the top-level arborescence.
Jul 6 2020, 3:32 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2481: Migrate dvcs loader tests code to pytest: D3431: Define common test helper function.
Jul 6 2020, 3:31 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added projects to T2481: Migrate dvcs loader tests code to pytest: Core Loader, Git loader, Mercurial loader, SVN Loader.
Jul 6 2020, 1:57 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
olasd closed T998: Mercurial loader sometimes creates snapshots that point to revisions that haven't been loaded as Resolved.

I expect this has been fixed now...

Jul 6 2020, 1:33 PM · Mercurial loader

May 19 2020

zack renamed T682: Ingest Google Code Mercurial repositories from Inject Google Code Mercurial repositories to Ingest Google Code Mercurial repositories.
May 19 2020, 9:56 AM · Archive coverage, Mercurial loader

Feb 6 2020

ardumont closed T1962: Mercurial loader sometimes hangs forever during hg clone as Resolved.
Feb 6 2020, 7:15 PM · Mercurial loader

Jan 29 2020

ardumont renamed T2211: Go beyond git expressivity from Go beyound git expressivity to Go beyond git expressivity.
Jan 29 2020, 6:43 PM · Mercurial loader, Storage manager, Data Model, Roadmap 2020

Jan 22 2020

vlorentz added projects to T2211: Go beyond git expressivity: Data Model, Storage manager, Mercurial loader.
Jan 22 2020, 4:38 PM · Mercurial loader, Storage manager, Data Model, Roadmap 2020

Oct 1 2019

ardumont edited P320 loader errors per loader type: ~/.config/swh/kibana/group-by.yml.
Oct 1 2019, 10:06 AM · Git loader, Mercurial loader, PyPI loader

Sep 10 2019

anlambert closed T1989: Mercurial visits are always eventful as Resolved by committing rDLDHG17ec90bac2d7: loader: Fix load status when no changes in repository since last visit.
Sep 10 2019, 2:17 PM · Mercurial loader

Sep 7 2019

olasd triaged T1989: Mercurial visits are always eventful as High priority.
Sep 7 2019, 6:25 PM · Mercurial loader

Aug 22 2019

olasd triaged T1962: Mercurial loader sometimes hangs forever during hg clone as High priority.
Aug 22 2019, 1:48 PM · Mercurial loader

Jul 3 2019

ardumont placed T682: Ingest Google Code Mercurial repositories up for grabs.
Jul 3 2019, 3:26 PM · Archive coverage, Mercurial loader

May 23 2019

anlambert closed T1729: Properly handle transplant_source entry in changeset extra as Resolved by committing rDLDHG10a791251e77: loader: Properly handle transplant_source in changeset extra fields.
May 23 2019, 2:27 PM · Mercurial loader

May 21 2019

anlambert updated the task description for T1729: Properly handle transplant_source entry in changeset extra.
May 21 2019, 3:20 PM · Mercurial loader
anlambert triaged T1729: Properly handle transplant_source entry in changeset extra as Normal priority.
May 21 2019, 3:18 PM · Mercurial loader

Feb 16 2019

ardumont added a comment to T906: mercurial loader: Debian package.

Heads up.

Feb 16 2019, 9:57 AM · System administration, Mercurial loader

Nov 1 2018

ardumont added a comment to T998: Mercurial loader sometimes creates snapshots that point to revisions that haven't been loaded.

Might be related to T1156.

Nov 1 2018, 10:06 AM · Mercurial loader

Oct 20 2018

ardumont edited P320 loader errors per loader type: ~/.config/swh/kibana/group-by.yml.
Oct 20 2018, 12:54 PM · Git loader, Mercurial loader, PyPI loader

Oct 19 2018

ardumont edited P320 loader errors per loader type: ~/.config/swh/kibana/group-by.yml.
Oct 19 2018, 10:20 AM · Git loader, Mercurial loader, PyPI loader
ardumont added projects to P320 loader errors per loader type: ~/.config/swh/kibana/group-by.yml: PyPI loader, Mercurial loader, Git loader.
Oct 19 2018, 9:54 AM · Git loader, Mercurial loader, PyPI loader

Oct 15 2018

ardumont created P315 querying mercurial loader errors.
Oct 15 2018, 2:33 PM · Mercurial loader

Oct 10 2018

ardumont added a comment to T954: Add tests to loader-mercurial.

(There is a verifier module which i don't use)

Oct 10 2018, 6:18 PM · Mercurial loader
ardumont closed T954: Add tests to loader-mercurial as Resolved.

And rDLDHG38b6ec659eee

Oct 10 2018, 2:45 PM · Mercurial loader

Oct 4 2018

zack removed a parent task for T329: hg / mercurial loader: T807: dogfooding: ingest the Software Heritage forge into the archive (via the canonical URLs).
Oct 4 2018, 11:51 AM · Mercurial loader

Oct 2 2018

ardumont closed T1189: Use mercurial tags as named pointer (referenced in the snapshot) as Invalid.

As per D409#8432 conclusion

Oct 2 2018, 5:54 PM · Mercurial loader

Oct 1 2018

zack raised the priority of T954: Add tests to loader-mercurial from Normal to High.

cc: @douardda , just because we discussed this today :)

Oct 1 2018, 7:39 PM · Mercurial loader

Sep 12 2018

ardumont triaged T1189: Use mercurial tags as named pointer (referenced in the snapshot) as Normal priority.
Sep 12 2018, 10:02 AM · Mercurial loader

Sep 11 2018

ardumont added a revision to T954: Add tests to loader-mercurial: D409: Use mercurial tags as named pointer (referenced in the snapshot).
Sep 11 2018, 6:24 PM · Mercurial loader

Aug 3 2018

ardumont added a comment to T682: Ingest Google Code Mercurial repositories.

First pass have been done complete a while back.

Aug 3 2018, 3:05 PM · Archive coverage, Mercurial loader
ardumont added a subtask for T682: Ingest Google Code Mercurial repositories: T1156: Fix release targets of already loaded mercurial type origins.
Aug 3 2018, 3:03 PM · Archive coverage, Mercurial loader
ardumont closed T329: hg / mercurial loader as Resolved.
Aug 3 2018, 3:03 PM · Mercurial loader
ardumont closed T329: hg / mercurial loader, a subtask of T682: Ingest Google Code Mercurial repositories, as Resolved.
Aug 3 2018, 3:03 PM · Archive coverage, Mercurial loader

Jul 26 2018

ardumont added a parent task for T1155: Mercurial loader: release target is invalid: T1156: Fix release targets of already loaded mercurial type origins.
Jul 26 2018, 3:17 PM · Mercurial loader

Jul 24 2018

anlambert closed T1155: Mercurial loader: release target is invalid as Resolved.

I forgot to reference the task in commit rDLDHGdb2803207a2934da4665379c12224f9eb90e8995 fixing the issue.

Jul 24 2018, 11:14 AM · Mercurial loader

Jul 19 2018

ardumont added a comment to T1155: Mercurial loader: release target is invalid.

to correct the revisions...

Jul 19 2018, 5:39 PM · Mercurial loader
anlambert added a comment to T1155: Mercurial loader: release target is invalid.

T1156 created for loading the hg origins once again

Jul 19 2018, 4:16 PM · Mercurial loader
zack added a comment to T1155: Mercurial loader: release target is invalid.

Thanks for spotting. We also need a separate task to correct the
revisions that were already loaded in the archive. Can you please file
it? (tag "archive content")

Jul 19 2018, 4:09 PM · Mercurial loader
anlambert triaged T1155: Mercurial loader: release target is invalid as Normal priority.
Jul 19 2018, 2:35 PM · Mercurial loader

Jun 19 2018

zack edited projects for T682: Ingest Google Code Mercurial repositories, added: Archive coverage; removed Archive content.
Jun 19 2018, 3:30 PM · Archive coverage, Mercurial loader

Mar 21 2018

ardumont added a comment to T682: Ingest Google Code Mercurial repositories.
$ cat ~/.config/swh/kibana/query.yml
indexes:
  - swh_workers-2018.03.*
Mar 21 2018, 3:07 PM · Archive coverage, Mercurial loader
ardumont added a comment to P239 latest mercurial loader issue.

Grunt, we are missing information again.
It was supposed to be fixed.

Mar 21 2018, 2:50 PM · Mercurial loader
ardumont created P239 latest mercurial loader issue.
Mar 21 2018, 2:45 PM · Mercurial loader
ardumont added a comment to T682: Ingest Google Code Mercurial repositories.

why no errors reported at all in logs (or logs for that matters..., removing all filters, this seems to stop around the 7th of march 2018)

Mar 21 2018, 2:29 PM · Archive coverage, Mercurial loader
ardumont added a comment to T682: Ingest Google Code Mercurial repositories.

Current status, the queue is empty.

Mar 21 2018, 12:31 PM · Archive coverage, Mercurial loader

Mar 16 2018

olasd placed T998: Mercurial loader sometimes creates snapshots that point to revisions that haven't been loaded up for grabs.
Mar 16 2018, 6:03 PM · Mercurial loader
olasd triaged T998: Mercurial loader sometimes creates snapshots that point to revisions that haven't been loaded as High priority.
Mar 16 2018, 6:03 PM · Mercurial loader

Mar 14 2018

ardumont added a comment to T682: Ingest Google Code Mercurial repositories.

Finally, rescheduled using swh-scheduler.
Heading towards T986.

Mar 14 2018, 3:25 PM · Archive coverage, Mercurial loader
ardumont added a comment to T682: Ingest Google Code Mercurial repositories.

As in https://forge.softwareheritage.org/T879#16396, a limit of 2Gib on dump size was used to separate origins.
The current lists are stored at:

Mar 14 2018, 2:17 PM · Archive coverage, Mercurial loader

Mar 9 2018

ardumont closed T964: 2018-02-16 worker disk full postmortem as Resolved.

Wrapping up:

  • Loaders (swh-worker@swh_loader_{something}.service) now are part of a systemd slice to limit their memory usage (up to 90%). [1]
  • Loaders can now use a /tmp dedicated to their systemd service. That permits, when restarting the service to automatically clean that /tmp. This is activated for svn, mercurial and deposit loaders. [2]
  • Sibling typed loader can clean up amongst themselves (if some are killed and did not have time to finish their job). [3]
  • Relatedly, loaders are now dealing properly with the prepare phase exploding (it did not clean up properly nor update the visit status). [4]
Mar 9 2018, 2:00 PM · Mercurial loader
ardumont closed T964: 2018-02-16 worker disk full postmortem, a subtask of T329: hg / mercurial loader, as Resolved.
Mar 9 2018, 2:00 PM · Mercurial loader
ardumont closed T982: failing worker consumes remaining tasks without processing them, a subtask of T964: 2018-02-16 worker disk full postmortem, as Resolved.
Mar 9 2018, 11:17 AM · Mercurial loader

Mar 7 2018

ardumont closed T985: loader*: Make prepare method resilient to error and origin visit status compliant as Resolved.
Mar 7 2018, 12:51 PM · Mercurial loader
ardumont closed T985: loader*: Make prepare method resilient to error and origin visit status compliant, a subtask of T964: 2018-02-16 worker disk full postmortem, as Resolved.
Mar 7 2018, 12:51 PM · Mercurial loader
ardumont added a comment to T985: loader*: Make prepare method resilient to error and origin visit status compliant.

The gist of this is:

  • separate a prepare_origin_visit method from prepare method
  • prepare_origin_visit is an adapter method to setup origin/visit data (loader dependent because we don't have the same parameter structure...). This could fail (prod issues) but in extreme cases.
  • prepare is a state dependent on the loader's logic, but independent from the origin preparation (this can fail and that's what this issue is all about).
Mar 7 2018, 12:51 PM · Mercurial loader
ardumont triaged T985: loader*: Make prepare method resilient to error and origin visit status compliant as Normal priority.
Mar 7 2018, 12:45 PM · Mercurial loader

Feb 24 2018

ardumont closed T976: google import: Clean up wrong revisions as Resolved.
Feb 24 2018, 5:32 PM · Archive content, Mercurial loader
ardumont closed T976: google import: Clean up wrong revisions, a subtask of T682: Ingest Google Code Mercurial repositories, as Resolved.
Feb 24 2018, 5:32 PM · Archive coverage, Mercurial loader

Feb 23 2018

ardumont changed the status of T976: google import: Clean up wrong revisions from Open to Work in Progress.

Status:

  • [DONE] backup
  • [IN-PROGRESS] Clean up in progress
Feb 23 2018, 10:30 AM · Archive content, Mercurial loader
ardumont changed the status of T976: google import: Clean up wrong revisions, a subtask of T682: Ingest Google Code Mercurial repositories, from Open to Work in Progress.
Feb 23 2018, 10:30 AM · Archive coverage, Mercurial loader

Feb 21 2018

ardumont added a comment to T976: google import: Clean up wrong revisions.

Thanks for the heads up.

Feb 21 2018, 6:46 PM · Archive content, Mercurial loader
olasd added a comment to T976: google import: Clean up wrong revisions.

FWIW the backup has now completed.

Feb 21 2018, 4:57 PM · Archive content, Mercurial loader
ardumont added a comment to T329: hg / mercurial loader.

It seems like the biggest problem is/was

Feb 21 2018, 3:52 PM · Mercurial loader
fiendish added a comment to T329: hg / mercurial loader.

Can we associate the name of the temporary storage directory for a load with that loader's pid, and then make every new loader instance compare existing temp storage dirs during init? If a storage directory exists for a process that does not exist (because the process was killed) then it can be deleted.

Feb 21 2018, 3:42 PM · Mercurial loader
fiendish added a comment to T329: hg / mercurial loader.

I worry that RAM is way more constrained than disk space is. It seems like the biggest problem is/was

Feb 21 2018, 3:27 PM · Mercurial loader
First
Prev
Next
About · Developer information · Legal