Page MenuHomeSoftware Heritage
Feed Advanced Search

Mon, Jul 27

ardumont closed T2481: Migrate dvcs loader tests code to pytest as Resolved.
Mon, Jul 27, 3:21 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a parent task for T2481: Migrate dvcs loader tests code to pytest: T2221: Development workflow & code quality.
Mon, Jul 27, 3:20 PM · SVN Loader, Mercurial loader, Git loader, Core Loader

Mon, Jul 20

ardumont closed T2483: tests: Make check-snapshot utility test function recursively check targetted object exists, a subtask of T2481: Migrate dvcs loader tests code to pytest, as Resolved.
Mon, Jul 20, 9:17 AM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont closed T2483: tests: Make check-snapshot utility test function recursively check targetted object exists as Resolved.
Mon, Jul 20, 9:17 AM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont closed T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin, a subtask of T2481: Migrate dvcs loader tests code to pytest, as Resolved.
Mon, Jul 20, 9:16 AM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont closed T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin as Resolved.
Mon, Jul 20, 9:16 AM · SVN Loader, Mercurial loader, Git loader, Core Loader

Fri, Jul 17

ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3551: tests: Reuse pytest fixtures from swh.loader.core.
Fri, Jul 17, 12:12 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3550: tests: Reuse pytest fixtures from swh.loader.core.
Fri, Jul 17, 12:04 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3549: tests: Reuse pytest fixtures from swh.loader.core.
Fri, Jul 17, 12:04 PM · SVN Loader, Mercurial loader, Git loader, Core Loader

Thu, Jul 16

ardumont closed T2488: Drop loader.core BaseLoaderTest and BaseLoaderStorageTest, a subtask of T2481: Migrate dvcs loader tests code to pytest, as Resolved.
Thu, Jul 16, 3:18 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont closed T2488: Drop loader.core BaseLoaderTest and BaseLoaderStorageTest as Resolved.
Thu, Jul 16, 3:18 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2488: Drop loader.core BaseLoaderTest and BaseLoaderStorageTest: D3522: tests: Drop no longer used loader base tests code.
Thu, Jul 16, 11:45 AM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3515: pytest_plugin: Avoid fixture client to declare optional dependency.
Thu, Jul 16, 10:30 AM · SVN Loader, Mercurial loader, Git loader, Core Loader

Fri, Jul 10

ardumont added revisions to T2483: tests: Make check-snapshot utility test function recursively check targetted object exists: D3502: check_snapshot: Check existence down to the first level directories, D3503: check_snapshot: Check existence down to contents.
Fri, Jul 10, 7:01 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont removed revisions from T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3503: check_snapshot: Check existence down to contents, D3502: check_snapshot: Check existence down to the first level directories.
Fri, Jul 10, 7:01 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3503: check_snapshot: Check existence down to contents.
Fri, Jul 10, 6:14 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3502: check_snapshot: Check existence down to the first level directories.
Fri, Jul 10, 5:40 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont reopened T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin, a subtask of T2481: Migrate dvcs loader tests code to pytest, as Open.
Fri, Jul 10, 3:44 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont reopened T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin as "Open".

Reopening as i'm still refactoring/cleaning up more modules.

Fri, Jul 10, 3:44 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3495: Expose a pytest_plugin module so other loaders can reuse for tests.
Fri, Jul 10, 3:42 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3492: setup: Do no expose the pytest-plugin, let modules require it.
Fri, Jul 10, 12:11 PM · SVN Loader, Mercurial loader, Git loader, Core Loader

Thu, Jul 9

anlambert triaged T2489: Git origin without smart transfer protocol support cannot be loaded as Normal priority.
Thu, Jul 9, 8:22 PM · Git loader
ardumont added a revision to T2483: tests: Make check-snapshot utility test function recursively check targetted object exists: D3478: loader.core: Make check_snapshot ensure snapshot object references exist.
Thu, Jul 9, 4:17 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3475: setup: Do no expose the pytest-plugin any longer.
Thu, Jul 9, 10:22 AM · SVN Loader, Mercurial loader, Git loader, Core Loader

Wed, Jul 8

ardumont added a revision to T2483: tests: Make check-snapshot utility test function recursively check targetted object exists: D3473: loader.core: check_snapshot: Allow to check using both Snapshot and dict objects.
Wed, Jul 8, 6:40 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2481: Migrate dvcs loader tests code to pytest: D3468: test_init: Use swh_storage fixture.
Wed, Jul 8, 4:52 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont closed T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin, a subtask of T2481: Migrate dvcs loader tests code to pytest, as Resolved.
Wed, Jul 8, 4:51 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont closed T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin as Resolved.
Wed, Jul 8, 4:51 PM · SVN Loader, Mercurial loader, Git loader, Core Loader

Tue, Jul 7

ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3453: tests: Use dedicated storage and scheduler fixtures.
Tue, Jul 7, 6:05 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin: D3452: Move sharable fixtures out of conftest into a dedicated pytest plugin.
Tue, Jul 7, 5:55 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2481: Migrate dvcs loader tests code to pytest: D3449: tests: Rename resources directory to data.
Tue, Jul 7, 4:44 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont closed T2487: Migrate hg loader tests code to pytest, a subtask of T2482: Migrate git loader tests code to pytest, as Resolved.
Tue, Jul 7, 4:38 PM · Git loader
ardumont triaged T2488: Drop loader.core BaseLoaderTest and BaseLoaderStorageTest as Normal priority.
Tue, Jul 7, 4:21 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont closed T2486: Add missing coverage to the check-snapshot utility check function (and drop dead code), a subtask of T2481: Migrate dvcs loader tests code to pytest, as Resolved.
Tue, Jul 7, 2:04 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont closed T2486: Add missing coverage to the check-snapshot utility check function (and drop dead code) as Resolved.
Tue, Jul 7, 2:04 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont renamed T2486: Add missing coverage to the check-snapshot utility check function (and drop dead code) from Add missing coverage to the check-snapshot utility check function (or drop dead code) to Add missing coverage to the check-snapshot utility check function (and drop dead code).
Tue, Jul 7, 2:04 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2486: Add missing coverage to the check-snapshot utility check function (and drop dead code): D3444: loaders.tests: Simplify and add coverage to check-snapshot.
Tue, Jul 7, 1:16 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont renamed T2486: Add missing coverage to the check-snapshot utility check function (and drop dead code) from Add missing coverage to the check-snapshot utility check function to Add missing coverage to the check-snapshot utility check function (or drop dead code).
Tue, Jul 7, 1:09 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a comment to T2486: Add missing coverage to the check-snapshot utility check function (and drop dead code).

In the end, it's more dead code since it's only code we pass into when the storage used is an in-memory instance.
This is no longer the case, tests are now using pg-storage instance.

Tue, Jul 7, 1:09 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont removed a revision from T2481: Migrate dvcs loader tests code to pytest: D3440: loader.hg: Initialize and refactor deprecated tests to use pytest fixture.
Tue, Jul 7, 1:08 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont closed T2482: Migrate git loader tests code to pytest as Resolved.
Tue, Jul 7, 1:08 PM · Git loader
ardumont closed T2482: Migrate git loader tests code to pytest, a subtask of T2481: Migrate dvcs loader tests code to pytest, as Resolved.
Tue, Jul 7, 1:08 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont triaged T2486: Add missing coverage to the check-snapshot utility check function (and drop dead code) as Normal priority.
Tue, Jul 7, 9:46 AM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2481: Migrate dvcs loader tests code to pytest: D3440: loader.hg: Initialize and refactor deprecated tests to use pytest fixture.
Tue, Jul 7, 9:32 AM · SVN Loader, Mercurial loader, Git loader, Core Loader

Mon, Jul 6

ardumont added a revision to T2482: Migrate git loader tests code to pytest: D3439: loader.svn.tests: Reuse helper function from swh.loader.core.
Mon, Jul 6, 6:23 PM · Git loader
ardumont added a revision to T2482: Migrate git loader tests code to pytest: D3438: tests: Reuse loader-core test utility function.
Mon, Jul 6, 6:10 PM · Git loader
ardumont triaged T2484: Move sharable fixtures out of conftest into a dedicated pytest plugin as Normal priority.
Mon, Jul 6, 3:57 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont triaged T2483: tests: Make check-snapshot utility test function recursively check targetted object exists as Normal priority.
Mon, Jul 6, 3:50 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2481: Migrate dvcs loader tests code to pytest: D3433: Move remaining common test utility functions to top-level arborescence.
Mon, Jul 6, 3:34 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2481: Migrate dvcs loader tests code to pytest: D3432: Move common test utility function to the top-level arborescence.
Mon, Jul 6, 3:32 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a revision to T2481: Migrate dvcs loader tests code to pytest: D3431: Define common test helper function.
Mon, Jul 6, 3:31 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added projects to T2481: Migrate dvcs loader tests code to pytest: Core Loader, Git loader, Mercurial loader, SVN Loader.
Mon, Jul 6, 1:57 PM · SVN Loader, Mercurial loader, Git loader, Core Loader
ardumont added a project to T2482: Migrate git loader tests code to pytest: Git loader.
Mon, Jul 6, 1:57 PM · Git loader

Jun 19 2020

olasd updated subscribers of T2459: skip exogenous branches when ingesting github/gitlab git repositories.

The heuristic you're talking about only applies for branches which name starts with refs/. All other branches are passed through unscathed, I think (which is a good thing, because most snapshots we generate as swh don't do refs/).

Jun 19 2020, 5:44 PM · Git loader
zack added a comment to T2459: skip exogenous branches when ingesting github/gitlab git repositories.

as a related data point, the current graph export code applies the following heuristic to decide which outbound edges from snapshot nodes to emit:

  • keep branch names starting with refs/heads/
  • keep branch names starting with refs/tags/
  • drop everything else
Jun 19 2020, 1:35 PM · Git loader
olasd closed T2410: Check and complete the gitorious.org import as Resolved.

We still need to try to ingest the zeq2 repo, but that can be done in a followup task.

Jun 19 2020, 10:20 AM · Git loader, Origin-Gitorious
zack updated the task description for T2459: skip exogenous branches when ingesting github/gitlab git repositories.
Jun 19 2020, 9:55 AM · Git loader
zack triaged T2459: skip exogenous branches when ingesting github/gitlab git repositories as Normal priority.
Jun 19 2020, 9:50 AM · Git loader

May 30 2020

olasd added a comment to T2410: Check and complete the gitorious.org import.

The following repositories failed to import. Their on-disk structure is either completely empty, or only contains refs (no actual git objects stored):

May 30 2020, 12:58 PM · Git loader, Origin-Gitorious

May 29 2020

olasd added a comment to T2410: Check and complete the gitorious.org import.

After the first (naive, I guess) pass, 1470 repositories are still missing.

May 29 2020, 5:16 PM · Git loader, Origin-Gitorious

May 19 2020

olasd changed the status of T2410: Check and complete the gitorious.org import from Open to Work in Progress.

The code for loading git repositories from disk hasn't been run in production in a while, so I've decided to run the imports of the missing repos manually.

May 19 2020, 5:02 PM · Git loader, Origin-Gitorious
olasd added a comment to T2410: Check and complete the gitorious.org import.

We also have a single origin with no full visit:

May 19 2020, 12:07 PM · Git loader, Origin-Gitorious
olasd added a comment to T2410: Check and complete the gitorious.org import.

After dumping all origins starting with https://gitorious.org/ in the archive:

May 19 2020, 12:04 PM · Git loader, Origin-Gitorious
rdicosmo triaged T2410: Check and complete the gitorious.org import as High priority.
May 19 2020, 9:49 AM · Git loader, Origin-Gitorious

Apr 28 2020

ardumont added a comment to T2373: staging: git loader: failure to ingest huge repository (e.g. nixpkgs).

Currently running this again with debug logs...

Apr 28 2020, 1:20 PM · Git loader
ardumont added a comment to T2373: staging: git loader: failure to ingest huge repository (e.g. nixpkgs).

Currently running this again with debug logs...
Thanks for the input.

Apr 28 2020, 12:10 PM · Git loader
olasd added a comment to T2373: staging: git loader: failure to ingest huge repository (e.g. nixpkgs).

Reading this again, and seeing that the workers have 16GB of RAM, there's something weird going on that's not related to the volume of the packfile (which is 2GB max).

Apr 28 2020, 11:53 AM · Git loader
olasd added a comment to T2373: staging: git loader: failure to ingest huge repository (e.g. nixpkgs).

The base logic of the git loader regarding packfiles hasn't really been touched since it was first implemented: it's never been really profiled/optimized with respect to its memory usage; This issue isn't specific to the staging infra, it's only more salient there because the workers have been made with tight constraints.

Apr 28 2020, 11:49 AM · Git loader

Apr 22 2020

ardumont added a comment to T2373: staging: git loader: failure to ingest huge repository (e.g. nixpkgs).

[2] I will add some swap to that node to check if that goes further with it.

Apr 22 2020, 4:31 PM · Git loader
ardumont updated the task description for T2373: staging: git loader: failure to ingest huge repository (e.g. nixpkgs).
Apr 22 2020, 3:37 PM · Git loader
ardumont renamed T2373: staging: git loader: failure to ingest huge repository (e.g. nixpkgs) from staging: loader git: failure to ingest repository to staging: git loader: failure to ingest huge repository (e.g. nixpkgs).
Apr 22 2020, 3:33 PM · Git loader
ardumont triaged T2373: staging: git loader: failure to ingest huge repository (e.g. nixpkgs) as Normal priority.
Apr 22 2020, 3:33 PM · Git loader

Apr 21 2020

zack closed T1195: git loader: fail to ingest our own hello world repository as Resolved.
Apr 21 2020, 11:46 AM · Git loader

Apr 15 2020

ardumont closed D3019: git.loader: fix failing origin visit update step due to uninitialized internal state variables.
Apr 15 2020, 11:49 AM · Git loader
anlambert accepted D3019: git.loader: fix failing origin visit update step due to uninitialized internal state variables.

I got the exact same situation when I updated the mercurial loader to swh-model objects.

Apr 15 2020, 11:42 AM · Git loader
swh-public-ci added a comment to D3019: git.loader: fix failing origin visit update step due to uninitialized internal state variables.

Build is green

Apr 15 2020, 10:27 AM · Git loader
ardumont updated the summary of D3019: git.loader: fix failing origin visit update step due to uninitialized internal state variables.
Apr 15 2020, 10:26 AM · Git loader
ardumont updated the diff for D3019: git.loader: fix failing origin visit update step due to uninitialized internal state variables.

Improve the git commit

Apr 15 2020, 10:25 AM · Git loader
ardumont retitled D3019: git.loader: fix failing origin visit update step due to uninitialized internal state variables from git.loader: Initialize internal state in __init__ to git.loader: fix failing origin visit update step due to uninitialized internal state variables.
Apr 15 2020, 10:24 AM · Git loader
ardumont updated the summary of D3019: git.loader: fix failing origin visit update step due to uninitialized internal state variables.
Apr 15 2020, 10:24 AM · Git loader

Jan 22 2020

olasd added a comment to T2242: GitHub loading optimization: skip repos with old enough updated_at/pushed_at timestamps.

I agree that this may be a useful optimization for some upstreams where getting the state of the remote repository is expensive.

Jan 22 2020, 1:25 PM · Git loader

Jan 21 2020

zack updated the task description for T2242: GitHub loading optimization: skip repos with old enough updated_at/pushed_at timestamps.
Jan 21 2020, 1:34 PM · Git loader
zack triaged T2242: GitHub loading optimization: skip repos with old enough updated_at/pushed_at timestamps as Normal priority.
Jan 21 2020, 1:33 PM · Git loader
zack created T2242: GitHub loading optimization: skip repos with old enough updated_at/pushed_at timestamps.
Jan 21 2020, 1:33 PM · Git loader

Nov 19 2019

ardumont added a comment to T2094: KeyError: 'content:add' in swh.loader.core.loader.

@douardda fixed that behavior in loader.core D2299

Nov 19 2019, 11:28 AM · Git loader
douardda closed T2094: KeyError: 'content:add' in swh.loader.core.loader as Resolved.

This has been fixed by cb42fea77070

Nov 19 2019, 11:26 AM · Git loader
ardumont added a comment to T2094: KeyError: 'content:add' in swh.loader.core.loader.

Reproduced.

Nov 19 2019, 10:53 AM · Git loader

Nov 15 2019

zack triaged T2094: KeyError: 'content:add' in swh.loader.core.loader as High priority.
Nov 15 2019, 11:23 PM · Git loader
robguinness updated the task description for T2094: KeyError: 'content:add' in swh.loader.core.loader.
Nov 15 2019, 6:36 PM · Git loader
robguinness created T2094: KeyError: 'content:add' in swh.loader.core.loader.
Nov 15 2019, 6:34 PM · Git loader

Nov 5 2019

moranegg added a comment to T2059: Generate (swh) releases from all git tags.

Note that this doesn't solve the question of pulling release notes from e.g. GitHub release pages, which is something that would need to be done by some other component (T17 comes to mind).

Nov 5 2019, 1:35 PM · Git loader
olasd updated the task description for T2059: Generate (swh) releases from all git tags.
Nov 5 2019, 12:00 PM · Git loader
olasd triaged T2059: Generate (swh) releases from all git tags as Normal priority.
Nov 5 2019, 11:58 AM · Git loader

Oct 1 2019

ardumont edited P320 loader errors per loader type: ~/.config/swh/kibana/group-by.yml.
Oct 1 2019, 10:06 AM · Git loader, Mercurial loader, PyPI loader

Sep 30 2019

ardumont added a comment to T1280: git origins: latest failure reports.

To ease the analysis, here is an aggregate of the 09/2019 latest failures:

Sep 30 2019, 7:47 PM · Git loader
ardumont added a comment to T1280: git origins: latest failure reports.

New dashboards with latest errors as of 09/2019 [1]

Sep 30 2019, 6:22 PM · Git loader

Sep 10 2019

olasd closed T1988: Upgrade dulwich on celery workers as Resolved.

I've backported dulwich 0.19.13-1 to our stretch repo, upgraded all workers and they're restarting.

Sep 10 2019, 12:10 PM · System administration, Git loader

Sep 7 2019

ardumont added a comment to T1988: Upgrade dulwich on celery workers .

And nice work on the investigation and the fix within dulwich ;)

Sep 7 2019, 9:41 AM · System administration, Git loader
ardumont added a project to T1988: Upgrade dulwich on celery workers : System administration.
Sep 7 2019, 9:41 AM · System administration, Git loader
anlambert triaged T1988: Upgrade dulwich on celery workers as Normal priority.
Sep 7 2019, 12:35 AM · System administration, Git loader