Page MenuHomeSoftware Heritage

Git loaderFolder
ActivePublic

Members

  • This project does not have any members.

Watchers

  • This project does not have any watchers.

Details

Recent Activity

Jan 22 2020

olasd added a comment to T2242: GitHub loading optimization: skip repos with old enough updated_at/pushed_at timestamps.

I agree that this may be a useful optimization for some upstreams where getting the state of the remote repository is expensive.

Jan 22 2020, 1:25 PM · Git loader

Jan 21 2020

zack updated the task description for T2242: GitHub loading optimization: skip repos with old enough updated_at/pushed_at timestamps.
Jan 21 2020, 1:34 PM · Git loader
zack triaged T2242: GitHub loading optimization: skip repos with old enough updated_at/pushed_at timestamps as Normal priority.
Jan 21 2020, 1:33 PM · Git loader
zack created T2242: GitHub loading optimization: skip repos with old enough updated_at/pushed_at timestamps.
Jan 21 2020, 1:33 PM · Git loader

Nov 19 2019

ardumont added a comment to T2094: KeyError: 'content:add' in swh.loader.core.loader.

@douardda fixed that behavior in loader.core D2299

Nov 19 2019, 11:28 AM · Git loader
douardda closed T2094: KeyError: 'content:add' in swh.loader.core.loader as Resolved.

This has been fixed by cb42fea77070

Nov 19 2019, 11:26 AM · Git loader
ardumont added a comment to T2094: KeyError: 'content:add' in swh.loader.core.loader.

Reproduced.

Nov 19 2019, 10:53 AM · Git loader

Nov 15 2019

zack triaged T2094: KeyError: 'content:add' in swh.loader.core.loader as High priority.
Nov 15 2019, 11:23 PM · Git loader
robguinness updated the task description for T2094: KeyError: 'content:add' in swh.loader.core.loader.
Nov 15 2019, 6:36 PM · Git loader
robguinness created T2094: KeyError: 'content:add' in swh.loader.core.loader.
Nov 15 2019, 6:34 PM · Git loader

Nov 5 2019

moranegg added a comment to T2059: Generate (swh) releases from all git tags.

Note that this doesn't solve the question of pulling release notes from e.g. GitHub release pages, which is something that would need to be done by some other component (T17 comes to mind).

Nov 5 2019, 1:35 PM · Git loader
olasd updated the task description for T2059: Generate (swh) releases from all git tags.
Nov 5 2019, 12:00 PM · Git loader
olasd triaged T2059: Generate (swh) releases from all git tags as Normal priority.
Nov 5 2019, 11:58 AM · Git loader

Oct 1 2019

ardumont edited P320 loader errors per loader type: ~/.config/swh/kibana/group-by.yml.
Oct 1 2019, 10:06 AM · Git loader, Mercurial loader, PyPI loader

Sep 30 2019

ardumont added a comment to T1280: git origins: latest failure reports.

To ease the analysis, here is an aggregate of the 09/2019 latest failures:

Sep 30 2019, 7:47 PM · Git loader
ardumont added a comment to T1280: git origins: latest failure reports.

New dashboards with latest errors as of 09/2019 [1]

Sep 30 2019, 6:22 PM · Git loader

Sep 10 2019

olasd closed T1988: Upgrade dulwich on celery workers as Resolved.

I've backported dulwich 0.19.13-1 to our stretch repo, upgraded all workers and they're restarting.

Sep 10 2019, 12:10 PM · System administration, Git loader

Sep 7 2019

ardumont added a comment to T1988: Upgrade dulwich on celery workers .

And nice work on the investigation and the fix within dulwich ;)

Sep 7 2019, 9:41 AM · System administration, Git loader
ardumont added a project to T1988: Upgrade dulwich on celery workers : System administration.
Sep 7 2019, 9:41 AM · System administration, Git loader
anlambert triaged T1988: Upgrade dulwich on celery workers as Normal priority.
Sep 7 2019, 12:35 AM · System administration, Git loader

Sep 6 2019

ardumont closed T1987: loader-git: failure when saving git pack as Resolved.
Sep 6 2019, 9:23 PM · Git loader
ardumont renamed T1987: loader-git: failure when saving git pack from loader-git: failure when trying to save git pack to loader-git: failure when saving git pack.
Sep 6 2019, 9:19 PM · Git loader
ardumont renamed T1987: loader-git: failure when saving git pack from loader-git: failure when trying to save data package to loader-git: failure when trying to save git pack.
Sep 6 2019, 9:18 PM · Git loader
ardumont renamed T1987: loader-git: failure when saving git pack from loader-git failure to loader-git: failure when trying to save data package.
Sep 6 2019, 2:33 PM · Git loader
ardumont updated the task description for T1987: loader-git: failure when saving git pack.
Sep 6 2019, 2:27 PM · Git loader
ardumont changed the status of T1987: loader-git: failure when saving git pack from Open to Work in Progress.
Sep 6 2019, 2:24 PM · Git loader
ardumont triaged T1987: loader-git: failure when saving git pack as High priority.
Sep 6 2019, 2:23 PM · Git loader

May 25 2019

zack closed T917: Git loader: update README for YAML-based syntax as Resolved.

This is done, I've forked off the part about consistently documenting configuration options to T1758.

May 25 2019, 5:15 PM · Git loader, Development documentation
zack updated the task description for T917: Git loader: update README for YAML-based syntax.
May 25 2019, 5:13 PM · Git loader, Development documentation

Feb 5 2019

olasd added a comment to T1514: MemoryError in loader-git.

That's a fairly large repo (as seen with how the content bundles get spread out to limit their size). It looks like it has some large directories (e.g. the .bugs directory looks like it has a lot of entries) so I'm not too surprised.

Feb 5 2019, 3:40 PM · Git loader
douardda triaged T1514: MemoryError in loader-git as Normal priority.
Feb 5 2019, 9:47 AM · Git loader

Jan 21 2019

anlambert added a comment to T1280: git origins: latest failure reports.

Errors of type dulwich.errors.NotGitRepository [1] are likely related to a bug in dulwich regarding redirected repository urls not correctly handled.
A pull request [2] has been submitted to fix that issue.

Jan 21 2019, 5:53 PM · Git loader

Dec 17 2018

ardumont raised the priority of T1219: add tests to git loader from High to Needs Triage.
Dec 17 2018, 1:56 PM · Sprint 2018 12, Git loader
ardumont moved T1219: add tests to git loader from in progress to done on the Sprint 2018 12 board.
Dec 17 2018, 1:56 PM · Sprint 2018 12, Git loader
ardumont closed T1219: add tests to git loader as Resolved.

Up to 85% now.

Dec 17 2018, 1:55 PM · Sprint 2018 12, Git loader
ardumont moved T1219: add tests to git loader from Backlog to in progress on the Sprint 2018 12 board.
Dec 17 2018, 12:02 PM · Sprint 2018 12, Git loader
ardumont added a project to T1219: add tests to git loader: Sprint 2018 12.
Dec 17 2018, 12:01 PM · Sprint 2018 12, Git loader
ardumont changed the status of T1219: add tests to git loader from Open to Work in Progress.
Dec 17 2018, 12:01 PM · Sprint 2018 12, Git loader

Dec 4 2018

vlorentz added a parent task for T1219: add tests to git loader: T1411: reach a minimum of 80% SLOC coverage across all components.
Dec 4 2018, 11:27 AM · Sprint 2018 12, Git loader

Nov 27 2018

zack placed T917: Git loader: update README for YAML-based syntax up for grabs.
Nov 27 2018, 12:17 PM · Git loader, Development documentation
zack added a parent task for T917: Git loader: update README for YAML-based syntax: T1388: Document the configuration system of each component.
Nov 27 2018, 12:17 PM · Git loader, Development documentation

Nov 16 2018

vlorentz added a revision to T1219: add tests to git loader: D665: Run git loader tests on BulkUpdater too..
Nov 16 2018, 12:29 PM · Sprint 2018 12, Git loader

Nov 14 2018

anlambert added a comment to T1280: git origins: latest failure reports.

Errors of type ValueError: year is out of range [1] are related to commit dates that can not be represented using standard datetime.datetime Python object (minyear = 0, maxyear = 9999).
See for instance:

Nov 14 2018, 5:13 PM · Git loader
anlambert added a comment to T1280: git origins: latest failure reports.

All the errors of type psycopg2.IntegrityError: duplicate key value violates unique constraint "content_pkey" [1] are all about sha1 collisions, mainly from repositories
testing the attack uncovered by SHAttered [2]

Nov 14 2018, 3:47 PM · Git loader
anlambert updated the task description for T1339: Handle malformed author and committer dates.
Nov 14 2018, 3:39 PM · Storage manager, Git loader
anlambert triaged T1342: Handle annotated tag with no tagger as Normal priority.
Nov 14 2018, 3:38 PM · Git loader
anlambert added a comment to T1339: Handle malformed author and committer dates.

Indeed, you're right the timezone offset is used to compute a revision identifier so even if its value is incorrect it should be stored anyway.

Nov 14 2018, 1:25 PM · Storage manager, Git loader
zack added a comment to T1339: Handle malformed author and committer dates.

The simplest solution would be to check if the computed timezone offset lies in the adequate bounds [UTC−14:00, UTC+14:00] and set it to 0 if not.

Nov 14 2018, 12:03 PM · Storage manager, Git loader
anlambert triaged T1339: Handle malformed author and committer dates as Normal priority.
Nov 14 2018, 11:57 AM · Storage manager, Git loader

Nov 9 2018

vlorentz reopened T1219: add tests to git loader as "Open".
Nov 9 2018, 5:49 PM · Sprint 2018 12, Git loader