Page MenuHomeSoftware Heritage

Core LoaderFolder
ActivePublic

Members

  • This project does not have any members.
  • View All

Watchers

  • This project does not have any watchers.
  • View All

Details

Description

loader plumbings used by other loaders

Recent Activity

Thu, Aug 4

ardumont changed the status of T4266: prod: Deploy swh-loader-core 3.5.0 on staging from Invalid to Resolved.
Thu, Aug 4, 4:10 PM · System administration, Core Loader

Jun 23 2022

vlorentz closed T1461: Add loader-related metrics to swh-loader-core as Resolved.

Closing this, as these metrics are now visible on https://grafana.softwareheritage.org/d/FqGC4zu7z/vlorentz-loader-metrics

Jun 23 2022, 10:59 AM · Core Loader, Metrics/monitoring

May 30 2022

vlorentz added a parent task for T3273: Use "fork" relationships to speed-up initial load of large repositories: T4283: Load https://github.com/chromium/chromium with a higher packfile size limit.
May 30 2022, 3:41 PM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader

May 23 2022

vlorentz closed T4266: prod: Deploy swh-loader-core 3.5.0 on staging as Invalid.
May 23 2022, 2:29 PM · System administration, Core Loader
vlorentz added a comment to T4266: prod: Deploy swh-loader-core 3.5.0 on staging.

oh you're right; this time status is undefined, not success

May 23 2022, 2:29 PM · System administration, Core Loader
ardumont moved T4266: prod: Deploy swh-loader-core 3.5.0 on staging from Backlog to Weekly backlog on the System administration board.
May 23 2022, 2:24 PM · System administration, Core Loader
ardumont added a comment to T4266: prod: Deploy swh-loader-core 3.5.0 on staging.

3.4.1 is deployed everywhere:

root@pergamon:~# clush -b -w @swh-workers -w @swh-workers-large -w @staging-workers 'dpkg -l python3-swh.loader.core'
---------------
worker[01-18],worker[0-3].internal.staging.swh.network (22)
---------------
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                    Version              Architecture Description
+++-=======================-====================-============-=================================
ii  python3-swh.loader.core 3.4.1-1~swh1~bpo10+1 all          Software Heritage Loader Core
May 23 2022, 2:24 PM · System administration, Core Loader
vlorentz triaged T4266: prod: Deploy swh-loader-core 3.5.0 on staging as Normal priority.
May 23 2022, 2:00 PM · System administration, Core Loader

May 13 2022

vlorentz added a revision to T3273: Use "fork" relationships to speed-up initial load of large repositories: D7831: Use all base snapshots in determine_wants().
May 13 2022, 3:23 PM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
ardumont closed T4238: Deploy latest loaders version (> v3.4) as Resolved.
May 13 2022, 11:47 AM · System administration, Core Loader
ardumont moved T4238: Deploy latest loaders version (> v3.4) from in-progress to deployed/landed/monitoring on the System administration board.
May 13 2022, 11:47 AM · System administration, Core Loader
ardumont updated the task description for T4238: Deploy latest loaders version (> v3.4).
May 13 2022, 11:47 AM · System administration, Core Loader
ardumont updated the task description for T4238: Deploy latest loaders version (> v3.4).
May 13 2022, 11:35 AM · System administration, Core Loader
ardumont updated the task description for T4238: Deploy latest loaders version (> v3.4).
May 13 2022, 10:42 AM · System administration, Core Loader
ardumont moved T4238: Deploy latest loaders version (> v3.4) from Weekly backlog to in-progress on the System administration board.
May 13 2022, 10:31 AM · System administration, Core Loader
ardumont updated the task description for T4238: Deploy latest loaders version (> v3.4).
May 13 2022, 10:23 AM · System administration, Core Loader
ardumont updated the task description for T4238: Deploy latest loaders version (> v3.4).
May 13 2022, 10:23 AM · System administration, Core Loader
ardumont moved T4238: Deploy latest loaders version (> v3.4) from Backlog to Weekly backlog on the System administration board.
May 13 2022, 10:19 AM · System administration, Core Loader
ardumont triaged T4238: Deploy latest loaders version (> v3.4) as High priority.
May 13 2022, 10:17 AM · System administration, Core Loader

May 10 2022

vlorentz added a comment to T3273: Use "fork" relationships to speed-up initial load of large repositories.

Currently can't do it on GitLab while logged out: https://gitlab.com/gitlab-org/gitlab/-/issues/361952

May 10 2022, 4:13 PM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader

May 3 2022

vlorentz triaged T4201: Support empty directories as Normal priority.

Yes; and they should be already supported in SWH (if some places don't, then it's a bug)

May 3 2022, 4:44 PM · Core Loader, BZR loader
vlorentz removed a subtask for T3273: Use "fork" relationships to speed-up initial load of large repositories: T2202: Collect extrinsic metadata.
May 3 2022, 11:16 AM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
vlorentz added subtasks for T3273: Use "fork" relationships to speed-up initial load of large repositories: T1740: fetch extrinsic origin metadata from GitHub, T2202: Collect extrinsic metadata.
May 3 2022, 11:16 AM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
vlorentz added a subtask for T3273: Use "fork" relationships to speed-up initial load of large repositories: T4219: Investigate why GitHub fork detection did not bring a speed-up.
May 3 2022, 11:15 AM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
vlorentz closed T4187: Pass forge type to loaders, a subtask of T4186: Allow loaders to fetch extrinsic metadata, as Resolved.
May 3 2022, 11:08 AM · Core Loader, Metadata workflow
vlorentz closed T4187: Pass forge type to loaders as Resolved.
May 3 2022, 11:08 AM · Core Loader, Metadata workflow
vlorentz closed T4186: Allow loaders to fetch extrinsic metadata as Resolved.
May 3 2022, 11:07 AM · Core Loader, Metadata workflow
vlorentz closed T4188: Make swh-loader-core run metadata fetchers before loading an origin, a subtask of T4186: Allow loaders to fetch extrinsic metadata, as Resolved.
May 3 2022, 11:07 AM · Core Loader, Metadata workflow
vlorentz closed T4188: Make swh-loader-core run metadata fetchers before loading an origin as Resolved.
May 3 2022, 11:07 AM · Core Loader, Metadata workflow

Apr 29 2022

ardumont closed T4206: prod: Deploy metadata loader v0.0.2, a subtask of T3273: Use "fork" relationships to speed-up initial load of large repositories, as Resolved.
Apr 29 2022, 11:27 AM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
ardumont closed T4204: prod: Deploy swh-scheduler 1.1.1, a subtask of T4186: Allow loaders to fetch extrinsic metadata, as Resolved.
Apr 29 2022, 11:27 AM · Core Loader, Metadata workflow
ardumont closed T4204: prod: Deploy swh-scheduler 1.1.1 as Resolved.
Apr 29 2022, 11:27 AM · System administration, Core Loader
ardumont closed T4205: prod: Deploy swh-loader-core 3.2.1, a subtask of T4204: prod: Deploy swh-scheduler 1.1.1, as Resolved.
Apr 29 2022, 11:27 AM · System administration, Core Loader
ardumont closed T4205: prod: Deploy swh-loader-core 3.2.1 as Resolved.
Apr 29 2022, 11:27 AM · System administration, Core Loader

Apr 28 2022

ardumont moved T4204: prod: Deploy swh-scheduler 1.1.1 from in-progress to deployed/landed/monitoring on the System administration board.
Apr 28 2022, 4:47 PM · System administration, Core Loader
ardumont moved T4205: prod: Deploy swh-loader-core 3.2.1 from in-progress to deployed/landed/monitoring on the System administration board.
Apr 28 2022, 4:47 PM · System administration, Core Loader
ardumont changed the status of T4206: prod: Deploy metadata loader v0.0.2, a subtask of T3273: Use "fork" relationships to speed-up initial load of large repositories, from Open to Work in Progress.
Apr 28 2022, 3:43 PM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
ardumont changed the status of T4205: prod: Deploy swh-loader-core 3.2.1 from Open to Work in Progress.
Apr 28 2022, 3:43 PM · System administration, Core Loader
ardumont changed the status of T4205: prod: Deploy swh-loader-core 3.2.1, a subtask of T4204: prod: Deploy swh-scheduler 1.1.1, from Open to Work in Progress.
Apr 28 2022, 3:43 PM · System administration, Core Loader
ardumont changed the status of T4204: prod: Deploy swh-scheduler 1.1.1, a subtask of T4186: Allow loaders to fetch extrinsic metadata, from Open to Work in Progress.
Apr 28 2022, 3:43 PM · Core Loader, Metadata workflow
ardumont changed the status of T4204: prod: Deploy swh-scheduler 1.1.1 from Open to Work in Progress.
Apr 28 2022, 3:43 PM · System administration, Core Loader
vlorentz edited projects for T3273: Use "fork" relationships to speed-up initial load of large repositories, added: Origin-GitHub; removed GitHub lister.
Apr 28 2022, 3:27 PM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
vlorentz edited projects for T3273: Use "fork" relationships to speed-up initial load of large repositories, added: Origin-GitLab; removed GitLab migration.
Apr 28 2022, 3:27 PM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
vlorentz added projects to T3273: Use "fork" relationships to speed-up initial load of large repositories: GitHub lister, GitLab migration.
Apr 28 2022, 3:27 PM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
vlorentz added a project to T3273: Use "fork" relationships to speed-up initial load of large repositories: Git loader.
Apr 28 2022, 3:26 PM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
vlorentz added a subtask for T3273: Use "fork" relationships to speed-up initial load of large repositories: T4206: prod: Deploy metadata loader v0.0.2.
Apr 28 2022, 3:26 PM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
vlorentz added a parent task for T4204: prod: Deploy swh-scheduler 1.1.1: T4206: prod: Deploy metadata loader v0.0.2.
Apr 28 2022, 3:25 PM · System administration, Core Loader
vlorentz triaged T4205: prod: Deploy swh-loader-core 3.2.1 as Normal priority.
Apr 28 2022, 3:24 PM · System administration, Core Loader
vlorentz reassigned T4204: prod: Deploy swh-scheduler 1.1.1 from vlorentz to ardumont.
Apr 28 2022, 3:23 PM · System administration, Core Loader
vlorentz triaged T4204: prod: Deploy swh-scheduler 1.1.1 as Normal priority.
Apr 28 2022, 3:23 PM · System administration, Core Loader