Page MenuHomeSoftware Heritage
Feed Advanced Search

May 3 2022

vlorentz removed a subtask for T3273: Use "fork" relationships to speed-up initial load of large repositories: T2202: Collect extrinsic metadata.
May 3 2022, 11:16 AM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
vlorentz added subtasks for T3273: Use "fork" relationships to speed-up initial load of large repositories: T1740: fetch extrinsic origin metadata from GitHub, T2202: Collect extrinsic metadata.
May 3 2022, 11:16 AM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
vlorentz added a subtask for T3273: Use "fork" relationships to speed-up initial load of large repositories: T4219: Investigate why GitHub fork detection did not bring a speed-up.
May 3 2022, 11:15 AM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
vlorentz closed T4187: Pass forge type to loaders, a subtask of T4186: Allow loaders to fetch extrinsic metadata, as Resolved.
May 3 2022, 11:08 AM · Core Loader, Metadata workflow
vlorentz closed T4187: Pass forge type to loaders as Resolved.
May 3 2022, 11:08 AM · Core Loader, Metadata workflow
vlorentz closed T4186: Allow loaders to fetch extrinsic metadata as Resolved.
May 3 2022, 11:07 AM · Core Loader, Metadata workflow
vlorentz closed T4188: Make swh-loader-core run metadata fetchers before loading an origin, a subtask of T4186: Allow loaders to fetch extrinsic metadata, as Resolved.
May 3 2022, 11:07 AM · Core Loader, Metadata workflow
vlorentz closed T4188: Make swh-loader-core run metadata fetchers before loading an origin as Resolved.
May 3 2022, 11:07 AM · Core Loader, Metadata workflow

Apr 29 2022

ardumont closed T4206: prod: Deploy metadata loader v0.0.2, a subtask of T3273: Use "fork" relationships to speed-up initial load of large repositories, as Resolved.
Apr 29 2022, 11:27 AM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
ardumont closed T4204: prod: Deploy swh-scheduler 1.1.1, a subtask of T4186: Allow loaders to fetch extrinsic metadata, as Resolved.
Apr 29 2022, 11:27 AM · Core Loader, Metadata workflow
ardumont closed T4204: prod: Deploy swh-scheduler 1.1.1 as Resolved.
Apr 29 2022, 11:27 AM · System administration, Core Loader
ardumont closed T4205: prod: Deploy swh-loader-core 3.2.1, a subtask of T4204: prod: Deploy swh-scheduler 1.1.1, as Resolved.
Apr 29 2022, 11:27 AM · System administration, Core Loader
ardumont closed T4205: prod: Deploy swh-loader-core 3.2.1 as Resolved.
Apr 29 2022, 11:27 AM · System administration, Core Loader

Apr 28 2022

ardumont moved T4204: prod: Deploy swh-scheduler 1.1.1 from in-progress to deployed/landed/monitoring on the System administration board.
Apr 28 2022, 4:47 PM · System administration, Core Loader
ardumont moved T4205: prod: Deploy swh-loader-core 3.2.1 from in-progress to deployed/landed/monitoring on the System administration board.
Apr 28 2022, 4:47 PM · System administration, Core Loader
ardumont changed the status of T4206: prod: Deploy metadata loader v0.0.2, a subtask of T3273: Use "fork" relationships to speed-up initial load of large repositories, from Open to Work in Progress.
Apr 28 2022, 3:43 PM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
ardumont changed the status of T4205: prod: Deploy swh-loader-core 3.2.1 from Open to Work in Progress.
Apr 28 2022, 3:43 PM · System administration, Core Loader
ardumont changed the status of T4205: prod: Deploy swh-loader-core 3.2.1, a subtask of T4204: prod: Deploy swh-scheduler 1.1.1, from Open to Work in Progress.
Apr 28 2022, 3:43 PM · System administration, Core Loader
ardumont changed the status of T4204: prod: Deploy swh-scheduler 1.1.1, a subtask of T4186: Allow loaders to fetch extrinsic metadata, from Open to Work in Progress.
Apr 28 2022, 3:43 PM · Core Loader, Metadata workflow
ardumont changed the status of T4204: prod: Deploy swh-scheduler 1.1.1 from Open to Work in Progress.
Apr 28 2022, 3:43 PM · System administration, Core Loader
vlorentz edited projects for T3273: Use "fork" relationships to speed-up initial load of large repositories, added: Origin-GitHub; removed GitHub lister.
Apr 28 2022, 3:27 PM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
vlorentz edited projects for T3273: Use "fork" relationships to speed-up initial load of large repositories, added: Origin-GitLab; removed GitLab migration.
Apr 28 2022, 3:27 PM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
vlorentz added projects to T3273: Use "fork" relationships to speed-up initial load of large repositories: GitHub lister, GitLab migration.
Apr 28 2022, 3:27 PM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
vlorentz added a project to T3273: Use "fork" relationships to speed-up initial load of large repositories: Git loader.
Apr 28 2022, 3:26 PM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
vlorentz added a subtask for T3273: Use "fork" relationships to speed-up initial load of large repositories: T4206: prod: Deploy metadata loader v0.0.2.
Apr 28 2022, 3:26 PM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
vlorentz added a parent task for T4204: prod: Deploy swh-scheduler 1.1.1: T4206: prod: Deploy metadata loader v0.0.2.
Apr 28 2022, 3:25 PM · System administration, Core Loader
vlorentz triaged T4205: prod: Deploy swh-loader-core 3.2.1 as Normal priority.
Apr 28 2022, 3:24 PM · System administration, Core Loader
vlorentz reassigned T4204: prod: Deploy swh-scheduler 1.1.1 from vlorentz to ardumont.
Apr 28 2022, 3:23 PM · System administration, Core Loader
vlorentz triaged T4204: prod: Deploy swh-scheduler 1.1.1 as Normal priority.
Apr 28 2022, 3:23 PM · System administration, Core Loader
ardumont edited P1352 4.612300505552681% whose ingestion is roughly done in staging.
Apr 28 2022, 10:32 AM · Core Loader
ardumont created P1352 4.612300505552681% whose ingestion is roughly done in staging.
Apr 28 2022, 10:31 AM · Core Loader

Apr 27 2022

anlambert added a revision to T4187: Pass forge type to loaders: D7704: tasks: Simplify implementation and add tests for listed origins.
Apr 27 2022, 5:11 PM · Core Loader, Metadata workflow
ardumont closed T4194: staging: Deploy swh-scheduler 1.1.0, a subtask of T4187: Pass forge type to loaders, as Resolved.
Apr 27 2022, 4:56 PM · Core Loader, Metadata workflow
Alphare created T4201: Support empty directories.
Apr 27 2022, 4:38 PM · Core Loader, BZR loader
vlorentz added revisions to T4186: Allow loaders to fetch extrinsic metadata: D7688: Reference the new swh.loader.metadata package, D7634: Add swh-loader-metadata package to the CI, D7599: Replace self.url with self.origin.url in package loaders.
Apr 27 2022, 4:14 PM · Core Loader, Metadata workflow
vlorentz added a revision to T4186: Allow loaders to fetch extrinsic metadata: D7701: cli: Pass metadata_fetcher_credentials from the config to the loader.
Apr 27 2022, 4:12 PM · Core Loader, Metadata workflow
anlambert added a revision to T4187: Pass forge type to loaders: D7702: tasks: Simplify implementation and add tests for listed origins.
Apr 27 2022, 4:10 PM · Core Loader, Metadata workflow
anlambert added a revision to T4187: Pass forge type to loaders: D7700: tasks: Simplify implementation and add tests for listed origins.
Apr 27 2022, 3:33 PM · Core Loader, Metadata workflow
anlambert added a revision to T4187: Pass forge type to loaders: D7699: tasks: Simplify implementation and add tests for listed origins.
Apr 27 2022, 3:24 PM · Core Loader, Metadata workflow
vlorentz claimed T3273: Use "fork" relationships to speed-up initial load of large repositories.
Apr 27 2022, 2:12 PM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
vlorentz added revisions to T3273: Use "fork" relationships to speed-up initial load of large repositories: D7691: Store the result of MetadataFetcher.get_parent_origins, D7695: Replace 'base_url' argument with 'self.parent_origins' attribute, D7663: Add method get_parent_origins().
Apr 27 2022, 2:07 PM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
vlorentz added a subtask for T4187: Pass forge type to loaders: T4194: staging: Deploy swh-scheduler 1.1.0.
Apr 27 2022, 11:25 AM · Core Loader, Metadata workflow
anlambert added a revision to T4187: Pass forge type to loaders: D7694: tasks: Simplify implementation and make visit_date parameter optional.
Apr 27 2022, 11:06 AM · Core Loader, Metadata workflow

Apr 26 2022

anlambert added a revision to T4187: Pass forge type to loaders: D7690: tasks: Fix and simplify implementation.
Apr 26 2022, 5:16 PM · Core Loader, Metadata workflow

Apr 22 2022

vlorentz added a revision to T4188: Make swh-loader-core run metadata fetchers before loading an origin: D7632: BaseLoader: Add hook to call metadata fetchers before loading an origin.
Apr 22 2022, 3:52 PM · Core Loader, Metadata workflow
vlorentz renamed T4188: Make swh-loader-core run metadata fetchers before loading an origin from Make swh-loader-core run metadata fetchers after loading an origin to Make swh-loader-core run metadata fetchers before loading an origin.
Apr 22 2022, 3:51 PM · Core Loader, Metadata workflow

Apr 21 2022

vlorentz added a revision to T4187: Pass forge type to loaders: D7628: Add a 'lister_instance_name' argument to all tasks created from ListedOrigin.
Apr 21 2022, 8:37 PM · Core Loader, Metadata workflow
vlorentz added revisions to T4187: Pass forge type to loaders: D7611: cvs: Update for swh.loader.core 3.0.0 and remove initialization boilerplate, D7612: git: Update for swh.loader.core 3.0.0 and remove initialization boilerplate, D7613: hg: Update for swh.loader.core 3.0.0 and remove initialization boilerplate, D7614: bzr: Update for swh.loader.core 3.0.0 and remove initialization boilerplate, D7615: svn: Update for swh.loader.core 3.0.0 and remove initialization boilerplate, D7610: BaseLoader: Add 'origin_url' argument and remove 'prepare_origin_visit' method, D7618: Make create_origin_task_dict a standalone function, D7621: Add a 'lister_name' argument to all tasks created from ListedOrigin.
Apr 21 2022, 12:39 PM · Core Loader, Metadata workflow
vlorentz triaged T4188: Make swh-loader-core run metadata fetchers before loading an origin as Normal priority.
Apr 21 2022, 9:05 AM · Core Loader, Metadata workflow
vlorentz triaged T4187: Pass forge type to loaders as Normal priority.
Apr 21 2022, 9:03 AM · Core Loader, Metadata workflow
vlorentz triaged T4186: Allow loaders to fetch extrinsic metadata as Normal priority.
Apr 21 2022, 9:02 AM · Core Loader, Metadata workflow

Feb 25 2022

ardumont moved T3973: Deploy swh.deposit v0.17 from Backlog to Deployed on the SWORD deposit board.
Feb 25 2022, 6:52 PM · System administration, Core Loader, SWORD deposit
ardumont closed T3973: Deploy swh.deposit v0.17 as Resolved.
Feb 25 2022, 2:16 PM · System administration, Core Loader, SWORD deposit
ardumont moved T3973: Deploy swh.deposit v0.17 from in-progress to deployed/landed/monitoring on the System administration board.
Feb 25 2022, 2:16 PM · System administration, Core Loader, SWORD deposit
ardumont updated the task description for T3973: Deploy swh.deposit v0.17.
Feb 25 2022, 2:16 PM · System administration, Core Loader, SWORD deposit
ardumont added a comment to T3973: Deploy swh.deposit v0.17.

After upgrading pergamon and debugging through sentry and cli, the deposit icinga check is back on track.
Triggered back the icinga checks there.

Feb 25 2022, 2:14 PM · System administration, Core Loader, SWORD deposit
ardumont updated the task description for T3973: Deploy swh.deposit v0.17.
Feb 25 2022, 2:13 PM · System administration, Core Loader, SWORD deposit
ardumont updated the task description for T3973: Deploy swh.deposit v0.17.
Feb 25 2022, 1:09 PM · System administration, Core Loader, SWORD deposit
ardumont updated the task description for T3973: Deploy swh.deposit v0.17.
Feb 25 2022, 12:47 PM · System administration, Core Loader, SWORD deposit
ardumont added a comment to T3973: Deploy swh.deposit v0.17.
  • Fix loader.core debian build [1]
Feb 25 2022, 12:47 PM · System administration, Core Loader, SWORD deposit
ardumont added a revision to T3973: Deploy swh.deposit v0.17: D7256: Ignore quilt .pc/ folder.
Feb 25 2022, 12:38 PM · System administration, Core Loader, SWORD deposit
ardumont added a revision to T3976: opam loader: Adapt for opam > 2.1: D7256: Ignore quilt .pc/ folder.
Feb 25 2022, 12:38 PM · Opam, Package Loader, Core Loader
ardumont updated the task description for T3976: opam loader: Adapt for opam > 2.1.
Feb 25 2022, 12:36 PM · Opam, Package Loader, Core Loader
ardumont added a comment to T3976: opam loader: Adapt for opam > 2.1.
In T3976#79630, @olasd wrote:

https://opam.ocaml.org/doc/FAQ.html#Why-does-opam-require-bwrap

"If needed, for special cases like unprivileged containers, sandboxing can be disabled on opam init with the --disable-sandboxing flag (only for non-initialised opam)".

I think that would make sense, as we never execute code from the opam root, we only read metadata files.

Feb 25 2022, 12:35 PM · Opam, Package Loader, Core Loader
ardumont updated the task description for T3976: opam loader: Adapt for opam > 2.1.
Feb 25 2022, 12:32 PM · Opam, Package Loader, Core Loader
olasd added a comment to T3976: opam loader: Adapt for opam > 2.1.

"If needed, for special cases like unprivileged containers, sandboxing can be disabled on opam init with the --disable-sandboxing flag (only for non-initialised opam)".

Feb 25 2022, 12:23 PM · Opam, Package Loader, Core Loader
ardumont triaged T3976: opam loader: Adapt for opam > 2.1 as Normal priority.
Feb 25 2022, 12:06 PM · Opam, Package Loader, Core Loader
ardumont added a revision to T3973: Deploy swh.deposit v0.17: D7255: loader/opam/tests: Do not run actual opam init command call.
Feb 25 2022, 10:09 AM · System administration, Core Loader, SWORD deposit

Feb 24 2022

ardumont added a revision to T3973: Deploy swh.deposit v0.17: D7248: opam/tests: Allow build to run the opam init step completely.
Feb 24 2022, 3:58 PM · System administration, Core Loader, SWORD deposit
ardumont updated the task description for T3973: Deploy swh.deposit v0.17.
Feb 24 2022, 3:22 PM · System administration, Core Loader, SWORD deposit
ardumont updated the task description for T3973: Deploy swh.deposit v0.17.
Feb 24 2022, 3:15 PM · System administration, Core Loader, SWORD deposit
ardumont added a comment to T3973: Deploy swh.deposit v0.17.
  • Fix deposit debian build
Feb 24 2022, 3:10 PM · System administration, Core Loader, SWORD deposit
ardumont updated the task description for T3973: Deploy swh.deposit v0.17.
Feb 24 2022, 3:09 PM · System administration, Core Loader, SWORD deposit
ardumont updated the task description for T3973: Deploy swh.deposit v0.17.
Feb 24 2022, 2:30 PM · System administration, Core Loader, SWORD deposit
ardumont updated the task description for T3973: Deploy swh.deposit v0.17.
Feb 24 2022, 12:08 PM · System administration, Core Loader, SWORD deposit
ardumont changed the status of T3973: Deploy swh.deposit v0.17 from Open to Work in Progress.
Feb 24 2022, 12:07 PM · System administration, Core Loader, SWORD deposit
ardumont added parent tasks for T3973: Deploy swh.deposit v0.17: T3971: swh.deposit.errors.ParserError: out of memory: line 1, column 0, T3677: Separate origin-source-code and provenance-metadata in the deposit.
Feb 24 2022, 10:20 AM · System administration, Core Loader, SWORD deposit
ardumont added a project to T3973: Deploy swh.deposit v0.17: System administration.
Feb 24 2022, 10:19 AM · System administration, Core Loader, SWORD deposit
ardumont updated the task description for T3973: Deploy swh.deposit v0.17.
Feb 24 2022, 10:19 AM · System administration, Core Loader, SWORD deposit
ardumont triaged T3973: Deploy swh.deposit v0.17 as Normal priority.
Feb 24 2022, 10:13 AM · System administration, Core Loader, SWORD deposit

Jan 18 2022

SupLinux added a comment to T3851: Investigate timeouts in the deposit loader in Docker.

ok, got it, we still want a patch for my first question, which is make timeout value could be configured

Jan 18 2022, 3:35 AM · Core Loader, SWORD deposit

Jan 17 2022

vlorentz added a comment to T3851: Investigate timeouts in the deposit loader in Docker.

Er yeah, the deposit isn't designed for archives this big. You should probably host your tarballs somewhere and point the archive loader to it, instead.

Jan 17 2022, 11:49 AM · Core Loader, SWORD deposit

Jan 15 2022

SupLinux added a comment to T3851: Investigate timeouts in the deposit loader in Docker.

Another problem is swh-deposit client, when I use below command to upload an large archive(16Gb size) , it will consume much memory which is more than 40Gb, this is also a big problem for client, I hope swh could automatically divide large archive:)

Jan 15 2022, 2:49 PM · Core Loader, SWORD deposit
SupLinux added a comment to T3851: Investigate timeouts in the deposit loader in Docker.

Thanks for help me open this issue, From my use usage scenario , I need to upload some package perhaps greater than 10G to deposit, thus will raise an timeout issue. I used this guide to deploy my environment(https://docs.softwareheritage.org/devel/getting-started.html#getting-started) , and my server configuration is 16core/64Gbram/200Gbdisk size, I hope this timeout value could changed by configuration file, because the upload time is depend on every user deployment environment.

Jan 15 2022, 2:34 PM · Core Loader, SWORD deposit
vlorentz triaged T3851: Investigate timeouts in the deposit loader in Docker as Normal priority.
Jan 15 2022, 11:10 AM · Core Loader, SWORD deposit

Dec 6 2021

ardumont updated the task description for T3765: Deploy latest swh.loader.core and swh.lister.
Dec 6 2021, 9:34 AM · System administration, Lister, Core Loader

Dec 3 2021

ardumont closed T3765: Deploy latest swh.loader.core and swh.lister as Resolved.
Dec 3 2021, 5:35 PM · System administration, Lister, Core Loader
ardumont moved T3765: Deploy latest swh.loader.core and swh.lister from deployed/landed/monitoring to Component upgrades on the System administration board.
Dec 3 2021, 5:35 PM · System administration, Lister, Core Loader
ardumont moved T3765: Deploy latest swh.loader.core and swh.lister from in-progress to deployed/landed/monitoring on the System administration board.
Dec 3 2021, 5:35 PM · System administration, Lister, Core Loader
ardumont added a parent task for T3765: Deploy latest swh.loader.core and swh.lister: T2400: Ingest current and historical Ubuntu releases.
Dec 3 2021, 5:29 PM · System administration, Lister, Core Loader
ardumont updated the task description for T3765: Deploy latest swh.loader.core and swh.lister.
Dec 3 2021, 5:13 PM · System administration, Lister, Core Loader
ardumont updated the task description for T3765: Deploy latest swh.loader.core and swh.lister.
Dec 3 2021, 5:11 PM · System administration, Lister, Core Loader
ardumont updated subscribers of T3765: Deploy latest swh.loader.core and swh.lister.

After some fighting to untangle the mess we had in the scheduling dbs:

  • wrong task type used
  • wrong data format in old entries
Dec 3 2021, 5:10 PM · System administration, Lister, Core Loader
ardumont changed the status of T3765: Deploy latest swh.loader.core and swh.lister from Open to Work in Progress.
Dec 3 2021, 3:57 PM · System administration, Lister, Core Loader
ardumont added a project to T3765: Deploy latest swh.loader.core and swh.lister: System administration.
Dec 3 2021, 3:57 PM · System administration, Lister, Core Loader
ardumont added a comment to T3765: Deploy latest swh.loader.core and swh.lister.

What a mess! The existing data both in staging and production are not in the expected
shape for the loader. Hence the issue of failing the load [1]

Dec 3 2021, 3:57 PM · System administration, Lister, Core Loader
ardumont updated the task description for T3765: Deploy latest swh.loader.core and swh.lister.
Dec 3 2021, 3:05 PM · System administration, Lister, Core Loader
ardumont updated the task description for T3765: Deploy latest swh.loader.core and swh.lister.
Dec 3 2021, 3:04 PM · System administration, Lister, Core Loader
ardumont updated the task description for T3765: Deploy latest swh.loader.core and swh.lister.
Dec 3 2021, 2:12 PM · System administration, Lister, Core Loader
ardumont updated the task description for T3765: Deploy latest swh.loader.core and swh.lister.
Dec 3 2021, 2:00 PM · System administration, Lister, Core Loader