Page MenuHomeSoftware Heritage

Core LoaderFolder
ActivePublic

Members

  • This project does not have any members.
  • View All

Watchers

  • This project does not have any watchers.
  • View All

Details

Description

loader plumbings used by other loaders

Recent Activity

Thu, Nov 4

vlorentz triaged T3704: Merge `assert_last_visit_matches` and `check_snapshot` and make them diff recursively as Low priority.
Thu, Nov 4, 6:58 PM · Core Loader

Sep 23 2021

vlorentz added a parent task for T3607: Document consistency guarantees of the loaders with respect to the storage: T3604: Document the architecture of all major packages/components.
Sep 23 2021, 3:00 PM · Data Model, Storage manager, Package Loader, Core Loader, Documentation
vlorentz triaged T3607: Document consistency guarantees of the loaders with respect to the storage as Normal priority.
Sep 23 2021, 3:00 PM · Data Model, Storage manager, Package Loader, Core Loader, Documentation

Sep 3 2021

vlorentz moved T1021: SWORD deposit of metadata about an existing SWH object from Backlog to Deployed on the SWORD deposit board.
Sep 3 2021, 11:39 AM · Core Loader, SWORD deposit
vlorentz closed T1021: SWORD deposit of metadata about an existing SWH object as Resolved.
Sep 3 2021, 11:38 AM · Core Loader, SWORD deposit

May 27 2021

anlambert closed T3347: Archive loader should have a snapshot append mode as Resolved by committing rDLDBASE584777f3a5d2: package/archive: Add snapshot_append parameter to ArchiveLoader.
May 27 2021, 2:04 PM · Save Code Now, Core Loader
anlambert added a revision to T3347: Archive loader should have a snapshot append mode: D5789: package/archive: Add snapshot_append parameter to ArchiveLoader.
May 27 2021, 11:52 AM · Save Code Now, Core Loader
ardumont added a comment to T3347: Archive loader should have a snapshot append mode.

Your diff sounds quite enough.

May 27 2021, 9:34 AM · Save Code Now, Core Loader

May 26 2021

anlambert added a comment to T3347: Archive loader should have a snapshot append mode.

Or even directly the archive loader's default behavior (append previously seen branch
from early snapshot/visit). As discussed, I'm wondering whether an archive loader (gnu
or cran [1]) would not benefit from always displaying previously seen branches (whether
they are still present in the current main api we list/visit or not [2]).

May 26 2021, 5:59 PM · Save Code Now, Core Loader
ardumont added a comment to T3347: Archive loader should have a snapshot append mode.

This could be implemented by adding a new option to the loader.

May 26 2021, 5:03 PM · Save Code Now, Core Loader
anlambert triaged T3347: Archive loader should have a snapshot append mode as Normal priority.
May 26 2021, 4:46 PM · Save Code Now, Core Loader
vlorentz changed the status of T2564: migrate existing revisions metadata extra_headers to actual extra_headers field, a subtask of T3090: Make loaders not rely on the 'metadata' column of the 'revision' table, from Work in Progress to Open.
May 26 2021, 11:26 AM · Core Loader

May 4 2021

KShivendu added projects to T3304: Kafka throws flush timeout error: Journal, Core Loader.
May 4 2021, 11:53 AM · Docker environment, Journal

Apr 28 2021

vlorentz changed the status of T2564: migrate existing revisions metadata extra_headers to actual extra_headers field, a subtask of T3090: Make loaders not rely on the 'metadata' column of the 'revision' table, from Open to Work in Progress.
Apr 28 2021, 12:43 PM · Core Loader

Apr 23 2021

olasd closed T2471: NPM package angular-ts-manage fails to be properly loaded as Resolved.

Now deployed in prod:

Apr 23 2021, 1:41 PM · Core Loader
ardumont added a comment to T2471: NPM package angular-ts-manage fails to be properly loaded.

Neat :)

Apr 23 2021, 1:34 PM · Core Loader
olasd added a comment to T2471: NPM package angular-ts-manage fails to be properly loaded.

New swh.loader.core deployed in staging.

Apr 23 2021, 1:20 PM · Core Loader

Apr 19 2021

vlorentz lowered the priority of T3273: Use "fork" relationships to speed-up initial load of large repositories from Normal to Low.
Apr 19 2021, 1:50 PM · Extrinsic metadata, Core Loader
vlorentz triaged T3273: Use "fork" relationships to speed-up initial load of large repositories as Normal priority.
Apr 19 2021, 1:49 PM · Extrinsic metadata, Core Loader
vlorentz claimed T2471: NPM package angular-ts-manage fails to be properly loaded.
Apr 19 2021, 12:43 PM · Core Loader
vlorentz added a parent task for T3138: Make loaders stop writing in the 'metadata' column of the 'revision' table: T2471: NPM package angular-ts-manage fails to be properly loaded.
Apr 19 2021, 12:43 PM · Core Loader
vlorentz edited subtasks for T2471: NPM package angular-ts-manage fails to be properly loaded, added: T3138: Make loaders stop writing in the 'metadata' column of the 'revision' table; removed: T3089: Remove the 'metadata' column of the 'revision' table.
Apr 19 2021, 12:43 PM · Core Loader

Apr 15 2021

vlorentz added a subtask for T3090: Make loaders not rely on the 'metadata' column of the 'revision' table: T2564: migrate existing revisions metadata extra_headers to actual extra_headers field.
Apr 15 2021, 3:15 PM · Core Loader
vlorentz closed T3090: Make loaders not rely on the 'metadata' column of the 'revision' table as Resolved.
Apr 15 2021, 3:15 PM · Core Loader
vlorentz closed T3138: Make loaders stop writing in the 'metadata' column of the 'revision' table, a subtask of T3090: Make loaders not rely on the 'metadata' column of the 'revision' table, as Resolved.
Apr 15 2021, 3:15 PM · Core Loader
vlorentz closed T3138: Make loaders stop writing in the 'metadata' column of the 'revision' table as Resolved.
Apr 15 2021, 3:15 PM · Core Loader
vlorentz closed T3139: Make loaders stop reading from the 'metadata' column of the 'revision' table, a subtask of T3138: Make loaders stop writing in the 'metadata' column of the 'revision' table, as Resolved.
Apr 15 2021, 3:15 PM · Core Loader
vlorentz closed T3139: Make loaders stop reading from the 'metadata' column of the 'revision' table as Resolved.
Apr 15 2021, 3:15 PM · Core Loader
vlorentz closed T3142: Make loaders write to the ExtId storage, a subtask of T3143: Migrate revision metadata to extid in the storage, as Resolved.
Apr 15 2021, 3:15 PM · System administration, Storage manager, Core Loader
vlorentz closed T3142: Make loaders write to the ExtId storage as Resolved.
Apr 15 2021, 3:15 PM · Core Loader

Apr 6 2021

vlorentz added a comment to T3143: Migrate revision metadata to extid in the storage.

if you remember the crash times (.zsh_history?), we could find a range of candidate SWHIDs...

Apr 6 2021, 5:12 PM · System administration, Storage manager, Core Loader
olasd closed T3143: Migrate revision metadata to extid in the storage, a subtask of T3139: Make loaders stop reading from the 'metadata' column of the 'revision' table, as Resolved.
Apr 6 2021, 4:53 PM · Core Loader
olasd closed T3143: Migrate revision metadata to extid in the storage as Resolved.

The migration script has now run to completion (took around a week).

Apr 6 2021, 4:53 PM · System administration, Storage manager, Core Loader
olasd added a revision to T3143: Migrate revision metadata to extid in the storage: D5430: Add sha512 as a valid field in dsc metadata.
Apr 6 2021, 4:48 PM · System administration, Storage manager, Core Loader
vlorentz added a revision to T3138: Make loaders stop writing in the 'metadata' column of the 'revision' table: D5425: HgBundle20Loader: Write nodeids to ExtID objects instead of Revision.
Apr 6 2021, 2:45 PM · Core Loader
vlorentz added a revision to T3139: Make loaders stop reading from the 'metadata' column of the 'revision' table: D5425: HgBundle20Loader: Write nodeids to ExtID objects instead of Revision.
Apr 6 2021, 2:45 PM · Core Loader
vlorentz added a revision to T3139: Make loaders stop reading from the 'metadata' column of the 'revision' table: D5424: HgLoaderFromDisk: Stop reading/writing Revision.metadata.
Apr 6 2021, 2:44 PM · Core Loader
vlorentz added a revision to T3138: Make loaders stop writing in the 'metadata' column of the 'revision' table: D5424: HgLoaderFromDisk: Stop reading/writing Revision.metadata.
Apr 6 2021, 2:44 PM · Core Loader
vlorentz closed T3140: Make loaders use ExtId to find what artifacts are already loaded, in addition to revision.metadata, a subtask of T3139: Make loaders stop reading from the 'metadata' column of the 'revision' table, as Resolved.
Apr 6 2021, 2:20 PM · Core Loader
vlorentz closed T3140: Make loaders use ExtId to find what artifacts are already loaded, in addition to revision.metadata as Resolved.
Apr 6 2021, 2:20 PM · Core Loader
vlorentz added a revision to T2471: NPM package angular-ts-manage fails to be properly loaded: D5423: package loaders: Stop reading/writing Revision.metadata.
Apr 6 2021, 2:18 PM · Core Loader
vlorentz added a revision to T3138: Make loaders stop writing in the 'metadata' column of the 'revision' table: D5423: package loaders: Stop reading/writing Revision.metadata.
Apr 6 2021, 2:18 PM · Core Loader
vlorentz added a revision to T3139: Make loaders stop reading from the 'metadata' column of the 'revision' table: D5423: package loaders: Stop reading/writing Revision.metadata.
Apr 6 2021, 2:18 PM · Core Loader

Mar 30 2021

olasd changed the status of T3143: Migrate revision metadata to extid in the storage, a subtask of T3139: Make loaders stop reading from the 'metadata' column of the 'revision' table, from Open to Work in Progress.
Mar 30 2021, 7:43 PM · Core Loader
olasd changed the status of T3143: Migrate revision metadata to extid in the storage from Open to Work in Progress.
Mar 30 2021, 7:43 PM · System administration, Storage manager, Core Loader
olasd added a comment to T3143: Migrate revision metadata to extid in the storage.

I've deployed the extid schema changes on all storages, and I've started the migration script on getty.

Mar 30 2021, 7:42 PM · System administration, Storage manager, Core Loader
vsellier added a project to T3143: Migrate revision metadata to extid in the storage: System administration.
Mar 30 2021, 5:26 PM · System administration, Storage manager, Core Loader
vlorentz added a project to T3143: Migrate revision metadata to extid in the storage: Storage manager.
Mar 30 2021, 4:57 PM · System administration, Storage manager, Core Loader
vlorentz added a comment to T3143: Migrate revision metadata to extid in the storage.

https://forge.softwareheritage.org/source/snippets/browse/master/vlorentz/migrate_extid.py

Mar 30 2021, 4:56 PM · System administration, Storage manager, Core Loader
vlorentz closed T3141: Make loaders pass ExtId objects between functions instead of "known_artifact", a subtask of T3140: Make loaders use ExtId to find what artifacts are already loaded, in addition to revision.metadata, as Resolved.
Mar 30 2021, 4:56 PM · Core Loader