Page MenuHomeSoftware Heritage
Feed Advanced Search

Feb 24 2022

ardumont triaged T3973: Deploy swh.deposit v0.17 as Normal priority.
Feb 24 2022, 10:13 AM · System administration, Core Loader, SWORD deposit

Jan 18 2022

SupLinux added a comment to T3851: Investigate timeouts in the deposit loader in Docker.

ok, got it, we still want a patch for my first question, which is make timeout value could be configured

Jan 18 2022, 3:35 AM · Core Loader, SWORD deposit

Jan 17 2022

vlorentz added a comment to T3851: Investigate timeouts in the deposit loader in Docker.

Er yeah, the deposit isn't designed for archives this big. You should probably host your tarballs somewhere and point the archive loader to it, instead.

Jan 17 2022, 11:49 AM · Core Loader, SWORD deposit

Jan 15 2022

SupLinux added a comment to T3851: Investigate timeouts in the deposit loader in Docker.

Another problem is swh-deposit client, when I use below command to upload an large archive(16Gb size) , it will consume much memory which is more than 40Gb, this is also a big problem for client, I hope swh could automatically divide large archive:)

Jan 15 2022, 2:49 PM · Core Loader, SWORD deposit
SupLinux added a comment to T3851: Investigate timeouts in the deposit loader in Docker.

Thanks for help me open this issue, From my use usage scenario , I need to upload some package perhaps greater than 10G to deposit, thus will raise an timeout issue. I used this guide to deploy my environment(https://docs.softwareheritage.org/devel/getting-started.html#getting-started) , and my server configuration is 16core/64Gbram/200Gbdisk size, I hope this timeout value could changed by configuration file, because the upload time is depend on every user deployment environment.

Jan 15 2022, 2:34 PM · Core Loader, SWORD deposit
vlorentz triaged T3851: Investigate timeouts in the deposit loader in Docker as Normal priority.
Jan 15 2022, 11:10 AM · Core Loader, SWORD deposit

Dec 6 2021

ardumont updated the task description for T3765: Deploy latest swh.loader.core and swh.lister.
Dec 6 2021, 9:34 AM · System administration, Lister, Core Loader

Dec 3 2021

ardumont closed T3765: Deploy latest swh.loader.core and swh.lister as Resolved.
Dec 3 2021, 5:35 PM · System administration, Lister, Core Loader
ardumont moved T3765: Deploy latest swh.loader.core and swh.lister from deployed/landed/monitoring to Component upgrades on the System administration board.
Dec 3 2021, 5:35 PM · System administration, Lister, Core Loader
ardumont moved T3765: Deploy latest swh.loader.core and swh.lister from in-progress to deployed/landed/monitoring on the System administration board.
Dec 3 2021, 5:35 PM · System administration, Lister, Core Loader
ardumont added a parent task for T3765: Deploy latest swh.loader.core and swh.lister: T2400: Ingest current and historical Ubuntu releases.
Dec 3 2021, 5:29 PM · System administration, Lister, Core Loader
ardumont updated the task description for T3765: Deploy latest swh.loader.core and swh.lister.
Dec 3 2021, 5:13 PM · System administration, Lister, Core Loader
ardumont updated the task description for T3765: Deploy latest swh.loader.core and swh.lister.
Dec 3 2021, 5:11 PM · System administration, Lister, Core Loader
ardumont updated subscribers of T3765: Deploy latest swh.loader.core and swh.lister.

After some fighting to untangle the mess we had in the scheduling dbs:

  • wrong task type used
  • wrong data format in old entries
Dec 3 2021, 5:10 PM · System administration, Lister, Core Loader
ardumont changed the status of T3765: Deploy latest swh.loader.core and swh.lister from Open to Work in Progress.
Dec 3 2021, 3:57 PM · System administration, Lister, Core Loader
ardumont added a project to T3765: Deploy latest swh.loader.core and swh.lister: System administration.
Dec 3 2021, 3:57 PM · System administration, Lister, Core Loader
ardumont added a comment to T3765: Deploy latest swh.loader.core and swh.lister.

What a mess! The existing data both in staging and production are not in the expected
shape for the loader. Hence the issue of failing the load [1]

Dec 3 2021, 3:57 PM · System administration, Lister, Core Loader
ardumont updated the task description for T3765: Deploy latest swh.loader.core and swh.lister.
Dec 3 2021, 3:05 PM · System administration, Lister, Core Loader
ardumont updated the task description for T3765: Deploy latest swh.loader.core and swh.lister.
Dec 3 2021, 3:04 PM · System administration, Lister, Core Loader
ardumont updated the task description for T3765: Deploy latest swh.loader.core and swh.lister.
Dec 3 2021, 2:12 PM · System administration, Lister, Core Loader
ardumont updated the task description for T3765: Deploy latest swh.loader.core and swh.lister.
Dec 3 2021, 2:00 PM · System administration, Lister, Core Loader
ardumont updated the task description for T3765: Deploy latest swh.loader.core and swh.lister.
Dec 3 2021, 12:30 PM · System administration, Lister, Core Loader
ardumont updated the task description for T3765: Deploy latest swh.loader.core and swh.lister.
Dec 3 2021, 12:29 PM · System administration, Lister, Core Loader
ardumont updated the task description for T3765: Deploy latest swh.loader.core and swh.lister.
Dec 3 2021, 12:26 PM · System administration, Lister, Core Loader
ardumont triaged T3765: Deploy latest swh.loader.core and swh.lister as Normal priority.
Dec 3 2021, 12:23 PM · System administration, Lister, Core Loader

Nov 4 2021

vlorentz triaged T3704: Merge `assert_last_visit_matches` and `check_snapshot` and make them diff recursively as Low priority.
Nov 4 2021, 6:58 PM · Core Loader

Sep 23 2021

vlorentz added a parent task for T3607: Document consistency guarantees of the loaders with respect to the storage: T3604: Document the architecture of all major packages/components.
Sep 23 2021, 3:00 PM · Data Model, Storage manager, Package Loader, Core Loader, Documentation
vlorentz triaged T3607: Document consistency guarantees of the loaders with respect to the storage as Normal priority.
Sep 23 2021, 3:00 PM · Data Model, Storage manager, Package Loader, Core Loader, Documentation

Sep 3 2021

vlorentz moved T1021: SWORD deposit of metadata about an existing SWH object from Backlog to Deployed on the SWORD deposit board.
Sep 3 2021, 11:39 AM · Core Loader, SWORD deposit
vlorentz closed T1021: SWORD deposit of metadata about an existing SWH object as Resolved.
Sep 3 2021, 11:38 AM · Core Loader, SWORD deposit

May 27 2021

anlambert closed T3347: Archive loader should have a snapshot append mode as Resolved by committing rDLDBASE584777f3a5d2: package/archive: Add snapshot_append parameter to ArchiveLoader.
May 27 2021, 2:04 PM · Save Code Now, Core Loader
anlambert added a revision to T3347: Archive loader should have a snapshot append mode: D5789: package/archive: Add snapshot_append parameter to ArchiveLoader.
May 27 2021, 11:52 AM · Save Code Now, Core Loader
ardumont added a comment to T3347: Archive loader should have a snapshot append mode.

Your diff sounds quite enough.

May 27 2021, 9:34 AM · Save Code Now, Core Loader

May 26 2021

anlambert added a comment to T3347: Archive loader should have a snapshot append mode.

Or even directly the archive loader's default behavior (append previously seen branch
from early snapshot/visit). As discussed, I'm wondering whether an archive loader (gnu
or cran [1]) would not benefit from always displaying previously seen branches (whether
they are still present in the current main api we list/visit or not [2]).

May 26 2021, 5:59 PM · Save Code Now, Core Loader
ardumont added a comment to T3347: Archive loader should have a snapshot append mode.

This could be implemented by adding a new option to the loader.

May 26 2021, 5:03 PM · Save Code Now, Core Loader
anlambert triaged T3347: Archive loader should have a snapshot append mode as Normal priority.
May 26 2021, 4:46 PM · Save Code Now, Core Loader
vlorentz changed the status of T2564: migrate existing revisions metadata extra_headers to actual extra_headers field, a subtask of T3090: Make loaders not rely on the 'metadata' column of the 'revision' table, from Work in Progress to Open.
May 26 2021, 11:26 AM · Core Loader

May 4 2021

KShivendu added projects to T3304: Kafka throws flush timeout error: Journal, Core Loader.
May 4 2021, 11:53 AM · Docker environment, Journal

Apr 28 2021

vlorentz changed the status of T2564: migrate existing revisions metadata extra_headers to actual extra_headers field, a subtask of T3090: Make loaders not rely on the 'metadata' column of the 'revision' table, from Open to Work in Progress.
Apr 28 2021, 12:43 PM · Core Loader

Apr 23 2021

olasd closed T2471: NPM package angular-ts-manage fails to be properly loaded as Resolved.

Now deployed in prod:

Apr 23 2021, 1:41 PM · Core Loader
ardumont added a comment to T2471: NPM package angular-ts-manage fails to be properly loaded.

Neat :)

Apr 23 2021, 1:34 PM · Core Loader
olasd added a comment to T2471: NPM package angular-ts-manage fails to be properly loaded.

New swh.loader.core deployed in staging.

Apr 23 2021, 1:20 PM · Core Loader

Apr 19 2021

vlorentz lowered the priority of T3273: Use "fork" relationships to speed-up initial load of large repositories from Normal to Low.
Apr 19 2021, 1:50 PM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
vlorentz triaged T3273: Use "fork" relationships to speed-up initial load of large repositories as Normal priority.
Apr 19 2021, 1:49 PM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
vlorentz claimed T2471: NPM package angular-ts-manage fails to be properly loaded.
Apr 19 2021, 12:43 PM · Core Loader
vlorentz added a parent task for T3138: Make loaders stop writing in the 'metadata' column of the 'revision' table: T2471: NPM package angular-ts-manage fails to be properly loaded.
Apr 19 2021, 12:43 PM · Core Loader
vlorentz edited subtasks for T2471: NPM package angular-ts-manage fails to be properly loaded, added: T3138: Make loaders stop writing in the 'metadata' column of the 'revision' table; removed: T3089: Remove the 'metadata' column of the 'revision' table.
Apr 19 2021, 12:43 PM · Core Loader

Apr 15 2021

vlorentz added a subtask for T3090: Make loaders not rely on the 'metadata' column of the 'revision' table: T2564: migrate existing revisions metadata extra_headers to actual extra_headers field.
Apr 15 2021, 3:15 PM · Core Loader
vlorentz closed T3090: Make loaders not rely on the 'metadata' column of the 'revision' table as Resolved.
Apr 15 2021, 3:15 PM · Core Loader
vlorentz closed T3138: Make loaders stop writing in the 'metadata' column of the 'revision' table, a subtask of T3090: Make loaders not rely on the 'metadata' column of the 'revision' table, as Resolved.
Apr 15 2021, 3:15 PM · Core Loader
vlorentz closed T3138: Make loaders stop writing in the 'metadata' column of the 'revision' table as Resolved.
Apr 15 2021, 3:15 PM · Core Loader
vlorentz closed T3139: Make loaders stop reading from the 'metadata' column of the 'revision' table, a subtask of T3138: Make loaders stop writing in the 'metadata' column of the 'revision' table, as Resolved.
Apr 15 2021, 3:15 PM · Core Loader
vlorentz closed T3139: Make loaders stop reading from the 'metadata' column of the 'revision' table as Resolved.
Apr 15 2021, 3:15 PM · Core Loader
vlorentz closed T3142: Make loaders write to the ExtId storage, a subtask of T3143: Migrate revision metadata to extid in the storage, as Resolved.
Apr 15 2021, 3:15 PM · System administration, Storage manager, Core Loader
vlorentz closed T3142: Make loaders write to the ExtId storage as Resolved.
Apr 15 2021, 3:15 PM · Core Loader

Apr 6 2021

vlorentz added a comment to T3143: Migrate revision metadata to extid in the storage.

if you remember the crash times (.zsh_history?), we could find a range of candidate SWHIDs...

Apr 6 2021, 5:12 PM · System administration, Storage manager, Core Loader
olasd closed T3143: Migrate revision metadata to extid in the storage, a subtask of T3139: Make loaders stop reading from the 'metadata' column of the 'revision' table, as Resolved.
Apr 6 2021, 4:53 PM · Core Loader
olasd closed T3143: Migrate revision metadata to extid in the storage as Resolved.

The migration script has now run to completion (took around a week).

Apr 6 2021, 4:53 PM · System administration, Storage manager, Core Loader
olasd added a revision to T3143: Migrate revision metadata to extid in the storage: D5430: Add sha512 as a valid field in dsc metadata.
Apr 6 2021, 4:48 PM · System administration, Storage manager, Core Loader
vlorentz added a revision to T3138: Make loaders stop writing in the 'metadata' column of the 'revision' table: D5425: HgBundle20Loader: Write nodeids to ExtID objects instead of Revision.
Apr 6 2021, 2:45 PM · Core Loader
vlorentz added a revision to T3139: Make loaders stop reading from the 'metadata' column of the 'revision' table: D5425: HgBundle20Loader: Write nodeids to ExtID objects instead of Revision.
Apr 6 2021, 2:45 PM · Core Loader
vlorentz added a revision to T3139: Make loaders stop reading from the 'metadata' column of the 'revision' table: D5424: HgLoaderFromDisk: Stop reading/writing Revision.metadata.
Apr 6 2021, 2:44 PM · Core Loader
vlorentz added a revision to T3138: Make loaders stop writing in the 'metadata' column of the 'revision' table: D5424: HgLoaderFromDisk: Stop reading/writing Revision.metadata.
Apr 6 2021, 2:44 PM · Core Loader
vlorentz closed T3140: Make loaders use ExtId to find what artifacts are already loaded, in addition to revision.metadata, a subtask of T3139: Make loaders stop reading from the 'metadata' column of the 'revision' table, as Resolved.
Apr 6 2021, 2:20 PM · Core Loader
vlorentz closed T3140: Make loaders use ExtId to find what artifacts are already loaded, in addition to revision.metadata as Resolved.
Apr 6 2021, 2:20 PM · Core Loader
vlorentz added a revision to T2471: NPM package angular-ts-manage fails to be properly loaded: D5423: package loaders: Stop reading/writing Revision.metadata.
Apr 6 2021, 2:18 PM · Core Loader
vlorentz added a revision to T3138: Make loaders stop writing in the 'metadata' column of the 'revision' table: D5423: package loaders: Stop reading/writing Revision.metadata.
Apr 6 2021, 2:18 PM · Core Loader
vlorentz added a revision to T3139: Make loaders stop reading from the 'metadata' column of the 'revision' table: D5423: package loaders: Stop reading/writing Revision.metadata.
Apr 6 2021, 2:18 PM · Core Loader

Mar 30 2021

olasd changed the status of T3143: Migrate revision metadata to extid in the storage, a subtask of T3139: Make loaders stop reading from the 'metadata' column of the 'revision' table, from Open to Work in Progress.
Mar 30 2021, 7:43 PM · Core Loader
olasd changed the status of T3143: Migrate revision metadata to extid in the storage from Open to Work in Progress.
Mar 30 2021, 7:43 PM · System administration, Storage manager, Core Loader
olasd added a comment to T3143: Migrate revision metadata to extid in the storage.

I've deployed the extid schema changes on all storages, and I've started the migration script on getty.

Mar 30 2021, 7:42 PM · System administration, Storage manager, Core Loader
vsellier added a project to T3143: Migrate revision metadata to extid in the storage: System administration.
Mar 30 2021, 5:26 PM · System administration, Storage manager, Core Loader
vlorentz added a project to T3143: Migrate revision metadata to extid in the storage: Storage manager.
Mar 30 2021, 4:57 PM · System administration, Storage manager, Core Loader
vlorentz added a comment to T3143: Migrate revision metadata to extid in the storage.

https://forge.softwareheritage.org/source/snippets/browse/master/vlorentz/migrate_extid.py

Mar 30 2021, 4:56 PM · System administration, Storage manager, Core Loader
vlorentz closed T3141: Make loaders pass ExtId objects between functions instead of "known_artifact", a subtask of T3140: Make loaders use ExtId to find what artifacts are already loaded, in addition to revision.metadata, as Resolved.
Mar 30 2021, 4:56 PM · Core Loader
vlorentz closed T3141: Make loaders pass ExtId objects between functions instead of "known_artifact", a subtask of T3142: Make loaders write to the ExtId storage, as Resolved.
Mar 30 2021, 4:56 PM · Core Loader
vlorentz closed T3141: Make loaders pass ExtId objects between functions instead of "known_artifact" as Resolved.

now it is. (the mercurial loader technically doesn't use ExtID internally, but it already passes nodeids, which are close enough)

Mar 30 2021, 4:56 PM · Core Loader

Mar 29 2021

vlorentz added a revision to T3142: Make loaders write to the ExtId storage: D5371: HgLoaderFromDisk: Read and write ExtIDs to find revisions already loaded.
Mar 29 2021, 2:11 PM · Core Loader
vlorentz added a revision to T3140: Make loaders use ExtId to find what artifacts are already loaded, in addition to revision.metadata: D5371: HgLoaderFromDisk: Read and write ExtIDs to find revisions already loaded.
Mar 29 2021, 2:11 PM · Core Loader
vlorentz added a revision to T3140: Make loaders use ExtId to find what artifacts are already loaded, in addition to revision.metadata: D5345: package.loader: Unnest loops in PackageLoader.load().
Mar 29 2021, 10:58 AM · Core Loader
vlorentz added a revision to T3142: Make loaders write to the ExtId storage: D5364: package.loader: Write to the ExtID storage.
Mar 29 2021, 10:55 AM · Core Loader
vlorentz reopened T3141: Make loaders pass ExtId objects between functions instead of "known_artifact", a subtask of T3140: Make loaders use ExtId to find what artifacts are already loaded, in addition to revision.metadata, as Open.
Mar 29 2021, 10:54 AM · Core Loader
vlorentz reopened T3141: Make loaders pass ExtId objects between functions instead of "known_artifact", a subtask of T3142: Make loaders write to the ExtId storage, as Open.
Mar 29 2021, 10:54 AM · Core Loader
vlorentz reopened T3141: Make loaders pass ExtId objects between functions instead of "known_artifact" as "Open".

actually, only solved for package loaders

Mar 29 2021, 10:54 AM · Core Loader

Mar 25 2021

vlorentz added a revision to T3140: Make loaders use ExtId to find what artifacts are already loaded, in addition to revision.metadata: D5349: package.loader: Lookup packages from the ExtID storage.
Mar 25 2021, 7:22 PM · Core Loader
vlorentz removed a parent task for T3142: Make loaders write to the ExtId storage: T3140: Make loaders use ExtId to find what artifacts are already loaded, in addition to revision.metadata.
Mar 25 2021, 7:22 PM · Core Loader
vlorentz removed a subtask for T3140: Make loaders use ExtId to find what artifacts are already loaded, in addition to revision.metadata: T3142: Make loaders write to the ExtId storage.
Mar 25 2021, 7:22 PM · Core Loader
vlorentz closed T3141: Make loaders pass ExtId objects between functions instead of "known_artifact", a subtask of T3140: Make loaders use ExtId to find what artifacts are already loaded, in addition to revision.metadata, as Resolved.
Mar 25 2021, 7:22 PM · Core Loader
vlorentz closed T3141: Make loaders pass ExtId objects between functions instead of "known_artifact", a subtask of T3142: Make loaders write to the ExtId storage, as Resolved.
Mar 25 2021, 7:22 PM · Core Loader
vlorentz closed T3141: Make loaders pass ExtId objects between functions instead of "known_artifact" as Resolved.
Mar 25 2021, 7:22 PM · Core Loader

Mar 22 2021

vlorentz added a subtask for T3142: Make loaders write to the ExtId storage: T3141: Make loaders pass ExtId objects between functions instead of "known_artifact".
Mar 22 2021, 1:28 PM · Core Loader
vlorentz added a parent task for T3141: Make loaders pass ExtId objects between functions instead of "known_artifact": T3142: Make loaders write to the ExtId storage.
Mar 22 2021, 1:28 PM · Core Loader

Mar 19 2021

vlorentz added revisions to T3141: Make loaders pass ExtId objects between functions instead of "known_artifact": D5285: debian: Make resolve_revision_from use the sha256 of the .dsc, D5286: debian: Split original_artifact parsing out of resolve_revision_from, D5287: npm, pypi: Split original_artifact parsing out of artifact_to_revision_id, D5288: nixguix: Split 'integrity' extraction out of resolve_revision_from, D5289: archive, cran: Replace 'artifact_identity' with extid to detect known packages, D5290: Deduplicate resolve_revision_from across package loaders.
Mar 19 2021, 2:48 PM · Core Loader

Mar 16 2021

vlorentz updated the task description for T3090: Make loaders not rely on the 'metadata' column of the 'revision' table.
Mar 16 2021, 5:53 PM · Core Loader
vlorentz added a parent task for T3142: Make loaders write to the ExtId storage: T3143: Migrate revision metadata to extid in the storage.
Mar 16 2021, 5:53 PM · Core Loader
vlorentz added a subtask for T3143: Migrate revision metadata to extid in the storage: T3142: Make loaders write to the ExtId storage.
Mar 16 2021, 5:53 PM · System administration, Storage manager, Core Loader
vlorentz triaged T3143: Migrate revision metadata to extid in the storage as Normal priority.
Mar 16 2021, 5:53 PM · System administration, Storage manager, Core Loader
vlorentz updated the task description for T3090: Make loaders not rely on the 'metadata' column of the 'revision' table.
Mar 16 2021, 12:33 PM · Core Loader
vlorentz triaged T3142: Make loaders write to the ExtId storage as Normal priority.
Mar 16 2021, 12:33 PM · Core Loader
vlorentz renamed T3140: Make loaders use ExtId to find what artifacts are already loaded, in addition to revision.metadata from Make loaders use ExtId to find what artefacts are already loaded, in addition to revision.metadata to Make loaders use ExtId to find what artifacts are already loaded, in addition to revision.metadata.
Mar 16 2021, 12:28 PM · Core Loader