Page MenuHomeSoftware Heritage
Feed All Stories

Sep 30 2022

ardumont closed D8581: Add Content Loader to ingest raw content file.
Sep 30 2022, 2:32 PM
ardumont committed rDLDBASEf774aba59e65: Add Content Loader to ingest raw content file (authored by ardumont).
Add Content Loader to ingest raw content file
Sep 30 2022, 2:32 PM
swh-public-ci added a comment to D8581: Add Content Loader to ingest raw content file.

Build is green

Sep 30 2022, 2:31 PM
ardumont requested review of D8584: Add Directory Loader to allow tarball ingestion as Directory.
Sep 30 2022, 2:31 PM
vlorentz accepted D8585: svn: Add support to basic subversion authentication in SvnRepo.

thx

Sep 30 2022, 2:30 PM
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Sep 30 2022, 2:29 PM · Data Model, Nixguix loader
ardumont updated the diff for D8581: Add Content Loader to ingest raw content file.

Rebase

Sep 30 2022, 2:27 PM
anlambert added inline comments to D8579: http_client: Add max_matching_nodes parameter to visit_nodes().
Sep 30 2022, 2:22 PM
anlambert accepted D8583: SimpleTraversal.visitNode: Remove redundant boolean.
Sep 30 2022, 2:18 PM
anlambert requested review of D8585: svn: Add support to basic subversion authentication in SvnRepo.
Sep 30 2022, 2:17 PM
anlambert added a project to T4590: swh-scanner - improve distribution, documentation and usability: Code scanner.
Sep 30 2022, 2:16 PM · Code scanner
anlambert added a project to T4592: swh-scanner - generic bug fixing, robustness and performance improvements: Code scanner.
Sep 30 2022, 2:16 PM · Code scanner
ardumont updated the summary of D8341: Add nixguix lister.
Sep 30 2022, 11:57 AM
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Sep 30 2022, 11:56 AM · Data Model, Nixguix loader
ardumont added inline comments to D8581: Add Content Loader to ingest raw content file.
Sep 30 2022, 11:56 AM
ardumont added a revision to T3781: Replace the Nixguix loader with a lister: D8584: Add Directory Loader to allow tarball ingestion as Directory.
Sep 30 2022, 11:56 AM · Data Model, Nixguix loader
vlorentz requested changes to D8566: Conda: Anaconda packages archive loader.
Sep 30 2022, 11:39 AM
vlorentz requested review of D8583: SimpleTraversal.visitNode: Remove redundant boolean.
Sep 30 2022, 11:35 AM
vlorentz added inline comments to D8580: Apply 'max_matching_nodes' restriction after 'return_types' filter.
Sep 30 2022, 11:30 AM
vlorentz added inline comments to D8580: Apply 'max_matching_nodes' restriction after 'return_types' filter.
Sep 30 2022, 11:23 AM
marmoute added a comment to T4590: swh-scanner - improve distribution, documentation and usability.

D8572 (patterns: adds a list of common patterns to exclude) should be attached to this task instead.

Sep 30 2022, 11:20 AM · Code scanner
marmoute added a comment to T4592: swh-scanner - generic bug fixing, robustness and performance improvements.

https://forge.softwareheritage.org/D8571 is also relevant for this Task

Sep 30 2022, 11:19 AM · Code scanner
marmoute closed D8571: from_disks: fix some of the pattern checking logic.
Sep 30 2022, 11:18 AM
marmoute committed rDMOD6a38c4ad047d: from_disks: fix some of the pattern checking logic (authored by marmoute).
from_disks: fix some of the pattern checking logic
Sep 30 2022, 11:18 AM
ardumont abandoned D8582: Allow more checksum computations in Content model.

Thanks! I'll close this.

Sep 30 2022, 10:52 AM
olasd added a comment to D8582: Allow more checksum computations in Content model.

It was initially to just compute that new hash at the same time as the standard ones we store but that's getting out of hand.

then use MultiHash directly in the Content loader, instead of Content.from_data

Sep 30 2022, 10:47 AM
olasd added a comment to D8582: Allow more checksum computations in Content model.

Why are the changes to the model object needed, instead of just hashing the file directly?

You mean using MultiHash directly. Yes, that feels more and more like the right way to do it.

It was initially to just compute that new hash at the same time as the standard ones we store but that's getting out of hand.

Sep 30 2022, 10:47 AM
vlorentz added a comment to D8582: Allow more checksum computations in Content model.

It was initially to just compute that new hash at the same time as the standard ones we store but that's getting out of hand.

Sep 30 2022, 10:46 AM
ardumont added a comment to D8582: Allow more checksum computations in Content model.

Model objects should map 1:1 with what is stored in the archive.

Sep 30 2022, 10:45 AM
vlorentz added a comment to D8582: Allow more checksum computations in Content model.
In D8582#223177, @olasd wrote:

Why are the changes to the model object needed, instead of just hashing the file directly?

Sep 30 2022, 10:45 AM
ardumont added a comment to D8582: Allow more checksum computations in Content model.

Why are the changes to the model object needed, instead of just hashing the file directly?

Sep 30 2022, 10:45 AM
ardumont added inline comments to D8582: Allow more checksum computations in Content model.
Sep 30 2022, 10:43 AM
olasd requested changes to D8582: Allow more checksum computations in Content model.

I don't agree with the idea of adding additional hashes, that will not be stored, to the Content model object. Model objects should map 1:1 with what is stored in the archive.

Sep 30 2022, 10:42 AM
vlorentz added inline comments to D8582: Allow more checksum computations in Content model.
Sep 30 2022, 10:37 AM
swh-public-ci added a comment to D8569: Add rubygems loader.

Build is green

Sep 30 2022, 10:30 AM
Alphare updated the diff for D8569: Add rubygems loader.

Use the built_at date instead of publication date

Sep 30 2022, 10:26 AM
swh-public-ci added a comment to D8581: Add Content Loader to ingest raw content file.

Build is green

Sep 30 2022, 10:26 AM
ardumont updated the diff for D8581: Add Content Loader to ingest raw content file.

Refactoring steps

Sep 30 2022, 10:22 AM
bchauvet added a revision to T4104: Ingest crates.io (Rust): D8199: Add a 'user' documentation entry for the Crates lister. Add myself as contributor.
Sep 30 2022, 10:21 AM · Crates loader, Crates lister, Archive coverage
bchauvet added a task to D8199: Add a 'user' documentation entry for the Crates lister. Add myself as contributor: T4104: Ingest crates.io (Rust).
Sep 30 2022, 10:21 AM
bchauvet added a revision to T4104: Ingest crates.io (Rust): D8206: crates: Add a developer documentation at module level.
Sep 30 2022, 10:20 AM · Crates loader, Crates lister, Archive coverage
bchauvet added a task to D8206: crates: Add a developer documentation at module level: T4104: Ingest crates.io (Rust).
Sep 30 2022, 10:20 AM
bchauvet added a revision to T4233: Ingest Arch Linux: D8259: arch: Extra_loader_arguments consistency.
Sep 30 2022, 10:20 AM · Arch loader, Arch Lister, Archive coverage
bchauvet added a task to D8259: arch: Extra_loader_arguments consistency: T4233: Ingest Arch Linux.
Sep 30 2022, 10:20 AM
franckbret added inline comments to D8379: Hackage: Loads Hackage Listed origins.
Sep 30 2022, 10:19 AM
swh-public-ci added a comment to D8581: Add Content Loader to ingest raw content file.

Build is green

Sep 30 2022, 10:17 AM
bchauvet added a revision to T4233: Ingest Arch Linux: D8264: Arch: Adapt the loader to manage 'arch_metadata' from 'extra_loader_arguments'.
Sep 30 2022, 10:16 AM · Arch loader, Arch Lister, Archive coverage
bchauvet added a task to D8264: Arch: Adapt the loader to manage 'arch_metadata' from 'extra_loader_arguments': T4233: Ingest Arch Linux.
Sep 30 2022, 10:16 AM
bchauvet added a revision to T4465: Ingest pub.dev (Dart, Flutter): D8287: Pub.dev lister for Dart and Flutter packages.
Sep 30 2022, 10:16 AM · PubDev loader, PubDev lister, Archive coverage
bchauvet added a task to D8287: Pub.dev lister for Dart and Flutter packages: T4465: Ingest pub.dev (Dart, Flutter).
Sep 30 2022, 10:16 AM
bchauvet added a task to D8296: Golang module loader: T4124: Golang support.
Sep 30 2022, 10:16 AM
bchauvet added a revision to T4124: Golang support: D8296: Golang module loader.
Sep 30 2022, 10:15 AM · Golang loader, Golang lister, Archive coverage
bchauvet added a revision to T4124: Golang support: D8298: Add incremental function to Golang Lister.
Sep 30 2022, 10:15 AM · Golang loader, Golang lister, Archive coverage
bchauvet added a task to D8298: Add incremental function to Golang Lister: T4124: Golang support.
Sep 30 2022, 10:15 AM
bchauvet added a revision to T4465: Ingest pub.dev (Dart, Flutter): D8305: PubDev, loads from pub.dev registry (Dart, Flutter).
Sep 30 2022, 10:14 AM · PubDev loader, PubDev lister, Archive coverage
bchauvet added a task to D8305: PubDev, loads from pub.dev registry (Dart, Flutter): T4465: Ingest pub.dev (Dart, Flutter).
Sep 30 2022, 10:14 AM
ardumont updated the diff for D8581: Add Content Loader to ingest raw content file.

Compute expected checksum to check integrity outside the loop

Sep 30 2022, 10:13 AM
bchauvet added a task to D8321: Pubdev, Change name of task method: T4465: Ingest pub.dev (Dart, Flutter).
Sep 30 2022, 10:13 AM
bchauvet added a revision to T4465: Ingest pub.dev (Dart, Flutter): D8321: Pubdev, Change name of task method.
Sep 30 2022, 10:13 AM · PubDev loader, PubDev lister, Archive coverage
swh-public-ci added a comment to D8379: Hackage: Loads Hackage Listed origins.

Build is green

Sep 30 2022, 10:13 AM
bchauvet added a revision to T4465: Ingest pub.dev (Dart, Flutter): D8330: Pubdev: Fix get_versions() failed with loose version names.
Sep 30 2022, 10:13 AM · PubDev loader, PubDev lister, Archive coverage
bchauvet added a task to D8330: Pubdev: Fix get_versions() failed with loose version names: T4465: Ingest pub.dev (Dart, Flutter).
Sep 30 2022, 10:13 AM
bchauvet added a revision to T4475: Ingest bower.io (Javascript package manager): D8333: Bower: List origins from registry.bower.io.
Sep 30 2022, 10:12 AM · Bower lister, Archive coverage
bchauvet added a task to D8333: Bower: List origins from registry.bower.io: T4475: Ingest bower.io (Javascript package manager).
Sep 30 2022, 10:12 AM
bchauvet added a revision to T4475: Ingest bower.io (Javascript package manager): D8336: Bower: Set VISIT_TYPE as 'git'.
Sep 30 2022, 10:11 AM · Bower lister, Archive coverage
bchauvet added a task to D8336: Bower: Set VISIT_TYPE as 'git': T4475: Ingest bower.io (Javascript package manager).
Sep 30 2022, 10:11 AM
bchauvet added a revision to T4597: Create a Hackage Lister: D8338: Hackage: List origins from hackage.haskell.org, The Haskell Package Repository.
Sep 30 2022, 10:10 AM · Hackage lister
bchauvet added a task to D8338: Hackage: List origins from hackage.haskell.org, The Haskell Package Repository: T4597: Create a Hackage Lister.
Sep 30 2022, 10:10 AM
bchauvet added a task to D8339: Arch: Add throttling retry for scrapping and resources download: T4233: Ingest Arch Linux.
Sep 30 2022, 10:09 AM
bchauvet added a revision to T4233: Ingest Arch Linux: D8339: Arch: Add throttling retry for scrapping and resources download.
Sep 30 2022, 10:09 AM · Arch loader, Arch Lister, Archive coverage
franckbret updated the diff for D8379: Hackage: Loads Hackage Listed origins.

Trust p_info.version instead of intrinsic_metadata["version"]

Sep 30 2022, 10:09 AM
bchauvet added a revision to T4598: Create a Hackage Loader: D8379: Hackage: Loads Hackage Listed origins.
Sep 30 2022, 10:06 AM · Hackage loader
bchauvet added a task to D8379: Hackage: Loads Hackage Listed origins: T4598: Create a Hackage Loader.
Sep 30 2022, 10:06 AM
bchauvet triaged T4598: Create a Hackage Loader as Normal priority.
Sep 30 2022, 10:06 AM · Hackage loader
bchauvet triaged T4597: Create a Hackage Lister as Normal priority.
Sep 30 2022, 10:06 AM · Hackage lister
bchauvet added a revision to T4104: Ingest crates.io (Rust): D8454: Crates.io: Add last_update for each version of a crate.
Sep 30 2022, 10:05 AM · Crates loader, Crates lister, Archive coverage
bchauvet added a task to D8454: Crates.io: Add last_update for each version of a crate: T4104: Ingest crates.io (Rust).
Sep 30 2022, 10:04 AM
bchauvet added a revision to T4595: Improve common tooling for loading: D8510: from_disk: only build a model object once.
Sep 30 2022, 10:04 AM
bchauvet added a task to D8510: from_disk: only build a model object once: T4595: Improve common tooling for loading.
Sep 30 2022, 10:04 AM
bchauvet added a revision to T4595: Improve common tooling for loading: D8512: various optimisation to the model validation logic..
Sep 30 2022, 10:03 AM
bchauvet added a task to D8512: various optimisation to the model validation logic.: T4595: Improve common tooling for loading.
Sep 30 2022, 10:03 AM
bchauvet added a revision to T4547: Create a Conda Lister: D8517: Conda: List origins from anaconda.com, the Package, dependency and environment management for any language.
Sep 30 2022, 10:03 AM · Conda lister
bchauvet added a task to D8517: Conda: List origins from anaconda.com, the Package, dependency and environment management for any language: T4547: Create a Conda Lister.
Sep 30 2022, 10:03 AM
bchauvet added a revision to T4595: Improve common tooling for loading: D8521: Use a Merkle discovery algorithm with archives.
Sep 30 2022, 10:02 AM
bchauvet added a task to D8521: Use a Merkle discovery algorithm with archives: T4595: Improve common tooling for loading.
Sep 30 2022, 10:02 AM
bchauvet added a revision to T4595: Improve common tooling for loading: D8525: from_disk: skip intermediate dictionnary creation when building model.
Sep 30 2022, 9:59 AM
bchauvet added a task to D8525: from_disk: skip intermediate dictionnary creation when building model: T4595: Improve common tooling for loading.
Sep 30 2022, 9:59 AM
bchauvet added a revision to T4595: Improve common tooling for loading: D8527: model: avoid another extra creation of Model object.
Sep 30 2022, 9:58 AM
bchauvet added a task to D8527: model: avoid another extra creation of Model object: T4595: Improve common tooling for loading.
Sep 30 2022, 9:58 AM
bchauvet added a task to D8528: Nuget: Lister for NuGet the package manager for .NET: T1718: Implement a NuGet(.NET) lister.
Sep 30 2022, 9:56 AM
bchauvet added a revision to T1718: Implement a NuGet(.NET) lister: D8528: Nuget: Lister for NuGet the package manager for .NET.
Sep 30 2022, 9:56 AM · Archive coverage
bchauvet added a revision to T1777: Rubygems Lister: D8529: RubyGems: List origins from https://rubygems.org.
Sep 30 2022, 9:55 AM · RubyGems lister, Archive coverage
bchauvet added a task to D8529: RubyGems: List origins from https://rubygems.org: T1777: Rubygems Lister.
Sep 30 2022, 9:55 AM
bchauvet added a revision to T4519: Create a Puppet Forge lister: D8531: Puppet: Lister for Puppet modules.
Sep 30 2022, 9:54 AM · Puppet lister
bchauvet added a task to D8531: Puppet: Lister for Puppet modules: T4519: Create a Puppet Forge lister.
Sep 30 2022, 9:53 AM
ardumont updated the task description for T4385: Federate prometheus instances through thanos.
Sep 30 2022, 9:52 AM · meta-task, System administration, Roadmap 2022
ardumont updated the task description for T4385: Federate prometheus instances through thanos.
Sep 30 2022, 9:52 AM · meta-task, System administration, Roadmap 2022
ardumont accepted D8571: from_disks: fix some of the pattern checking logic.

I went for a minimal changes for that untested code as I am mostly planning a full
overhaul of that area. So I did not spent to effort to add new test (the code isn't
currently directly tested either as far as I understand).

Sep 30 2022, 9:49 AM
bchauvet added a revision to T4595: Improve common tooling for loading: D8538: Setup async interface for discovery module.
Sep 30 2022, 9:46 AM
bchauvet added a task to D8538: Setup async interface for discovery module: T4595: Improve common tooling for loading.
Sep 30 2022, 9:46 AM