Making an async swh-storage (or RPC client for our framework, in general) turns out the not be worth it, threads are just as efficient and don't require complex hacks

Nov 22 2021, 10:37 AM · Vault

vlorentz added a task to D6650: git_bare: Fetch directories concurrently, using threads.: T3731: Try parallelizing directory_get_entries with an async swh-storage client.

Nov 22 2021, 10:35 AM

vlorentz added revisions to T3731: Try parallelizing directory_get_entries with an async swh-storage client: D6650: git_bare: Fetch directories concurrently, using threads., D6651: WIP: Add AsyncRPCClient, D6652: WIP: Add AsyncRemoteStorage, D6653: git_bare: Add an alternative using asyncio instead of threads..

Nov 22 2021, 10:35 AM · Vault

ardumont closed T3716: Migrate giverny from stretch to buster, a subtask of T3579: Meta-task: upgrade infrastructure to Debian Bullseye, as Resolved.

Nov 22 2021, 10:00 AM · System administration (Component upgrades)

ardumont closed T3716: Migrate giverny from stretch to buster as Resolved.

Nov 22 2021, 10:00 AM · System administration (Component upgrades)

ardumont moved T3716: Migrate giverny from stretch to buster from In progress to Done on the System administration (Component upgrades) board.

Nov 22 2021, 10:00 AM · System administration (Component upgrades)

ardumont updated the task description for T3716: Migrate giverny from stretch to buster.

Nov 22 2021, 9:59 AM · System administration (Component upgrades)

dachary committed rDOPH9266eaa6ead4: create and lookup a Read Shard with a perfect hash (authored by dachary).

create and lookup a Read Shard with a perfect hash

Nov 22 2021, 9:34 AM

dachary closed D6424: Perfect hashmap C implementation.

Nov 22 2021, 9:34 AM

dachary committed rDOPHed6bad56a505: C stub compiled via cffi and tested in python (authored by dachary).

C stub compiled via cffi and tested in python

Nov 22 2021, 9:34 AM

vlorentz requested changes to D6396: Implement maven jar source files loader.

Looks like we're almost done.

Nov 22 2021, 1:39 AM

Nov 21 2021

borisbaldassari added a comment to D6396: Implement maven jar source files loader.

In D6396#173266, @vlorentz wrote:

We also tweaked the values of various fields to be consistent across loaders (especially the message), see: https://docs.softwareheritage.org/devel/swh-loader-core/package-loader-specifications.html#release-fields

Hi @vlorentz, I've fixed the code for the new release object and for the loader fields. I also updated the loaders spec documentation. Good for me.

Nov 21 2021, 9:12 AM

swh-public-ci added a comment to D6396: Implement maven jar source files loader.

Build is green

Nov 21 2021, 9:10 AM

borisbaldassari updated the diff for D6396: Implement maven jar source files loader.

maven-loader: fixes after ardumont's review in D6396
maven-loader: fix release instead of revision

Nov 21 2021, 9:08 AM

Nov 20 2021

swh-public-ci added a comment to D6396: Implement maven jar source files loader.

Build is green

Nov 20 2021, 7:25 PM

borisbaldassari updated the diff for D6396: Implement maven jar source files loader.

maven jar-loader: Initalise files.
maven-loader: fixes after ardumont's review in D6396
maven-loader: Fix copyright in header
maven-loader: Fix extId to include url and timestamp
maven-loader: fix release instead of revision

Nov 20 2021, 7:22 PM

vlorentz triaged T3744: Add a checklist at the end of the package-loader tutorial as High priority.

Nov 20 2021, 1:50 PM · Package Loader, Documentation

vlorentz triaged T3743: Update the package-loader tutorial to reference releases instead of revisions as High priority.

Nov 20 2021, 1:49 PM · Package Loader, Documentation

borisbaldassari added a comment to D6396: Implement maven jar source files loader.

In D6396#173266, @vlorentz wrote:

In D6396#173260, @borisbaldassari wrote:

Arg, changes on the master now break the build, throwing "NotImplementedError: build_release".
What happened to build_release, any idea?

I replaced "build_revision" with "build_release" in the core loader. You need to rename your "build_revision" function, and change the object it returns.

We also tweaked the values of various fields to be consistent across loaders (especially the message), see: https://docs.softwareheritage.org/devel/swh-loader-core/package-loader-specifications.html#release-fields

Nov 20 2021, 11:09 AM

vlorentz added a comment to D6396: Implement maven jar source files loader.

oops, wrong link, that's the old fields. See https://forge.softwareheritage.org/D6629#change-PjtKhodzSu0H for the list of changes

Nov 20 2021, 10:50 AM

vlorentz added a comment to D6396: Implement maven jar source files loader.

In D6396#173260, @borisbaldassari wrote:

Arg, changes on the master now break the build, throwing "NotImplementedError: build_release".
What happened to build_release, any idea?

Nov 20 2021, 10:49 AM

borisbaldassari added a comment to D6396: Implement maven jar source files loader.

Arg, changes on the master now break the build, throwing "NotImplementedError: build_release".
What happened to build_release, any idea?

Nov 20 2021, 10:46 AM

borisbaldassari added a comment to D6395: lister: Add new maven lister.

In D6395#173250, @ardumont wrote:

I'd say if we are confident enough it does the right thing and it's well covered (I
think it is right now) you should be able to rebase all your diffs on latest master. And
then "land" [1] them!

Fine. I'll wait for your review and land the diff when you tell me to then.

Nov 20 2021, 10:25 AM

Harbormaster failed remote builds in B25088: Diff 24211 for D6396: Implement maven jar source files loader!

Nov 20 2021, 10:14 AM

swh-public-ci added a comment to D6396: Implement maven jar source files loader.

Build has FAILED

Nov 20 2021, 10:14 AM

borisbaldassari updated the diff for D6396: Implement maven jar source files loader.

maven-loader: Fix extId to include url and timestamp

Nov 20 2021, 10:12 AM

pilarutama updated pilarutama.

Nov 20 2021, 6:30 AM

Nov 19 2021

olasd accepted D6424: Perfect hashmap C implementation.

Thanks a lot!

Nov 19 2021, 5:54 PM

seirl committed rDSEA150cbbca19bb: setup.py: use yarnpkg instead of yarn if present in PATH (authored by seirl).

setup.py: use yarnpkg instead of yarn if present in PATH

Nov 19 2021, 5:53 PM

seirl closed T3742: yarn called in swh-search setup.py but not present in developer setup docs as Resolved.

Fixed in rDDOC55cdfd9ee957f57cf91b0f6932cc941d2887d933

Nov 19 2021, 5:29 PM · Archive search

seirl committed rDDOC55cdfd9ee957: Add yarnpkg dependency to developer-setup (authored by seirl).

Add yarnpkg dependency to developer-setup

Nov 19 2021, 5:28 PM

rdicosmo committed rMSLD7453840b4219: Update SWH foundation image (authored by rdicosmo).

Update SWH foundation image

Nov 19 2021, 5:22 PM

rdicosmo committed rMSLD40fde893f950: Draft presentation public part UNESCO 2021 (authored by rdicosmo).

Draft presentation public part UNESCO 2021

Nov 19 2021, 5:22 PM

rdicosmo committed rMSLD5f83808c96c9: Add walkthrough (authored by rdicosmo).

Add walkthrough

Nov 19 2021, 5:22 PM

seirl triaged T3742: yarn called in swh-search setup.py but not present in developer setup docs as Normal priority.

Nov 19 2021, 5:17 PM · Archive search

vsellier triaged T3741: swh-search - upgrade elasticsearch backend as Normal priority.

Nov 19 2021, 5:01 PM · System administration (Component upgrades), Archive search

ardumont added a comment to D6395: lister: Add new maven lister.

I will let you know when the full chain is executed.

Well, if works well enough to fill up my vm's hard drive. I'm not sure how to check
the actual loaded content, but it took Gigabytes silently running behind my back.

Nov 19 2021, 4:51 PM

seirl added a subtask for T3739: swh-graph: Remove SWHID -> Node ID mapping, use MPH instead: T3740: swh-graph: Translate node IDs on the Java side, not Python side.

Nov 19 2021, 4:44 PM · Compressed graph service

seirl added a parent task for T3740: swh-graph: Translate node IDs on the Java side, not Python side: T3739: swh-graph: Remove SWHID -> Node ID mapping, use MPH instead.

Nov 19 2021, 4:44 PM · Compressed graph service

seirl triaged T3740: swh-graph: Translate node IDs on the Java side, not Python side as High priority.

Nov 19 2021, 4:42 PM · Compressed graph service

seirl triaged T3739: swh-graph: Remove SWHID -> Node ID mapping, use MPH instead as High priority.

Nov 19 2021, 4:40 PM · Compressed graph service

vsellier committed rSPSITE8b58820956ae: Increase the number of workers of the staging objstorage (authored by vsellier).

Increase the number of workers of the staging objstorage

Nov 19 2021, 4:16 PM

vsellier closed D6663: varnish: specify the authentication type in case of 401.

Nov 19 2021, 4:05 PM

vsellier committed rSPSITE5c17ed204a18: varnish: specify the authentication type in case of 401 (authored by vsellier).

varnish: specify the authentication type in case of 401

Nov 19 2021, 4:05 PM

vsellier added inline comments to D6663: varnish: specify the authentication type in case of 401.

Nov 19 2021, 4:01 PM

vsellier updated the diff for D6663: varnish: specify the authentication type in case of 401.

remove useless response code update

Nov 19 2021, 4:01 PM

ardumont added a comment to T3733: Develop a "Add forge now" mock-up.

Listing and submission view with a standard user (or anonymous):

Nov 19 2021, 2:57 PM · Roadmap 2021, Web app

olasd accepted D6663: varnish: specify the authentication type in case of 401.

Nov 19 2021, 2:19 PM

olasd added inline comments to D6663: varnish: specify the authentication type in case of 401.

Nov 19 2021, 2:19 PM

vsellier requested review of D6663: varnish: specify the authentication type in case of 401.

Nov 19 2021, 2:13 PM

vsellier added a revision to T3621: Create a production read-only objstorage: D6663: varnish: specify the authentication type in case of 401.

Nov 19 2021, 2:13 PM · System administration

ardumont updated the task description for T3716: Migrate giverny from stretch to buster.

Nov 19 2021, 12:13 PM · System administration (Component upgrades)

anlambert closed D6662: loader: Apply SvnLoaderFromRemoteDump optimization only for stale repos.

Nov 19 2021, 12:00 PM

anlambert closed D6661: svn: Ensure proper incremental loading of a repository.

Nov 19 2021, 12:00 PM

anlambert committed rDLDSVNed0728c8b02e: svn: Ensure proper incremental loading of a repository (authored by anlambert).

svn: Ensure proper incremental loading of a repository

Nov 19 2021, 12:00 PM

anlambert committed rDLDSVN0e725edbc31a: loader: Apply SvnLoaderFromRemoteDump optimization only for stale repos (authored by anlambert).

loader: Apply SvnLoaderFromRemoteDump optimization only for stale repos

Nov 19 2021, 12:00 PM

anlambert closed D6660: loader: Ensure to reload from first revision when history got altered.

Nov 19 2021, 12:00 PM

anlambert committed rDLDSVNc9aaa314df27: loader: Ensure to reload from first revision when history got altered (authored by anlambert).

loader: Ensure to reload from first revision when history got altered

Nov 19 2021, 12:00 PM

ardumont accepted D6662: loader: Apply SvnLoaderFromRemoteDump optimization only for stale repos.

Nov 19 2021, 11:56 AM

vsellier closed D6659: logstash: puppetize the main logstash configuration file.

Nov 19 2021, 11:54 AM

vsellier committed rSPSITE53eef5fef7fc: logstash: puppetize the main logstash configuration file (authored by vsellier).

logstash: puppetize the main logstash configuration file

Nov 19 2021, 11:54 AM

ardumont accepted D6661: svn: Ensure proper incremental loading of a repository.

Nice catch.

Nov 19 2021, 11:51 AM

ardumont accepted D6660: loader: Ensure to reload from first revision when history got altered.

Nov 19 2021, 11:47 AM

ardumont accepted D6659: logstash: puppetize the main logstash configuration file.

Nov 19 2021, 11:47 AM

vsellier updated the task description for T3738: Replace failing disks on db1 and storage1 (before the end of february 2022).

Nov 19 2021, 11:37 AM · System administration

vsellier added a project to T3738: Replace failing disks on db1 and storage1 (before the end of february 2022): System administration.

Nov 19 2021, 11:35 AM · System administration

vsellier triaged T3738: Replace failing disks on db1 and storage1 (before the end of february 2022) as Normal priority.

Nov 19 2021, 11:35 AM · System administration

anlambert requested review of D6662: loader: Apply SvnLoaderFromRemoteDump optimization only for stale repos.

Nov 19 2021, 10:55 AM

anlambert requested review of D6661: svn: Ensure proper incremental loading of a repository.

Nov 19 2021, 10:54 AM

anlambert requested review of D6660: loader: Ensure to reload from first revision when history got altered.

Nov 19 2021, 10:53 AM

ardumont added a comment to T3716: Migrate giverny from stretch to buster.

And now giverny seems to refuse to boot back. Stopping a bit that part as I cannot check further yet.

Nov 19 2021, 10:40 AM · System administration (Component upgrades)

ardumont renamed T3716: Migrate giverny from stretch to buster from Migrate giverny from stretch to bullseye to Migrate giverny from stretch to buster.

Nov 19 2021, 10:39 AM · System administration (Component upgrades)