Page MenuHomeSoftware Heritage
Feed All Stories

Sep 15 2021

ardumont updated the diff for D6249: Allow filtering extids per extid_version/extid_type when reading.

Adapt according to suggestions.

Sep 15 2021, 5:17 PM
ardumont added inline comments to D6249: Allow filtering extids per extid_version/extid_type when reading.
Sep 15 2021, 5:15 PM
vsellier added a comment to T3573: [cassandra] directory and content read benchmarks.

Test of a the new D6269 patch:

Sep 15 2021, 5:12 PM · System administration, Storage manager
ardumont closed D6262: test: Explicit that 2 visits without change ends up with no snapshot.
Sep 15 2021, 5:05 PM
ardumont committed rDLDHGbf39050051ec: test: Explicit that 2 visits without change ends up with no snapshot (authored by ardumont).
test: Explicit that 2 visits without change ends up with no snapshot
Sep 15 2021, 5:05 PM
ardumont added inline comments to D6268: mercurial: Build snapshot on visits.
Sep 15 2021, 5:03 PM
swh-public-ci added a comment to D6268: mercurial: Build snapshot on visits.

Build is green

Sep 15 2021, 5:02 PM
swh-public-ci added a comment to D6240: Use extids to filter out already seen revisions across hg origins.

Build is green

Sep 15 2021, 5:01 PM
swh-public-ci added a comment to D6262: test: Explicit that 2 visits without change ends up with no snapshot.

Build is green

Sep 15 2021, 4:59 PM
ardumont updated the diff for D6268: mercurial: Build snapshot on visits.

Rebase

Sep 15 2021, 4:59 PM
ardumont updated the diff for D6240: Use extids to filter out already seen revisions across hg origins.

Rebase

Sep 15 2021, 4:59 PM
ardumont updated the diff for D6262: test: Explicit that 2 visits without change ends up with no snapshot.

Adapt according to suggestion

Sep 15 2021, 4:57 PM
swh-public-ci added a comment to D6268: mercurial: Build snapshot on visits.

Build is green

Sep 15 2021, 4:57 PM
ardumont updated the diff for D6268: mercurial: Build snapshot on visits.

Fix the remaining suggestion (i finally got it)

Sep 15 2021, 4:55 PM
ardumont added inline comments to D6268: mercurial: Build snapshot on visits.
Sep 15 2021, 4:52 PM
swh-public-ci added a comment to D6268: mercurial: Build snapshot on visits.

Build is green

Sep 15 2021, 4:52 PM
ardumont updated the diff for D6268: mercurial: Build snapshot on visits.

Fix/revert almost all that got said.

Sep 15 2021, 4:49 PM
ardumont added inline comments to D6268: mercurial: Build snapshot on visits.
Sep 15 2021, 4:48 PM
ardumont added inline comments to D6268: mercurial: Build snapshot on visits.
Sep 15 2021, 4:43 PM
anlambert closed D6271: tarball: Add support for uncompressing crate files.
Sep 15 2021, 4:32 PM
anlambert committed rDCORE0ef57a4dd346: tarball: Add support for uncompressing crate files (authored by anlambert).
tarball: Add support for uncompressing crate files
Sep 15 2021, 4:32 PM
aeviso requested review of D6273: Remove remote storage based on `swh.core.api.RPCClient`.
Sep 15 2021, 4:30 PM
aeviso retitled D6165: Add new RabbitMQ-based client/server API from [WIP] Add new RabbitMQ-based client/server API to Add new RabbitMQ-based client/server API.
Sep 15 2021, 4:26 PM
swh-public-ci added a comment to D6165: Add new RabbitMQ-based client/server API.

Build is green

Sep 15 2021, 4:25 PM
aeviso added a reviewer for D6272: Remove remote storage based on `swh.core.api.RPCClient`: olasd.
Sep 15 2021, 4:25 PM
swh-public-ci added a comment to D6271: tarball: Add support for uncompressing crate files.

Build is green

Sep 15 2021, 4:25 PM
aeviso requested review of D6272: Remove remote storage based on `swh.core.api.RPCClient`.
Sep 15 2021, 4:23 PM
anlambert updated the diff for D6271: tarball: Add support for uncompressing crate files.

Simply use of str.endswith

Sep 15 2021, 4:23 PM
swh-public-ci added a comment to D6270: pypi/loader: Filter out sdist archives not of interest.

Build is green

Sep 15 2021, 4:23 PM
anlambert added inline comments to D6271: tarball: Add support for uncompressing crate files.
Sep 15 2021, 4:22 PM
aeviso updated the diff for D6165: Add new RabbitMQ-based client/server API.
  • Add new RabbitMQ-based client/server API
  • Rework ProvenanceStorageRabbitMQWorker to handle connection loss
  • Improve server/client shoutdown logic
Sep 15 2021, 4:22 PM
anlambert updated the diff for D6270: pypi/loader: Filter out sdist archives not of interest.

Remove no longer needed call to any

Sep 15 2021, 4:21 PM
swh-public-ci added a comment to D6270: pypi/loader: Filter out sdist archives not of interest.

Build is green

Sep 15 2021, 4:20 PM
anlambert updated the diff for D6270: pypi/loader: Filter out sdist archives not of interest.

Simplify some code

Sep 15 2021, 4:18 PM
vlorentz accepted D6271: tarball: Add support for uncompressing crate files.
Sep 15 2021, 4:14 PM
swh-public-ci added a comment to D6165: Add new RabbitMQ-based client/server API.

Build is green

Sep 15 2021, 4:05 PM
vlorentz requested changes to D6249: Allow filtering extids per extid_version/extid_type when reading.
Sep 15 2021, 4:03 PM
aeviso updated the diff for D6165: Add new RabbitMQ-based client/server API.

rebase

Sep 15 2021, 4:02 PM
anlambert requested review of D6271: tarball: Add support for uncompressing crate files.
Sep 15 2021, 3:56 PM
anlambert requested review of D6270: pypi/loader: Filter out sdist archives not of interest.
Sep 15 2021, 3:42 PM
anlambert added a revision to T3575: Filter out sdist archives that are not of interest: D6270: pypi/loader: Filter out sdist archives not of interest.
Sep 15 2021, 3:39 PM · PyPI loader
olasd accepted D6262: test: Explicit that 2 visits without change ends up with no snapshot.

It's a bit weird to accept a diff with an explicit reference to a bug. :P

Sep 15 2021, 3:15 PM
olasd requested changes to D6268: mercurial: Build snapshot on visits.
Sep 15 2021, 3:12 PM
swh-public-ci added a comment to D6268: mercurial: Build snapshot on visits.

Build is green

Sep 15 2021, 3:04 PM
ardumont added inline comments to D6268: mercurial: Build snapshot on visits.
Sep 15 2021, 3:03 PM
ardumont updated the diff for D6268: mercurial: Build snapshot on visits.

Rebase with the missing test added back and adapted.

Sep 15 2021, 3:02 PM
swh-public-ci added a comment to D6240: Use extids to filter out already seen revisions across hg origins.

Build is green

Sep 15 2021, 3:02 PM
swh-public-ci added a comment to D6240: Use extids to filter out already seen revisions across hg origins.

Build is green

Sep 15 2021, 3:01 PM
ardumont updated the diff for D6240: Use extids to filter out already seen revisions across hg origins.

Use correct range of commits

Sep 15 2021, 3:00 PM
ardumont updated the diff for D6240: Use extids to filter out already seen revisions across hg origins.

Add back dropped test by mistake (rebase messed up probably)

Sep 15 2021, 2:59 PM
ardumont requested review of D6268: mercurial: Build snapshot on visits.
Sep 15 2021, 2:57 PM
ardumont added a revision to T3571: mercurial loader: Fix snapshot creation: D6268: mercurial: Build snapshot on visits.
Sep 15 2021, 2:55 PM · Mercurial loader
vsellier added a comment to T3573: [cassandra] directory and content read benchmarks.

2 flame graphs of the previous directory_ls:

  • one-by-one

first run (cache cold):

Sep 15 2021, 2:55 PM · System administration, Storage manager
anlambert renamed T3575: Filter out sdist archives that are not of interest from Filter out sdist archives that cannot be extracted by swh.core.tarball.uncompress to Filter out sdist archives that are not of interest.
Sep 15 2021, 2:52 PM · PyPI loader
anlambert abandoned D6267: tarball: Add support for uncompressing egg files.

Apparently, decision has been made to not archive egg files so abandoning this.

Sep 15 2021, 2:49 PM
anlambert closed T3576: Add support for Python eggs file processing as Invalid.

Apparently we decided not to archive them so better filtering those files out as proposed in T3575.

Sep 15 2021, 2:47 PM · PyPI loader
anlambert updated the task description for T3576: Add support for Python eggs file processing .
Sep 15 2021, 2:47 PM · PyPI loader
Harbormaster failed remote builds in B23626: Diff 22693 for D6249: Allow filtering extids per extid_version/extid_type when reading!
Sep 15 2021, 2:46 PM
swh-public-ci added a comment to D6249: Allow filtering extids per extid_version/extid_type when reading.

Build was aborted

Sep 15 2021, 2:46 PM
vsellier added a comment to T3573: [cassandra] directory and content read benchmarks.

This is the results of the different runs:

Sep 15 2021, 2:40 PM · System administration, Storage manager
ardumont updated the diff for D6249: Allow filtering extids per extid_version/extid_type when reading.

Drop the extra index and use the optimization started in D6263 (it works now)

Sep 15 2021, 2:25 PM
anlambert requested review of D6267: tarball: Add support for uncompressing egg files.
Sep 15 2021, 2:23 PM
vlorentz created P1165 cassandra extid allow filtering.
Sep 15 2021, 2:23 PM
anlambert added a revision to T3576: Add support for Python eggs file processing : D6267: tarball: Add support for uncompressing egg files.
Sep 15 2021, 2:22 PM · PyPI loader
vlorentz closed D6261: Add the whole traceback in error messages..
Sep 15 2021, 2:03 PM
vlorentz closed D6258: Add support for custom SMTP configuration.
Sep 15 2021, 2:03 PM
vlorentz committed rDVAU523f91583935: Add support for custom SMTP configuration (authored by vlorentz).
Add support for custom SMTP configuration
Sep 15 2021, 2:03 PM
vlorentz committed rDVAU3681a72c561d: Add the whole traceback in error messages. (authored by vlorentz).
Add the whole traceback in error messages.
Sep 15 2021, 2:03 PM
swh-public-ci added a comment to D6261: Add the whole traceback in error messages..

Build is green

Sep 15 2021, 2:02 PM
aeviso added a reviewer for D6165: Add new RabbitMQ-based client/server API: olasd.
Sep 15 2021, 2:02 PM
aeviso added a reviewer for D6256: Add StatsD support to `ArchiveInterface` implementations: jayeshv.
Sep 15 2021, 2:01 PM
aeviso added a reviewer for D6266: Add StatsD support to the main algorithms of the module: jayeshv.
Sep 15 2021, 2:01 PM
vlorentz updated the diff for D6261: Add the whole traceback in error messages..

fix commit range

Sep 15 2021, 2:00 PM
anlambert updated the task description for T3576: Add support for Python eggs file processing .
Sep 15 2021, 2:00 PM · PyPI loader
vlorentz added a revision to T3566: git-fsck errors in the git-bare cooker are not properly reported: D6261: Add the whole traceback in error messages..
Sep 15 2021, 2:00 PM · Vault
vlorentz updated the summary of D6261: Add the whole traceback in error messages..
Sep 15 2021, 2:00 PM
anlambert updated the task description for T3576: Add support for Python eggs file processing .
Sep 15 2021, 2:00 PM · PyPI loader
anlambert triaged T3576: Add support for Python eggs file processing as Normal priority.
Sep 15 2021, 1:58 PM · PyPI loader
ardumont accepted D6261: Add the whole traceback in error messages..

Careful on the range of commits, i see duplicated stuff from the other diff about smtp setup (the virtual vcfg configuration part).

Sep 15 2021, 1:57 PM
ardumont accepted D6258: Add support for custom SMTP configuration.
Sep 15 2021, 1:56 PM
swh-public-ci added a comment to D6165: Add new RabbitMQ-based client/server API.

Build is green

Sep 15 2021, 1:53 PM
swh-public-ci added a comment to D6254: package/tests/test_utils: Remove code duplication.

Build is green

Sep 15 2021, 1:52 PM
swh-public-ci added a comment to D6252: package/utils: Improve downloaded filename extraction.

Build is green

Sep 15 2021, 1:52 PM
vlorentz requested review of D6261: Add the whole traceback in error messages..
Sep 15 2021, 1:51 PM
aeviso requested review of D6266: Add StatsD support to the main algorithms of the module.
Sep 15 2021, 1:49 PM
swh-public-ci added a comment to D6256: Add StatsD support to `ArchiveInterface` implementations.

Build is green

Sep 15 2021, 1:49 PM
anlambert closed D6264: tarball: Try to get archive format before unpacking it.
Sep 15 2021, 1:48 PM
anlambert committed rDCORE632673f6bd1b: tarball: Try to get archive format before unpacking it (authored by anlambert).
tarball: Try to get archive format before unpacking it
Sep 15 2021, 1:48 PM
aeviso updated the diff for D6165: Add new RabbitMQ-based client/server API.
  • Add new RabbitMQ-based client/server API
  • Split set method's requests into several queues on server side
  • Remove get queues and have client read from ProvenanceStorage directly
  • Add support for relation_add to the RabbitMQ server
  • Refactor server to use multiple sub-processes instead of threads
  • Rework ProvenanceStorageRabbitMQWorker to handle connection loss
  • Remove old client/server storage based on swh.core.api.RPCClient
  • Improve connection error handling on both client and server side
  • Switch to use a topic exchange instead of a direct one on remote backend
  • Improve timeout while waiting for response handling on client side
  • Improve server/client shoutdown logic
Sep 15 2021, 1:47 PM
anlambert updated the diff for D6254: package/tests/test_utils: Remove code duplication.

Rebase

Sep 15 2021, 1:46 PM
anlambert updated the diff for D6252: package/utils: Improve downloaded filename extraction.

Update: Add extracted filename extra sanitization pass.

Sep 15 2021, 1:46 PM
swh-public-ci added a comment to D6264: tarball: Try to get archive format before unpacking it.

Build is green

Sep 15 2021, 1:45 PM
aeviso updated the summary of D6165: Add new RabbitMQ-based client/server API.
Sep 15 2021, 1:45 PM
aeviso retitled D6256: Add StatsD support to `ArchiveInterface` implementations from Add StatsD support on `ArchiveInterface` implementations to Add StatsD support to `ArchiveInterface` implementations.
Sep 15 2021, 1:44 PM
vlorentz closed D6260: vault: Only show the first status line.
Sep 15 2021, 1:43 PM
vlorentz committed rDWAPPS3a5f6087f1ba: vault: Only show the first status line (authored by vlorentz).
vault: Only show the first status line
Sep 15 2021, 1:43 PM
aeviso updated the diff for D6256: Add StatsD support to `ArchiveInterface` implementations.

rebase

Sep 15 2021, 1:43 PM
anlambert updated the diff for D6264: tarball: Try to get archive format before unpacking it.

Rebase

Sep 15 2021, 1:42 PM
anlambert closed D6265: tarball: Run unzip in quiet mode.
Sep 15 2021, 1:41 PM
anlambert committed rDCOREe45733e9b47d: tarball: Run unzip in quiet mode (authored by anlambert).
tarball: Run unzip in quiet mode
Sep 15 2021, 1:41 PM