Page MenuHomeSoftware Heritage
Feed All Stories

Sep 15 2021

anlambert triaged T3575: Filter out sdist archives that are not of interest as Normal priority.
Sep 15 2021, 1:40 PM · PyPI loader
ardumont accepted D6265: tarball: Run unzip in quiet mode.
Sep 15 2021, 12:35 PM
ardumont accepted D6264: tarball: Try to get archive format before unpacking it.
Sep 15 2021, 12:35 PM
anlambert requested review of D6265: tarball: Run unzip in quiet mode.
Sep 15 2021, 12:14 PM
anlambert requested review of D6264: tarball: Try to get archive format before unpacking it.
Sep 15 2021, 12:11 PM
vsellier created P1164 flush cassandra buffers.
Sep 15 2021, 12:04 PM
olasd changed the status of T3574: Upgrade hedgedoc to 1.9.0 from Open to Work in Progress.
Sep 15 2021, 11:45 AM · System administration
olasd accepted D6138: package/utils: Handle downloads for urls with missing schema.

Thanks for this change!

Sep 15 2021, 11:42 AM
ardumont added inline comments to D6240: Use extids to filter out already seen revisions across hg origins.
Sep 15 2021, 11:20 AM
vsellier added a comment to T3573: [cassandra] directory and content read benchmarks.

The directory_ls and indirectly the get_content performace was tested with this small script: P1163
A cold restart (all buffer cleared, cassandra restarted) is done between each tests (P1164)

Sep 15 2021, 11:20 AM · System administration, Storage manager
vsellier renamed T3573: [cassandra] directory and content read benchmarks from [cassandra] directory and content read benchmarkss to [cassandra] directory and content read benchmarks.
Sep 15 2021, 11:19 AM · System administration, Storage manager
vsellier created P1163 directory_ls.py.
Sep 15 2021, 11:18 AM
vsellier changed the status of T3573: [cassandra] directory and content read benchmarks from Open to Work in Progress.
Sep 15 2021, 11:11 AM · System administration, Storage manager
swh-public-ci added a comment to D6240: Use extids to filter out already seen revisions across hg origins.

Build is green

Sep 15 2021, 11:09 AM
ardumont updated the diff for D6240: Use extids to filter out already seen revisions across hg origins.

Rebase on top of D6262

Sep 15 2021, 11:08 AM
olasd created P1162 Full visits with null snapshots.
Sep 15 2021, 11:05 AM
olasd added inline comments to D6240: Use extids to filter out already seen revisions across hg origins.
Sep 15 2021, 11:05 AM
swh-public-ci added a comment to D6254: package/tests/test_utils: Remove code duplication.

Build is green

Sep 15 2021, 11:03 AM
anlambert updated the diff for D6254: package/tests/test_utils: Remove code duplication.

Rebase

Sep 15 2021, 11:01 AM
ardumont added inline comments to D6240: Use extids to filter out already seen revisions across hg origins.
Sep 15 2021, 10:58 AM
ardumont added a project to T3572: mercurial loader: Refactor / clean up old implementations and rename appropriately the official one: Mercurial loader.
Sep 15 2021, 10:54 AM · Mercurial loader
ardumont added a project to T3571: mercurial loader: Fix snapshot creation: Mercurial loader.
Sep 15 2021, 10:53 AM · Mercurial loader
ardumont triaged T3572: mercurial loader: Refactor / clean up old implementations and rename appropriately the official one as Normal priority.
Sep 15 2021, 10:50 AM · Mercurial loader
ardumont triaged T3571: mercurial loader: Fix snapshot creation as High priority.
Sep 15 2021, 10:42 AM · Mercurial loader
ardumont created T3571: mercurial loader: Fix snapshot creation.
Sep 15 2021, 10:42 AM · Mercurial loader
ardumont added inline comments to D6262: test: Explicit that 2 visits without change ends up with no snapshot.
Sep 15 2021, 10:40 AM
swh-public-ci added a comment to D6262: test: Explicit that 2 visits without change ends up with no snapshot.

Build is green

Sep 15 2021, 10:34 AM
olasd added inline comments to D6262: test: Explicit that 2 visits without change ends up with no snapshot.
Sep 15 2021, 10:33 AM
ardumont updated the diff for D6262: test: Explicit that 2 visits without change ends up with no snapshot.

Explicit that the snapshot is null

Sep 15 2021, 10:32 AM
ardumont requested review of D6262: test: Explicit that 2 visits without change ends up with no snapshot.
Sep 15 2021, 10:29 AM
swh-public-ci added a comment to D6249: Allow filtering extids per extid_version/extid_type when reading.

Build is green

Sep 15 2021, 10:10 AM
ardumont added inline comments to D6240: Use extids to filter out already seen revisions across hg origins.
Sep 15 2021, 10:10 AM
ardumont updated the diff for D6249: Allow filtering extids per extid_version/extid_type when reading.

Forgot to catch the RemoteException case for the edge cases of the endpoint adaptation

Sep 15 2021, 10:03 AM
vsellier closed T3476: One of the system disks of beaubourg is out of order, a subtask of T3444: 26/07/2021: Unstuck infrastructure outage then post-mortem, as Resolved.
Sep 15 2021, 8:32 AM · System administration
vsellier closed T3476: One of the system disks of beaubourg is out of order as Resolved.

The disk was received Monday and replaced Thuesday by Christophe from the DSI.
The raid card automatically launch the raid rebuild. Everything is ok now.

root@beaubourg:~#  megacli -PDList -aALL
...
Sep 15 2021, 8:32 AM · System administration

Sep 14 2021

vlorentz closed D6201: Add an overview of the metadata workflow.
Sep 14 2021, 8:18 PM
vlorentz committed rDDOC5f92841cb0e9: Add an overview of the metadata workflow (authored by vlorentz).
Add an overview of the metadata workflow
Sep 14 2021, 8:18 PM
Harbormaster failed remote builds in B23601: Diff 22667 for D6249: Allow filtering extids per extid_version/extid_type when reading!
Sep 14 2021, 7:17 PM
swh-public-ci added a comment to D6249: Allow filtering extids per extid_version/extid_type when reading.

Build has FAILED

Sep 14 2021, 7:17 PM
ardumont updated the diff for D6249: Allow filtering extids per extid_version/extid_type when reading.

Fix tests hopefully

Sep 14 2021, 7:10 PM
Harbormaster failed remote builds in B23600: Diff 22666 for D6249: Allow filtering extids per extid_version/extid_type when reading!
Sep 14 2021, 6:54 PM
swh-public-ci added a comment to D6249: Allow filtering extids per extid_version/extid_type when reading.

Build has FAILED

Sep 14 2021, 6:54 PM
ardumont accepted D6260: vault: Only show the first status line.
Sep 14 2021, 6:53 PM
ardumont accepted D6252: package/utils: Improve downloaded filename extraction.
Sep 14 2021, 6:53 PM
ardumont retitled D6249: Allow filtering extids per extid_version/extid_type when reading from Allow filtering extids per extid_version when reading to Allow filtering extids per extid_version/extid_type when reading.
Sep 14 2021, 6:50 PM
ardumont accepted D6254: package/tests/test_utils: Remove code duplication.
Sep 14 2021, 6:49 PM
ardumont updated the diff for D6249: Allow filtering extids per extid_version/extid_type when reading.

Add extid_get_from_target to filter correctly on extid_type and extid_version.

Sep 14 2021, 6:47 PM
ardumont added inline comments to D6240: Use extids to filter out already seen revisions across hg origins.
Sep 14 2021, 6:31 PM
vlorentz requested review of D6260: vault: Only show the first status line.
Sep 14 2021, 6:09 PM
vlorentz closed D6259: Make swh-vault use mailhog.
Sep 14 2021, 5:56 PM
vlorentz committed rDENV5e31155f91b9: Make swh-vault use mailhog (authored by vlorentz).
Make swh-vault use mailhog
Sep 14 2021, 5:56 PM
anlambert accepted D6258: Add support for custom SMTP configuration.
Sep 14 2021, 5:33 PM
anlambert accepted D6259: Make swh-vault use mailhog.
Sep 14 2021, 5:32 PM
vlorentz requested review of D6258: Add support for custom SMTP configuration.
Sep 14 2021, 5:27 PM
vlorentz requested review of D6259: Make swh-vault use mailhog.
Sep 14 2021, 5:26 PM
olasd added inline comments to D6240: Use extids to filter out already seen revisions across hg origins.
Sep 14 2021, 5:25 PM
vlorentz closed D6257: Make mailhog available via nginx.
Sep 14 2021, 5:23 PM
vlorentz committed rDENVe768fe1e222c: Make mailhog available via nginx (authored by vlorentz).
Make mailhog available via nginx
Sep 14 2021, 5:23 PM
ardumont added inline comments to D6240: Use extids to filter out already seen revisions across hg origins.
Sep 14 2021, 5:22 PM
swh-public-ci added a comment to D6240: Use extids to filter out already seen revisions across hg origins.

Build is green

Sep 14 2021, 5:19 PM
ardumont added a comment to T3563: Analyze and make the bitbucket ingestion faster.

By the way, i forgot:

Sep 14 2021, 5:18 PM · System administration, Mercurial loader
ardumont updated the diff for D6240: Use extids to filter out already seen revisions across hg origins.

Drop no longer needed conditional

Sep 14 2021, 5:17 PM
ardumont updated the summary of D6240: Use extids to filter out already seen revisions across hg origins.
Sep 14 2021, 5:09 PM
anlambert accepted D6257: Make mailhog available via nginx.
Sep 14 2021, 5:06 PM
swh-public-ci added a comment to D6240: Use extids to filter out already seen revisions across hg origins.

Build is green

Sep 14 2021, 4:55 PM
ardumont updated the diff for D6240: Use extids to filter out already seen revisions across hg origins.

Adapt docstring and rework commit message.

Sep 14 2021, 4:54 PM
vlorentz claimed T75: Check integrity of directories, revisions, and releases.
Sep 14 2021, 4:50 PM · Archive content, Restricted Project
vlorentz requested review of D6257: Make mailhog available via nginx.
Sep 14 2021, 4:45 PM
swh-public-ci added a comment to D6240: Use extids to filter out already seen revisions across hg origins.

Build is green

Sep 14 2021, 4:20 PM
ardumont added inline comments to D6240: Use extids to filter out already seen revisions across hg origins.
Sep 14 2021, 4:19 PM
ardumont updated the diff for D6240: Use extids to filter out already seen revisions across hg origins.

Actually filter through snapshot then on the remaining data set, filter out through
extids mapping.

Sep 14 2021, 4:18 PM
swh-public-ci added a comment to D6252: package/utils: Improve downloaded filename extraction.

Build is green

Sep 14 2021, 4:17 PM
anlambert updated the diff for D6252: package/utils: Improve downloaded filename extraction.
  • Use single regexp and strip quotes and spaces
  • Handle UTF-8 encoding defined in rfc5987
  • Add more test cases
Sep 14 2021, 4:15 PM
aeviso closed D6255: Generalize types for `content_add` and `directory_add` in the storage interface.
Sep 14 2021, 3:54 PM
aeviso committed rDPROV3383cae57ef7: Generalize types for `content_add` and `directory_add` in the storage interface (authored by aeviso).
Generalize types for `content_add` and `directory_add` in the storage interface
Sep 14 2021, 3:54 PM
jayeshv accepted D6255: Generalize types for `content_add` and `directory_add` in the storage interface.
Sep 14 2021, 3:49 PM
aeviso requested review of D6256: Add StatsD support to `ArchiveInterface` implementations.
Sep 14 2021, 3:39 PM
swh-public-ci added a comment to D6240: Use extids to filter out already seen revisions across hg origins.

Build is green

Sep 14 2021, 3:36 PM
ardumont updated the diff for D6240: Use extids to filter out already seen revisions across hg origins.

Possibly forgotten comment changes

Sep 14 2021, 3:32 PM
swh-public-ci added a comment to D6165: Add new RabbitMQ-based client/server API.

Build is green

Sep 14 2021, 3:26 PM
ardumont added inline comments to D6240: Use extids to filter out already seen revisions across hg origins.
Sep 14 2021, 3:26 PM
aeviso updated the diff for D6165: Add new RabbitMQ-based client/server API.
  • Improve timeout while waiting for response handling on client side
Sep 14 2021, 3:23 PM
swh-public-ci added a comment to D6165: Add new RabbitMQ-based client/server API.

Build is green

Sep 14 2021, 3:14 PM
vlorentz added a comment to D6252: package/utils: Improve downloaded filename extraction.

Update: Also handle quoted filename in content-disposition header parsing.

Sep 14 2021, 3:13 PM
aeviso requested review of D6255: Generalize types for `content_add` and `directory_add` in the storage interface.
Sep 14 2021, 3:12 PM
vlorentz requested changes to D6252: package/utils: Improve downloaded filename extraction.

The filename should be sanitized. What about using just the extension, and only if it matches [a-zA-Z0-9_-]+?

Sep 14 2021, 3:10 PM
aeviso updated the diff for D6165: Add new RabbitMQ-based client/server API.
  • Add new RabbitMQ-based client/server API
  • Split set method's requests into several queues on server side
  • Remove get queues and have client read from ProvenanceStorage directly
  • Add support for relation_add to the RabbitMQ server
  • Refactor server to use multiple sub-processes instead of threads
  • Rework ProvenanceStorageRabbitMQWorker to handle connection loss
  • Remove old client/server storage based on swh.core.api.RPCClient
  • Improve connection error handling on both client and server side
  • Switch to use a topic exchange instead of a direct one on remote backend
Sep 14 2021, 3:09 PM
anlambert created P1161 (An Untitled Masterwork).
Sep 14 2021, 3:08 PM
olasd accepted D6252: package/utils: Improve downloaded filename extraction.

I've suggested some refinements to the regexps, and I'd like the tests to have some more "adversarial" examples (e.g. filenames with spaces, headers with out of order fields, etc.), but this is still a good improvement over the status quo!

Sep 14 2021, 3:06 PM
swh-public-ci added a comment to D6165: Add new RabbitMQ-based client/server API.

Build is green

Sep 14 2021, 2:56 PM
aeviso updated the diff for D6165: Add new RabbitMQ-based client/server API.
  • Split set method's requests into several queues on server side
  • Remove get queues and have client read from ProvenanceStorage directly
  • Add support for relation_add to the RabbitMQ server
  • Refactor server to use multiple sub-processes instead of threads
  • Rework ProvenanceStorageRabbitMQWorker to handle connection loss
  • Remove old client/server storage based on swh.core.api.RPCClient
  • Improve connection error handling on both client and server side
  • Generalize types for content_add and directory_add in the storage interface
  • Switch to use a topic exchange instead of a direct one on remote backend
Sep 14 2021, 2:53 PM
ardumont updated the summary of D6240: Use extids to filter out already seen revisions across hg origins.
Sep 14 2021, 2:51 PM
vlorentz accepted D6253: add CVS loader to the swh-loader.rst index.

Can you move it before git so it's sorted alphabetically?

Sep 14 2021, 2:47 PM
swh-public-ci added a comment to D6254: package/tests/test_utils: Remove code duplication.

Build is green

Sep 14 2021, 2:32 PM
swh-public-ci added a comment to D6252: package/utils: Improve downloaded filename extraction.

Build is green

Sep 14 2021, 2:31 PM
anlambert updated the diff for D6254: package/tests/test_utils: Remove code duplication.

Rebase

Sep 14 2021, 2:29 PM
anlambert updated the diff for D6252: package/utils: Improve downloaded filename extraction.

Update: Also handle quoted filename in content-disposition header parsing.

Sep 14 2021, 2:28 PM
swh-public-ci added a comment to D6240: Use extids to filter out already seen revisions across hg origins.

Build is green

Sep 14 2021, 2:26 PM
ardumont updated the diff for D6240: Use extids to filter out already seen revisions across hg origins.

Add back dropped test (stash mix up)

Sep 14 2021, 2:23 PM
swh-public-ci added a comment to D6240: Use extids to filter out already seen revisions across hg origins.

Build is green

Sep 14 2021, 2:18 PM