Page MenuHomeSoftware Heritage
Feed All Stories

Oct 8 2021

olasd closed D6447: buffer: add some debug logging for number of objects sent.
Oct 8 2021, 5:27 PM
olasd committed rDSTO3441f68985ae: buffer: add some debug logging for number of objects sent (authored by olasd).
buffer: add some debug logging for number of objects sent
Oct 8 2021, 5:27 PM
ardumont closed D6438: docs: Explain task type registering to complete the save forge doc.
Oct 8 2021, 5:17 PM
ardumont committed rDLS04dc6280916a: docs: Explain task type registering to complete the save forge doc (authored by ardumont).
docs: Explain task type registering to complete the save forge doc
Oct 8 2021, 5:17 PM
ardumont closed D6428: docs: Add a save forge documentation.
Oct 8 2021, 5:17 PM
ardumont committed rDLS1a9c08c93feb: docs: Add a save forge documentation (authored by ardumont).
docs: Add a save forge documentation
Oct 8 2021, 5:17 PM
Harbormaster failed to build B24335: rDSTO7c5b0ec15e40: buffer: add a threshold for the number of revision parents in one batch for rDSTO7c5b0ec15e40: buffer: add a threshold for the number of revision parents in one batch!
Oct 8 2021, 5:14 PM
vsellier added a revision to T3621: Create a production read-only objstorage: D6448: Deploy a read-only objstorage on moma.
Oct 8 2021, 5:14 PM · System administration
vsellier requested review of D6448: Deploy a read-only objstorage on moma.
Oct 8 2021, 5:14 PM
swh-public-ci added a comment to D6447: buffer: add some debug logging for number of objects sent.

Build is green

Oct 8 2021, 5:12 PM
olasd closed D6446: buffer: add a threshold for the estimated size of revision and release batches.
Oct 8 2021, 5:05 PM
olasd committed rDSTOb6040142fe72: buffer: add a threshold for the estimated size of revision and release batches (authored by olasd).
buffer: add a threshold for the estimated size of revision and release batches
Oct 8 2021, 5:05 PM
Harbormaster failed remote builds in B24337: Diff 23421 for D6447: buffer: add some debug logging for number of objects sent!
Oct 8 2021, 5:03 PM
swh-public-ci added a comment to D6447: buffer: add some debug logging for number of objects sent.

Build has FAILED

Oct 8 2021, 5:03 PM
swh-public-ci added a comment to D6446: buffer: add a threshold for the estimated size of revision and release batches.

Build is green

Oct 8 2021, 5:03 PM
olasd updated the diff for D6447: buffer: add some debug logging for number of objects sent.

rebase

Oct 8 2021, 4:54 PM
olasd updated the diff for D6446: buffer: add a threshold for the estimated size of revision and release batches.

Fix revision -> release typo in release_add flush call

Oct 8 2021, 4:54 PM
ardumont accepted D6446: buffer: add a threshold for the estimated size of revision and release batches.

lgtm, providing the typo is fixed ;)

Oct 8 2021, 4:53 PM
olasd closed D6445: buffer: add a threshold for the number of revision parents in one batch.
Oct 8 2021, 4:53 PM
olasd committed rDSTO7c5b0ec15e40: buffer: add a threshold for the number of revision parents in one batch (authored by olasd).
buffer: add a threshold for the number of revision parents in one batch
Oct 8 2021, 4:53 PM
ardumont added inline comments to D6446: buffer: add a threshold for the estimated size of revision and release batches.
Oct 8 2021, 4:50 PM
ardumont accepted D6447: buffer: add some debug logging for number of objects sent.
Oct 8 2021, 4:49 PM
ardumont added a comment to D6447: buffer: add some debug logging for number of objects sent.

And i assume they were helpful ;)

Oct 8 2021, 4:49 PM
swh-public-ci added a comment to D6441: Deprecate /browse/origin/log/ URLs.

Build is green

Oct 8 2021, 4:46 PM
Harbormaster failed remote builds in B24333: Diff 23417 for D6441: Deprecate /browse/origin/log/ URLs!
Oct 8 2021, 4:44 PM
swh-public-ci added a comment to D6441: Deprecate /browse/origin/log/ URLs.

Build has FAILED

Oct 8 2021, 4:44 PM
jayeshv updated the diff for D6441: Deprecate /browse/origin/log/ URLs.

Diff for log routes

Oct 8 2021, 4:31 PM
olasd requested review of D6447: buffer: add some debug logging for number of objects sent.
Oct 8 2021, 4:23 PM
jayeshv updated the diff for D6441: Deprecate /browse/origin/log/ URLs.

WIP: Many URLs defined in the origin view (urlconf) are redundant.
This fix redirects /origin/branches?origin_url=<url> and
/origin/<url>/branches to /snapshot-branches/

Oct 8 2021, 4:22 PM
swh-public-ci added a comment to D6442: Extract the path slicing logic in a dedicated PathSlicer class.

Build is green

Oct 8 2021, 4:21 PM
ardumont added a comment to T3632: Investigate the ContentDisallowed exception.

It still appears in sentry.
Although it seems specific to staging (which is a hint of a divergence of some sort).

Oct 8 2021, 4:20 PM · Scheduling utilities
douardda updated the diff for D6442: Extract the path slicing logic in a dedicated PathSlicer class.

Better docstrings and kill a few map()

Oct 8 2021, 4:19 PM
swh-public-ci added a comment to D6438: docs: Explain task type registering to complete the save forge doc.

Build is green

Oct 8 2021, 4:10 PM
swh-public-ci added a comment to D6428: docs: Add a save forge documentation.

Build is green

Oct 8 2021, 4:10 PM
ardumont accepted D6445: buffer: add a threshold for the number of revision parents in one batch.
Oct 8 2021, 4:10 PM
olasd requested review of D6446: buffer: add a threshold for the estimated size of revision and release batches.
Oct 8 2021, 4:08 PM
ardumont added a project to T3632: Investigate the ContentDisallowed exception: Scheduling utilities.
Oct 8 2021, 4:08 PM · Scheduling utilities
olasd requested review of D6445: buffer: add a threshold for the number of revision parents in one batch.
Oct 8 2021, 4:08 PM
ardumont updated the diff for D6438: docs: Explain task type registering to complete the save forge doc.

Rebase

Oct 8 2021, 4:08 PM
ardumont added inline comments to D6428: docs: Add a save forge documentation.
Oct 8 2021, 4:08 PM
ardumont updated the diff for D6428: docs: Add a save forge documentation.

Fix typo

Oct 8 2021, 4:07 PM
Harbormaster failed to build B24324: rDSTO5edc0ba7ac12: buffer: add a threshold for the number of directory entries in one batch for rDSTO5edc0ba7ac12: buffer: add a threshold for the number of directory entries in one batch!
Oct 8 2021, 4:07 PM
swh-public-ci added a comment to D6438: docs: Explain task type registering to complete the save forge doc.

Build is green

Oct 8 2021, 4:07 PM
swh-public-ci added a comment to D6428: docs: Add a save forge documentation.

Build is green

Oct 8 2021, 4:07 PM
zack triaged T3639: prepare quote for "granet2", next gen swh-graph compression server as High priority.
Oct 8 2021, 4:04 PM · System administration
ardumont updated the diff for D6438: docs: Explain task type registering to complete the save forge doc.

Rebase

Oct 8 2021, 4:04 PM
ardumont updated the diff for D6428: docs: Add a save forge documentation.

Adapt according to suggestion

Oct 8 2021, 4:03 PM
olasd added a comment to T3625: Reduce git loader memory footprint.
In T3625#71799, @olasd wrote:

While we're at it, we should probably be adding some thresholds in the buffer proxy for:

  • cumulated length of messages for revisions and releases
Oct 8 2021, 4:02 PM · Git loader
ardumont committed rDSNIP357946d60f8c: Autoscale workers depending on number of messages in queues (authored by ardumont).
Autoscale workers depending on number of messages in queues
Oct 8 2021, 4:01 PM
ardumont committed rDSNIPe2d6fc879bc2: config-map: Change pool policy to prefork and drop --events (authored by ardumont).
config-map: Change pool policy to prefork and drop --events
Oct 8 2021, 4:01 PM
olasd added a revision to T3625: Reduce git loader memory footprint: D6445: buffer: add a threshold for the number of revision parents in one batch.
Oct 8 2021, 4:01 PM · Git loader
olasd added a revision to T3625: Reduce git loader memory footprint: D6446: buffer: add a threshold for the estimated size of revision and release batches.
Oct 8 2021, 3:58 PM · Git loader
ardumont added a comment to T3632: Investigate the ContentDisallowed exception.

Since i don't really see how to check that, I was in a mood to just mark it as
resolve in sentry until sentry shout "regression" (or not hopefully)

Oct 8 2021, 3:57 PM · Scheduling utilities
olasd closed D6443: buffer: add a threshold for the number of directory entries in one batch.
Oct 8 2021, 3:56 PM
olasd committed rDSTO5edc0ba7ac12: buffer: add a threshold for the number of directory entries in one batch (authored by olasd).
buffer: add a threshold for the number of directory entries in one batch
Oct 8 2021, 3:56 PM
olasd committed rDSTOabe95b34a2f9: filter: add filtering for release_add (authored by olasd).
filter: add filtering for release_add
Oct 8 2021, 3:56 PM
olasd committed rDSTOc52b7b667911: filter: do not call the underlying functions if there's nothing to add (authored by olasd).
filter: do not call the underlying functions if there's nothing to add
Oct 8 2021, 3:56 PM
olasd closed D6427: swh.storage filter/buffer improvements.
Oct 8 2021, 3:56 PM
olasd committed rDSTO5d5d4c941eac: buffer: Ensure that we don't send data from empty buffers (authored by olasd).
buffer: Ensure that we don't send data from empty buffers
Oct 8 2021, 3:56 PM
douardda added inline comments to D6442: Extract the path slicing logic in a dedicated PathSlicer class.
Oct 8 2021, 3:53 PM
douardda closed D6444: docker: configure and document the APP evironment variable for celery.
Oct 8 2021, 3:51 PM
douardda committed rDENVeefd5e532124: docker: configure and document the APP evironment variable for celery (authored by douardda).
docker: configure and document the APP evironment variable for celery
Oct 8 2021, 3:51 PM
ardumont accepted D6444: docker: configure and document the APP evironment variable for celery.
Oct 8 2021, 3:49 PM
ardumont accepted D6443: buffer: add a threshold for the number of directory entries in one batch.

Thanks.

Oct 8 2021, 3:46 PM
douardda accepted D6443: buffer: add a threshold for the number of directory entries in one batch.

Thx

Oct 8 2021, 3:46 PM
ardumont updated the task description for T3592: POC elastic worker infrastructure.
Oct 8 2021, 3:41 PM · System administration
douardda updated the diff for D6444: docker: configure and document the APP evironment variable for celery.

be a bit more consistent...

Oct 8 2021, 3:29 PM
douardda added a comment to D6410: Allow application/x-msgpack deserialization again.

as @vlorentz pointed out [1], this change should be irrelevant though...

[1] https://github.com/celery/kombu/blob/master/kombu/serialization.py#L369-L372

does not seem to be the proper fix.

FTR, using the celery cli tool directly from a development venv to interact with the celery server running in the docker compose test setup (as described there ) used to work ok, but not any more.

One have to specify the app, like:

celery --app=swh.scheduler.celery_backend.config.app status

[edit] I use celery 5.1.2 in my venv.

Oct 8 2021, 3:27 PM
douardda requested review of D6444: docker: configure and document the APP evironment variable for celery.
Oct 8 2021, 3:25 PM
olasd requested review of D6443: buffer: add a threshold for the number of directory entries in one batch.
Oct 8 2021, 3:21 PM
ardumont accepted D6427: swh.storage filter/buffer improvements.
Oct 8 2021, 3:14 PM
douardda added a comment to T3632: Investigate the ContentDisallowed exception.

Unless I'm mistaken, this error does not appear in sentry any more, right?

Oct 8 2021, 3:06 PM · Scheduling utilities
olasd added a revision to T3625: Reduce git loader memory footprint: D6443: buffer: add a threshold for the number of directory entries in one batch.
Oct 8 2021, 3:06 PM · Git loader
douardda added a comment to D6410: Allow application/x-msgpack deserialization again.

as @vlorentz pointed out [1], this change should be irrelevant though...

[1] https://github.com/celery/kombu/blob/master/kombu/serialization.py#L369-L372

Oct 8 2021, 3:04 PM
douardda accepted D6427: swh.storage filter/buffer improvements.

looks fine to me

Oct 8 2021, 2:53 PM
douardda accepted D6428: docs: Add a save forge documentation.

Ok but see my 2 (nitpicky) comments

Oct 8 2021, 2:49 PM
vlorentz added inline comments to D6442: Extract the path slicing logic in a dedicated PathSlicer class.
Oct 8 2021, 2:47 PM
olasd added inline comments to D6442: Extract the path slicing logic in a dedicated PathSlicer class.
Oct 8 2021, 2:46 PM
douardda accepted D6431: Rename imports of swh.model.identifiers to fix deprecation warnings..

LGTM thx

Oct 8 2021, 2:46 PM
douardda added a comment to T3104: Persistent readonly perfect hash table.

[FTR] See https://forge.softwareheritage.org/source/swh-perfecthash/

Oct 8 2021, 2:43 PM · Object storage (RedHat collaboration)
swh-public-ci added a comment to D6427: swh.storage filter/buffer improvements.

Build is green

Oct 8 2021, 2:42 PM
Harbormaster failed remote builds in B24316: Diff 23399 for D6427: swh.storage filter/buffer improvements!
Oct 8 2021, 2:34 PM
swh-public-ci added a comment to D6427: swh.storage filter/buffer improvements.

Build was aborted

Oct 8 2021, 2:34 PM
vlorentz added a comment to T3636: Make the opam loader write extrinsic metadata.

(this task is a dependency of T3638, because author != committer in revisions created by opam, and releases don't have a "committer" field, so switching to releases would lose this data)

Oct 8 2021, 2:33 PM · Extrinsic metadata, Opam
vlorentz added a subtask for T3638: Make package loaders create releases objects instead of revisions: T3636: Make the opam loader write extrinsic metadata.
Oct 8 2021, 2:32 PM · Package Loader, Data Model, Archive content
vlorentz added a parent task for T3636: Make the opam loader write extrinsic metadata: T3638: Make package loaders create releases objects instead of revisions.
Oct 8 2021, 2:32 PM · Extrinsic metadata, Opam
vlorentz added projects to T3638: Make package loaders create releases objects instead of revisions: Data Model, Package Loader.
Oct 8 2021, 2:30 PM · Package Loader, Data Model, Archive content
vlorentz triaged T3638: Make package loaders create releases objects instead of revisions as Normal priority.
Oct 8 2021, 2:30 PM · Package Loader, Data Model, Archive content
vlorentz edited projects for T3636: Make the opam loader write extrinsic metadata, added: Extrinsic metadata; removed Metadata workflow.
Oct 8 2021, 2:26 PM · Extrinsic metadata, Opam
vlorentz added a project to T3637: Make the opam loader incremental: Opam.
Oct 8 2021, 2:26 PM · Opam
vlorentz updated the task description for T3636: Make the opam loader write extrinsic metadata.
Oct 8 2021, 2:26 PM · Extrinsic metadata, Opam
vlorentz triaged T3637: Make the opam loader incremental as Normal priority.
Oct 8 2021, 2:25 PM · Opam
vlorentz triaged T3636: Make the opam loader write extrinsic metadata as Normal priority.
Oct 8 2021, 2:24 PM · Extrinsic metadata, Opam
dachary added a comment to D6424: Perfect hashmap C implementation.

I'm curious to know where the libcmph-dev dependency was added?

Oct 8 2021, 2:24 PM
vsellier added a comment to T3621: Create a production read-only objstorage.

rSENV646f62805ef564bceed4d3a4d84d8fb6890f2d19 declares the new certificate for the vagrant tests (wrong task on the commit message)

Oct 8 2021, 2:21 PM · System administration
vsellier committed rSENV646f62805ef5: Add read-only storage self-signed certificate (authored by vsellier).
Add read-only storage self-signed certificate
Oct 8 2021, 2:20 PM
ardumont added a project to T3635: git loader: enable "partial" global deduplication of revisions via the extid mapping table: Git loader.
Oct 8 2021, 2:17 PM · Git loader
olasd published D6427: swh.storage filter/buffer improvements for review.

I'll split off the new buffer thresholds in a new diff. This diff now only contains the (small) improvements to the buffer/filter proxies

Oct 8 2021, 2:15 PM
swh-public-ci added a comment to D6442: Extract the path slicing logic in a dedicated PathSlicer class.

Build is green

Oct 8 2021, 2:13 PM
ardumont triaged T3635: git loader: enable "partial" global deduplication of revisions via the extid mapping table as Normal priority.
Oct 8 2021, 2:12 PM · Git loader