Page MenuHomeSoftware Heritage
Feed All Stories

Oct 8 2021

douardda updated the diff for D6442: Extract the path slicing logic in a dedicated PathSlicer class.

allow the pathslicer to be a noop (with an empty slicing)

Oct 8 2021, 2:12 PM
douardda requested review of D6442: Extract the path slicing logic in a dedicated PathSlicer class.
Oct 8 2021, 2:10 PM
ardumont added a comment to T3625: Reduce git loader memory footprint.

I concur with this analysis btw

Oct 8 2021, 2:00 PM · Git loader
ardumont accepted D6440: Rename remaining references to swh.model.identifiers..
Oct 8 2021, 1:54 PM
ardumont accepted D6433: serializers: Prepare rename of 'identifiers_enum' to 'swhids_enum'..
Oct 8 2021, 1:53 PM
swh-public-ci added a comment to D6438: docs: Explain task type registering to complete the save forge doc.

Build is green

Oct 8 2021, 1:47 PM
swh-public-ci added a comment to D6428: docs: Add a save forge documentation.

Build is green

Oct 8 2021, 1:46 PM
ardumont updated the diff for D6438: docs: Explain task type registering to complete the save forge doc.

Adapt according to suggestion

Oct 8 2021, 1:44 PM
ardumont updated the diff for D6428: docs: Add a save forge documentation.

Fix cli wrap around ``

Oct 8 2021, 1:43 PM
jayeshv requested review of D6441: Deprecate /browse/origin/log/ URLs.
Oct 8 2021, 12:54 PM
vlorentz committed rDSNIP498edfa2cae7: analyze_consistency_failures.py: misc fixes (authored by vlorentz).
analyze_consistency_failures.py: misc fixes
Oct 8 2021, 12:35 PM
vlorentz committed rDSNIP1e7a1a14c64b: analyze_consistency_failures.py: Avoid cloning/fetching the same origins over… (authored by vlorentz).
analyze_consistency_failures.py: Avoid cloning/fetching the same origins over…
Oct 8 2021, 12:35 PM
jayeshv added a revision to T3608: Deprecate most of the /browse/origin/.* URLs: D6441: Deprecate /browse/origin/log/ URLs.
Oct 8 2021, 12:33 PM · Web app
vlorentz requested review of D6440: Rename remaining references to swh.model.identifiers..
Oct 8 2021, 12:13 PM
vlorentz accepted D6438: docs: Explain task type registering to complete the save forge doc.
Oct 8 2021, 11:10 AM
olasd accepted D6439: Make workers send task events only when required.

Thanks!

Oct 8 2021, 10:56 AM
ardumont added a revision to T3458: save code now: Requests are not getting updated from time to time: D6439: Make workers send task events only when required.
Oct 8 2021, 10:53 AM · Save Code Now
ardumont updated the summary of D6439: Make workers send task events only when required.
Oct 8 2021, 10:53 AM
ardumont added inline comments to D6439: Make workers send task events only when required.
Oct 8 2021, 10:51 AM
ardumont updated the diff for D6439: Make workers send task events only when required.

Adapt according to last discussion ^

Oct 8 2021, 10:50 AM
ardumont retitled D6439: Make workers send task events only when required from Only activate worker task events on worker requiring it to Make workers send task events only when required.
Oct 8 2021, 10:50 AM
ardumont abandoned D6408: Stop sending next-gen scheduled task results to scheduler listener.
Oct 8 2021, 9:31 AM

Oct 7 2021

ardumont planned changes to D6439: Make workers send task events only when required.
18:25 <+ardumont> the test plan is a bit huge due to the octo-diff
18:27 <+ardumont> (i have still some doubt about whether i need to adapt the scheduler code a bit like D6405, but with `if self.send_events:` instead)
18:27 -- Notice(swhbot): D6405 (author: ardumont, Abandoned) on swh-scheduler: Respect task configuration to allow ignoring task result event <https://forge.softwareheritage.org/D6405>
18:28 <+olasd> you shouldn't, no. self.events should just become a noop
18:28 <+ardumont> (well, i mean with the right predicate)
18:28 <+ardumont> ack, thx
18:29 <+olasd> as for the diff, this is going to break monitoring (the ping-restart script) and all the stuff that depends on swh-worker@xxx.service. I think a gentler approach would be using an environment variable to set the --events flag or not
18:29 <+olasd> inside the existing swh-worker@.service
18:30 <+olasd> e.g. having a CELERY_WORKER_EXTRA_ARGS= environment variable that would be empty by default
18:31 <+ardumont> yeah, the breaking part saddens me a bit
18:31 <+olasd> and set to --events for the worker instances that need it
18:31 <+ardumont> ah interesting
18:31 <+ardumont> sounds way simpler, thx
18:31 <+olasd> (and add that envvar to the command line)
18:31 <+ardumont> thx
18:31 <+ardumont> miam
18:31 <+ardumont> i'll have a stab at it tomorrow
18:31 <+olasd> sorry I'm not writing that to the diff, my mouse is behaving and I'm about to head back home
18:31 <+olasd> feel free to c/p it :P
18:32 <+ardumont> yeah, thx, it's all good
Oct 7 2021, 6:32 PM
ardumont retitled D6439: Make workers send task events only when required from wip: Only activate worker task events on worker requiring it to Only activate worker task events on worker requiring it.
Oct 7 2021, 6:23 PM
ardumont updated the diff for D6439: Make workers send task events only when required.

Fix typos

Oct 7 2021, 6:22 PM
ardumont retitled D6439: Make workers send task events only when required from Only activate worker task events on worker requiring it to wip: Only activate worker task events on worker requiring it.
Oct 7 2021, 6:19 PM
ardumont updated the diff for D6439: Make workers send task events only when required.

Use correct repository

Oct 7 2021, 6:15 PM
ardumont updated the diff for D6439: Make workers send task events only when required.

Align code and diff

Oct 7 2021, 6:15 PM
ardumont updated the summary of D6439: Make workers send task events only when required.
Oct 7 2021, 6:14 PM
ardumont updated the diff for D6439: Make workers send task events only when required.

Align diff description and the actual code

Oct 7 2021, 6:09 PM
ardumont retitled D6439: Make workers send task events only when required from wip: Only activate worker tasks events for the relevant workers to Only activate worker task events on worker requiring it.
Oct 7 2021, 6:08 PM
ardumont updated the summary of D6439: Make workers send task events only when required.
Oct 7 2021, 6:06 PM
ardumont added inline comments to D6439: Make workers send task events only when required.
Oct 7 2021, 5:29 PM
ardumont updated the diff for D6439: Make workers send task events only when required.

More typos to fix

Oct 7 2021, 5:15 PM
ardumont updated the diff for D6439: Make workers send task events only when required.

Fix config name typos

Oct 7 2021, 5:13 PM
ardumont added a comment to D6408: Stop sending next-gen scheduled task results to scheduler listener.

Superseded by D6439 (provided it works)

Oct 7 2021, 4:51 PM
ardumont abandoned D6405: Respect task configuration to allow ignoring task result event.

Superseded by D6439 (provided it works)

Oct 7 2021, 4:51 PM
ardumont retitled D6439: Make workers send task events only when required from Only activate worker tasks events for the relevant workers to wip: Only activate worker tasks events for the relevant workers.
Oct 7 2021, 4:50 PM
ardumont requested review of D6439: Make workers send task events only when required.
Oct 7 2021, 4:49 PM
ardumont added a comment to D6408: Stop sending next-gen scheduled task results to scheduler listener.
In D6408#166674, @olasd wrote:

Rather than doing this, we should probably disable worker task events altogether (that is, run celery worker without the --events/--task-events flag)

Oct 7 2021, 4:45 PM
ardumont renamed T3629: doc: Add a "how to save a forge" as in how it's currently done from doc: Add a "how to save a forge" today to doc: Add a "how to save a forge" as in how it's currently done.
Oct 7 2021, 3:51 PM · Documentation
ardumont updated the summary of D6428: docs: Add a save forge documentation.
Oct 7 2021, 3:51 PM
ardumont requested review of D6438: docs: Explain task type registering to complete the save forge doc.
Oct 7 2021, 3:46 PM
swh-public-ci added a comment to D6428: docs: Add a save forge documentation.

Build is green

Oct 7 2021, 3:46 PM
vlorentz closed D6437: Rename imports of swh.model.identifiers to fix deprecation warnings..
Oct 7 2021, 3:43 PM
vlorentz committed rDLDBASE9d89bf86b045: Rename imports of swh.model.identifiers to fix deprecation warnings. (authored by vlorentz).
Rename imports of swh.model.identifiers to fix deprecation warnings.
Oct 7 2021, 3:43 PM
ardumont added a revision to T3629: doc: Add a "how to save a forge" as in how it's currently done: D6438: docs: Explain task type registering to complete the save forge doc.
Oct 7 2021, 3:43 PM · Documentation
ardumont updated the diff for D6428: docs: Add a save forge documentation.

Adapt links

Oct 7 2021, 3:43 PM
anlambert accepted D6437: Rename imports of swh.model.identifiers to fix deprecation warnings..
Oct 7 2021, 3:37 PM
vlorentz requested review of D6437: Rename imports of swh.model.identifiers to fix deprecation warnings..
Oct 7 2021, 3:35 PM
swh-public-ci added a comment to D6428: docs: Add a save forge documentation.

Build is green

Oct 7 2021, 3:33 PM
swh-public-ci added a comment to D6428: docs: Add a save forge documentation.

Build is green

Oct 7 2021, 3:30 PM
ardumont updated the diff for D6428: docs: Add a save forge documentation.

Rebase

Oct 7 2021, 3:30 PM
ardumont abandoned D6426: docs: Rename run a new lister doc into develop a new lister.

no

Oct 7 2021, 3:29 PM
ardumont updated the diff for D6428: docs: Add a save forge documentation.

Rewrote sentences

Oct 7 2021, 3:27 PM
vlorentz committed rDMFCDd30d00eca8a4: Rename imports of swh.model.identifiers to fix deprecation warnings. (authored by vlorentz).
Rename imports of swh.model.identifiers to fix deprecation warnings.
Oct 7 2021, 3:26 PM
swh-public-ci added a comment to D6428: docs: Add a save forge documentation.

Build is green

Oct 7 2021, 3:26 PM
ardumont retitled D6428: docs: Add a save forge documentation from docs: Transpose intranet docs on deploying a lister into sphinx doc to docs: Add a save forge documentation.
Oct 7 2021, 3:24 PM
ardumont updated the diff for D6428: docs: Add a save forge documentation.

Just concentrate on save forge for now

Oct 7 2021, 3:23 PM
vlorentz committed rDGRPHca7b39def2bc: Rename imports of swh.model.identifiers to fix deprecation warnings. (authored by vlorentz).
Rename imports of swh.model.identifiers to fix deprecation warnings.
Oct 7 2021, 3:22 PM
anlambert closed D6436: Replace use of deprecated swh.model.identifiers module.
Oct 7 2021, 3:17 PM
anlambert committed rDWAPPS264c0bc84bee: Replace use of deprecated swh.model.identifiers module (authored by anlambert).
Replace use of deprecated swh.model.identifiers module
Oct 7 2021, 3:17 PM
vlorentz accepted D6436: Replace use of deprecated swh.model.identifiers module.
Oct 7 2021, 3:15 PM
anlambert requested review of D6436: Replace use of deprecated swh.model.identifiers module.
Oct 7 2021, 3:12 PM
vlorentz committed rDSNIP973275745d7b: analyze_consistency_failures.py: Clone all linux forks in the same repo (saves… (authored by vlorentz).
analyze_consistency_failures.py: Clone all linux forks in the same repo (saves…
Oct 7 2021, 3:08 PM
vlorentz committed rDSNIP2bbf8e6c6e7a: analyze_consistency_failures.py: Add support for releases + mitigate FD leaks (authored by vlorentz).
analyze_consistency_failures.py: Add support for releases + mitigate FD leaks
Oct 7 2021, 3:08 PM
vlorentz committed rDSNIP541a8e0ab749: analyze_consistency_failures.py: Fix some remaining bugs in revision handling (authored by vlorentz).
analyze_consistency_failures.py: Fix some remaining bugs in revision handling
Oct 7 2021, 3:08 PM
vlorentz committed rDWCLI861beef16434: Rename imports of swh.model.identifiers to fix deprecation warnings. (authored by vlorentz).
Rename imports of swh.model.identifiers to fix deprecation warnings.
Oct 7 2021, 2:40 PM
vlorentz committed rDLDSVN5454c2601a04: Remove reference to swh.model.identifiers, it's deprecated (authored by vlorentz).
Remove reference to swh.model.identifiers, it's deprecated
Oct 7 2021, 2:29 PM
vlorentz closed D6435: Replace imports/references to swh.model.identifiers, it's deprecated..
Oct 7 2021, 2:28 PM
vlorentz committed rDTSCN0fb9280de794: Replace imports/references to swh.model.identifiers, it's deprecated. (authored by vlorentz).
Replace imports/references to swh.model.identifiers, it's deprecated.
Oct 7 2021, 2:28 PM
ardumont added a comment to T3487: Installation of the new provenance server.

*old comment not submitted*

Oct 7 2021, 2:19 PM · System administration
vlorentz added a comment to T3627: Consider dropping pull request references from the git loader ingestion.

An alternative to annotating synthetic refs: add a "type" or "forge_type" attribute to snapshots.

Oct 7 2021, 2:10 PM · Git loader
olasd added a comment to T3627: Consider dropping pull request references from the git loader ingestion.

Ah, another question I've been thinking about: should we go back to existing visits of git repositories and give them a new, pruned snapshot? Our data model now allows it: we can just append a new final OriginVisitStatus pointing at a pruned snapshot.

Oct 7 2021, 12:46 PM · Git loader
olasd added a comment to D6424: Perfect hashmap C implementation.

@dachary It'd be nice if you could describe what this is about in the commit message and
the diff description (if you actually provide a commit description, then when you create
the diff, the commit message is used as a description bootstrap). I know it's more work
for you but it happens that:

  1. it helps the reviewers to have some context directly here (without having to follow

between a multitude of tasks. FYI, I've followed through the task but it's not enough,
i need to also dig in that arborescence of tasks).

  1. is also how we are doing that in every other modules ;)
  1. the curious could learn a thing or 2 even if they don't do a proper review.

Please and thanks in advance.

Cheers,

Oct 7 2021, 12:39 PM
olasd added a comment to T3627: Consider dropping pull request references from the git loader ingestion.
In T3627#71809, @zack wrote:

Thanks for your feedback @olasd. I see three main arguments raised there: (1) the raciness of archiving those data via other means (= related forks), (2) the completeness of our canvassing of synthetic refs, (3) annotating rather than not archiving "synthetic" refs.

For (1), sure, it's racy, hence we could lose stuff that gets removed from GitHub before we have the time to archive it. But this is a drop in the ocean in comparison with our lag/backlog.

Oct 7 2021, 12:15 PM · Git loader
ardumont added a comment to D6424: Perfect hashmap C implementation.

@dachary It'd be nice if you could describe what this is about in the commit message and
the diff description (if you actually provide a commit description, then when you create
the diff, the commit message is used as a description bootstrap). I know it's more work
for you but it happens that:

Oct 7 2021, 12:14 PM
vlorentz requested review of D6433: serializers: Prepare rename of 'identifiers_enum' to 'swhids_enum'..
Oct 7 2021, 12:13 PM
vlorentz requested review of D6431: Rename imports of swh.model.identifiers to fix deprecation warnings..
Oct 7 2021, 12:13 PM
vlorentz closed D6429: docs: Update for the new API + remove references to deprecated module swh.model.identifiers.
Oct 7 2021, 12:08 PM
vlorentz committed rDVAUbd9862af5165: docs: Update for the new API + remove references to deprecated module swh.model. (authored by vlorentz).
docs: Update for the new API + remove references to deprecated module swh.model.
Oct 7 2021, 12:08 PM
anlambert closed D6434: faq: Fix broken links related to Web API.
Oct 7 2021, 12:06 PM
anlambert committed rDDOC60f9bb1d6d91: faq: Fix broken links related to Web API (authored by anlambert).
faq: Fix broken links related to Web API
Oct 7 2021, 12:06 PM
ardumont accepted D6434: faq: Fix broken links related to Web API.
Oct 7 2021, 12:06 PM
anlambert accepted D6435: Replace imports/references to swh.model.identifiers, it's deprecated..
Oct 7 2021, 12:06 PM
ardumont added a comment to T3338: Load the archived bitbucket mercurial repositories.

I'm gonna attend to this soon.

Oct 7 2021, 12:05 PM · System administration, Mercurial loader
anlambert accepted D6429: docs: Update for the new API + remove references to deprecated module swh.model.identifiers.

Looks good to me.

Oct 7 2021, 12:04 PM
vlorentz requested review of D6435: Replace imports/references to swh.model.identifiers, it's deprecated..
Oct 7 2021, 12:00 PM
anlambert requested review of D6434: faq: Fix broken links related to Web API.
Oct 7 2021, 11:56 AM
anlambert closed D6432: keycloak: Add swh.web.api.save_origin role to swh-web client.
Oct 7 2021, 11:35 AM
anlambert committed rSPSITE8bf99a574a06: keycloak: Add swh.web.api.save_origin role to swh-web client (authored by anlambert).
keycloak: Add swh.web.api.save_origin role to swh-web client
Oct 7 2021, 11:35 AM
anlambert added a comment to T3608: Deprecate most of the /browse/origin/.* URLs.

Yes we can, the snapshot should be provided as query parameter in other views in that case.

Thanks. Then we can delete most of the code in https://forge.softwareheritage.org/source/swh-web/browse/master/swh/web/browse/snapshot_context.py

Testing will be a bit tricky, I will try to add some generic cypress test.

Oct 7 2021, 11:27 AM · Web app
ardumont accepted D6432: keycloak: Add swh.web.api.save_origin role to swh-web client.
Oct 7 2021, 11:21 AM
ardumont retitled D6432: keycloak: Add swh.web.api.save_origin role to swh-web client from keycloak: Add swh.web.api.save_origin role to swh-web client Related to D6425 to keycloak: Add swh.web.api.save_origin role to swh-web client.
Oct 7 2021, 11:20 AM
anlambert requested review of D6432: keycloak: Add swh.web.api.save_origin role to swh-web client.
Oct 7 2021, 11:19 AM
olasd added a comment to T3487: Installation of the new provenance server.

rSPSITE6a233452cd48 fixed the prometheus node exporter.

Oct 7 2021, 11:17 AM · System administration
anlambert closed D6425: origin_save: Lift save request creation restrictions with permission.
Oct 7 2021, 11:16 AM
anlambert committed rDWAPPS0da4e46ae3c1: origin_save: Lift save request creation restrictions with permission (authored by anlambert).
origin_save: Lift save request creation restrictions with permission
Oct 7 2021, 11:16 AM
jayeshv added a comment to T3608: Deprecate most of the /browse/origin/.* URLs.

Yes we can, the snapshot should be provided as query parameter in other views in that case.

Oct 7 2021, 11:12 AM · Web app
anlambert added a comment to T3608: Deprecate most of the /browse/origin/.* URLs.

@anlambert do you think we can deprecate following routes as well? I think they can be redirected to the corresponding swh/web/browse/views/<object_type>.py routes.

  • /snapshot/(?P<snapshot_id>[0-9a-f]+)/content/
  • /snapshot/(?P<snapshot_id>[0-9a-f]+)/directory/(?P<path>.+)
  • / snapshot/(?P<snapshot_id>[0-9a-f]+)/directory/
  • `/snapshot/(?P<snapshot_id>[0-9a-f]+)/content/(?P<path>.+)/'

we can delete a lot of code from snapshot_context.py module then

Oct 7 2021, 11:09 AM · Web app