Page MenuHomeSoftware Heritage
Feed All Stories

Oct 5 2021

aeviso committed rDPROV246e55f9b7e3: Split `Provenance::flush` method in two (one per layer) (authored by aeviso).
Split `Provenance::flush` method in two (one per layer)
Oct 5 2021, 12:05 PM
aeviso committed rDPROVf0210c3753c3: Add `open`/`close` methods to both `ProvenanceInterface` and… (authored by aeviso).
Add `open`/`close` methods to both `ProvenanceInterface` and…
Oct 5 2021, 12:05 PM
aeviso closed D6273: Remove remote storage based on `swh.core.api.RPCClient`.
Oct 5 2021, 12:05 PM
aeviso committed rDPROV172e327c2588: Remove remote provenance storage based on `swh.core.api.RPCClient` (authored by aeviso).
Remove remote provenance storage based on `swh.core.api.RPCClient`
Oct 5 2021, 12:05 PM
vlorentz committed rDSNIP279cb4a99207: analyze_consistency_failures.py: Add more edge cases, normalize digest keys… (authored by vlorentz).
analyze_consistency_failures.py: Add more edge cases, normalize digest keys…
Oct 5 2021, 11:55 AM
vlorentz committed rDSNIP315248b5eb68: analyze_consistency_failures.py: Cache swh-graph output, it's too slow (authored by vlorentz).
analyze_consistency_failures.py: Cache swh-graph output, it's too slow
Oct 5 2021, 11:55 AM
ardumont triaged T3632: Investigate the ContentDisallowed exception as Normal priority.
Oct 5 2021, 11:54 AM · Scheduling utilities
Harbormaster failed remote builds in B24214: Diff 23300 for D6339: Add support for remote backend on existing storage tests!
Oct 5 2021, 11:49 AM
swh-public-ci added a comment to D6339: Add support for remote backend on existing storage tests.

Build was aborted

Oct 5 2021, 11:49 AM
anlambert triaged T3631: Make snapshot query parameter mandatory when browsing objects in an origin context as Normal priority.
Oct 5 2021, 11:44 AM · Web app
vsellier added a comment to T3630: staging - journal0 needs more space.

Actions to perform for the migration:

  • add the role::swh_kafka_broker role to the new server in puppet and deploy
  • stop the staging workers
  • stop kafka on journal0
  • rsync the content from journal0 to the new server
  • update the staging configurations to use the new server as journal
  • update the NAT public staging for staging journal on the firewall (https://192.168.50.1/firewall_nat.php)
Oct 5 2021, 11:36 AM · System administration
aeviso added a comment to D6339: Add support for remote backend on existing storage tests.

IMHO This diff should be squashed in D6165 (it's really part of the work adding the rabbitmq-based backend).

Oct 5 2021, 11:35 AM
ardumont closed D6407: Adapt logrotate configuration so extra directory is also logrotated.
Oct 5 2021, 11:32 AM
ardumont committed rSPSITE2a007a23921c: Adapt logrotate configuration for webapp instances (authored by ardumont).
Adapt logrotate configuration for webapp instances
Oct 5 2021, 11:32 AM
ardumont updated subscribers of D6382: make "yarn install" non-fatal in swh-web entrypoint.

@douardda you had a fight with this recently i believe, what antoine is expliciting might interest you

Oct 5 2021, 11:31 AM
vlorentz closed D6400: Fix label of 'Extrinsic metadata' buttons.
Oct 5 2021, 11:29 AM
vlorentz committed rDWAPPS333ca8259c74: Fix label of 'Extrinsic metadata' buttons (authored by vlorentz).
Fix label of 'Extrinsic metadata' buttons
Oct 5 2021, 11:29 AM
vsellier triaged T3630: staging - journal0 needs more space as High priority.
Oct 5 2021, 11:28 AM · System administration
aeviso updated the diff for D6339: Add support for remote backend on existing storage tests.

rebase

Oct 5 2021, 11:27 AM
jayeshv added a comment to T3608: Deprecate most of the /browse/origin/.* URLs.

The issue is, the redirected url (/browse/directory/<sha1>?origin_url=<>) will call the same function (get_snapshot_context) again. This is true for all the urls.
Do you see some other efficient alternative to fix this ticket?

@jayeshv, all costly storage requests result (getting origin visits, snapshot content and size) performed by the get_snapshot_context
function are cached at the django level so no need to worry about performance here as it has already been handled.

Oct 5 2021, 11:26 AM · Web app
vsellier accepted D6407: Adapt logrotate configuration so extra directory is also logrotated.

LGTM thanks

Oct 5 2021, 11:26 AM
aeviso updated the summary of D6357: Split `Provenance::flush` method in two (one per layer).
Oct 5 2021, 11:25 AM
aeviso added a comment to D6334: Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`.

looks ok to me. Just one question, why do you need __future__.annotation?

Oct 5 2021, 11:24 AM
ardumont updated the diff for D6407: Adapt logrotate configuration so extra directory is also logrotated.

Update after the correct rebase

Oct 5 2021, 11:23 AM
vlorentz closed D6387: type_validator: Re-allow subclasses.
Oct 5 2021, 11:22 AM
vlorentz committed rDMOD916627e1b504: type_validator: Re-allow subclasses (authored by vlorentz).
type_validator: Re-allow subclasses
Oct 5 2021, 11:22 AM
ardumont updated the diff for D6407: Adapt logrotate configuration so extra directory is also logrotated.

Adapt according to suggestion

Oct 5 2021, 11:22 AM
ardumont updated the diff for D6407: Adapt logrotate configuration so extra directory is also logrotated.

Use postrotate stanza

Oct 5 2021, 11:21 AM
ardumont added inline comments to D6407: Adapt logrotate configuration so extra directory is also logrotated.
Oct 5 2021, 11:21 AM
ardumont updated the test plan for D6407: Adapt logrotate configuration so extra directory is also logrotated.
Oct 5 2021, 11:21 AM
vsellier added inline comments to D6407: Adapt logrotate configuration so extra directory is also logrotated.
Oct 5 2021, 11:11 AM
anlambert added inline comments to D6382: make "yarn install" non-fatal in swh-web entrypoint.
Oct 5 2021, 11:06 AM
anlambert requested changes to D6382: make "yarn install" non-fatal in swh-web entrypoint.

Looks good to me apart the warning message that is not correct.

Oct 5 2021, 11:02 AM
ardumont updated the diff for D6407: Adapt logrotate configuration so extra directory is also logrotated.

Rebase

Oct 5 2021, 11:01 AM
ardumont updated the diff for D6407: Adapt logrotate configuration so extra directory is also logrotated.

Push variables at the top level manifest

Oct 5 2021, 11:00 AM
ardumont updated the test plan for D6407: Adapt logrotate configuration so extra directory is also logrotated.
Oct 5 2021, 11:00 AM
ardumont updated the diff for D6407: Adapt logrotate configuration so extra directory is also logrotated.

Use correct repository

Oct 5 2021, 10:57 AM
ardumont updated the diff for D6407: Adapt logrotate configuration so extra directory is also logrotated.

Adapt after discussing with @vsellier

Oct 5 2021, 10:56 AM
vsellier closed D6406: provenance: Configure the postgresql max_connections.
Oct 5 2021, 10:56 AM
vsellier committed rSPSITEefb36f766516: provenance: Configure the postgresql max_connections (authored by vsellier).
provenance: Configure the postgresql max_connections
Oct 5 2021, 10:56 AM
douardda closed D6403: docker: use a dedicated container for the cron-like job of swh-web.
Oct 5 2021, 10:56 AM
douardda committed rDENVaf0a2af3e7c8: docker: use a dedicated container for the cron-like job of swh-web (authored by douardda).
docker: use a dedicated container for the cron-like job of swh-web
Oct 5 2021, 10:56 AM
douardda closed D6402: docker: Do not limit the list of task types handled by swh-scheduler-runner-priority.
Oct 5 2021, 10:56 AM
douardda committed rDENVebb07bdae059: docker: Do not limit the list of task types handled by swh-scheduler-runner… (authored by douardda).
docker: Do not limit the list of task types handled by swh-scheduler-runner…
Oct 5 2021, 10:56 AM
ardumont updated the test plan for D6407: Adapt logrotate configuration so extra directory is also logrotated.
Oct 5 2021, 10:54 AM
anlambert accepted D6407: Adapt logrotate configuration so extra directory is also logrotated.

+1, thanks !

Oct 5 2021, 10:50 AM
ardumont added a revision to T3313: Web API: per-user accounting: D6407: Adapt logrotate configuration so extra directory is also logrotated.
Oct 5 2021, 10:47 AM · System administration, Web app
ardumont updated the summary of D6407: Adapt logrotate configuration so extra directory is also logrotated.
Oct 5 2021, 10:47 AM
douardda added a comment to D6165: Add new RabbitMQ-based client/server API.

Also there is no real value in keeping 3 revisions: the last 2 revisions actually improve/modify the code from the first revision.

Oct 5 2021, 10:47 AM
douardda added a comment to D6339: Add support for remote backend on existing storage tests.

this should be squashed with the previous diff, and still my previous question about .gitignore

Oct 5 2021, 10:45 AM
douardda accepted D6165: Add new RabbitMQ-based client/server API.

As others (and I) said, this must come with actual documentation.
As is, I have hard time understanding how this actually works (even after reading the document in hedgdoc).

Oct 5 2021, 10:39 AM
anlambert added a comment to T3608: Deprecate most of the /browse/origin/.* URLs.

The issue is, the redirected url (/browse/directory/<sha1>?origin_url=<>) will call the same function (get_snapshot_context) again. This is true for all the urls.
Do you see some other efficient alternative to fix this ticket?

Oct 5 2021, 10:32 AM · Web app
douardda updated the diff for D6403: docker: use a dedicated container for the cron-like job of swh-web.

indent...

Oct 5 2021, 10:01 AM
douardda updated the diff for D6403: docker: use a dedicated container for the cron-like job of swh-web.

improve entrypoint script to properly handle a SIGTERM

Oct 5 2021, 9:59 AM
ardumont closed D6404: Clean up missed unused module from early clean up.
Oct 5 2021, 9:36 AM
ardumont committed rDLDHG8cbf370e0360: Clean up missed unused module from early clean up (authored by ardumont).
Clean up missed unused module from early clean up
Oct 5 2021, 9:36 AM
douardda accepted D6334: Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`.
Oct 5 2021, 9:34 AM
douardda added a comment to D6334: Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`.

looks ok to me. Just one question, why do you need __future__.annotation?

Oct 5 2021, 9:34 AM
ardumont updated the summary of D6408: Stop sending next-gen scheduled task results to scheduler listener.
Oct 5 2021, 9:22 AM
ardumont updated the test plan for D6405: Respect task configuration to allow ignoring task result event.
Oct 5 2021, 9:21 AM
ardumont updated the summary of D6407: Adapt logrotate configuration so extra directory is also logrotated.
Oct 5 2021, 9:21 AM
ardumont added a revision to T3458: save code now: Requests are not getting updated from time to time: D6408: Stop sending next-gen scheduled task results to scheduler listener.
Oct 5 2021, 9:18 AM · Save Code Now
ardumont requested review of D6408: Stop sending next-gen scheduled task results to scheduler listener.
Oct 5 2021, 9:18 AM
ardumont added a reviewer for D6407: Adapt logrotate configuration so extra directory is also logrotated: System administration.
Oct 5 2021, 9:15 AM
ardumont updated the test plan for D6407: Adapt logrotate configuration so extra directory is also logrotated.
Oct 5 2021, 9:14 AM
ardumont requested review of D6407: Adapt logrotate configuration so extra directory is also logrotated.
Oct 5 2021, 9:13 AM

Oct 4 2021

ardumont updated subscribers of T3625: Reduce git loader memory footprint.

One good question got raised on the mailing list thread both by @douardda and @stsp.

Oct 4 2021, 6:45 PM · Git loader
ardumont added a comment to T3458: save code now: Requests are not getting updated from time to time.

*very old comment not pushed for some reason*

Oct 4 2021, 6:38 PM · Save Code Now
ardumont accepted D6406: provenance: Configure the postgresql max_connections.

neat.

Oct 4 2021, 6:26 PM
vsellier added a revision to T3487: Installation of the new provenance server: D6406: provenance: Configure the postgresql max_connections.
Oct 4 2021, 5:59 PM · System administration
vsellier requested review of D6406: provenance: Configure the postgresql max_connections.
Oct 4 2021, 5:59 PM
ardumont updated subscribers of D6403: docker: use a dedicated container for the cron-like job of swh-web.

@anlambert might be interested as well ;)

Oct 4 2021, 5:55 PM
ardumont requested review of D6405: Respect task configuration to allow ignoring task result event.
Oct 4 2021, 5:52 PM
ardumont added a revision to T3458: save code now: Requests are not getting updated from time to time: D6405: Respect task configuration to allow ignoring task result event.
Oct 4 2021, 5:49 PM · Save Code Now
ardumont requested review of D6404: Clean up missed unused module from early clean up.
Oct 4 2021, 5:48 PM
ardumont added a revision to T3612: Clean up mercurial loader code: D6404: Clean up missed unused module from early clean up.
Oct 4 2021, 5:46 PM · Mercurial loader
ardumont accepted D6403: docker: use a dedicated container for the cron-like job of swh-web.
Oct 4 2021, 5:44 PM
ardumont accepted D6403: docker: use a dedicated container for the cron-like job of swh-web.
Oct 4 2021, 5:44 PM
ardumont updated subscribers of D6403: docker: use a dedicated container for the cron-like job of swh-web.

Nice! Fine with the current implem as it's the same as before in a dedicated container so better.

Oct 4 2021, 5:44 PM
ardumont accepted D6402: docker: Do not limit the list of task types handled by swh-scheduler-runner-priority.

lgtm

Oct 4 2021, 5:35 PM
douardda requested review of D6403: docker: use a dedicated container for the cron-like job of swh-web.
Oct 4 2021, 5:33 PM
douardda requested review of D6402: docker: Do not limit the list of task types handled by swh-scheduler-runner-priority.
Oct 4 2021, 5:33 PM
ardumont updated the task description for T3629: doc: Add a "how to save a forge" as in how it's currently done.
Oct 4 2021, 5:27 PM · Documentation
ardumont retitled D6401: Filter out pull request related branches from Filter out pull requests related branches to Filter out pull request related branches.
Oct 4 2021, 5:25 PM
swh-public-ci added a comment to D6339: Add support for remote backend on existing storage tests.

Build was aborted

Oct 4 2021, 5:22 PM
ardumont accepted D6400: Fix label of 'Extrinsic metadata' buttons.
Oct 4 2021, 5:18 PM
ardumont triaged T3629: doc: Add a "how to save a forge" as in how it's currently done as Normal priority.
Oct 4 2021, 4:57 PM · Documentation
ardumont added a project to T3628: doc: Add a "how to test your own loader in docker" section: Documentation.
Oct 4 2021, 4:55 PM · Documentation
ardumont triaged T3628: doc: Add a "how to test your own loader in docker" section as Normal priority.
Oct 4 2021, 4:55 PM · Documentation
Harbormaster failed remote builds in B24195: Diff 23278 for D6339: Add support for remote backend on existing storage tests!
Oct 4 2021, 4:51 PM
swh-public-ci added a comment to D6339: Add support for remote backend on existing storage tests.

Build was aborted

Oct 4 2021, 4:51 PM
zack added a comment to T3627: Consider dropping pull request references from the git loader ingestion.

No, the snippet mentioned filters out refs whose name starts with refs/pulls and finishes with /merge
(i realize i made a typo in the main description..., it's now fixed)

Oct 4 2021, 4:37 PM · Git loader
aeviso updated the diff for D6339: Add support for remote backend on existing storage tests.

rebase

Oct 4 2021, 4:30 PM
ardumont requested review of D6401: Filter out pull request related branches.
Oct 4 2021, 4:20 PM
ardumont added a comment to T3627: Consider dropping pull request references from the git loader ingestion.

@zack Maybe the diff ^ will clarify a bit ;)

Oct 4 2021, 4:20 PM · Git loader
Harbormaster failed remote builds in B24193: Diff 23276 for D6339: Add support for remote backend on existing storage tests!
Oct 4 2021, 4:18 PM
swh-public-ci added a comment to D6339: Add support for remote backend on existing storage tests.

Build was aborted

Oct 4 2021, 4:18 PM
ardumont added a revision to T3627: Consider dropping pull request references from the git loader ingestion: D6401: Filter out pull request related branches.
Oct 4 2021, 4:18 PM · Git loader
ardumont added a comment to T3627: Consider dropping pull request references from the git loader ingestion.

According to the snippet referenced by @ardumont, all branch names starting with refs/pull/ should be filtered out.

Oct 4 2021, 4:11 PM · Git loader
ardumont updated the task description for T3627: Consider dropping pull request references from the git loader ingestion.
Oct 4 2021, 4:11 PM · Git loader