Page MenuHomeSoftware Heritage
Feed All Stories

Sep 21 2021

vsellier retitled D6308: Add a documentation page to list the services urls from WIP - add a documentation page to list the services urls to Add a documentation page to list the services urls.
Sep 21 2021, 7:52 AM · System administration

Sep 20 2021

borisbaldassari added a comment to D6133: maven-lister: initialise lister..

I'm not done yet but here is first review on my side.

Thanks for your time and interest.

Sep 20 2021, 9:49 PM
vsellier updated the diff for D6308: Add a documentation page to list the services urls.

complete other tools section

Sep 20 2021, 9:45 PM · System administration
vsellier updated the diff for D6308: Add a documentation page to list the services urls.
  • fix a couple of staging urls
  • add the production urls
Sep 20 2021, 7:42 PM · System administration
vsellier added a comment to D6308: Add a documentation page to list the services urls.

Do you reference only the staging part?

Sep 20 2021, 6:23 PM · System administration
vsellier updated the diff for D6308: Add a documentation page to list the services urls.

fix typos

Sep 20 2021, 6:20 PM · System administration
ardumont added a comment to D6308: Add a documentation page to list the services urls.

ok, nice start.
Do you reference only the staging part?

Sep 20 2021, 6:16 PM · System administration
olasd requested changes to D6310: opam: Move the state initialization into the get_pages method.

If we want to add this, we should also add it to test_pattern.py tests (which check that the scaffolding actually runs as expected). The documentation of the main pattern class also needs to be updated to add a reference to the new method.

Sep 20 2021, 6:16 PM
ardumont added a comment to T3563: Analyze and make the bitbucket ingestion faster.

I've patched the systemd swh-worker@loader_oneshot to actually lift --autoscale 10,20
from celery cli. It's actually holding fine. And that coupled with the filtering server
side makes for a huge bump in speed. The archive db does not seem to mind at all.

Sep 20 2021, 6:13 PM · System administration, Mercurial loader
ardumont accepted D6311: admin/origin_save: Parse optional note from rejection POST request body.
Sep 20 2021, 6:09 PM
ardumont planned changes to D6305: opam: Install and maintain up-to-date shared opam root directories.

Given the D6306 rework, this requires a bit of change (but nothing too drastic).

Sep 20 2021, 6:07 PM
anlambert requested review of D6311: admin/origin_save: Parse optional note from rejection POST request body.
Sep 20 2021, 6:04 PM
swh-public-ci added a comment to D6312: opam: Make the instance optional and derived from the url.

Build is green

Sep 20 2021, 6:00 PM
swh-public-ci added a comment to D6310: opam: Move the state initialization into the get_pages method.

Build is green

Sep 20 2021, 5:59 PM
swh-public-ci added a comment to D6309: common/models: Add note text field to SaveOriginRequest model.

Build is green

Sep 20 2021, 5:59 PM
swh-public-ci added a comment to D6306: opam: Allow defining where to actually install the opam_root folder.

Build is green

Sep 20 2021, 5:58 PM
ardumont updated the summary of D6306: opam: Allow defining where to actually install the opam_root folder.
Sep 20 2021, 5:57 PM
ardumont added a revision to T3590: opam loader: Ensure required opam state is shared amongst ingestion/listing runs: D6312: opam: Make the instance optional and derived from the url.
Sep 20 2021, 5:57 PM · Archive coverage, Opam
ardumont updated the summary of D6312: opam: Make the instance optional and derived from the url.
Sep 20 2021, 5:57 PM
ardumont updated the diff for D6312: opam: Make the instance optional and derived from the url.

Rebase

Sep 20 2021, 5:56 PM
ardumont updated the diff for D6310: opam: Move the state initialization into the get_pages method.

Rebase

Sep 20 2021, 5:56 PM
ardumont updated the diff for D6306: opam: Allow defining where to actually install the opam_root folder.
  • Adapt to be multi instance aware
  • Rebase on top of other diffs
Sep 20 2021, 5:54 PM
ardumont requested review of D6312: opam: Make the instance optional and derived from the url.
Sep 20 2021, 5:52 PM
anlambert added a revision to T3256: Propose reason for rejecting a save code now: D6311: admin/origin_save: Parse optional note from rejection POST request body.
Sep 20 2021, 5:48 PM · Save Code Now, Easy hack, Web app
ardumont requested review of D6310: opam: Move the state initialization into the get_pages method.
Sep 20 2021, 5:47 PM
anlambert updated the diff for D6309: common/models: Add note text field to SaveOriginRequest model.

Add missing doc about new note field in apidoc for /origin/save/

Sep 20 2021, 5:43 PM
anlambert committed rDWAPPS670090538797: package.json: Upgrade dependencies (authored by anlambert).
package.json: Upgrade dependencies
Sep 20 2021, 5:08 PM
swh-public-ci added a comment to D6300: Capture missing revision <-> hgnode-id scenario in a xfail test.

Build is green

Sep 20 2021, 5:02 PM
ardumont updated the diff for D6300: Capture missing revision <-> hgnode-id scenario in a xfail test.

Adapt test docstring into a todo since it's actually a todo

Sep 20 2021, 5:00 PM
ardumont added a comment to D6306: opam: Allow defining where to actually install the opam_root folder.

LGTM, but how is the new opam_root option expected to be set (in production I mean)?

Sep 20 2021, 4:57 PM
douardda accepted D6306: opam: Allow defining where to actually install the opam_root folder.

LGTM, but how is the new opam_root option expected to be set (in production I mean)?

Sep 20 2021, 4:46 PM
douardda requested changes to D6133: maven-lister: initialise lister..

I'm not done yet but here is first review on my side.

Sep 20 2021, 4:33 PM
ardumont accepted D6309: common/models: Add note text field to SaveOriginRequest model.
Sep 20 2021, 4:21 PM
anlambert committed rDWAPPSe6a022d41ceb: templates/show-swhids: Fix typo in SWHIDs documentation URL (authored by anlambert).
templates/show-swhids: Fix typo in SWHIDs documentation URL
Sep 20 2021, 4:16 PM
anlambert requested review of D6309: common/models: Add note text field to SaveOriginRequest model.
Sep 20 2021, 4:15 PM
anlambert added a revision to T3256: Propose reason for rejecting a save code now: D6309: common/models: Add note text field to SaveOriginRequest model.
Sep 20 2021, 4:00 PM · Save Code Now, Easy hack, Web app
vlorentz added a comment to T75: Check integrity of directories, revisions, and releases.

swh:1:dir:880034219b47a123b97161de4e0d3301746cec75 (from https://github.com/kentnl/Dist-Zilla-PluginBundle-Author-KENTNL) has a single entry; we stored it with mode 40000 on that entry, but the original mode is 40755.

Sep 20 2021, 3:59 PM · Archive content, Restricted Project
vsellier planned changes to D6308: Add a documentation page to list the services urls.
Sep 20 2021, 3:57 PM · System administration
vsellier requested review of D6308: Add a documentation page to list the services urls.
Sep 20 2021, 3:57 PM · System administration
vsellier added a revision to T3408: Provide read-only access to production servers: D6308: Add a documentation page to list the services urls.
Sep 20 2021, 3:57 PM · System administration
vlorentz renamed T3591: Create a list/inventory of the hostname/port of all services from Create an list of the hostname/port of all services to Create a list/inventory of the hostname/port of all services.
Sep 20 2021, 3:23 PM · Documentation, System administration
vlorentz triaged T3591: Create a list/inventory of the hostname/port of all services as Low priority.
Sep 20 2021, 3:16 PM · Documentation, System administration
ardumont requested review of D6306: opam: Allow defining where to actually install the opam_root folder.
Sep 20 2021, 3:13 PM
ardumont updated the summary of D6305: opam: Install and maintain up-to-date shared opam root directories.
Sep 20 2021, 3:11 PM
ardumont updated the summary of D6305: opam: Install and maintain up-to-date shared opam root directories.
Sep 20 2021, 3:11 PM
ardumont added a revision to T3590: opam loader: Ensure required opam state is shared amongst ingestion/listing runs: D6306: opam: Allow defining where to actually install the opam_root folder.
Sep 20 2021, 3:10 PM · Archive coverage, Opam
ardumont requested review of D6305: opam: Install and maintain up-to-date shared opam root directories.
Sep 20 2021, 3:05 PM
ardumont added a revision to T3590: opam loader: Ensure required opam state is shared amongst ingestion/listing runs: D6305: opam: Install and maintain up-to-date shared opam root directories.
Sep 20 2021, 3:05 PM · Archive coverage, Opam
ardumont triaged T3590: opam loader: Ensure required opam state is shared amongst ingestion/listing runs as Normal priority.
Sep 20 2021, 2:51 PM · Archive coverage, Opam
ardumont closed T3468: staging: current opam loading issues, a subtask of T3424: Opam support, as Resolved.
Sep 20 2021, 2:44 PM · Archive coverage, Opam
ardumont closed T3468: staging: current opam loading issues as Resolved.
Sep 20 2021, 2:44 PM · System administration, Opam
ardumont moved T3468: staging: current opam loading issues from code-review/await-feedback/pause to deployed/landed/monitoring on the System administration board.
Sep 20 2021, 2:44 PM · System administration, Opam
ardumont moved T3468: staging: current opam loading issues from in-progress to code-review/await-feedback/pause on the System administration board.
Sep 20 2021, 2:44 PM · System administration, Opam
ardumont changed the status of T3468: staging: current opam loading issues from Open to Work in Progress.
Sep 20 2021, 2:44 PM · System administration, Opam
ardumont changed the status of T3468: staging: current opam loading issues, a subtask of T3424: Opam support, from Open to Work in Progress.
Sep 20 2021, 2:44 PM · Archive coverage, Opam
ardumont moved T3468: staging: current opam loading issues from Backlog to Weekly backlog on the System administration board.
Sep 20 2021, 2:44 PM · System administration, Opam
ardumont added a project to T3468: staging: current opam loading issues: System administration.
Sep 20 2021, 2:44 PM · System administration, Opam
ardumont added a comment to T3468: staging: current opam loading issues.

Deployed the multiple fixes we did with @anlambert.
That and @mclovin's upstream fixes. We should be good.

Sep 20 2021, 2:42 PM · System administration, Opam
ardumont updated the summary of D6300: Capture missing revision <-> hgnode-id scenario in a xfail test.
Sep 20 2021, 2:28 PM
vsellier closed D6304: swh-counters: use the right docker override file name.
Sep 20 2021, 1:52 PM
vsellier committed rDENVc232b0c6e10e: swh-counters: use the right docker override file name (authored by vsellier).
swh-counters: use the right docker override file name
Sep 20 2021, 1:52 PM
vsellier closed D6303: swh-web: fix the metadata backend configuration in the swh-search override.
Sep 20 2021, 1:52 PM
vsellier committed rDENVa0939bff7344: swh-web: fix the metadata backend configuration in the swh-search override (authored by vsellier).
swh-web: fix the metadata backend configuration in the swh-search override
Sep 20 2021, 1:52 PM
ardumont added a comment to D6300: Capture missing revision <-> hgnode-id scenario in a xfail test.

italic text

Sep 20 2021, 1:52 PM
ardumont accepted D6304: swh-counters: use the right docker override file name.

oops ;)

Sep 20 2021, 1:51 PM
ardumont accepted D6303: swh-web: fix the metadata backend configuration in the swh-search override.
Sep 20 2021, 1:51 PM
vsellier requested review of D6304: swh-counters: use the right docker override file name.
Sep 20 2021, 1:49 PM
vlorentz added inline comments to D6300: Capture missing revision <-> hgnode-id scenario in a xfail test.
Sep 20 2021, 1:48 PM
vsellier requested review of D6303: swh-web: fix the metadata backend configuration in the swh-search override.
Sep 20 2021, 1:42 PM
vsellier added a revision to T3433: Deploy swh.search v0.10/v0.11: D6303: swh-web: fix the metadata backend configuration in the swh-search override.
Sep 20 2021, 1:42 PM · System administration, Archive search
ardumont created P1170 opam root taking around 214Mib.
Sep 20 2021, 12:30 PM · Opam
ardumont updated the summary of D6300: Capture missing revision <-> hgnode-id scenario in a xfail test.
Sep 20 2021, 12:23 PM
rdicosmo moved T3537: Blog post Obsidian NLNet from Restricted Project Column to Restricted Project Column on the Unknown Object (Project) board.
Sep 20 2021, 12:21 PM · Unknown Object (Project)
ardumont moved T3563: Analyze and make the bitbucket ingestion faster from in-progress to code-review/await-feedback/pause on the System administration board.
Sep 20 2021, 12:11 PM · System administration, Mercurial loader
ardumont added a comment to T3563: Analyze and make the bitbucket ingestion faster.
10:02:40 softwareheritage@belvedere:5432=> select now(), count(distinct url) from origin o inner join origin_visit ov on o.id=ov.origin where o.url like 'https://bitbucket.org/%' and ov.type='hg';
+------------------------------+--------+
|             now              | count  |
+------------------------------+--------+
| 2021-09-20 10:04:30.89072+00 | 280995 |
+------------------------------+--------+
(1 row)
Sep 20 2021, 12:11 PM · System administration, Mercurial loader
ardumont added a comment to T3563: Analyze and make the bitbucket ingestion faster.

Deployed the loader mercurial v2.3 (with filtering server side).
As expected, less time is spent in the method fetching the new changesets.

Sep 20 2021, 12:08 PM · System administration, Mercurial loader
ardumont moved T3589: Deploy swh.vault v1.2 from Backlog to Weekly backlog on the System administration board.
Sep 20 2021, 12:06 PM · System administration, Vault
ardumont added a project to T3589: Deploy swh.vault v1.2: System administration.
Sep 20 2021, 12:06 PM · System administration, Vault
ardumont moved T3588: Deploy swh.loader.git v1.0 from Backlog to Weekly backlog on the System administration board.
Sep 20 2021, 11:59 AM · System administration, Git loader
ardumont added a project to T3588: Deploy swh.loader.git v1.0: System administration.
Sep 20 2021, 11:59 AM · System administration, Git loader
ardumont added a comment to T3588: Deploy swh.loader.git v1.0.

Fix [1] first

Sep 20 2021, 11:58 AM · System administration, Git loader
ardumont triaged T3589: Deploy swh.vault v1.2 as Normal priority.
Sep 20 2021, 11:56 AM · System administration, Vault
douardda closed T1510: Have a look at openAPI and decide whether we want to follow these specs, a subtask of T1805: Public API v2, as Resolved.
Sep 20 2021, 11:54 AM · meta-task, Web app
douardda closed T1510: Have a look at openAPI and decide whether we want to follow these specs as Resolved.
Sep 20 2021, 11:54 AM · Web app
douardda closed T2196: Batch APIs as Wontfix.

not useful as a dedicated task, see T1805 for the main discussion one on this subject

Sep 20 2021, 11:54 AM · Roadmap 2020
douardda closed T2196: Batch APIs, a subtask of T2194: Archive Integration (Web API), as Wontfix.
Sep 20 2021, 11:54 AM · Roadmap 2021, meta-task
vlorentz added a task to D6281: converters: Recompute hashes and check they match the originals: T3135: Improve integrity of ingested content.
Sep 20 2021, 11:05 AM
vlorentz added a revision to T3135: Improve integrity of ingested content: D6281: converters: Recompute hashes and check they match the originals.
Sep 20 2021, 11:05 AM · Storage manager, Roadmap 2021, meta-task
jayeshv added a comment to D6291: WIP - Unit tests for the mongo backend.

I'm not really convinced about adding this test, it essentially recreates situations that are already tested in the test_provenance_storage test (but there it is done for all backends at once).
Also, performing direct queries to the mongodb object breaks the abstraction layer which means any future refactoring will require reimplementing this tests.
I rather design test to be independent of the actual implementation, so that they check that the class behave as expected from a semantic point of view.

Sep 20 2021, 10:55 AM
aeviso requested changes to D6291: WIP - Unit tests for the mongo backend.

I'm not really convinced about adding this test, it essentially recreates situations that are already tested in the test_provenance_storage test (but there it is done for all backends at once).
Also, performing direct queries to the mongodb object breaks the abstraction layer which means any future refactoring will require reimplementing this tests.
I rather design test to be independent of the actual implementation, so that they check that the class behave as expected from a semantic point of view.

Sep 20 2021, 10:31 AM
ardumont added a comment to D6300: Capture missing revision <-> hgnode-id scenario in a xfail test.

anomad-d is the user-repository name. It's a repository which presents an anomaly in some way. That anomaly makes the ingestion fail.

Sep 20 2021, 10:10 AM
douardda requested changes to D6300: Capture missing revision <-> hgnode-id scenario in a xfail test.

I don't understand what exactly is (not) tested here. What does "anomad-d" stand for BTW?

Sep 20 2021, 9:59 AM
douardda accepted D6220: Added test only method info in the interface doc strings.
Sep 20 2021, 9:49 AM
swh-public-ci added a comment to D6291: WIP - Unit tests for the mongo backend.

Build is green

Sep 20 2021, 8:22 AM
jayeshv updated the diff for D6291: WIP - Unit tests for the mongo backend.

string changes

Sep 20 2021, 8:18 AM

Sep 19 2021

rdicosmo moved T3162: Services page iconography from Restricted Project Column to Restricted Project Column on the Unknown Object (Project) board.
Sep 19 2021, 11:28 AM · Unknown Object (Project)
rdicosmo moved T3536: Blog post Easter Eggs NLNet from Restricted Project Column to Restricted Project Column on the Unknown Object (Project) board.
Sep 19 2021, 11:26 AM · Unknown Object (Project)
rdicosmo added a comment to T3536: Blog post Easter Eggs NLNet.

The agreement has been signed, so we can move forward and publish the blog post.

Sep 19 2021, 11:24 AM · Unknown Object (Project)

Sep 18 2021

zack added a project to T3425: Opam loader: Archive coverage.
Sep 18 2021, 8:29 AM · Archive coverage, Opam
zack added a project to T3358: Opam lister: Archive coverage.
Sep 18 2021, 8:29 AM · Archive coverage, Lister