vsellier retitled D7985: [provenance] Implement a naive archive multiplexer from Declare the missing swh-graph dependency to [provenance] Implement a naive archive multiplexer.

Jun 15 2022, 10:57 AM · Provenance database

Jun 14 2022

vsellier added a revision to T4315: [provenance] Fallback to swh-storage if a revision or its parent is not found in swh-graph: D7985: [provenance] Implement a naive archive multiplexer.

Jun 14 2022, 12:23 PM · Provenance database

Jun 8 2022

vsellier closed T4314: [provenance] HistoryGraph building is failing when reaching the first revision as Resolved.

fixed with the workaround in D7972

Jun 8 2022, 11:51 PM · Provenance database

vsellier renamed T4315: [provenance] Fallback to swh-storage if a revision or its parent is not found in swh-graph from [provenance] Fallback to swh-storage if a revision is not found in swh-graph to [provenance] Fallback to swh-storage if a revision or its parent is not found in swh-graph.

Jun 8 2022, 2:17 PM · Provenance database

vsellier triaged T4315: [provenance] Fallback to swh-storage if a revision or its parent is not found in swh-graph as Normal priority.

Jun 8 2022, 2:16 PM · Provenance database

vsellier added a revision to T4314: [provenance] HistoryGraph building is failing when reaching the first revision: D7972: Support empty response from swh-graph.

Jun 8 2022, 1:59 PM · Provenance database

vsellier triaged T4314: [provenance] HistoryGraph building is failing when reaching the first revision as Normal priority.

Jun 8 2022, 12:43 AM · Provenance database

Jun 7 2022

vsellier triaged T4313: [provenance] some process are oom killed as Normal priority.

Jun 7 2022, 8:25 PM · System administration, Provenance database

Jun 4 2022

vsellier closed T4302: provenance: origin processed several time by different clients as Invalid.

This issue was a false lead, different snapshots are declared for the same origin letting me believe there were some duplicates.
I completely missed it and fall head first into the trap

Jun 4 2022, 8:19 AM · Provenance database

Jun 3 2022

vsellier added a comment to T4302: provenance: origin processed several time by different clients.

after some ingestion time, it seems the first analysis is wrong:

(provenance) provenance-client01:~$ sort ~/origin-client.log |grep Processing | uniq -c  | sort -n | tail -n 10
      6 INFO:swh.provenance.origin:Processing origin https://bitbucket.org/360factors/workflow-conditions.git
      6 INFO:swh.provenance.origin:Processing origin https://bitbucket.org/4s/cda-document-viewer.git
      6 INFO:swh.provenance.origin:Processing origin https://bitbucket.org/4tic/goc.git
      6 INFO:swh.provenance.origin:Processing origin https://bitbucket.org/ALEXks/sapfor_2017.git
      6 INFO:swh.provenance.origin:Processing origin https://bitbucket.org/ASIWeb/testcomplete.git
      7 INFO:swh.provenance.origin:Processing origin https://anonhg.netbsd.org/xsrc-draft/
      7 INFO:swh.provenance.origin:Processing origin https://anonhg.netbsd.org/xsrc-public/
      8 INFO:swh.provenance.origin:Processing origin https://anonhg.netbsd.org/xsrc/
     13 INFO:swh.provenance.origin:Processing origin https://bitbucket.org/9front/plan9front
     20 INFO:swh.provenance.origin:Processing origin https://anongit.kde.org/kdenlive.git

Jun 3 2022, 11:46 AM · Provenance database

vsellier added a revision to T4302: provenance: origin processed several time by different clients: D7956: Provenance: Docker for origin layer + logging improvement.

Jun 3 2022, 11:44 AM · Provenance database

Jun 2 2022

vsellier added a comment to T4302: provenance: origin processed several time by different clients.

It's not as bad as expected, it seems only 2 clients are proceeding the same origin at the same time:

Jun 2 2022, 5:34 PM · Provenance database

vsellier triaged T4302: provenance: origin processed several time by different clients as Normal priority.

Jun 2 2022, 3:40 PM · Provenance database

May 25 2022

ardumont added a revision to T4230: [provenance] pip deploy the stack on the ovh server: D7900: Update manifest to exclude swh/provenance/tools.

May 25 2022, 5:20 PM · Provenance database, System administration

May 11 2022

vsellier changed the status of T4230: [provenance] pip deploy the stack on the ovh server from Open to Work in Progress.

May 11 2022, 10:37 AM · Provenance database, System administration

Apr 19 2022

vsellier added a watcher for Provenance database: vsellier.

Apr 19 2022, 4:36 PM

Mar 31 2022

bchauvet moved T3112: Provenance index for the full archive from Backlog to Work in progress on the Roadmap 2022 board.

Mar 31 2022, 4:17 PM · Roadmap 2022, Provenance database, Roadmap 2021, meta-task

Mar 25 2022

bchauvet raised the priority of T3112: Provenance index for the full archive from Normal to High.

Mar 25 2022, 5:27 PM · Roadmap 2022, Provenance database, Roadmap 2021, meta-task

Mar 23 2022

bchauvet added a project to T3112: Provenance index for the full archive: Roadmap 2022.

Mar 23 2022, 5:07 PM · Roadmap 2022, Provenance database, Roadmap 2021, meta-task

Dec 6 2021

ardumont added projects to T3772: Fix default collation configuration on staging databases: Provenance database, Staging environment, System administration.

Dec 6 2021, 6:02 PM · System administration, Staging environment, Provenance database

Nov 4 2021

jayeshv added a comment to T3557: Run experiments against the MongoDB backend .

2M with unique sha1 index (Embedded arrays with bulk write on content and directory)
(Exited after 542,000 of memory errors)

Nov 4 2021, 10:52 AM · Provenance database

jayeshv added a comment to T3557: Run experiments against the MongoDB backend .

Postgres: 10k with sha1 index (Embedded arrays with bulk write on content and directory)
python client.py -n 4 -C config.yml

Nov 4 2021, 10:33 AM · Provenance database

jayeshv added a comment to T3557: Run experiments against the MongoDB backend .

Postgres: 10k with sha1 index (Embedded arrays without bulk write)
python client.py -n 4 -C config.yml

Nov 4 2021, 10:12 AM · Provenance database

Sep 6 2021

jayeshv renamed T3561: Implement the initial MonogDB backend with a simple data model from Implement the initial MonogDB backend to Implement the initial MonogDB backend with a simple data model.

Sep 6 2021, 2:14 PM · Provenance database

jayeshv closed T3561: Implement the initial MonogDB backend with a simple data model, a subtask of T3431: Implement a MongoDB backend for SWH-provenance , as Resolved.

Sep 6 2021, 2:13 PM · Provenance database

jayeshv closed T3561: Implement the initial MonogDB backend with a simple data model as Resolved.

Initial version is merged to master.

Sep 6 2021, 2:13 PM · Provenance database

jayeshv triaged T3561: Implement the initial MonogDB backend with a simple data model as Normal priority.

Sep 6 2021, 2:12 PM · Provenance database

jayeshv added a comment to T3557: Run experiments against the MongoDB backend .

First test 07/09/2021
Against the simple/experimental data model version. (https://forge.softwareheritage.org/rDPROV3e009a2f77de1d4d00eb52f838537c7af327f010)
Computer: Dell 7400

Sep 6 2021, 10:41 AM · Provenance database

jayeshv triaged T3557: Run experiments against the MongoDB backend as Normal priority.