Page MenuHomeSoftware Heritage
Feed Advanced Search

May 5 2021

douardda closed D5671: Make postgresql's origin_add not raise an error in case of conflict.
May 5 2021, 12:44 PM
douardda committed rDSTO77ef651d9582: Make postgresql's origin_add not raise an error in case of conflict (authored by douardda).
Make postgresql's origin_add not raise an error in case of conflict
May 5 2021, 12:43 PM
douardda accepted D5656: Add a naive git bare cooker.
May 5 2021, 12:40 PM
douardda accepted D5670: Stop storing authority/fetcher metadata..
May 5 2021, 12:34 PM
douardda updated the diff for D5671: Make postgresql's origin_add not raise an error in case of conflict.

rebased

May 5 2021, 12:33 PM
douardda added inline comments to D5671: Make postgresql's origin_add not raise an error in case of conflict.
May 5 2021, 12:30 PM
douardda closed D3334: Add a new TenaciousProxyStorage.
May 5 2021, 12:18 PM
douardda committed rDSTOffb38f71d9b6: Add a new TenaciousProxyStorage (authored by douardda).
Add a new TenaciousProxyStorage
May 5 2021, 12:18 PM
douardda updated the diff for D3334: Add a new TenaciousProxyStorage.

remove dead code and improve a bit the commit message

May 5 2021, 11:59 AM
douardda added inline comments to D5671: Make postgresql's origin_add not raise an error in case of conflict.
May 5 2021, 11:56 AM
douardda added inline comments to D3334: Add a new TenaciousProxyStorage.
May 5 2021, 11:49 AM
douardda created P1034 (An Untitled Masterwork).
May 5 2021, 10:05 AM

May 4 2021

douardda added inline comments to D5656: Add a naive git bare cooker.
May 4 2021, 6:17 PM
douardda accepted D5659: Make the SourceForge lister incremental.

besides the type aliasing statements that I find a bit confusing, LGTM.

May 4 2021, 5:17 PM
douardda accepted D5628: Replace old loader with the new one.

I'm not very fond of the name of all this (from_disk, HgLoaderFromDisk, etc.) since it makes people think it cannot deal with remote hg repos (which the HgLoaderFromDisk does, but this is not really advertized, as far as I can tell).

May 4 2021, 4:53 PM
douardda added inline comments to D5628: Replace old loader with the new one.
May 4 2021, 4:42 PM
douardda requested review of D5671: Make postgresql's origin_add not raise an error in case of conflict.
May 4 2021, 4:23 PM
douardda updated the diff for D3334: Add a new TenaciousProxyStorage.

Answer (most) vlorentz comments/remark

May 4 2021, 3:32 PM
douardda added inline comments to D3334: Add a new TenaciousProxyStorage.
May 4 2021, 3:00 PM
douardda updated the diff for D3334: Add a new TenaciousProxyStorage.

rebase and improve logging messages

May 4 2021, 12:10 PM
douardda triaged T3303: Replaying REMD topics is broken as High priority.
May 4 2021, 10:11 AM · Mirror

May 3 2021

douardda added inline comments to D5646: Make buffer and validate proxy storage also handle other object types.
May 3 2021, 3:34 PM
douardda added a comment to D5648: Add a bit of logging in the buffer proxy storage.

What's the motivation?

May 3 2021, 3:33 PM
douardda accepted D5582: cassandra: Add 'allow_overwrite' option, to allow updating objects.
May 3 2021, 2:14 PM

Apr 29 2021

douardda updated the diff for D5648: Add a bit of logging in the buffer proxy storage.

rebase

Apr 29 2021, 4:19 PM
douardda updated the diff for D5646: Make buffer and validate proxy storage also handle other object types.

and MetadataXXX

Apr 29 2021, 4:19 PM
douardda updated the diff for D5648: Add a bit of logging in the buffer proxy storage.

rebase

Apr 29 2021, 4:02 PM
douardda updated the diff for D5646: Make buffer and validate proxy storage also handle other object types.

and the origin_visit_status_add() specific implementation as well...

Apr 29 2021, 4:01 PM
douardda requested review of D5648: Add a bit of logging in the buffer proxy storage.
Apr 29 2021, 3:55 PM
douardda published D5646: Make buffer and validate proxy storage also handle other object types for review.
Apr 29 2021, 3:47 PM
douardda committed rDSTO92d551a4a515: Normalize all Storage.xxx_add() methods to return a summary (authored by douardda).
Normalize all Storage.xxx_add() methods to return a summary
Apr 29 2021, 3:08 PM
douardda closed D5643: Normalize all Storage.xxx_add() methods to return a summary.
Apr 29 2021, 3:08 PM
douardda committed rDSTOff7ecb4b8445: Properly annotate output of Storage.xxx_add() methods as Dict[str, int] (authored by douardda).
Properly annotate output of Storage.xxx_add() methods as Dict[str, int]
Apr 29 2021, 3:08 PM
douardda requested review of D5643: Normalize all Storage.xxx_add() methods to return a summary.
Apr 29 2021, 12:58 PM
douardda requested changes to D5582: cassandra: Add 'allow_overwrite' option, to allow updating objects.

I'm not fond of the 'check_missing' name for the argument, but would prefer 'allow_overwrite'.

Apr 29 2021, 9:53 AM

Apr 28 2021

douardda closed D5634: Add a fixer for ExtrinsicRawMetadata.
Apr 28 2021, 3:00 PM
douardda committed rDSTO98804f9e1262: Add a fixer for ExtrinsicRawMetadata (authored by douardda).
Add a fixer for ExtrinsicRawMetadata
Apr 28 2021, 3:00 PM
douardda added inline comments to D5634: Add a fixer for ExtrinsicRawMetadata.
Apr 28 2021, 2:16 PM
douardda updated the diff for D5634: Add a fixer for ExtrinsicRawMetadata.

remove the pprint from the doctest

Apr 28 2021, 2:14 PM
douardda added inline comments to D5634: Add a fixer for ExtrinsicRawMetadata.
Apr 28 2021, 2:11 PM
douardda requested review of D5634: Add a fixer for ExtrinsicRawMetadata.
Apr 28 2021, 12:44 PM
douardda created P1027 (An Untitled Masterwork).
Apr 28 2021, 12:34 PM
douardda created P1026 (An Untitled Masterwork).
Apr 28 2021, 12:27 PM
douardda created P1025 (An Untitled Masterwork).
Apr 28 2021, 12:23 PM

Apr 27 2021

douardda closed D5589: Fix swh_model_data hardcoded id values.
Apr 27 2021, 10:42 AM
douardda committed rDMOD446bd2b167c3: Fix swh_model_data hardcoded id values (authored by douardda).
Fix swh_model_data hardcoded id values
Apr 27 2021, 10:42 AM
douardda added a comment to D5589: Fix swh_model_data hardcoded id values.

if you are convinced it's a report covering issue, ok then ;)

Apr 27 2021, 10:41 AM
douardda accepted D5584: cassandra: Add a test of a 'complex' migration, with a PK update.

I would not say all this is crystal clear to me, but overall looks fine.

Apr 27 2021, 10:34 AM
douardda added inline comments to D5582: cassandra: Add 'allow_overwrite' option, to allow updating objects.
Apr 27 2021, 10:26 AM
douardda accepted D5614: tarball: properly normalize perms for all extracted files.

lgtm

Apr 27 2021, 10:16 AM
douardda added a comment to D5614: tarball: properly normalize perms for all extracted files.

Why only 0o100644 and 0o100755? I don't think we should make the package loaders discard this information just because Git does.

Apr 27 2021, 10:16 AM
douardda added inline comments to D5589: Fix swh_model_data hardcoded id values.
Apr 27 2021, 10:12 AM

Apr 26 2021

douardda updated the diff for D3334: Add a new TenaciousProxyStorage.

Rebased and updated to current HEAD

Apr 26 2021, 4:23 PM
douardda committed rDSTO2c477ec442b7: Fix storage_data hardcoded id values (authored by douardda).
Fix storage_data hardcoded id values
Apr 26 2021, 4:19 PM
douardda closed D5587: Fix storage_data hardcoded id values.
Apr 26 2021, 4:19 PM
douardda added inline comments to D5589: Fix swh_model_data hardcoded id values.
Apr 26 2021, 4:06 PM
douardda requested review of D5587: Fix storage_data hardcoded id values.
Apr 26 2021, 2:35 PM
douardda added inline comments to D5589: Fix swh_model_data hardcoded id values.
Apr 26 2021, 2:29 PM
douardda committed rDENVf9991bc8ed2b: Remove most of the README content and point to the Developer setup page (authored by douardda).
Remove most of the README content and point to the Developer setup page
Apr 26 2021, 1:57 PM
douardda closed D5585: Remove most of the README content and point to the Developer setup page.
Apr 26 2021, 1:57 PM
douardda updated the diff for D5585: Remove most of the README content and point to the Developer setup page.

rebase

Apr 26 2021, 1:55 PM

Apr 23 2021

douardda requested review of D5589: Fix swh_model_data hardcoded id values.
Apr 23 2021, 5:27 PM
douardda updated the diff for D5585: Remove most of the README content and point to the Developer setup page.

also add lik to the docker page

Apr 23 2021, 12:10 PM
douardda requested review of D5585: Remove most of the README content and point to the Developer setup page.
Apr 23 2021, 11:57 AM
douardda added a comment to T3283: Create a vm to test the mirror environment.

it works! thx

Apr 23 2021, 11:06 AM · System administration

Apr 22 2021

douardda created P1014 (An Untitled Masterwork).
Apr 22 2021, 5:37 PM
douardda created P1013 (An Untitled Masterwork).
Apr 22 2021, 5:33 PM
douardda updated the task description for T3281: Create a list of known test/buggy repos and use them in loader/storage tests.
Apr 22 2021, 2:59 PM
douardda added a subtask for T1957: Handling missing DAG nodes: T3282: Add support for "uninterpreted upstream object" in SWH model and storage.
Apr 22 2021, 2:44 PM · Data Model
douardda added a parent task for T3282: Add support for "uninterpreted upstream object" in SWH model and storage: T1957: Handling missing DAG nodes.
Apr 22 2021, 2:44 PM · Data Model
douardda reopened T3282: Add support for "uninterpreted upstream object" in SWH model and storage as "Open".

actually no, it's not the same...

Apr 22 2021, 2:43 PM · Data Model
douardda added a comment to T1957: Handling missing DAG nodes.

Examples of such missing objects are revisions with attributes that cannot fit the current data model, e.g. out of range dates. We have example of such revisions in kafka, as mentionned in T3200 and T3170.

Apr 22 2021, 2:39 PM · Data Model
douardda closed T3282: Add support for "uninterpreted upstream object" in SWH model and storage as Wontfix.

Same as T1957

Apr 22 2021, 2:37 PM · Data Model
douardda renamed T3282: Add support for "uninterpreted upstream object" in SWH model and storage from Add support for "uninterpreted upstream object" in our model and storage to Add support for "uninterpreted upstream object" in SWH model and storage.
Apr 22 2021, 2:33 PM · Data Model
douardda created T3282: Add support for "uninterpreted upstream object" in SWH model and storage.
Apr 22 2021, 2:32 PM · Data Model
douardda triaged T3281: Create a list of known test/buggy repos and use them in loader/storage tests as Normal priority.
Apr 22 2021, 12:25 PM
douardda added a comment to T3200: Mirror: year is out of range.

These revisions are probably coming from https://gitlab.com/gitlab-org/gitlab-test (or a clone)

Apr 22 2021, 12:10 PM · Mirror
douardda added a comment to T3200: Mirror: year is out of range.

Ah fun, one of the revisions with this pb, on staging (ba3343bc4fa403a8dfbfcab7fc1a8c29ee34bd69) seems to have been crafted by https://gitlab.com/gitlab-org/gitlab-foss/-/blob/staging-26-fix_add_deploy_key_spec/spec/models/merge_request_diff_commit_spec.rb

Apr 22 2021, 12:05 PM · Mirror

Apr 21 2021

douardda added a comment to T3170: Revisions in the journal with out of range dates.

Note that none of their parent revisions can be found either in the archive (one invalid revision in a set of ingested revisions prevent any of them being inserted in the database I suppose, but they are already inserted in kafka at this moment).

Apr 21 2021, 7:08 PM · Data Model, Journal
douardda added a comment to T3200: Mirror: year is out of range.

See T3170 (error generated by the same invalid kafka messages).

Apr 21 2021, 6:58 PM · Mirror
douardda created P1010 (An Untitled Masterwork).
Apr 21 2021, 10:27 AM

Apr 20 2021

douardda closed T3229: Update "software architecture" image as Resolved by committing rDDOCbf30a9270a87: Update the general architecture diagram.
Apr 20 2021, 5:27 PM · Documentation
douardda closed D5558: Update the general architecture diagram.
Apr 20 2021, 5:27 PM
douardda committed rDDOCbf30a9270a87: Update the general architecture diagram (authored by douardda).
Update the general architecture diagram
Apr 20 2021, 5:27 PM
douardda added a comment to T3087: Implement support for takedown notices (infra, admin tools, workflow).

So what about exports of the archive available on git-annex?

Apr 20 2021, 11:09 AM · Roadmap 2022, meta-task, Roadmap 2021, Web app
douardda added a comment to T3246: Document takedown request processing workflow.

do we also intent to have a takedown topic on kafka?

Apr 20 2021, 11:08 AM · Archive content
douardda added a comment to T1481: add metric to monitor "save code now" efficiency.

Note that there is the same transient vs cumulative discrepency on the "Accepted requests" graph.

Apr 20 2021, 11:06 AM · Save Code Now, System administration, Metrics/monitoring
douardda added a comment to T1481: add metric to monitor "save code now" efficiency.

I think the "submitted requests per visit type / status" graph should be split in 2 parts. Both accepted and rejected are cumulative values that will indefinitely grow, while pending are transient value aiming at staying near zero, so it makes no sense to have them on the same graph.

Apr 20 2021, 11:02 AM · Save Code Now, System administration, Metrics/monitoring
douardda added a comment to T1481: add metric to monitor "save code now" efficiency.

I think the "submitted requests per visit type / status" graph should be split in 2 parts. Both accepted and rejected are cumulative values that will indefinitely grow, while pending are transient value aiming at staying near zero, so it makes no sense to have them on the same graph.

Apr 20 2021, 11:00 AM · Save Code Now, System administration, Metrics/monitoring
douardda added a comment to T3084: Fast track save code now requests.

is there a grafana dashboard dedicated to this queue?

Apr 20 2021, 10:55 AM · System administration, Web app
douardda changed the status of T3227: DB Schema link broken in docs under swh-storage. from Duplicate to Resolved.
Apr 20 2021, 10:52 AM · Easy hack, Documentation
douardda requested review of D5558: Update the general architecture diagram.
Apr 20 2021, 10:51 AM
douardda added a revision to T3229: Update "software architecture" image: D5558: Update the general architecture diagram.
Apr 20 2021, 10:51 AM · Documentation
douardda closed D5553: Update black to 20.8b1.
Apr 20 2021, 9:44 AM
douardda committed rDPROV8009af31a914: Update black to 20.8b1 (authored by douardda).
Update black to 20.8b1
Apr 20 2021, 9:44 AM

Apr 19 2021

douardda updated subscribers of D5553: Update black to 20.8b1.

Looks good to me. It's true the black version we are using is quite outdated.

We should upgrade black in all other swh repositories in the same manner then.

Apr 19 2021, 5:46 PM
douardda requested review of D5553: Update black to 20.8b1.
Apr 19 2021, 5:06 PM
douardda committed rDPROV982e2c1a2a9a: Add synthetic test files for the mindepth=2 heuristic (authored by douardda).
Add synthetic test files for the mindepth=2 heuristic
Apr 19 2021, 4:56 PM
douardda closed D5389: Improve tests.
Apr 19 2021, 4:56 PM
douardda closed D5388: Also test the provenance db with ArchiveStorage.
Apr 19 2021, 4:56 PM