Page MenuHomeSoftware Heritage

olasd (Nicolas Dandrimont)
UserAdministrator

Projects (8)

User Details

User Since
Sep 7 2015, 3:25 PM (333 w, 3 d)
Roles
Administrator

Recent Activity

Yesterday

olasd added a comment to T3887: Storing multiple authors in Revisions and Releases.
In T3887#77951, @zack wrote:

Do you foresee any issue in adding extra_headers to releases as well, other than "someone should do it"?

Thu, Jan 27, 5:54 PM · SWORD deposit, Data Model, BZR loader
olasd added a comment to T3887: Storing multiple authors in Revisions and Releases.

Now that I've written it out loud, of course, Releases don't have extra_headers so the package loaders can't make use of this workaround/hack for now.

Thu, Jan 27, 5:34 PM · SWORD deposit, Data Model, BZR loader
olasd renamed T3887: Storing multiple authors in Revisions and Releases from Storing multiple authors to Storing multiple authors in Revisions and Releases.
Thu, Jan 27, 5:30 PM · SWORD deposit, Data Model, BZR loader
olasd added a comment to T3887: Storing multiple authors in Revisions and Releases.

From merged tasks, this would also be useful for some package loaders, e.g. npm, that support multiple authors in their packaging metadata.

Thu, Jan 27, 5:29 PM · SWORD deposit, Data Model, BZR loader
olasd merged task T1645: Associating multiple authors to a revision into T3887: Storing multiple authors in Revisions and Releases.
Thu, Jan 27, 5:28 PM · Data Model
olasd merged task T3284: Support for multiple revision authors? into T3887: Storing multiple authors in Revisions and Releases.
Thu, Jan 27, 5:28 PM · Data Model
olasd merged tasks T1645: Associating multiple authors to a revision, T3284: Support for multiple revision authors? into T3887: Storing multiple authors in Revisions and Releases.
Thu, Jan 27, 5:28 PM · SWORD deposit, Data Model, BZR loader
olasd lowered the priority of T3887: Storing multiple authors in Revisions and Releases from Normal to Wishlist.
Thu, Jan 27, 5:27 PM · SWORD deposit, Data Model, BZR loader
olasd added a comment to T3887: Storing multiple authors in Revisions and Releases.

Practically, we could be storing the metadata on additional authors *now* in the extra_headers field (make them a bunch of (b'author', b'XXX <yyy@zzz.ttt>') entries). Of course, that doesn't solve the question of presenting the information.

Thu, Jan 27, 5:26 PM · SWORD deposit, Data Model, BZR loader
olasd added a comment to T3888: Storing associated bugs.

Is this information fully intrinsic, or can it be modified without the revision id changing ?

Thu, Jan 27, 5:15 PM · BZR loader
olasd added inline comments to D7021: Add graph dataset reading classes (orc+edges).
Thu, Jan 27, 3:36 PM

Wed, Jan 26

olasd created P1267 Command-Line Input.
Wed, Jan 26, 9:46 PM
olasd committed rSPSITEb4bc8a45a68a: Refresh key for deb.sury.org (authored by olasd).
Refresh key for deb.sury.org
Wed, Jan 26, 5:36 PM
olasd committed rDSNIP0455611094ee: recover_corrupt_objects: clean imports (authored by olasd).
recover_corrupt_objects: clean imports
Wed, Jan 26, 1:02 PM
olasd committed rDSNIP5e20eee1b438: recover_corrupt_objects: use logging instead of tqdm (authored by olasd).
recover_corrupt_objects: use logging instead of tqdm
Wed, Jan 26, 1:02 PM
olasd committed rDSNIP871da6ca0a09: recover_corrupt_objects: use stream_results instead of stream_results_optional (authored by olasd).
recover_corrupt_objects: use stream_results instead of stream_results_optional
Wed, Jan 26, 1:02 PM

Tue, Jan 25

olasd committed rSPSITE89642e23c829: unattended_upgrades: Allow the new codename for debian-security (authored by olasd).
unattended_upgrades: Allow the new codename for debian-security
Tue, Jan 25, 8:24 PM
olasd committed rDSNIP99a3e4997191: recover_corrupt_objects: make assertions more useful (authored by olasd).
recover_corrupt_objects: make assertions more useful
Tue, Jan 25, 6:27 PM
olasd added inline comments to D7035: Automate weekly-planning script.
Tue, Jan 25, 5:00 PM
olasd committed rDSNIP27369c56cac9: recover_corrupt_objects: Remove spurious select (authored by olasd).
recover_corrupt_objects: Remove spurious select
Tue, Jan 25, 4:52 PM
olasd committed rDSNIP44fe78774db2: recover_corrupt_objects: Pull config from a configfile instead of the cli (authored by olasd).
recover_corrupt_objects: Pull config from a configfile instead of the cli
Tue, Jan 25, 4:52 PM
olasd added a comment to D7029: Filter out extids targeting non-existing releases.

I wonder if that'd be worth a warning. May end up being a bit noisy though.

Tue, Jan 25, 3:52 PM
olasd accepted D7033: the desired key len is 32 for sha256.

Thanks! (and sorry for the hash algo ping-pong)

Tue, Jan 25, 3:46 PM
olasd accepted D6957: Add recover_corrupt_objects.py.

I'm not 100% convinced we need to recheck the objects at every addition (within a transaction that can still fail to commit) instead of afterwards, but it doesn't /hurt/ either. We'll make a full pass on all objects again later anyway.

Tue, Jan 25, 1:43 PM

Mon, Jan 24

olasd added inline comments to D7024: Fix directory_add to actually insert the manifest + add directory_get_raw_manifest.
Mon, Jan 24, 2:27 PM
olasd accepted D7024: Fix directory_add to actually insert the manifest + add directory_get_raw_manifest.

Oof. I guess that's one more reason to be wary of TODOs in tests.

Mon, Jan 24, 1:32 PM
olasd triaged T3879: Replace ntp with systemd-timesyncd (or chrony) for time synchronization across all the infra as Low priority.
Mon, Jan 24, 1:28 PM · System administration

Fri, Jan 21

olasd accepted D7020: Configure the kafka clusters environment for the consumer lag exporter.

Neat, thanks.

Fri, Jan 21, 7:01 PM
olasd added inline comments to D6979: Document the mirror credentials management.
Fri, Jan 21, 5:39 PM
olasd accepted D6986: kafka: add a script to create the kafka credentials.

thanks!

Fri, Jan 21, 5:38 PM
olasd accepted D7016: Refactor clone with timout util from hg loader.
Fri, Jan 21, 4:51 PM
olasd accepted D7008: Stop using the deprecated 'TimestampWithTimezone.offset' attribute.

Thanks!

Fri, Jan 21, 2:49 PM
olasd accepted D7007: Stop using the deprecated 'TimestampWithTimezone.offset' attribute.

Thanks!

Fri, Jan 21, 2:48 PM
olasd accepted D7006: Stop using the deprecated 'TimestampWithTimezone.offset' attribute.

Thanks!

Fri, Jan 21, 2:47 PM
olasd accepted D7009: Consider unauthorized access to origin as a not found visit status.

On most forges, 403 errors are used in place of 404 errors (so you're not able to do discovery of private repositories by checking the return code). I think treating them as not found is correct.

Fri, Jan 21, 2:43 PM
olasd accepted D6937: Remove 'offset' and 'negative_utc'.

Either way, I think this is good to go, thanks!

Fri, Jan 21, 2:01 PM
olasd added a comment to D6937: Remove 'offset' and 'negative_utc'.

We still query the fields out of the Postgres database (to ignore them in the db -> model object conversion), right? Should we stop doing that too?

Fri, Jan 21, 1:53 PM
olasd accepted D7005: Add method 'TimestampWithTimezone.offset_minutes'.

Sounds good to me, thanks!

Fri, Jan 21, 1:39 PM
olasd accepted D6984: Refactor clone with timout util from hg loader.

Thanks!

Fri, Jan 21, 11:31 AM
olasd accepted D6944: Add a few statsd metrics in the kafka journal client.
Fri, Jan 21, 11:28 AM

Thu, Jan 20

olasd accepted D6991: Fix direct sql query for directories to the archive.
Thu, Jan 20, 5:58 PM
olasd published D6991: Fix direct sql query for directories to the archive for review.
Thu, Jan 20, 5:58 PM
olasd accepted D6989: Add a Statsd.status_gauge() context manager.

Awesome, thanks!

Thu, Jan 20, 3:27 PM
olasd added a comment to D5646: Make buffer and validate proxy storage also handle other object types.

The buffer bits look reasonable to me. I'm not so sure about the validator bits, as we're not actually storing the id fields for these objects, afaik?

Thu, Jan 20, 3:25 PM
olasd closed T3487: Installation of the new provenance server as Resolved.
Thu, Jan 20, 11:47 AM · System administration
olasd closed D6983: Add provenance-client01 virtual machine.
Thu, Jan 20, 11:22 AM
olasd committed rSPREee8a56b4df35: Add provenance-client01 virtual machine (authored by olasd).
Add provenance-client01 virtual machine
Thu, Jan 20, 11:22 AM
olasd updated the diff for D6983: Add provenance-client01 virtual machine.

Update tfstate after provisioning the machine

Thu, Jan 20, 11:22 AM
olasd added a comment to D6986: kafka: add a script to create the kafka credentials.

Could we create this script, for each kafka cluster, on the kafka management host rather than on individual brokers, so we can have all the management in a single place?

Thu, Jan 20, 11:02 AM
olasd added a comment to D6983: Add provenance-client01 virtual machine.

lgtm

Note: jsyk, the inventory entry [1] references 40G for that node and the default is 32G (as per the test plan).

[1] https://inventory.internal.softwareheritage.org/virtualization/virtual-machines/108/edit/

Thu, Jan 20, 10:25 AM
olasd added a comment to D6984: Refactor clone with timout util from hg loader.

Any chance you could add a couple of tests for this?

Thu, Jan 20, 10:12 AM
olasd committed rSPREde0278bd78dd: proxmox/terraform: Make the vmid variable properly optional (authored by olasd).
proxmox/terraform: Make the vmid variable properly optional
Thu, Jan 20, 8:50 AM
olasd closed D6982: proxmox/terraform: Make the vmid variable properly optional.
Thu, Jan 20, 8:50 AM
olasd committed rSPRE686b74e25834: Mark kelvingrove's cpu as host instead of kvm64 (authored by olasd).
Mark kelvingrove's cpu as host instead of kvm64
Thu, Jan 20, 8:50 AM
olasd closed D6981: Mark kelvingrove's cpu as host instead of kvm64.
Thu, Jan 20, 8:50 AM

Wed, Jan 19

olasd added inline comments to D6957: Add recover_corrupt_objects.py.
Wed, Jan 19, 11:21 PM
olasd closed T3819: Deploy swh.model 4.1.0 / swh.storage 0.41.0 to production as Resolved.
Wed, Jan 19, 7:12 PM · System administration
olasd closed T3819: Deploy swh.model 4.1.0 / swh.storage 0.41.0 to production, a subtask of T3752: Store/represent time offsets as strings, as Resolved.
Wed, Jan 19, 7:12 PM · Data Model, Storage manager
olasd updated the task description for T3819: Deploy swh.model 4.1.0 / swh.storage 0.41.0 to production.
Wed, Jan 19, 7:12 PM · System administration
olasd added a comment to T3819: Deploy swh.model 4.1.0 / swh.storage 0.41.0 to production.
softwareheritage=#   alter table revision
    add constraint revision_date_offset_not_null
    check (date is null or date_offset_bytes is not null) not valid,
    add constraint revision_committer_date_offset_not_null
    check (committer_date is null or committer_date_offset_bytes is not null) not valid;
ALTER TABLE
Wed, Jan 19, 7:11 PM · System administration
olasd requested review of D6983: Add provenance-client01 virtual machine.
Wed, Jan 19, 6:25 PM
olasd requested review of D6982: proxmox/terraform: Make the vmid variable properly optional.
Wed, Jan 19, 6:24 PM
olasd requested review of D6981: Mark kelvingrove's cpu as host instead of kvm64.
Wed, Jan 19, 6:23 PM
olasd committed rDDOCb2a68a9d2605: Add hints on how to deploy the storage db externally (authored by olasd).
Add hints on how to deploy the storage db externally
Wed, Jan 19, 4:04 PM
olasd closed D6969: Update mirror docker docs following a walkthrough.
Wed, Jan 19, 4:04 PM
olasd committed rDDOC2b8d86d713fb: Update mirror docker docs following a walkthrough (authored by olasd).
Update mirror docker docs following a walkthrough
Wed, Jan 19, 4:04 PM
olasd updated the diff for D6969: Update mirror docker docs following a walkthrough.

Gotta love rebasing

Wed, Jan 19, 3:47 PM
olasd committed rCDFP18d94c295557: Migrate volume-dependent service placement constraints to labels (authored by olasd).
Migrate volume-dependent service placement constraints to labels
Wed, Jan 19, 3:42 PM
olasd committed rCDFP35cd316296af: Gitignore the graph-replayer and content-replayer configs (authored by olasd).
Gitignore the graph-replayer and content-replayer configs
Wed, Jan 19, 3:42 PM
olasd closed D6978: Update docker stack configuration to match documentation changes.
Wed, Jan 19, 3:42 PM
olasd committed rCDFPc26ab911ae13: Make the default storage db flavor "mirror" (authored by olasd).
Make the default storage db flavor "mirror"
Wed, Jan 19, 3:42 PM
olasd removed a reviewer for D6978: Update docker stack configuration to match documentation changes: Reviewers.
Wed, Jan 19, 3:40 PM
olasd updated the diff for D6969: Update mirror docker docs following a walkthrough.

Add hints on how to deploy the storage db externally

Wed, Jan 19, 3:30 PM
olasd requested review of D6978: Update docker stack configuration to match documentation changes.
Wed, Jan 19, 2:56 PM
olasd added a revision to T3829: Document mirror - how to create and deploy a mirror from scratch: D6978: Update docker stack configuration to match documentation changes.
Wed, Jan 19, 2:56 PM · Mirror
olasd closed D6968: Rename secrets using a swh-mirror prefix.
Wed, Jan 19, 2:54 PM
olasd committed rCDFP6e574b79e7c6: Rename secrets using a swh-mirror prefix (authored by olasd).
Rename secrets using a swh-mirror prefix
Wed, Jan 19, 2:54 PM
olasd updated the diff for D6969: Update mirror docker docs following a walkthrough.

*shrug*

Wed, Jan 19, 2:44 PM
olasd triaged T3863: Allow loading sensitive configuration values from distinct, more restricted, files as Normal priority.
Wed, Jan 19, 2:41 PM · Core & foundations
olasd requested review of D6969: Update mirror docker docs following a walkthrough.
Wed, Jan 19, 12:36 PM

Tue, Jan 18

olasd requested review of D6968: Rename secrets using a swh-mirror prefix.
Tue, Jan 18, 6:04 PM
olasd added a comment to D6944: Add a few statsd metrics in the kafka journal client.

The code for the gauges feels like something that would be usefully handled with a context manager.

Tue, Jan 18, 2:54 PM
olasd accepted D6884: Add support for the rdkafka 'stats_cb' config option in JournalClient.

I'm kinda wondering if this import stuff should move to a common module - I think we do kind of the same thing with entrypoints?

Tue, Jan 18, 1:42 PM
olasd added a comment to T3819: Deploy swh.model 4.1.0 / swh.storage 0.41.0 to production.

All revisions (supposedly) have been migrated to bytes offsets. I'll wait for the ongoing base backup and vacuum to complete before adding the constraints on the production database.

Tue, Jan 18, 12:28 PM · System administration
olasd merged task T2449: Consider switching timestamp offset storage to strings/byte arrays into T3752: Store/represent time offsets as strings.
Tue, Jan 18, 12:27 PM · Storage manager, Data Model
olasd merged T2449: Consider switching timestamp offset storage to strings/byte arrays into T3752: Store/represent time offsets as strings.
Tue, Jan 18, 12:27 PM · Data Model, Storage manager
olasd added a subtask for T3752: Store/represent time offsets as strings: T3819: Deploy swh.model 4.1.0 / swh.storage 0.41.0 to production.
Tue, Jan 18, 12:26 PM · Data Model, Storage manager
olasd added a parent task for T3819: Deploy swh.model 4.1.0 / swh.storage 0.41.0 to production: T3752: Store/represent time offsets as strings.
Tue, Jan 18, 12:26 PM · System administration
olasd updated the task description for T3819: Deploy swh.model 4.1.0 / swh.storage 0.41.0 to production.
Tue, Jan 18, 12:26 PM · System administration
olasd accepted D6956: model: Add support for more edge cases in _parse_offset_bytes.

Great, thanks!

Tue, Jan 18, 11:01 AM
olasd added a comment to D6958: postgres: Add indices to keep track of objects with a raw_manifest.

Sounds good. I don't think we need to make them unique though (and it might make stuff confusing if writes get rejected by this index rather than the primary one)

Tue, Jan 18, 11:00 AM

Mon, Jan 17

olasd accepted D6939: Stop passing 'offset' and 'negative_utc' to TimestampWithTimezone().

Thanks!

Mon, Jan 17, 4:44 PM
olasd added inline comments to D6939: Stop passing 'offset' and 'negative_utc' to TimestampWithTimezone().
Mon, Jan 17, 4:30 PM
olasd closed D6858: requirements-test: Drop pre-commit.
Mon, Jan 17, 4:27 PM
olasd committed rDCOREd374b6002955: requirements-test: Drop pre-commit (authored by olasd).
requirements-test: Drop pre-commit
Mon, Jan 17, 4:27 PM
olasd updated the diff for D6858: requirements-test: Drop pre-commit.

Rebase

Mon, Jan 17, 4:25 PM
olasd accepted D6894: converters: Write object_bytes and raw_manifest on revisions and releases.

This all looks fine to me, thanks!

Mon, Jan 17, 4:22 PM

Fri, Jan 14

olasd accepted D6923: converters: Write raw_manifest of Directory objects.

Awesome, thanks.

Fri, Jan 14, 6:29 PM
olasd added inline comments to D6939: Stop passing 'offset' and 'negative_utc' to TimestampWithTimezone().
Fri, Jan 14, 5:50 PM
olasd accepted D6940: tests: Use 'offset_bytes' instead of 'offset'/'negative_utc'.

Great, thanks!

Fri, Jan 14, 5:49 PM