Page MenuHomeSoftware Heritage
Feed Advanced Search

Mar 3 2022

olasd added inline comments to D7280: Properly split optional per-backend requirements.
Mar 3 2022, 1:49 PM
olasd requested review of D7280: Properly split optional per-backend requirements.
Mar 3 2022, 1:28 PM
olasd committed rDOPHfb106d8a3d78: Add explicit dependency on python3-cffi (authored by olasd).
Add explicit dependency on python3-cffi
Mar 3 2022, 12:08 PM
olasd committed rDOPH8f7ea5af2ac1: Rebuild for new cffi/dh-python3 (authored by olasd).
Rebuild for new cffi/dh-python3
Mar 3 2022, 11:58 AM
olasd committed rDOPH0c39e35bc671: Make sure _hash_cffi.so is gitignored (authored by olasd).
Make sure _hash_cffi.so is gitignored
Mar 3 2022, 11:54 AM

Mar 2 2022

olasd added a comment to D7278: deposit: Check created metadata is visible from the public API.

what values should we give to the new provider-url argument in the current icinga checks for staging and production? (the web-url one is easier :P)

Mar 2 2022, 6:15 PM
olasd accepted D7279: cache: Remove unused method add_stream.

kthxbye

Mar 2 2022, 6:14 PM
olasd added inline comments to D7272: Use flask instead of aiohttp as RPC server..
Mar 2 2022, 5:52 PM
olasd added inline comments to D7272: Use flask instead of aiohttp as RPC server..
Mar 2 2022, 5:08 PM
olasd accepted D7272: Use flask instead of aiohttp as RPC server..
Mar 2 2022, 5:06 PM
olasd added a comment to D7272: Use flask instead of aiohttp as RPC server..

I think this all looks good, thank you! Obviously, the consistency with other RPCs is much appreciated.

Mar 2 2022, 5:05 PM
olasd accepted D7277: Rename and document constants in objstorage.py.

Great, thanks!

Mar 2 2022, 4:57 PM
olasd accepted D7269: server: Group multiple object ids in the same HTTP chunk.

I really wish this had some kind of encapsulation/separator, but I guess with using raw bytes that's not really practical...

Mar 2 2022, 3:26 PM
olasd accepted D7270: Update for the new output format of the Deposit's API..

So, this is all matching the currently deployed version of swh-deposit, correct?

Mar 2 2022, 1:51 PM
olasd accepted D7268: client: Do not depend on how the server chunks the response.

Cool, thanks (see inline comment).

Mar 2 2022, 12:59 PM
olasd added a comment to D7268: client: Do not depend on how the server chunks the response.

@vlorentz looks like you didn't push the right commit in this diff update

Mar 2 2022, 11:38 AM
olasd accepted D7273: Add test to enforce all objstorage backends follow the interface.

Great, thanks.

Mar 2 2022, 11:29 AM
olasd accepted D7275: utils: Add a new 'iter_chunks' function.

Thanks

Mar 2 2022, 11:28 AM

Mar 1 2022

olasd added a comment to D7268: client: Do not depend on how the server chunks the response.

Does the get_stream method guarantee the size of the received chunks? By default I would assume that it chunks contents using the passed size as a maximum.

Mar 1 2022, 5:02 PM
olasd accepted D7266: RPCClient: Make methods {get,post}{,_stream} protected.

Thanks, that's been an itch previously.

Mar 1 2022, 4:47 PM
olasd accepted D7271: RPCServerApp: Add hooks to add behaviors to generated methods.

Looks useful, thanks.

Mar 1 2022, 4:41 PM
olasd accepted D7267: Remove method add_stream from the RPC API..

Cool, thanks

Mar 1 2022, 4:37 PM
olasd committed rSPSITE01417f37ab39: Add account for bchauvet (authored by olasd).
Add account for bchauvet
Mar 1 2022, 11:52 AM

Feb 28 2022

olasd closed D7265: Move check_url_match_provider to api.checks instead of utils.
Feb 28 2022, 7:05 PM
olasd committed rDDEPdc1ea97a1ca4: Move check_url_match_provider to api.checks instead of utils (authored by olasd).
Move check_url_match_provider to api.checks instead of utils
Feb 28 2022, 7:05 PM
olasd added a revision to T3983: Add monitoring of vault bundle downloads: D7264: vault: add monitoring of bundle downloads.
Feb 28 2022, 6:21 PM · System administration, Monitoring
olasd added a member for Reviewers: compay2k.
Feb 28 2022, 11:24 AM
olasd added a member for Developers: compay2k.
Feb 28 2022, 11:24 AM

Feb 25 2022

olasd added a comment to T3976: opam loader: Adapt for opam > 2.1.

"If needed, for special cases like unprivileged containers, sandboxing can be disabled on opam init with the --disable-sandboxing flag (only for non-initialised opam)".

Feb 25 2022, 12:23 PM · Opam, Package Loader, Core Loader

Feb 22 2022

olasd accepted D7217: postgresql/howto-connect: Update swh-deposit guest access.
Feb 22 2022, 2:53 PM

Feb 5 2022

olasd committed rSPSITE38328da817d1: keycloak: add swh.web.mailmap client role (authored by olasd).
keycloak: add swh.web.mailmap client role
Feb 5 2022, 12:35 AM
olasd committed rDWAPPS4e6a487cdac3: Add a sync_mailmaps management command (authored by olasd).
Add a sync_mailmaps management command
Feb 5 2022, 12:11 AM
olasd committed rDWAPPS0e517e6b17ae: requirements-test: Pin pytest to < 7.0.0 (authored by anlambert).
requirements-test: Pin pytest to < 7.0.0
Feb 5 2022, 12:11 AM
olasd committed rDWAPPSb32e24e920b0: Merge tag 'v0.0.356' (only containing cherry-picked changes from the master… (authored by olasd).
Merge tag 'v0.0.356' (only containing cherry-picked changes from the master…
Feb 5 2022, 12:11 AM
olasd committed rSPSITE594ea9a93953: webapp: Put pgpass settings in the right order (authored by olasd).
webapp: Put pgpass settings in the right order
Feb 5 2022, 12:09 AM
olasd committed rSPSITE12083c6ab83e: webapp: drop spaces in pg_service.conf inifile (authored by olasd).
webapp: drop spaces in pg_service.conf inifile
Feb 5 2022, 12:04 AM

Feb 4 2022

olasd committed rSPSITE201357bf46eb: webapp: Ensure pgpass and pg_service files are present (authored by olasd).
webapp: Ensure pgpass and pg_service files are present
Feb 4 2022, 11:58 PM
olasd committed rSPSITEf9f065684ed0: webapp: add timer for sync_mailmaps operation (authored by olasd).
webapp: add timer for sync_mailmaps operation
Feb 4 2022, 11:49 PM
olasd committed rSPSITE3a645e70a718: Randomize the save code now status update (authored by olasd).
Randomize the save code now status update
Feb 4 2022, 11:49 PM
olasd committed rSPSITEa8d2885938ca: webapp: simplify templates and puppet manifest for cron timer units (authored by olasd).
webapp: simplify templates and puppet manifest for cron timer units
Feb 4 2022, 11:49 PM
olasd committed rSPSITEa2e9547db0cd: webapp: make the timer enablement logic false by default (authored by olasd).
webapp: make the timer enablement logic false by default
Feb 4 2022, 11:49 PM
olasd committed rSPSITE20e105fcc308: webapp: drop some legacy, cleaned up resources (authored by olasd).
webapp: drop some legacy, cleaned up resources
Feb 4 2022, 11:49 PM
olasd committed rSPPRIVCa7119551fe19: Add swh::deploy::webapp::sync_mailmaps::db::password (authored by olasd).
Add swh::deploy::webapp::sync_mailmaps::db::password
Feb 4 2022, 11:21 PM
olasd closed D7090: Add a sync_mailmaps management command.
Feb 4 2022, 9:05 PM
olasd committed rDWAPPS39a32f958e1c: Add a sync_mailmaps management command (authored by olasd).
Add a sync_mailmaps management command
Feb 4 2022, 9:05 PM
olasd updated the diff for D7090: Add a sync_mailmaps management command.

Appease the sphinx deities

Feb 4 2022, 8:41 PM
olasd updated the diff for D7090: Add a sync_mailmaps management command.

Deferred auto_now fields don't get updated, apparently.

Feb 4 2022, 8:34 PM
olasd updated the diff for D7090: Add a sync_mailmaps management command.

hello? is it me you're looking for?

Feb 4 2022, 7:46 PM
olasd updated the diff for D7090: Add a sync_mailmaps management command.

Harden against the race condition @vlorentz noticed

Feb 4 2022, 7:44 PM
olasd added a comment to D7090: Add a sync_mailmaps management command.

I just checked, and the update is racy. (on postgresql only; it's not on sqlite since sqlite does not allow this kind of concurrency, at least not by default)

Feb 4 2022, 7:09 PM
olasd added a comment to D7090: Add a sync_mailmaps management command.

By the way, I just realized the time filter is only on the filter_to_disable query, not on filter_to_update. How come?

Feb 4 2022, 6:30 PM
olasd added a comment to D7090: Add a sync_mailmaps management command.

Shouldn't storage_db.cursor() be used as a context manager to make sure transactions committed/rolledback appropriately?

Feb 4 2022, 5:39 PM
olasd updated the diff for D7090: Add a sync_mailmaps management command.

Apply comments from @vlorentz

Feb 4 2022, 5:38 PM
olasd requested review of D7090: Add a sync_mailmaps management command.
Feb 4 2022, 3:44 PM
olasd added a comment to D7069: Add typing to revision_walker.py and make the state a dataclass.

I see you have to always ignore the typing when manipulating self._state.revs_to_visit, would it make more sense to make it an attribute of the child classes directly?

Feb 4 2022, 11:26 AM

Feb 3 2022

olasd committed rDLDBASEb3579ed07b63: Remove most manual Person() parsing in favor of Person.from_fullname (authored by olasd).
Remove most manual Person() parsing in favor of Person.from_fullname
Feb 3 2022, 2:14 PM
olasd closed D7072: Remove most manual Person() parsing in favor of Person.from_fullname.
Feb 3 2022, 2:14 PM
olasd requested review of D7072: Remove most manual Person() parsing in favor of Person.from_fullname.
Feb 3 2022, 1:33 PM
olasd committed rDLDBASE87de0140c10f: Add missing py.typed markers in swh.loader subpackages (authored by olasd).
Add missing py.typed markers in swh.loader subpackages
Feb 3 2022, 12:13 PM
olasd added a comment to T3906: Create in keycloak the swh-deposit-moderator role.

On top of the role, we need to store a mapping (I assume, in the swh-deposit database?) of which clients the moderator is allowed to view. Since we'll have to handle this information, is a new keycloak role *also* needed?

Feb 3 2022, 11:45 AM · Web app, SWORD deposit
olasd accepted D7068: mountpoints: remove useless default mountpoints.

Thanks

Feb 3 2022, 11:39 AM
olasd added inline comments to D6344: Non-incremental, non-delta Bazaar/Breezy loader.
Feb 3 2022, 10:39 AM

Feb 2 2022

olasd added a comment to T3891: Migrate sentry to admin vlan.

Looks like at least some parts of staging don't have access to sentry anymore; on storage1, for instance, https connections to sentry.softwareheritage.org just hang.

Feb 2 2022, 10:28 PM · System administration
olasd accepted D7067: git_bare: Use raw_manifest when available.

Nice, thanks!

Feb 2 2022, 6:56 PM
olasd added a comment to D7063: Make 'swh db init' fill the dbversion table.

I'm not sure I like this change. What is the advantage of this over keeping an insert into dbversion ... statement in the SQL files, which would be used when the database is initialized?

Feb 2 2022, 6:36 PM
olasd added a comment to D7062: Add support for dbversion and dbmodule handling in swh db init.

Thanks, this looks like a good step forward!

Feb 2 2022, 6:24 PM
olasd added inline comments to D7063: Make 'swh db init' fill the dbversion table.
Feb 2 2022, 5:16 PM

Feb 1 2022

olasd accepted D7057: Fix `swh_provenance_relation_add_from_temp` to use new index.

Cheers!

Feb 1 2022, 3:26 PM
olasd abandoned D7056: Use the new digest(path, 'sha1') index on location for joins.

In favor of D7057

Feb 1 2022, 2:32 PM
olasd requested review of D7056: Use the new digest(path, 'sha1') index on location for joins.
Feb 1 2022, 2:27 PM
olasd committed rDSTO4544d7ca8e9e: Introduce a new displayname field for persons in the PostgreSQL storage (authored by olasd).
Introduce a new displayname field for persons in the PostgreSQL storage
Feb 1 2022, 2:05 PM
olasd closed D7051: Introduce a new displayname field for persons in the PostgreSQL storage.
Feb 1 2022, 2:05 PM
olasd committed rDSTO97caa933c848: Make test_release_add_get_arbitrary non-flaky (authored by olasd).
Make test_release_add_get_arbitrary non-flaky
Feb 1 2022, 2:05 PM
olasd added a comment to D7051: Introduce a new displayname field for persons in the PostgreSQL storage.

Am I right in understanding that there will be a process in which users ask for some
changes (about their display name) and then we are modifying (in db) the new displayname
field in the Person table [1]? And then the reading part will simply use that
displayname (if present) over the fields we used to use for display.

Feb 1 2022, 2:05 PM
olasd updated the diff for D7051: Introduce a new displayname field for persons in the PostgreSQL storage.
  • (fallout of D7053) Make test_release_add_get_arbitrary non-flaky
  • add suggested comments for NULL name/email
  • add tests for revision_log
Feb 1 2022, 1:04 PM
olasd added inline comments to D7055: storage01.euwest.azure: Only keep the gunicorn-swh-storage service.
Feb 1 2022, 12:51 PM
olasd accepted D7054: developer-setup: Mention the current mypy version we use.

Meh, but I guess there's no real way around it...

Feb 1 2022, 10:30 AM
olasd committed rDSTOf868f3c8ad46: Mostly use normalized Person objects in tests (authored by olasd).
Mostly use normalized Person objects in tests
Feb 1 2022, 10:28 AM
olasd closed D7053: Incremental Person fullname normalization changes.
Feb 1 2022, 10:28 AM
olasd committed rDSTOd4ddd41535d0: postgresql: Use Person.from_fullname if name and email are None (authored by olasd).
postgresql: Use Person.from_fullname if name and email are None
Feb 1 2022, 10:28 AM

Jan 31 2022

olasd requested review of D7053: Incremental Person fullname normalization changes.
Jan 31 2022, 10:16 PM
olasd updated the diff for D7051: Introduce a new displayname field for persons in the PostgreSQL storage.

Add tests + rebase on D7053

Jan 31 2022, 10:09 PM
olasd requested review of D7051: Introduce a new displayname field for persons in the PostgreSQL storage.
Jan 31 2022, 7:01 PM
olasd added inline comments to D7049: weekly-planning: Use curl instead of httpie.
Jan 31 2022, 12:41 PM

Jan 29 2022

olasd added a comment to T3609: SWHIDv2: List issues with SWHIDv1 that should be fixed.

I've thought of mentioning here a couple of issues that I've seen come up again over the past few days:

Jan 29 2022, 10:16 PM · Roadmap 2020, Data Model, Web app, Roadmap 2021

Jan 28 2022

olasd added inline comments to D6344: Non-incremental, non-delta Bazaar/Breezy loader.
Jan 28 2022, 4:20 PM
olasd added a comment to D7039: Update the debian local package building section.
In D7039#183023, @olasd wrote:

Thanks.

--build-dep-resolver=aptitude should only be used when building with an extra-repository which has a non-default priority, that is only when using a -backports suite (so only for the bullseye and buster instructions). It should probably be documented in the list of "useful options" rather than as the default.

But does it hurt to use it by default?

Jan 28 2022, 4:14 PM

Jan 27 2022

olasd added a comment to T3887: Storing multiple authors in Revisions and Releases.
In T3887#77951, @zack wrote:

Do you foresee any issue in adding extra_headers to releases as well, other than "someone should do it"?

Jan 27 2022, 5:54 PM · SWORD deposit, Data Model, BZR loader
olasd added a comment to T3887: Storing multiple authors in Revisions and Releases.

Now that I've written it out loud, of course, Releases don't have extra_headers so the package loaders can't make use of this workaround/hack for now.

Jan 27 2022, 5:34 PM · SWORD deposit, Data Model, BZR loader
olasd renamed T3887: Storing multiple authors in Revisions and Releases from Storing multiple authors to Storing multiple authors in Revisions and Releases.
Jan 27 2022, 5:30 PM · SWORD deposit, Data Model, BZR loader
olasd added a comment to T3887: Storing multiple authors in Revisions and Releases.

From merged tasks, this would also be useful for some package loaders, e.g. npm, that support multiple authors in their packaging metadata.

Jan 27 2022, 5:29 PM · SWORD deposit, Data Model, BZR loader
olasd merged task T1645: Associating multiple authors to a revision into T3887: Storing multiple authors in Revisions and Releases.
Jan 27 2022, 5:28 PM · Data Model
olasd merged task T3284: Support for multiple revision authors? into T3887: Storing multiple authors in Revisions and Releases.
Jan 27 2022, 5:28 PM · Data Model
olasd merged tasks T1645: Associating multiple authors to a revision, T3284: Support for multiple revision authors? into T3887: Storing multiple authors in Revisions and Releases.
Jan 27 2022, 5:28 PM · SWORD deposit, Data Model, BZR loader
olasd lowered the priority of T3887: Storing multiple authors in Revisions and Releases from Normal to Wishlist.
Jan 27 2022, 5:27 PM · SWORD deposit, Data Model, BZR loader
olasd added a comment to T3887: Storing multiple authors in Revisions and Releases.

Practically, we could be storing the metadata on additional authors *now* in the extra_headers field (make them a bunch of (b'author', b'XXX <yyy@zzz.ttt>') entries). Of course, that doesn't solve the question of presenting the information.

Jan 27 2022, 5:26 PM · SWORD deposit, Data Model, BZR loader
olasd added a comment to T3888: Storing associated bugs.

Is this information fully intrinsic, or can it be modified without the revision id changing ?

Jan 27 2022, 5:15 PM · BZR loader
olasd added inline comments to D7021: Add graph dataset reading classes (orc+edges).
Jan 27 2022, 3:36 PM

Jan 26 2022

olasd created P1267 Command-Line Input.
Jan 26 2022, 9:46 PM
olasd committed rSPSITEb4bc8a45a68a: Refresh key for deb.sury.org (authored by olasd).
Refresh key for deb.sury.org
Jan 26 2022, 5:36 PM