Page MenuHomeSoftware Heritage
Feed Advanced Search

Oct 5 2021

douardda updated the diff for D6403: docker: use a dedicated container for the cron-like job of swh-web.

indent...

Oct 5 2021, 10:01 AM
douardda updated the diff for D6403: docker: use a dedicated container for the cron-like job of swh-web.

improve entrypoint script to properly handle a SIGTERM

Oct 5 2021, 9:59 AM
douardda accepted D6334: Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`.
Oct 5 2021, 9:34 AM
douardda added a comment to D6334: Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`.

looks ok to me. Just one question, why do you need __future__.annotation?

Oct 5 2021, 9:34 AM

Oct 4 2021

douardda requested review of D6403: docker: use a dedicated container for the cron-like job of swh-web.
Oct 4 2021, 5:33 PM
douardda requested review of D6402: docker: Do not limit the list of task types handled by swh-scheduler-runner-priority.
Oct 4 2021, 5:33 PM
douardda accepted D6387: type_validator: Re-allow subclasses.

Oh well...

Oct 4 2021, 4:04 PM
douardda created P1195 (An Untitled Masterwork).
Oct 4 2021, 3:31 PM
douardda added a comment to T3611: Define the mapping for Bazaar repositories/branches to the SWH data model.

Ideally this doc would (briefly) describe how bazaar works and how it is different from already supported DVCS, then document chosen the "mapping" of the bzr model into swh (especially mentioning what is lost during this).

Oct 4 2021, 11:43 AM · Data Model, BZR loader
douardda added a comment to T3104: Persistent readonly perfect hash table.

@douardda

SWH I guess: I don't see the difference whether it's embedded in swh-objstorage, winery or a dedicated package.

If I understand correctly, you're suggesting that I create a package at the same level as https://forge.softwareheritage.org/source/puppet-swh-site/, right ? For instance https://forge.softwareheritage.org/source/swh-perfecthash/ by following the instructions from the documentation.

So does it make sense to use this package instead of reimplementing one? What's the catch?

In addition to being unmaintained,

Oct 4 2021, 11:39 AM · Object storage (RedHat collaboration)
douardda added a comment to T3611: Define the mapping for Bazaar repositories/branches to the SWH data model.

Would it be possible to add a "conception documentation" included in the docs/ of the BZR loader repo? (possibly with D6344 or as a standalone diff)?

Oct 4 2021, 10:48 AM · Data Model, BZR loader

Oct 1 2021

douardda added a comment to T3104: Persistent readonly perfect hash table.

Wouldn't it make sense to put the cffi-based cmph wrapper in a dedicated python module/project (not necessarily under the swh namespace)?

It would but who would maintain it in the long run ?

Oct 1 2021, 3:19 PM · Object storage (RedHat collaboration)
douardda added a comment to D6339: Add support for remote backend on existing storage tests.

IMHO This diff should be squashed in D6165 (it's really part of the work adding the rabbitmq-based backend).

Oct 1 2021, 2:39 PM
douardda accepted D6272: Remove remote storage based on `swh.core.api.RPCClient`.

as @olasd should be squashed, but meh

Oct 1 2021, 2:32 PM
douardda accepted D6273: Remove remote storage based on `swh.core.api.RPCClient`.
Oct 1 2021, 2:30 PM
douardda requested changes to D6334: Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`.

Look to me that this open/close interface really should come with a context manager.

Oct 1 2021, 2:07 PM
douardda accepted D6358: Make old StatsD metrics style compliant with the rest of the module.

I still think it's best to use the wrapped function name as "method" but meh

Oct 1 2021, 1:59 PM

Sep 30 2021

douardda accepted D6376: SvnLoaderFromRemoteDump: Fix failed visit which should be an uneventful visit.

Looks ok (not sure I really understand the fix however, more precisely, what was the purpose of the revision_start != 1 condition), but I really don't understand the commit message:

Sep 30 2021, 9:40 AM

Sep 29 2021

douardda created P1184 (An Untitled Masterwork).
Sep 29 2021, 4:19 PM
douardda added a comment to T3104: Persistent readonly perfect hash table.

Ideally, since the perfecthash feature will be needed only for a specific objstorage backend, it should be an optional dependency.

Wouldn't it make sense to put the cffi-based cmph wrapper in a dedicated python module/project (not necessarily under the swh namespace)?

Or use this one maybe https://github.com/GregBowyer/cmph-cffi ?

Source for the cmph-cffi package in pypi seems to be https://github.com/venkateshks/cmph-cffi (well at least there are tags in there)

Sep 29 2021, 12:00 PM · Object storage (RedHat collaboration)
douardda added a comment to T3104: Persistent readonly perfect hash table.

Ideally, since the perfecthash feature will be needed only for a specific objstorage backend, it should be an optional dependency.

Sep 29 2021, 11:55 AM · Object storage (RedHat collaboration)
douardda committed rDSTO113088ab065b: replay: add type annotation for process_replay_objects() (authored by douardda).
replay: add type annotation for process_replay_objects()
Sep 29 2021, 11:26 AM
douardda committed rDSTO9a3589f24cae: replay: fix raw_extrinsic_metadata insertion and type annotation (authored by douardda).
replay: fix raw_extrinsic_metadata insertion and type annotation
Sep 29 2021, 11:26 AM
douardda closed D6370: replay: fix raw_extrinsic_metadata insertion and type annotation.
Sep 29 2021, 11:26 AM
douardda committed rDSTO21aff2d1cbab: replay: fix annotation of collision_aware_content_add() (authored by douardda).
replay: fix annotation of collision_aware_content_add()
Sep 29 2021, 11:26 AM
douardda updated the diff for D6370: replay: fix raw_extrinsic_metadata insertion and type annotation.

add type annotation for process_replay_objects()

Sep 29 2021, 11:02 AM
douardda updated the diff for D6370: replay: fix raw_extrinsic_metadata insertion and type annotation.

rebase

Sep 29 2021, 10:49 AM
douardda accepted D6341: model: Replace attrs-strict with stricter validation.

Test coverage looks fairly complete, thx

Sep 29 2021, 10:45 AM

Sep 28 2021

douardda created P1181 (An Untitled Masterwork).
Sep 28 2021, 4:54 PM
douardda created P1179 (An Untitled Masterwork).
Sep 28 2021, 4:46 PM

Sep 27 2021

douardda added inline comments to D6353: Add StatsD support to graph submodule.
Sep 27 2021, 5:00 PM
douardda added inline comments to D6358: Make old StatsD metrics style compliant with the rest of the module.
Sep 27 2021, 4:55 PM
douardda accepted D6351: Add StatsD support to provenance backend.
Sep 27 2021, 4:50 PM
douardda added a comment to T3487: Installation of the new provenance server.
  • postgresql:13
    • 1000 parallel connections allowed
Sep 27 2021, 4:39 PM · System administration
douardda added inline comments to D6340: opam: Define a initialize_opam_root parameter for opam loader.
Sep 27 2021, 3:29 PM
douardda closed D6347: docker: use a dedicated container for memcached.
Sep 27 2021, 3:15 PM
douardda committed rDENV9dd8181b0107: docker: use a dedicated container for memcached (authored by douardda).
docker: use a dedicated container for memcached
Sep 27 2021, 3:15 PM
douardda added inline comments to D6353: Add StatsD support to graph submodule.
Sep 27 2021, 2:57 PM
douardda accepted D6353: Add StatsD support to graph submodule.
Sep 27 2021, 2:56 PM
douardda added a comment to D6352: Add StatsD support to provenance storage implementations.

Might be possible to simplify this a bit using a similar approach to https://forge.softwareheritage.org/source/swh-storage/browse/master/swh/storage/metrics.py$16-26

Sep 27 2021, 2:55 PM
douardda accepted D6352: Add StatsD support to provenance storage implementations.
Sep 27 2021, 2:53 PM
douardda requested changes to D6351: Add StatsD support to provenance backend.

Overall looks ok to me but:

Sep 27 2021, 2:32 PM
douardda added inline comments to D6348: Clarify local/remote heads type as those are hexadecimal bytes str.
Sep 27 2021, 12:11 PM
douardda closed D6346: docker: use dsn connection string in web.yml.
Sep 27 2021, 12:02 PM
douardda committed rDENV27ddb8a16f3b: docker: use dsn connection string in web.yml (authored by douardda).
docker: use dsn connection string in web.yml
Sep 27 2021, 12:02 PM
douardda closed D6345: docker: do not override the DJANGO_SETTINGS_MODULE in swh-web/entrypoint.sh.
Sep 27 2021, 12:02 PM
douardda committed rDENV6c4f3c5f41fc: docker: do not override the DJANGO_SETTINGS_MODULE in swh-web/entrypoint.sh (authored by douardda).
docker: do not override the DJANGO_SETTINGS_MODULE in swh-web/entrypoint.sh
Sep 27 2021, 12:02 PM
douardda added a comment to D6347: docker: use a dedicated container for memcached.

Why?

Sep 27 2021, 12:00 PM
douardda updated the diff for D6347: docker: use a dedicated container for memcached.

Rebase

Sep 27 2021, 11:51 AM
douardda updated the diff for D6346: docker: use dsn connection string in web.yml.

Better commit message

Sep 27 2021, 11:51 AM
douardda updated the diff for D6345: docker: do not override the DJANGO_SETTINGS_MODULE in swh-web/entrypoint.sh.

typos

Sep 27 2021, 11:51 AM
douardda added a comment to D6346: docker: use dsn connection string in web.yml.

I don't see why (I'm guessing for simplification), but ok

Sep 27 2021, 11:45 AM
douardda accepted D6349: Update archive changelog about the opam.ocaml.org instance.
Sep 27 2021, 11:30 AM
douardda requested review of D6347: docker: use a dedicated container for memcached.
Sep 27 2021, 10:13 AM
douardda requested review of D6346: docker: use dsn connection string in web.yml.
Sep 27 2021, 10:12 AM
douardda requested review of D6345: docker: do not override the DJANGO_SETTINGS_MODULE in swh-web/entrypoint.sh.
Sep 27 2021, 10:11 AM
douardda added a comment to D6341: model: Replace attrs-strict with stricter validation.

Looks fine to me, but it needs some extensive tests indeed.

Sep 27 2021, 9:42 AM

Sep 24 2021

douardda closed D6336: Naive attempt to add support for dsn url config style for production db.
Sep 24 2021, 3:33 PM
douardda committed rDWAPPSdd6dde3e44cd: Naive attempt to add support for dsn url config style for production db (authored by douardda).
Naive attempt to add support for dsn url config style for production db
Sep 24 2021, 3:33 PM
douardda closed D6335: Wrap long lines in the README file.
Sep 24 2021, 3:33 PM
douardda committed rDWAPPS15b0e84456ae: Wrap long lines in the README file (authored by douardda).
Wrap long lines in the README file
Sep 24 2021, 3:33 PM
douardda updated the diff for D6336: Naive attempt to add support for dsn url config style for production db.

use types-psycopg2 instead of ignore it in mymy.ini

Sep 24 2021, 2:14 PM
douardda added inline comments to D6336: Naive attempt to add support for dsn url config style for production db.
Sep 24 2021, 2:10 PM
douardda added inline comments to D6336: Naive attempt to add support for dsn url config style for production db.
Sep 24 2021, 2:08 PM
douardda requested review of D6336: Naive attempt to add support for dsn url config style for production db.
Sep 24 2021, 12:18 PM
douardda requested review of D6335: Wrap long lines in the README file.
Sep 24 2021, 12:18 PM
douardda accepted D6330: Deprecate identifiers.py.

fine for me

Sep 24 2021, 9:28 AM
douardda accepted D6332: Move SWHID-related tests to test_swhids.py.

lgtm

Sep 24 2021, 9:26 AM
douardda accepted D6333: Add module-level docstrings..

thx a lot

Sep 24 2021, 9:25 AM

Sep 23 2021

douardda accepted D6322: Add bazaar as supported revision type.

LGTM

Sep 23 2021, 10:17 AM

Sep 22 2021

douardda created P1173 (An Untitled Masterwork).
Sep 22 2021, 6:15 PM
douardda added a comment to D6316: opam: Share opam root directory even on multiple instances.

You may use fcntl.flock for this

I mean using an empty (lock) file in the opam_root directory.

Sep 22 2021, 2:14 PM
douardda added a comment to D6316: opam: Share opam root directory even on multiple instances.

You may use fcntl.flock for this

Sep 22 2021, 2:12 PM
douardda added a comment to D6316: opam: Share opam root directory even on multiple instances.

Following what i said in the loader diff, i'm actually closing this.
Ack on the lock folder but i won't attend to it immediately.

[1] D6318

As i was wrong in my implementation of the loader implementation and @aleo made me realize, i've fixed it.
So now that lister diff becomes relevant again, so claimed it back.

I think there was already a problem before, but since we have now more chance to hit it, I'd really like the opam_init process to lock the directory when running opam commands.

It's a great idea but i've no idea how to actually do that though.

Maybe adding --safe flag [1] during the command that actually list the packages would be enough instead.
I've actually added that for the loader [2] (for the command that also read information)

[1]

--safe, --readonly
    Make sure nothing will be automatically updated or rewritten. Useful for calling from completion scripts, for example. Will fail whenever such an operation is needed ; also avoids waiting for locks, skips interactive
    questions and overrides the $OPAMDEBUG variable. This is equivalent to set environment variable $OPAMSAFE.

[2] D6318

Sep 22 2021, 2:11 PM
douardda accepted D6308: Add a documentation page to list the services urls.

LGTM (not checked everything is accurate nor there are obvious missing services, but it's a huge improvement as is, thx)

Sep 22 2021, 11:42 AM · System administration
douardda added a comment to T1805: Public API v2.

Items 5, 6, 7 aka pagination, auth and batches - I believe these come naturally with item 4 (specification wise)

They don't. OpenAPI is a specification to describe APIs, and it contains absolutely nothing about pagination or batches.

Sep 22 2021, 11:36 AM · meta-task, Web app
douardda added a comment to D6316: opam: Share opam root directory even on multiple instances.

I think there was already a problem before, but since we have now more chance to hit it, I'd really like the opam_init process to lock the directory when running opam commands.

Sep 22 2021, 11:16 AM
douardda added inline comments to D6133: maven-lister: initialise lister..
Sep 22 2021, 11:08 AM
douardda added a comment to D6133: maven-lister: initialise lister..

It would be nice to have a README fil in swh/lister/maven/tests/data explaining what the data files are, where they come from, how they have been generated, etc.

Sep 22 2021, 10:52 AM
douardda added a comment to D6165: Add new RabbitMQ-based client/server API.

What is the reason for this change? Is it more efficient assign requests to workers based on ID rather than randomly?

Sep 22 2021, 10:48 AM
douardda added inline comments to D6308: Add a documentation page to list the services urls.
Sep 22 2021, 10:43 AM · System administration
douardda accepted D6317: opam: Initialize opam root directory outside the constructor.
Sep 22 2021, 10:38 AM
douardda accepted D6300: Capture missing revision <-> hgnode-id scenario in a xfail test.
Sep 22 2021, 10:37 AM

Sep 21 2021

douardda added a comment to D6133: maven-lister: initialise lister..

some more :-)

Sep 21 2021, 11:38 AM
douardda added inline comments to D6133: maven-lister: initialise lister..
Sep 21 2021, 11:14 AM
douardda added inline comments to D6310: opam: Move the state initialization into the get_pages method.
Sep 21 2021, 11:04 AM

Sep 20 2021

douardda accepted D6306: opam: Allow defining where to actually install the opam_root folder.

LGTM, but how is the new opam_root option expected to be set (in production I mean)?

Sep 20 2021, 4:46 PM
douardda requested changes to D6133: maven-lister: initialise lister..

I'm not done yet but here is first review on my side.

Sep 20 2021, 4:33 PM
douardda closed T1510: Have a look at openAPI and decide whether we want to follow these specs, a subtask of T1805: Public API v2, as Resolved.
Sep 20 2021, 11:54 AM · meta-task, Web app
douardda closed T1510: Have a look at openAPI and decide whether we want to follow these specs as Resolved.
Sep 20 2021, 11:54 AM · Web app
douardda closed T2196: Batch APIs as Wontfix.

not useful as a dedicated task, see T1805 for the main discussion one on this subject

Sep 20 2021, 11:54 AM · Roadmap 2020
douardda closed T2196: Batch APIs, a subtask of T2194: Archive Integration (Web API), as Wontfix.
Sep 20 2021, 11:54 AM · Roadmap 2021, meta-task
douardda requested changes to D6300: Capture missing revision <-> hgnode-id scenario in a xfail test.

I don't understand what exactly is (not) tested here. What does "anomad-d" stand for BTW?

Sep 20 2021, 9:59 AM
douardda accepted D6220: Added test only method info in the interface doc strings.
Sep 20 2021, 9:49 AM

Sep 16 2021

douardda added inline comments to D6281: converters: Recompute hashes and check they match the originals.
Sep 16 2021, 5:26 PM
douardda committed rDENV57ad032071ff: docker: document some useful kafka management commands in the README file (authored by douardda).
docker: document some useful kafka management commands in the README file
Sep 16 2021, 4:15 PM
douardda closed D6277: Improve docker/README a bit.
Sep 16 2021, 4:15 PM
douardda committed rDENVe24535cc0064: docker: wrap long cli command lines in the README file (authored by douardda).
docker: wrap long cli command lines in the README file
Sep 16 2021, 4:15 PM
douardda updated the diff for D6277: Improve docker/README a bit.

fix indentation (tab->ws) and a few typos

Sep 16 2021, 4:10 PM
douardda requested review of D6277: Improve docker/README a bit.
Sep 16 2021, 11:00 AM