Page MenuHomeSoftware Heritage
Feed Advanced Search

Sep 24 2020

douardda updated the diff for D4029: Attempt to fix dependency-related hell in tox.

update ci message

Sep 24 2020, 11:26 AM
douardda added a comment to D4029: Attempt to fix dependency-related hell in tox.

commit message! ;)

Sep 24 2020, 11:18 AM
douardda created D4029: Attempt to fix dependency-related hell in tox.
Sep 24 2020, 11:13 AM

Sep 23 2020

douardda committed rDSEA7b9a254fc016: Adapt cli declaration entrypoint to swh.core 0.3 (authored by douardda).
Adapt cli declaration entrypoint to swh.core 0.3
Sep 23 2020, 6:40 PM
douardda committed rDTSCNf10d3d87cbcb: Adapt cli declaration entrypoint to swh.core 0.3 (authored by douardda).
Adapt cli declaration entrypoint to swh.core 0.3
Sep 23 2020, 6:27 PM
douardda committed rDTPL19683b52773f: Adapt cli declaration entrypoint to swh.core 0.3 (authored by douardda).
Adapt cli declaration entrypoint to swh.core 0.3
Sep 23 2020, 6:21 PM
douardda committed rDOBJSfd1cc6dc71c8: Adapt cli declaration entrypoint to swh.core 0.3 (authored by douardda).
Adapt cli declaration entrypoint to swh.core 0.3
Sep 23 2020, 6:18 PM
douardda updated the diff for D4022: Rename mirror-related things from replica to mirror.

fix a couple of sedisms

Sep 23 2020, 6:12 PM
douardda created D4022: Rename mirror-related things from replica to mirror.
Sep 23 2020, 6:07 PM
douardda committed rDLDBASE7ed1927b9968: Adapt cli declaration entrypoint to swh.core 0.3 (authored by douardda).
Adapt cli declaration entrypoint to swh.core 0.3
Sep 23 2020, 5:50 PM
douardda committed rDLSd10f78d80c95: Adapt cli declaration entrypoint to swh.core 0.3 (authored by douardda).
Adapt cli declaration entrypoint to swh.core 0.3
Sep 23 2020, 5:42 PM
douardda committed rDCIDX8c4b33ba2c74: Adapt cli declaration entrypoint to swh.core 0.3 (authored by douardda).
Adapt cli declaration entrypoint to swh.core 0.3
Sep 23 2020, 5:26 PM
douardda committed rDICP750a15e4f9ae: Adapt cli declaration entrypoint to swh.core 0.3 (authored by douardda).
Adapt cli declaration entrypoint to swh.core 0.3
Sep 23 2020, 5:13 PM
douardda committed rDGRPHc293d0726a7b: Adapt cli declaration entrypoint to swh.core 0.3 (authored by douardda).
Adapt cli declaration entrypoint to swh.core 0.3
Sep 23 2020, 5:04 PM
douardda committed rDDEP3416abd01019: Adapt cli declaration entrypoint to swh.core 0.3 (authored by douardda).
Adapt cli declaration entrypoint to swh.core 0.3
Sep 23 2020, 4:37 PM
douardda committed rDDATASET6205fdee730b: Adapt cli declaration entrypoint to swh.core 0.3 (authored by douardda).
Adapt cli declaration entrypoint to swh.core 0.3
Sep 23 2020, 4:36 PM
douardda committed rDSTOc97b23bcd78b: Adapt cli declaration entrypoint to swh.core 0.3 (authored by douardda).
Adapt cli declaration entrypoint to swh.core 0.3
Sep 23 2020, 4:35 PM
douardda closed D4002: Adapt cli declaration entrypoint to swh.core 0.3.
Sep 23 2020, 4:35 PM
douardda updated the diff for D4002: Adapt cli declaration entrypoint to swh.core 0.3.

rebase

Sep 23 2020, 4:29 PM
douardda committed rCDFJ4e78deb8b89c: Add libfuse3-dev in the docker (authored by douardda).
Add libfuse3-dev in the docker
Sep 23 2020, 3:45 PM
douardda closed D4021: Add libfuse3-dev in the docker.
Sep 23 2020, 3:45 PM
douardda created D4021: Add libfuse3-dev in the docker.
Sep 23 2020, 3:44 PM
douardda committed rCJSWHf46ad02ee918: Add swh-fuse (authored by douardda).
Add swh-fuse
Sep 23 2020, 2:59 PM
douardda closed D4019: Add swh-fuse.
Sep 23 2020, 2:59 PM
douardda created D4019: Add swh-fuse.
Sep 23 2020, 2:52 PM
douardda added a comment to D3997: Add support for database flavors to `swh db-init`.

LGTM, just a couple of wishes (and we are not even close to xmas!)

Sep 23 2020, 2:20 PM
douardda accepted D4014: pytest_plugin: Change dbname to storage to avoid clash in tests.
Sep 23 2020, 1:11 PM · Storage manager
douardda accepted D4015: pytest_plugin: Order the fixture definitions in dependency order.
Sep 23 2020, 1:10 PM
douardda accepted D4016: pytest_plugin: Reuse swh_storage_postgresql connection string.
Sep 23 2020, 1:10 PM
douardda added a comment to D4014: pytest_plugin: Change dbname to storage to avoid clash in tests.

it's unclear to me if the hunk in swh_storage_backend_config is related with the fix or not.
isn't the very last diff hunk enough for the job?

Sep 23 2020, 11:56 AM · Storage manager

Sep 22 2020

douardda abandoned D3902: cli: do not add sub sub commands to the main swh cli group.
Sep 22 2020, 2:41 PM
douardda committed rDCORE08788aa3783c: Do not automagically add click cmd from swh.cli.subcommands (authored by douardda).
Do not automagically add click cmd from swh.cli.subcommands
Sep 22 2020, 2:40 PM
douardda closed D3930: Do not automagically add click cmd from swh.cli.subcommands.
Sep 22 2020, 2:39 PM
douardda created D4002: Adapt cli declaration entrypoint to swh.core 0.3.
Sep 22 2020, 12:06 PM
douardda committed rDSTO30cdb78f17eb: Drop the -swh- part of sql files (authored by douardda).
Drop the -swh- part of sql files
Sep 22 2020, 10:32 AM
douardda committed rDSTO915575d5b50a: Rename 10-swh-init.sql as 10-superuser-init.sql (authored by douardda).
Rename 10-swh-init.sql as 10-superuser-init.sql
Sep 22 2020, 10:32 AM
douardda closed D4001: Rename 10-swh-init.sql as 10-superuser-init.sql.
Sep 22 2020, 10:32 AM
douardda created D4001: Rename 10-swh-init.sql as 10-superuser-init.sql.
Sep 22 2020, 10:21 AM
douardda updated the diff for D3930: Do not automagically add click cmd from swh.cli.subcommands.

rebase

Sep 22 2020, 9:40 AM
douardda committed rDCORE56d505f34a96: Split db-init cli in 2 (init and create) and move them under 'swh db' (authored by douardda).
Split db-init cli in 2 (init and create) and move them under 'swh db'
Sep 22 2020, 9:37 AM
douardda closed D4000: Split db-init cli in 2 (init and create) and move them under 'swh db'.
Sep 22 2020, 9:37 AM
douardda updated the diff for D4000: Split db-init cli in 2 (init and create) and move them under 'swh db'.

fix docstring (thx ardumont)

Sep 22 2020, 9:36 AM

Sep 21 2020

douardda added inline comments to D4000: Split db-init cli in 2 (init and create) and move them under 'swh db'.
Sep 21 2020, 5:21 PM
douardda updated the diff for D4000: Split db-init cli in 2 (init and create) and move them under 'swh db'.

several fixes and add a --template option to 'swh db create'

Sep 21 2020, 5:19 PM
douardda updated the diff for D4000: Split db-init cli in 2 (init and create) and move them under 'swh db'.

better ci mesg

Sep 21 2020, 4:42 PM
douardda created D4000: Split db-init cli in 2 (init and create) and move them under 'swh db'.
Sep 21 2020, 4:36 PM
douardda accepted D3997: Add support for database flavors to `swh db-init`.
Sep 21 2020, 2:25 PM
douardda added a comment to D3997: Add support for database flavors to `swh db-init`.

I don't understand how this is supposed to work.

How does the flavor "name" is taken in consideration? Is it supposed to select "flavored" versions of sql files? What's the pattern ? As written, I don't understand how the flavor "name" is used besides being written the dbflavor table.

Oh now I get it (after reading D3981)

It's the presence of a xxx-flavor.sql that triggers the insertion/update of the flavor given as cli argument in the dbflavor table, which then specify how the other sql files are executed (thx to /if statements in there).

Sep 21 2020, 11:57 AM
douardda added a comment to D3997: Add support for database flavors to `swh db-init`.

I don't understand how this is supposed to work.

How does the flavor "name" is taken in consideration? Is it supposed to select "flavored" versions of sql files? What's the pattern ? As written, I don't understand how the flavor "name" is used besides being written the dbflavor table.

Sep 21 2020, 11:56 AM
douardda committed rDSTO67ee86b8e03f: Warn about skipped_content sneaking the 'content' topics (authored by douardda).
Warn about skipped_content sneaking the 'content' topics
Sep 21 2020, 10:44 AM
douardda committed rDSTO8de656451239: Small fix in the graph replayer to prevent a wrong warning (authored by douardda).
Small fix in the graph replayer to prevent a wrong warning
Sep 21 2020, 10:44 AM
douardda closed D3993: Small fix in the graph replayer to prevent a wrong warning.
Sep 21 2020, 10:44 AM
douardda updated the diff for D3993: Small fix in the graph replayer to prevent a wrong warning.

typo (thx anlambert)

Sep 21 2020, 10:44 AM
douardda added a comment to D3981: Support different database flavors in the SQL scripts.

what about the deduplication flavor? it's not described in the ci message. Also this description of flavors should be documented somewhere (I mean somewhere else than the ci message)

Sep 21 2020, 10:39 AM
douardda added a comment to D3997: Add support for database flavors to `swh db-init`.

I don't understand how this is supposed to work.

Sep 21 2020, 10:25 AM
douardda added a comment to D3998: migrate_extrinsic_metadata: add support for guessing the origin of more PyPI packages from filenames..

I'm not sure I understand what's a stake here, but it really looks odd.

Sep 21 2020, 10:17 AM
douardda added a comment to D3999: blueprint: detail when deposit updates are allowed..

just my 2c. do we want to allow deposit update in the ready state or do we want to be able to get back from ready to partial?

Sep 21 2020, 10:11 AM

Sep 18 2020

douardda committed rCDFPe101f10fcb50: Major cleanup of the config files and better documentation (authored by douardda).
Major cleanup of the config files and better documentation
Sep 18 2020, 4:36 PM
douardda committed rCDFPb1e109b5aa2e: Refactor base/entrypoint.sh to add support for a fix-mirror command (authored by douardda).
Refactor base/entrypoint.sh to add support for a fix-mirror command
Sep 18 2020, 4:36 PM
douardda committed rCDFP9d3a0dffed4c: Do not publish built images to the docker hub by default (authored by douardda).
Do not publish built images to the docker hub by default
Sep 18 2020, 4:36 PM
douardda created D3993: Small fix in the graph replayer to prevent a wrong warning.
Sep 18 2020, 4:33 PM

Sep 17 2020

douardda committed rDSTO469c38c01943: Make origin_add() handle multiple occurences of an origin properly (authored by douardda).
Make origin_add() handle multiple occurences of an origin properly
Sep 17 2020, 4:57 PM
douardda closed D3975: Make origin_add() handle multiple occurences of an origin properly.
Sep 17 2020, 4:57 PM
douardda updated the diff for D3975: Make origin_add() handle multiple occurences of an origin properly.

typos (thx ardumont) + better ci message (thx vlorentz)

Sep 17 2020, 4:49 PM
douardda added a comment to D3975: Make origin_add() handle multiple occurences of an origin properly.

Why do we need this?

because of

Sep 17 2020, 4:46 PM
douardda updated the diff for D3975: Make origin_add() handle multiple occurences of an origin properly.

slightly better comments

Sep 17 2020, 3:01 PM
douardda added a comment to D3975: Make origin_add() handle multiple occurences of an origin properly.
In D3975#98089, @olasd wrote:

It took me a bit to understand what the sorteds were achieving (dedup the list while keeping it in the same order), so I wouldn't mind if you turned them into an explicit loop, or at least if you added a comment.

Sep 17 2020, 2:51 PM
douardda updated the diff for D3930: Do not automagically add click cmd from swh.cli.subcommands.

fix db/test_cli

Sep 17 2020, 2:28 PM
douardda updated the diff for D3930: Do not automagically add click cmd from swh.cli.subcommands.

rebase

Sep 17 2020, 2:09 PM
douardda created D3975: Make origin_add() handle multiple occurences of an origin properly.
Sep 17 2020, 2:04 PM
douardda updated the diff for D3930: Do not automagically add click cmd from swh.cli.subcommands.

rebase

Sep 17 2020, 12:31 PM
douardda added inline comments to D3930: Do not automagically add click cmd from swh.cli.subcommands.
Sep 17 2020, 12:05 PM
douardda added a comment to T2373: git loader OOM when loading huge repository.

FTR, in a test setup I made a few days ago on docker, I had a git loader crunching ~28GB of RES mem (on 32 available on that machine). Not sure which repo it was ingesting, but it was on codeberg.

Sep 17 2020, 9:10 AM · Git loader

Sep 16 2020

douardda added a comment to T2589: expose swh-graph API at archive.s.o/api/1/graph/.

FTR, in a previous life, I've set up a json web token auth validation in varnish.

Sep 16 2020, 2:58 PM · System administration, Web app, Compressed graph service
douardda created P770 tb in the mirror.
Sep 16 2020, 1:12 PM

Sep 15 2020

douardda created P765 (An Untitled Masterwork).
Sep 15 2020, 12:12 PM
douardda triaged T2592: Latest version of kafka-manager in docker venv is a bit broken as Normal priority.
Sep 15 2020, 10:40 AM · Development environment

Sep 14 2020

douardda added a comment to D3900: Add standard config support and HTTP auth token for swh-scanner.

Almost good but there is still something that puzzles me regarding global configuration handling. Let me explain.

What we want to achieve here is to share Web API URL and authentication token across several swh components (swh-scanner and swh-web-client at the moment)
without duplicating it in multiple configuration files. So basically we want to put that configuration in the global.yml file.

Sep 14 2020, 1:40 PM
douardda accepted D3900: Add standard config support and HTTP auth token for swh-scanner.

nitpicking: the config management part does need to be in the same git revision as the auth support.

Sep 14 2020, 1:26 PM
douardda accepted D3906: Add edge labelling prototype.

LGTM, as far as I can tell (but I cannot tell that much TBH)

Sep 14 2020, 1:14 PM
douardda requested changes to D3931: refactor non-cli code out of cli in swh-scanner.

Just as small thing: please use imperative mood in the commit message. The subject is ok, but the "moved [...]" is not.
Also it would make more sense to explicitly refers to cli.py instead of just cli in the subject, i.e.:

Sep 14 2020, 12:32 PM
douardda closed T2575: Investigate if/how we could improve `swh` cli command startup time as Resolved.

Let's consider this done for now. Still not "great" loading time, but much better than before.

Sep 14 2020, 10:03 AM · Development environment

Sep 11 2020

douardda updated the diff for D3930: Do not automagically add click cmd from swh.cli.subcommands.

fix db/test_cli

Sep 11 2020, 5:21 PM
douardda created D3930: Do not automagically add click cmd from swh.cli.subcommands.
Sep 11 2020, 4:33 PM
douardda committed rDOBJSRPL2c21474347f9: Rename cli imported from objstorage as objstorage_cli_group (authored by douardda).
Rename cli imported from objstorage as objstorage_cli_group
Sep 11 2020, 4:17 PM
douardda closed D3918: Rename cli imported from objstorage as objstorage_cli_group.
Sep 11 2020, 4:17 PM
douardda updated the diff for D3918: Rename cli imported from objstorage as objstorage_cli_group.

forgot setup.py

Sep 11 2020, 3:52 PM
douardda retitled D3918: Rename cli imported from objstorage as objstorage_cli_group from Rename cli imported from objstorage as cli_objstorage_group to Rename cli imported from objstorage as objstorage_cli_group.
Sep 11 2020, 3:34 PM
douardda retitled D3918: Rename cli imported from objstorage as objstorage_cli_group from Rename cli imported from objstorage as objstorage_group to Rename cli imported from objstorage as cli_objstorage_group.
Sep 11 2020, 3:33 PM
douardda updated the diff for D3918: Rename cli imported from objstorage as objstorage_cli_group.

also fix test_cli

Sep 11 2020, 3:30 PM
douardda updated the diff for D3918: Rename cli imported from objstorage as objstorage_cli_group.

rename as objstorage_cli_group

Sep 11 2020, 3:22 PM
douardda added a comment to D3918: Rename cli imported from objstorage as objstorage_cli_group.

What about objstorage_cli instead?

Sep 11 2020, 3:19 PM
douardda committed rDCIDX2ef7177bc2e2: cli: speedup the `swh` cli command startup time (authored by douardda).
cli: speedup the `swh` cli command startup time
Sep 11 2020, 3:11 PM
douardda closed D3913: cli: speedup the `swh` cli command startup time.
Sep 11 2020, 3:11 PM
douardda updated the diff for D3913: cli: speedup the `swh` cli command startup time.

fix test_cli

Sep 11 2020, 3:08 PM
douardda committed rDSTOed55e9c775d6: Load the "fast" hypothesis profile by default (authored by douardda).
Load the "fast" hypothesis profile by default
Sep 11 2020, 2:54 PM
douardda closed D3925: Load the "fast" hypothesis profile by default.
Sep 11 2020, 2:54 PM
douardda updated the task description for T2585: Properly handle deleted/no longer available origins.
Sep 11 2020, 2:38 PM · Lister, Core Loader
douardda triaged T2585: Properly handle deleted/no longer available origins as Normal priority.
Sep 11 2020, 2:38 PM · Lister, Core Loader
douardda closed T1342: Handle annotated tag with no tagger, a subtask of T1280: git origins: latest failure reports, as Resolved.
Sep 11 2020, 2:30 PM · Git loader