Page MenuHomeSoftware Heritage
Feed All Stories

Fri, Jul 1

douardda triaged T4370: Refactor the origin visit data model (aka get rid of the OriginVisit model object) as High priority.
Fri, Jul 1, 4:35 PM · Storage manager, Data Model
anlambert triaged T4369: canonicalize gitlab urls in origin API as Normal priority.
Fri, Jul 1, 3:24 PM · Web app
anlambert added a comment to T4369: canonicalize gitlab urls in origin API.

For the record, we recently disabled any origin URL processing in the Web API part (we were checking an origin URL with and without
trailing slash, but this should only be performed in the Web UI as the Web API should be as dumb as possible, see D7988).

Fri, Jul 1, 3:23 PM · Web app
douardda requested review of D8066: Add the 'flavor' argument to db_utils.initialize_database_for_module().
Fri, Jul 1, 3:03 PM
jayeshv closed D8065: Add type annotations for base_connection.
Fri, Jul 1, 3:00 PM
jayeshv committed rDGQLe4d4a30b1f36: Add type annotations for base_connection (authored by jayeshv).
Add type annotations for base_connection
Fri, Jul 1, 3:00 PM
anlambert accepted D8065: Add type annotations for base_connection.

Looks good to me.

Fri, Jul 1, 2:58 PM
swh-public-ci added a comment to D8065: Add type annotations for base_connection.

Build is green

Fri, Jul 1, 2:54 PM
jayeshv updated the diff for D8065: Add type annotations for base_connection.

Address review comments

Fri, Jul 1, 2:52 PM
anlambert added inline comments to D8065: Add type annotations for base_connection.
Fri, Jul 1, 2:13 PM
Harbormaster failed remote builds in B30196: Diff 29104 for D8047: Indexer for Packagist(composer.json)!
Fri, Jul 1, 1:05 PM
swh-public-ci added a comment to D8047: Indexer for Packagist(composer.json).

Build was aborted

Fri, Jul 1, 1:05 PM
VickyMerzOwn updated the diff for D8047: Indexer for Packagist(composer.json).

Excludes author.role from composer.py

Fri, Jul 1, 12:54 PM
swh-public-ci added a comment to D8065: Add type annotations for base_connection.

Build is green

Fri, Jul 1, 12:20 PM
jayeshv updated the diff for D8065: Add type annotations for base_connection.

Add type annotation on _get_paged_result functions

Fri, Jul 1, 12:17 PM
jayeshv requested review of D8065: Add type annotations for base_connection.
Fri, Jul 1, 11:55 AM
jayeshv added a revision to T4261: Swh-graphql Code cleanup: D8065: Add type annotations for base_connection.
Fri, Jul 1, 11:53 AM · GraphQL API
rdicosmo added a comment to T4369: canonicalize gitlab urls in origin API.

Nice catch.... actually, the GitLab API uses the project slug (e.g.: checkscale-gitlab/git-wtf) without the .git, we should make sure we can handle this same slug

Fri, Jul 1, 11:52 AM · Web app
bchauvet created T4369: canonicalize gitlab urls in origin API.
Fri, Jul 1, 11:15 AM · Web app
douardda triaged T4368: Loosen "foreign key" validation in storages used as mirror ingestion endpoint as High priority.
Fri, Jul 1, 11:13 AM · Storage manager
douardda created T4368: Loosen "foreign key" validation in storages used as mirror ingestion endpoint.
Fri, Jul 1, 11:13 AM · Storage manager
jayeshv updated the task description for T4365: GraphQL: Use the right date time serialization.
Fri, Jul 1, 10:54 AM · GraphQL API
jayeshv updated the task description for T4366: GraphQL: add revision log to the revision type.
Fri, Jul 1, 10:54 AM · GraphQL API
douardda added a comment to T2309: Add support for other hash algo than sha1 in current objstorage implementation.

do you have in mind to make the actual hash used as primary key in an objstorage a configuration of said storage instance? e.g. create a pathslicer or s3 objstorage using sha256 is just a matter of configuration of the objstorage?

Fri, Jul 1, 10:38 AM · Object storage
douardda added a comment to T2309: Add support for other hash algo than sha1 in current objstorage implementation.

do you have in mind to make the actual hash used as primary key in an objstorage a configuration of said storage instance? e.g. create a pathslicer or s3 objstorage using sha256 is just a matter of configuration of the objstorage?

Fri, Jul 1, 10:34 AM · Object storage
douardda added inline comments to D8029: Start introducing composite ObjId in the interface.
Fri, Jul 1, 10:22 AM
ardumont committed rCJSWH0613c0c9d3df: README.md: Improve gogs setup instructions (authored by ardumont).
README.md: Improve gogs setup instructions
Fri, Jul 1, 10:02 AM
ardumont committed rCJSWHa8a26a77f28a: Ignore docker-compose.override.yml (authored by ardumont).
Ignore docker-compose.override.yml
Fri, Jul 1, 10:02 AM
swh-public-ci added a comment to D8047: Indexer for Packagist(composer.json).

Build is green

Fri, Jul 1, 9:06 AM
VickyMerzOwn updated the diff for D8047: Indexer for Packagist(composer.json).

Excludes codemeta-v1 column from composer.csv

Fri, Jul 1, 9:01 AM
vlorentz added inline comments to D8047: Indexer for Packagist(composer.json).
Fri, Jul 1, 8:39 AM

Thu, Jun 30

vsellier committed rCJSWHb0e07c673ec0: wip - add a forge to host the local Changes (authored by vsellier).
wip - add a forge to host the local Changes
Thu, Jun 30, 11:25 PM
swh-public-ci added a comment to D8047: Indexer for Packagist(composer.json).

Build is green

Thu, Jun 30, 11:18 PM
VickyMerzOwn updated the diff for D8047: Indexer for Packagist(composer.json).

Corrects Copyright year

Thu, Jun 30, 11:13 PM
swh-public-ci added a comment to D8047: Indexer for Packagist(composer.json).

Build is green

Thu, Jun 30, 10:52 PM
VickyMerzOwn updated the diff for D8047: Indexer for Packagist(composer.json).

Adds license to composer.py
Excludes redundant columns from composer.csv

Thu, Jun 30, 10:46 PM
VickyMerzOwn added inline comments to D8047: Indexer for Packagist(composer.json).
Thu, Jun 30, 10:45 PM
vlorentz added inline comments to D8047: Indexer for Packagist(composer.json).
Thu, Jun 30, 10:31 PM
swh-public-ci added a comment to D8047: Indexer for Packagist(composer.json).

Build is green

Thu, Jun 30, 10:00 PM
VickyMerzOwn updated the diff for D8047: Indexer for Packagist(composer.json).

Undo changes to crosswalk.csv

Thu, Jun 30, 9:54 PM
swh-public-ci added a comment to D8047: Indexer for Packagist(composer.json).

Build is green

Thu, Jun 30, 9:40 PM
vlorentz added inline comments to D8047: Indexer for Packagist(composer.json).
Thu, Jun 30, 9:38 PM
swh-public-ci added a comment to D8047: Indexer for Packagist(composer.json).

Build is green

Thu, Jun 30, 9:38 PM
VickyMerzOwn updated the diff for D8047: Indexer for Packagist(composer.json).

Undo changes to crosswalk.csv

Thu, Jun 30, 9:34 PM
VickyMerzOwn updated the diff for D8047: Indexer for Packagist(composer.json).

Composer mappings use a separate table

Thu, Jun 30, 9:32 PM
seirl committed rDGRPH555e3b6d3e11: rpc_server: use shlex.quote() to print command (authored by seirl).
rpc_server: use shlex.quote() to print command
Thu, Jun 30, 8:49 PM
seirl committed rDGRPHa55edc1f88d5: Remove shlex.join() call (3.8 only) (authored by seirl).
Remove shlex.join() call (3.8 only)
Thu, Jun 30, 8:33 PM
Harbormaster failed to build B30184: rDGRPH441aacc6aeeb: Update sux4j to 5.4.0 for rDGRPH441aacc6aeeb: Update sux4j to 5.4.0!
Thu, Jun 30, 8:15 PM
Harbormaster failed to build B30185: rDGRPH94418d4944a8: GRPC: use port 50091 by default for rDGRPH94418d4944a8: GRPC: use port 50091 by default!
Thu, Jun 30, 8:15 PM
seirl committed rDGRPH94418d4944a8: GRPC: use port 50091 by default (authored by seirl).
GRPC: use port 50091 by default
Thu, Jun 30, 8:07 PM
seirl committed rDGRPH441aacc6aeeb: Update sux4j to 5.4.0 (authored by seirl).
Update sux4j to 5.4.0
Thu, Jun 30, 8:07 PM
VickyMerzOwn added inline comments to D8047: Indexer for Packagist(composer.json).
Thu, Jun 30, 7:57 PM
vlorentz requested changes to D8047: Indexer for Packagist(composer.json).
Thu, Jun 30, 7:43 PM
swh-public-ci added a comment to D8047: Indexer for Packagist(composer.json).

Build is green

Thu, Jun 30, 7:15 PM
seirl requested review of D8064: swh-graph: rename services (now production-ready, no longer dev).
Thu, Jun 30, 7:15 PM
VickyMerzOwn updated the diff for D8047: Indexer for Packagist(composer.json).

Adds composer mappings to metadata dictionary

Thu, Jun 30, 7:09 PM
Harbormaster failed to build B30181: rDGRPH26e52b389fb1: rpc_server.py: terminate, then kill server on timeout for rDGRPH26e52b389fb1: rpc_server.py: terminate, then kill server on timeout!
Thu, Jun 30, 7:06 PM
seirl committed rDGRPH26e52b389fb1: rpc_server.py: terminate, then kill server on timeout (authored by seirl).
rpc_server.py: terminate, then kill server on timeout
Thu, Jun 30, 6:59 PM
seirl added a comment to T3259: Gracefully handle a client closing the connection in the middle of a response being streamed.

GRPC update: cancelling a GRPC stream works fine, but it doesn't seem like aiohttp is doing that when the HTTP stream is closed.

Thu, Jun 30, 6:41 PM · Compressed graph service
ardumont committed rCJSWHb0872b4ec80b: wip: Start swh-apps/update-dependencies job (authored by ardumont).
wip: Start swh-apps/update-dependencies job
Thu, Jun 30, 5:56 PM
ardumont committed rCJSWH6ce55e57218b: wip: Start swh-apps/update-dependencies job (authored by ardumont).
wip: Start swh-apps/update-dependencies job
Thu, Jun 30, 5:46 PM
ardumont committed rCJSWHdfac84b3435f: wip: Start swh-apps/update-dependencies job (authored by ardumont).
wip: Start swh-apps/update-dependencies job
Thu, Jun 30, 5:43 PM
ardumont committed rCJSWHe41c157273e8: wip: Start swh-apps/update-dependencies job (authored by ardumont).
wip: Start swh-apps/update-dependencies job
Thu, Jun 30, 5:41 PM
ardumont committed rCJSWHd25fc9bb8b51: wip: Start swh-apps/update-dependencies job (authored by ardumont).
wip: Start swh-apps/update-dependencies job
Thu, Jun 30, 5:37 PM
seirl closed T4316: Push of swh-graph to pypi is broken as Resolved.

We requested a larger quota here: https://github.com/pypa/pypi-support/issues/1998

Thu, Jun 30, 5:37 PM · System administration, Compressed graph service
seirl committed rDGRPHf2f9e8ba69ef: DumpProperties: fix output for tag names (authored by seirl).
DumpProperties: fix output for tag names
Thu, Jun 30, 5:18 PM
vsellier committed rCJSWH3d562f112c91: wip - poc the swh-apps pipeline (authored by vsellier).
wip - poc the swh-apps pipeline
Thu, Jun 30, 5:13 PM
vsellier committed rCJSWH278f12b744b1: wip - poc the swh-apps pipeline (authored by vsellier).
wip - poc the swh-apps pipeline
Thu, Jun 30, 5:10 PM
ardumont closed D8063: README.md: Adapt docker instructions.
Thu, Jun 30, 5:06 PM
ardumont committed rCJSWHdf549fba24d0: README.md: Adapt docker instructions (authored by ardumont).
README.md: Adapt docker instructions
Thu, Jun 30, 5:06 PM
ardumont requested review of D8063: README.md: Adapt docker instructions.
Thu, Jun 30, 5:06 PM
seirl closed D8061: Re-add swh/graph/client.py for backward-compatibility.
Thu, Jun 30, 4:48 PM
seirl committed rDGRPH3b9685923021: Re-add swh/graph/client.py for backward-compatibility (authored by seirl).
Re-add swh/graph/client.py for backward-compatibility
Thu, Jun 30, 4:48 PM
vsellier closed D8062: fix a typo on the production objstorage vhost.
Thu, Jun 30, 4:48 PM
vsellier committed rSPSITE57fb33253ca7: fix a typo on the production objstorage vhost (authored by vsellier).
fix a typo on the production objstorage vhost
Thu, Jun 30, 4:47 PM
anlambert accepted D8061: Re-add swh/graph/client.py for backward-compatibility.

Great, thanks !

Thu, Jun 30, 4:47 PM
seirl requested review of D8061: Re-add swh/graph/client.py for backward-compatibility.
Thu, Jun 30, 4:44 PM
ardumont accepted D8062: fix a typo on the production objstorage vhost.

fix typo for fun and profit! ;)

Thu, Jun 30, 4:42 PM
olasd accepted D8062: fix a typo on the production objstorage vhost.

lol

Thu, Jun 30, 4:41 PM
vsellier requested review of D8062: fix a typo on the production objstorage vhost.
Thu, Jun 30, 4:40 PM
jayeshv renamed T4367: GraphQL: Fix issue in pagination 'after' argument from GraphQL: Fix issue in pagination to GraphQL: Fix issue in pagination 'after' argument.
Thu, Jun 30, 4:33 PM · GraphQL API
jayeshv triaged T4367: GraphQL: Fix issue in pagination 'after' argument as Low priority.
Thu, Jun 30, 4:32 PM · GraphQL API
seirl closed T2100: Bootstrap Debian packaging for swh.graph, a subtask of T3168: Proper deployment of swh-graph with debian package, as Wontfix.
Thu, Jun 30, 4:21 PM · Compressed graph service, Puppet recipes
seirl closed T2100: Bootstrap Debian packaging for swh.graph as Wontfix.

We are migrating away from Debian packages as a deployment tool, closing this as WONTFIX.

Thu, Jun 30, 4:21 PM · Compressed graph service
seirl closed T3168: Proper deployment of swh-graph with debian package as Wontfix.

We are migrating away from Debian packages as a deployment tool, closing this as WONTFIX.

Thu, Jun 30, 4:20 PM · Compressed graph service, Puppet recipes
seirl closed T2081: swh-graph: "Cannot open client FIFO" when answering HEAD requests as Resolved.

No longer happens after the GRPC migration:

Thu, Jun 30, 4:19 PM · Compressed graph service
jayeshv triaged T4366: GraphQL: add revision log to the revision type as Normal priority.
Thu, Jun 30, 4:17 PM · GraphQL API
seirl closed T2103: (Debian) package py4j, a subtask of T2100: Bootstrap Debian packaging for swh.graph, as Wontfix.
Thu, Jun 30, 4:16 PM · Compressed graph service
seirl closed T2103: (Debian) package py4j as Wontfix.

We removed the Py4J dependency by migrating to GRPC.

Thu, Jun 30, 4:16 PM · Compressed graph service
seirl closed T3301: graph: add test for the "algo" parameter of walk() as Wontfix.

We no longer support multiple algorithms for shortest path requests.

Thu, Jun 30, 4:16 PM · Easy hack, Compressed graph service
jayeshv triaged T4365: GraphQL: Use the right date time serialization as Normal priority.
Thu, Jun 30, 4:15 PM · GraphQL API
seirl closed T3623: Run swh-graph with gunicorn to support multiple/parallel requests as Resolved.

Obsoleted by the migration to GRPC. Now we use GRPC's threading model, with a threadpool configurable by passing --threads to the Java service. By default, nproc is used.

Thu, Jun 30, 4:12 PM · Compressed graph service, System administration
seirl closed D8059: Generate .labelobl files to speed-up labelled graph loading.
Thu, Jun 30, 4:11 PM
seirl committed rDGRPH32f2626a7d0a: Generate .labelobl files to speed-up labelled graph loading (authored by seirl).
Generate .labelobl files to speed-up labelled graph loading
Thu, Jun 30, 4:11 PM
seirl closed T4113: Review border case of empty response for `visit_nodes` as Resolved.

Obsoleted by the migration to GRPC. We no longer create iterators on the decoded stream of a UNIX pipeline, we directly use GRPC stream iterators.

Thu, Jun 30, 4:11 PM · Compressed graph service
swh-public-ci added a comment to D8059: Generate .labelobl files to speed-up labelled graph loading.

Build is green

Thu, Jun 30, 4:10 PM
vsellier closed D8057: Add a docker environment to test the job-builder inside jenkins.
Thu, Jun 30, 4:09 PM
swh-public-ci added a comment to D8059: Generate .labelobl files to speed-up labelled graph loading.

Build has FAILED

Thu, Jun 30, 4:09 PM
vsellier committed rCJSWH21e47db56cb5: Add a docker environment to test the job-builder inside jenkins (authored by vsellier).
Add a docker environment to test the job-builder inside jenkins
Thu, Jun 30, 4:09 PM
Harbormaster failed remote builds in B30170: Diff 29081 for D8059: Generate .labelobl files to speed-up labelled graph loading!
Thu, Jun 30, 4:09 PM