Page MenuHomeSoftware Heritage
Feed Advanced Search

May 31 2022

ardumont requested review of D7928: Deploy new origin intrinsic metadata journal client indexer.
May 31 2022, 4:11 PM
ardumont added a revision to T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1: D7928: Deploy new origin intrinsic metadata journal client indexer.
May 31 2022, 4:11 PM · System administration, Indexer, Metadata workflow
ardumont accepted D7926: tests: use stock pytest_postgresql factory function.
May 31 2022, 4:08 PM
ardumont updated the task description for T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1.
May 31 2022, 3:53 PM · System administration, Indexer, Metadata workflow
ardumont updated the task description for T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1.
May 31 2022, 3:42 PM · System administration, Indexer, Metadata workflow
ardumont accepted D7925: Recreate the staging-worker cluster.
May 31 2022, 3:40 PM
ardumont updated the task description for T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1.
May 31 2022, 3:33 PM · System administration, Indexer, Metadata workflow
ardumont renamed T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1 from staging: Deploy new origin intrinsic metadata journal client indexer to staging: Deploy new origin intrinsic metadata journal client indexer v1.1.
May 31 2022, 3:27 PM · System administration, Indexer, Metadata workflow
ardumont accepted D7923: Rewrite origin_head.py as a normal function instead of an indexer.

one suggestion inline.

May 31 2022, 3:04 PM
ardumont closed D7885: docker: Install required ssh/config so migration can push to gitlab.
May 31 2022, 2:54 PM
ardumont closed D7866: gitlab: Allow gitlab rail command to run without ssh.
May 31 2022, 2:54 PM
ardumont committed R263:45dc0f03d962: docker: Install required ssh/config so migration can push to gitlab (authored by ardumont).
docker: Install required ssh/config so migration can push to gitlab
May 31 2022, 2:54 PM
ardumont committed R263:cdecc3dc007c: gitlab: Allow gitlab rails command to run without ssh (authored by ardumont).
gitlab: Allow gitlab rails command to run without ssh
May 31 2022, 2:54 PM
ardumont closed D7865: Allow remote mysql connection.
May 31 2022, 2:54 PM
ardumont committed R263:64e042052da0: Allow remote mysql connection (authored by ardumont).
Allow remote mysql connection
May 31 2022, 2:53 PM
ardumont closed D7858: Allow main script executions within container.
May 31 2022, 2:53 PM
ardumont committed R263:d269fbc44a13: Allow main script executions within container (authored by ardumont).
Allow main script executions within container
May 31 2022, 2:53 PM
ardumont closed D7857: Install Dockerfile to allow containerized script execution.
May 31 2022, 2:53 PM
ardumont committed R263:ef75d7099967: Install Dockerfile to allow containerized script execution (authored by ardumont).
Install Dockerfile to allow containerized script execution
May 31 2022, 2:53 PM
ardumont added a comment to T4264: Add photos to https://mybox.inria.fr/.

(Use imperative form in tasks, it mostly matches how we name task in the forge ;)

May 31 2022, 2:39 PM · Unknown Object (Project)
ardumont renamed T4264: Add photos to https://mybox.inria.fr/ from To add photos to https://mybox.inria.fr/ to Add photos to https://mybox.inria.fr/.
May 31 2022, 2:38 PM · Unknown Object (Project)
ardumont accepted D7920: tests: use stock pytest_postgresql factory function.

one remark, I'd be neat you add the related task id on those diffs (if there is one).

May 31 2022, 2:37 PM
ardumont accepted D7921: Convert test_origin_head from unittest to pytest.
May 31 2022, 2:36 PM
ardumont accepted D7919: tests: use stock pytest_postgresql postgresql factory function.
May 31 2022, 2:24 PM
ardumont closed T4228: scrubber: Investigate the apparent lock (staging) as Resolved.
May 31 2022, 2:22 PM · Archive integrity, System administration
ardumont added a comment to T4228: scrubber: Investigate the apparent lock (staging).

No more idle in transaction queries and corrupt_object still growing [1].
We can close this.

May 31 2022, 2:22 PM · Archive integrity, System administration
ardumont closed T4284: scrubber does not comply to what's expected by the swh db tooling as Resolved.
May 31 2022, 2:18 PM · Archive integrity
ardumont added a comment to T4284: scrubber does not comply to what's expected by the swh db tooling.

Scrubber got finally adapted so migration script is now ok [1]
Closing this.

May 31 2022, 2:18 PM · Archive integrity
ardumont added a project to T4284: scrubber does not comply to what's expected by the swh db tooling: Archive integrity.
May 31 2022, 2:17 PM · Archive integrity
ardumont accepted D7908: Refactor functional tests to use existing dataset.

A couple of remarks/suggestions inline.

May 31 2022, 2:15 PM
ardumont accepted D7918: pytest_plugin: use the stock pytest_postgresql postgresql factory.
May 31 2022, 2:12 PM
ardumont added a comment to D7918: pytest_plugin: use the stock pytest_postgresql postgresql factory.

instead of swh-core's postgresql_fact one, since we actually do not
use its custom features any more in swh-storage.

Well, initially the feature used was the reset between tables to make
it faster than it was (and it worked 'cause it was hellishly slow to drop
and create tables again).

So, now storage uses the template to make the tests go as fast, is that it?

yes, when using the load argument of the postgresql_proc factory, it is used to initialize the template database used to create the db for each test.
For this to work, the dbname used when calling the postgresql_proc factory function must be the same as the one used to call the postgresql factory (so the removal of dbname="storage" in this diff; just use default db names unless specifically needed).

May 31 2022, 2:12 PM
ardumont accepted D7915: cli: Add support for running "all" indexers in the journal client.
May 31 2022, 2:07 PM
ardumont added a comment to D7918: pytest_plugin: use the stock pytest_postgresql postgresql factory.

instead of swh-core's postgresql_fact one, since we actually do not
use its custom features any more in swh-storage.

May 31 2022, 2:05 PM
ardumont accepted D7917: Add missing __init__.py in proxies/.
May 31 2022, 2:01 PM
ardumont added a comment to T4228: scrubber: Investigate the apparent lock (staging).

We now have corrupt object being stored in the db:

11:26:40 swh-scrubber@db1:5432=> select count(*) from corrupt_object ;
+-------+
| count |
+-------+
| 59238 |
+-------+
(1 row)
May 31 2022, 11:29 AM · Archive integrity, System administration
ardumont planned changes to D7913: db: Grant read access to guest user on all tables of the schema.
May 31 2022, 11:28 AM
ardumont added a comment to D7913: db: Grant read access to guest user on all tables of the schema.

With @douardda, we agreed on irc to postpone this diff a bit the time david found back how to properly update the postgres template (to integrate the guest user).
So the part spawning back the pytest-plugin postgresql_fact becomes unnecessary [1]

May 31 2022, 11:28 AM
ardumont added a comment to T4228: scrubber: Investigate the apparent lock (staging).

And v0.0.6 tagged and deployed.

May 31 2022, 11:26 AM · Archive integrity, System administration
ardumont closed D7914: Wrap query in transaction.
May 31 2022, 11:15 AM
ardumont committed rDSCRUB31bcec6000a8: Wrap queries in transaction (authored by ardumont).
Wrap queries in transaction
May 31 2022, 11:15 AM
ardumont updated the diff for D7914: Wrap query in transaction.

Reword commit message

May 31 2022, 11:10 AM
ardumont updated the test plan for D7914: Wrap query in transaction.
May 31 2022, 11:02 AM
ardumont updated the test plan for D7914: Wrap query in transaction.
May 31 2022, 11:01 AM
ardumont retitled D7914: Wrap query in transaction from wip: Wrap query in transaction to Wrap query in transaction.
May 31 2022, 11:00 AM
ardumont added a comment to T4228: scrubber: Investigate the apparent lock (staging).

With [1] applied, the insert query got executed!

May 31 2022, 10:58 AM · Archive integrity, System administration
ardumont added a comment to T4228: scrubber: Investigate the apparent lock (staging).

Stopping all services and letting run only 1 checker service.

May 31 2022, 10:58 AM · Archive integrity, System administration
ardumont requested review of D7914: Wrap query in transaction.
May 31 2022, 10:39 AM
ardumont added a revision to T4228: scrubber: Investigate the apparent lock (staging): D7914: Wrap query in transaction.
May 31 2022, 10:36 AM · Archive integrity, System administration
ardumont updated the summary of D7913: db: Grant read access to guest user on all tables of the schema.
May 31 2022, 9:57 AM
ardumont updated the summary of D7913: db: Grant read access to guest user on all tables of the schema.
May 31 2022, 9:50 AM
ardumont requested review of D7913: db: Grant read access to guest user on all tables of the schema.
May 31 2022, 9:48 AM
ardumont added a revision to T4228: scrubber: Investigate the apparent lock (staging): D7913: db: Grant read access to guest user on all tables of the schema.
May 31 2022, 9:19 AM · Archive integrity, System administration
ardumont added a comment to T4228: scrubber: Investigate the apparent lock (staging).

Nothing bulged, same id process...
(Well, new ones appeared but old ones are still waiting for something)

May 31 2022, 9:05 AM · Archive integrity, System administration

May 30 2022

ardumont added a comment to T4228: scrubber: Investigate the apparent lock (staging).

We have plenty of processes stuck with "idle in transaction". According to [1], this means
"waiting for client inside a BEGIN block", so there might be issues in the scrubber code
[2]?

May 30 2022, 6:48 PM · Archive integrity, System administration
ardumont added a comment to T4228: scrubber: Investigate the apparent lock (staging).

There seem to still exist staling queries [1] and nothing gets written to the db [2]:

May 30 2022, 6:35 PM · Archive integrity, System administration
ardumont added a comment to T4228: scrubber: Investigate the apparent lock (staging).

Finally, after installing postgresql-client-11 and stopping scrubber services [1]:

May 30 2022, 6:10 PM · Archive integrity, System administration
ardumont added a comment to T4228: scrubber: Investigate the apparent lock (staging).

It was still not ok as the upgrade tooling also expects the sql/upgrades/<final-version>.sql to exist and be packaged.
There was nothing there hence [1]

May 30 2022, 6:00 PM · Archive integrity, System administration
ardumont committed rDCORE749ac571a757: docs/db: Update datastore requirement (authored by ardumont).
docs/db: Update datastore requirement
May 30 2022, 5:58 PM
ardumont closed D7912: Reference the scrubber db model upgrade (from version 1 to 2).
May 30 2022, 5:58 PM
ardumont committed rDSCRUBf24850469c02: Reference the scrubber db model upgrade (from version 1 to 2) (authored by ardumont).
Reference the scrubber db model upgrade (from version 1 to 2)
May 30 2022, 5:58 PM
ardumont added a revision to T4228: scrubber: Investigate the apparent lock (staging): D7912: Reference the scrubber db model upgrade (from version 1 to 2).
May 30 2022, 5:54 PM · Archive integrity, System administration
ardumont closed T4280: Unstuck swh-web master build failing as Resolved.

Thanks again!

May 30 2022, 5:47 PM · Web app
ardumont updated subscribers of T4280: Unstuck swh-web master build failing .

Build is back to green thanks to you @vlorentz.

May 30 2022, 5:38 PM · Web app
ardumont closed D7910: Allow module to specify another config key than their module name.
May 30 2022, 5:36 PM
ardumont committed rDCORE240ae09ddade: Allow module to specify another config key than their module name (authored by ardumont).
Allow module to specify another config key than their module name
May 30 2022, 5:36 PM
ardumont closed D7909: cli.db: Reword ignore sentence.
May 30 2022, 5:36 PM
ardumont committed rDCORE9b62406fd36f: cli.db: Reword ignore sentence (authored by ardumont).
cli.db: Reword ignore sentence
May 30 2022, 5:36 PM
ardumont closed D7911: Recursive include the swh/scrubber/sql folder.
May 30 2022, 5:36 PM
ardumont committed rDSCRUB2041b0e579be: Recursive include the swh/scrubber/sql folder (authored by ardumont).
Recursive include the swh/scrubber/sql folder
May 30 2022, 5:36 PM
ardumont added a revision to T4284: scrubber does not comply to what's expected by the swh db tooling: D7911: Recursive include the swh/scrubber/sql folder.
May 30 2022, 5:31 PM · Archive integrity
ardumont added a comment to T4284: scrubber does not comply to what's expected by the swh db tooling.

How great, now, we got the following with D7910 applied:

May 30 2022, 5:29 PM · Archive integrity
ardumont requested review of D7910: Allow module to specify another config key than their module name.
May 30 2022, 5:27 PM
ardumont added a revision to T4284: scrubber does not comply to what's expected by the swh db tooling: D7910: Allow module to specify another config key than their module name.
May 30 2022, 5:25 PM · Archive integrity
ardumont requested review of D7909: cli.db: Reword ignore sentence.
May 30 2022, 5:10 PM
ardumont closed D7907: cli.db: Use attribute current_version instead of undeclared getter.
May 30 2022, 5:06 PM
ardumont committed rDCORE5cda0ca62601: cli.db: Use attribute current_version instead of undeclared getter (authored by ardumont).
cli.db: Use attribute current_version instead of undeclared getter
May 30 2022, 5:06 PM
ardumont updated the test plan for D7907: cli.db: Use attribute current_version instead of undeclared getter.
May 30 2022, 5:05 PM
ardumont updated the summary of D7907: cli.db: Use attribute current_version instead of undeclared getter.
May 30 2022, 5:01 PM
ardumont updated the summary of D7907: cli.db: Use attribute current_version instead of undeclared getter.
May 30 2022, 4:58 PM
ardumont requested review of D7907: cli.db: Use attribute current_version instead of undeclared getter.
May 30 2022, 4:58 PM
ardumont added a revision to T4284: scrubber does not comply to what's expected by the swh db tooling: D7907: cli.db: Use attribute current_version instead of undeclared getter.
May 30 2022, 4:43 PM · Archive integrity
ardumont added a revision to T4228: scrubber: Investigate the apparent lock (staging): D7906: db: Provide a default get_current_version method to db classes.
May 30 2022, 4:14 PM · Archive integrity, System administration
ardumont updated the task description for T4284: scrubber does not comply to what's expected by the swh db tooling.
May 30 2022, 4:11 PM · Archive integrity
ardumont updated the task description for T4284: scrubber does not comply to what's expected by the swh db tooling.
May 30 2022, 4:10 PM · Archive integrity
ardumont updated the task description for T4284: scrubber does not comply to what's expected by the swh db tooling.
May 30 2022, 4:06 PM · Archive integrity
ardumont triaged T4284: scrubber does not comply to what's expected by the swh db tooling as Normal priority.
May 30 2022, 4:03 PM · Archive integrity
ardumont committed rSPSITEa37141a1f49f: common/common: Use postgresql over local configuration key (authored by ardumont).
common/common: Use postgresql over local configuration key
May 30 2022, 3:54 PM
ardumont closed D7905: Unify factory to use keyword 'postgresql' over deprecated 'local'.
May 30 2022, 3:43 PM
ardumont committed rDSCRUBf21145a901aa: Unify factory to use keyword 'postgresql' over deprecated 'local' (authored by ardumont).
Unify factory to use keyword 'postgresql' over deprecated 'local'
May 30 2022, 3:43 PM
ardumont closed D7904: db: Bump to version 2.
May 30 2022, 3:42 PM
ardumont committed rDSCRUB8f6409eb0e49: db: Bump to version 2 (authored by ardumont).
db: Bump to version 2
May 30 2022, 3:42 PM
ardumont updated the diff for D7905: Unify factory to use keyword 'postgresql' over deprecated 'local'.

Add tests over the factory change

May 30 2022, 3:39 PM
ardumont retitled D7905: Unify factory to use keyword 'postgresql' over deprecated 'local' from Use postgresql for factory to Unify factory to use keyword 'postgresql' over deprecated 'local'.
May 30 2022, 3:39 PM
ardumont requested review of D7905: Unify factory to use keyword 'postgresql' over deprecated 'local'.
May 30 2022, 3:21 PM
ardumont added a revision to T4228: scrubber: Investigate the apparent lock (staging): D7905: Unify factory to use keyword 'postgresql' over deprecated 'local'.
May 30 2022, 3:18 PM · Archive integrity, System administration
ardumont requested review of D7904: db: Bump to version 2.
May 30 2022, 3:16 PM
ardumont closed D7903: cli.db: Fix help message typo.
May 30 2022, 3:13 PM
ardumont committed rDCOREf18016af5c46: cli.db: Fix help message typo (authored by ardumont).
cli.db: Fix help message typo
May 30 2022, 3:13 PM
ardumont added a revision to T4228: scrubber: Investigate the apparent lock (staging): D7904: db: Bump to version 2.
May 30 2022, 3:12 PM · Archive integrity, System administration