Page MenuHomeSoftware Heritage
Feed Advanced Search

Jul 11 2022

ardumont added a comment to T4387: Scrubber processes getting killed by OOM killer.

After upgrading packages, and reboot.
This seems to have increased a bit its memory to something more sensible.
Let's see if the problem disappears altogether now.

Jul 11 2022, 2:21 PM · System administration, Datastore Scrubber
ardumont accepted D8103: rlog, cvs2gitdump: Fix handling of revision number greater than one.
Jul 11 2022, 2:20 PM
ardumont added inline comments to D8105: Install zfs and docker on the cassandra node to prepare the cass operator tests.
Jul 11 2022, 2:18 PM
ardumont accepted D8105: Install zfs and docker on the cassandra node to prepare the cass operator tests.

but otherwise, lgtm ;)

Jul 11 2022, 2:16 PM
ardumont added inline comments to D8103: rlog, cvs2gitdump: Fix handling of revision number greater than one.
Jul 11 2022, 2:09 PM
ardumont added a comment to D8105: Install zfs and docker on the cassandra node to prepare the cass operator tests.

one question inline.

Jul 11 2022, 2:03 PM
ardumont added a comment to D8106: weekly-planning: Remove 'template' tag from generated documents.

or you could edit on the template directly? [1]

Jul 11 2022, 2:01 PM
ardumont accepted D8104: loader: Use utf-8 instead of ascii to decode rsync output.
Jul 11 2022, 1:53 PM
ardumont triaged T4387: Scrubber processes getting killed by OOM killer as Normal priority.
Jul 11 2022, 10:50 AM · System administration, Datastore Scrubber

Jul 8 2022

ardumont updated the task description for T4385: Federate prometheus instances through thanos.
Jul 8 2022, 5:07 PM · meta-task, System administration, Roadmap 2022
ardumont updated the task description for T4385: Federate prometheus instances through thanos.
Jul 8 2022, 5:06 PM · meta-task, System administration, Roadmap 2022
ardumont updated the task description for T4385: Federate prometheus instances through thanos.
Jul 8 2022, 4:46 PM · meta-task, System administration, Roadmap 2022
ardumont updated the diff for D8097: thanos: Configure gateway service to expose historical data.

Adapt according to suggestion.

Jul 8 2022, 4:40 PM
ardumont updated the summary of D8097: thanos: Configure gateway service to expose historical data.
Jul 8 2022, 4:40 PM
ardumont accepted D8102: Fix the load_and_check_config() function to support the "postgresql" cls value.

one comment inline

Jul 8 2022, 2:37 PM
ardumont added inline comments to D8102: Fix the load_and_check_config() function to support the "postgresql" cls value.
Jul 8 2022, 2:34 PM
ardumont planned changes to D8097: thanos: Configure gateway service to expose historical data.

btw:

11:19 <+olasd> ardumont: we want the gateway services to run on the central Thanos host, don't we?
11:19 <+olasd> (to reduce the number of open ports and cross-tenant dependencies)
11:46 <+ardumont> Not sure we can? It wants access to the disk store? I'll double check
11:50 <+olasd> there's no disk store
11:50 <+olasd> only azure objstorage
Jul 8 2022, 2:21 PM
ardumont updated the summary of D8097: thanos: Configure gateway service to expose historical data.
Jul 8 2022, 11:14 AM
ardumont updated the task description for T4385: Federate prometheus instances through thanos.
Jul 8 2022, 11:07 AM · meta-task, System administration, Roadmap 2022
ardumont requested review of D8097: thanos: Configure gateway service to expose historical data.
Jul 8 2022, 11:06 AM
ardumont added a revision to T4385: Federate prometheus instances through thanos: D8097: thanos: Configure gateway service to expose historical data.
Jul 8 2022, 11:06 AM · meta-task, System administration, Roadmap 2022
ardumont updated the task description for T4385: Federate prometheus instances through thanos.
Jul 8 2022, 10:30 AM · meta-task, System administration, Roadmap 2022
ardumont accepted D8081: backfill: Add support for directories with duplicated entries.

Assuming the build it depends on is landed, tagged and released and this build goes green.

Jul 8 2022, 10:04 AM
ardumont accepted D8080: model: Add Directory.from_possibly_duplicated_entries factory.
Jul 8 2022, 10:03 AM
ardumont added a comment to D8080: model: Add Directory.from_possibly_duplicated_entries factory.

lgtm, one typo inline.

Jul 8 2022, 10:03 AM
ardumont added inline comments to D8081: backfill: Add support for directories with duplicated entries.
Jul 8 2022, 9:42 AM

Jul 7 2022

ardumont updated the diff for D8092: Configure thanos query service to new dedicated node.

Use collected resources to generate the configuration file for thanos query service

Jul 7 2022, 6:39 PM
ardumont added inline comments to D8092: Configure thanos query service to new dedicated node.
Jul 7 2022, 5:39 PM
ardumont updated the diff for D8092: Configure thanos query service to new dedicated node.

wip: Rework logic to instantiate thanos query to lookup datastore (out of collected
resources and statically declared resources)

Jul 7 2022, 5:37 PM
ardumont updated the diff for D8092: Configure thanos query service to new dedicated node.

Address the simple part of the comments (had a bit of a fight with vagrant):

  • sidecar: dedicate http and grpc port
  • sidecar: expose only internal ip
  • query: dedicate http and grpc port
  • systemd service: use dynamic user
  • stores list: use full $service:$port string
Jul 7 2022, 3:59 PM
ardumont committed rSENV8f6702e1b1fd: Vagrantfile: Reference thanos node (authored by ardumont).
Vagrantfile: Reference thanos node
Jul 7 2022, 3:52 PM
ardumont committed rSENV6e74e4f17525: rsync puppet facts (authored by ardumont).
rsync puppet facts
Jul 7 2022, 3:47 PM
ardumont added inline comments to D8092: Configure thanos query service to new dedicated node.
Jul 7 2022, 3:18 PM
ardumont updated the task description for T4385: Federate prometheus instances through thanos.
Jul 7 2022, 3:12 PM · meta-task, System administration, Roadmap 2022
ardumont accepted D8095: Add minimum version requirement for pytest.
Jul 7 2022, 2:53 PM
ardumont closed D8089: Provision thanos query node.
Jul 7 2022, 2:52 PM
ardumont committed rSPRE0c3856d06d93: Provision thanos query node in the admin vlan (authored by ardumont).
Provision thanos query node in the admin vlan
Jul 7 2022, 2:52 PM
ardumont updated the diff for D8092: Configure thanos query service to new dedicated node.

Revert wrong update

Jul 7 2022, 2:50 PM
ardumont updated the diff for D8089: Provision thanos query node.

Update with the right diff id

Jul 7 2022, 2:49 PM
ardumont updated the diff for D8092: Configure thanos query service to new dedicated node.

Update

Jul 7 2022, 2:48 PM
ardumont accepted D8094: Declare the kubernetes cluster and management nodes for cassandra.
Jul 7 2022, 2:12 PM
ardumont accepted D8093: Add missing dependency on types-cryptography.
Jul 7 2022, 2:06 PM
ardumont accepted D8091: loader: Ensure to strip trailing slash from origin URL.
Jul 7 2022, 2:06 PM
ardumont updated the task description for T4385: Federate prometheus instances through thanos.
Jul 7 2022, 12:01 PM · meta-task, System administration, Roadmap 2022
ardumont updated the test plan for D8092: Configure thanos query service to new dedicated node.
Jul 7 2022, 11:59 AM
ardumont added inline comments to D8092: Configure thanos query service to new dedicated node.
Jul 7 2022, 11:58 AM
ardumont added inline comments to D8092: Configure thanos query service to new dedicated node.
Jul 7 2022, 11:57 AM
ardumont updated the test plan for D8092: Configure thanos query service to new dedicated node.
Jul 7 2022, 11:56 AM
ardumont added a comment to D8089: Provision thanos query node.

I've no idea if the cpu/memory/disk spec are large enough or not, I didn't find the info on the thanos documentation

Jul 7 2022, 11:54 AM
ardumont updated the test plan for D8092: Configure thanos query service to new dedicated node.
Jul 7 2022, 11:50 AM
ardumont updated the test plan for D8092: Configure thanos query service to new dedicated node.
Jul 7 2022, 11:49 AM
ardumont requested review of D8092: Configure thanos query service to new dedicated node.
Jul 7 2022, 11:48 AM
ardumont added a revision to T4385: Federate prometheus instances through thanos: D8092: Configure thanos query service to new dedicated node.
Jul 7 2022, 11:48 AM · meta-task, System administration, Roadmap 2022
ardumont accepted D8082: Document codemeta.make_absolute_uri().
Jul 7 2022, 10:00 AM

Jul 6 2022

ardumont updated the task description for T4385: Federate prometheus instances through thanos.
Jul 6 2022, 6:19 PM · meta-task, System administration, Roadmap 2022
ardumont updated the diff for D8089: Provision thanos query node.

Move that node to the admin vlan

Jul 6 2022, 6:12 PM
ardumont updated the summary of D8089: Provision thanos query node.
Jul 6 2022, 6:03 PM
ardumont updated the diff for D8089: Provision thanos query node.

Update reserved ip address

Jul 6 2022, 6:03 PM
ardumont updated the task description for T4385: Federate prometheus instances through thanos.
Jul 6 2022, 5:52 PM · meta-task, System administration, Roadmap 2022
ardumont requested review of D8089: Provision thanos query node.
Jul 6 2022, 5:48 PM
ardumont added a revision to T4385: Federate prometheus instances through thanos: D8089: Provision thanos query node.
Jul 6 2022, 5:48 PM · meta-task, System administration, Roadmap 2022
ardumont triaged T4385: Federate prometheus instances through thanos as High priority.
Jul 6 2022, 5:46 PM · meta-task, System administration, Roadmap 2022
ardumont accepted D8088: add_forge_now: Display pre in request history only for non empty text.
Jul 6 2022, 5:46 PM
ardumont added a comment to D8086: Fix loading of CVS repositories with non valid UTF-8 paths.

Its syntax is more convenient and it automatically deals with encoding. eg. wtpath = os.path.join(os.fsencode(self.tempdir_path), path) becomes wtpath = self.tempdir_path / path with pathlib

I see, maybe we could handle such change in another diff ? If I recall correctly, the subversion loader implementation could also be migrated to pathlib.

Jul 6 2022, 5:17 PM
ardumont renamed T4383: Test full phabricator migration to gitlab from Test full phabricator migratio to gitlab to Test full phabricator migration to gitlab.
Jul 6 2022, 2:43 PM · System administration, GitLab migration, Roadmap 2020
ardumont added a comment to T4383: Test full phabricator migration to gitlab.

I just reused the current last migration run and added the swh-objstorage to the list of
repositories (not yet migrated). And the migration just broke on that repository.

Jul 6 2022, 2:43 PM · System administration, GitLab migration, Roadmap 2020
ardumont triaged T4383: Test full phabricator migration to gitlab as High priority.
Jul 6 2022, 2:42 PM · System administration, GitLab migration, Roadmap 2020
ardumont added a comment to P1401 hal deposit user information in staging.
14:21:51 swh-deposit@db1:5432=> select * from deposit_client dc inner join auth_user u on user_ptr_id=u.id  where u.username='hal-preprod' ;
+-[ RECORD 1 ]-+--------------------------------------------------------------------------------+
| user_ptr_id  | 5                                                                              |
| collections  | {4}                                                                            |
| provider_url | https://inria.halpreprod.archives-ouvertes.fr/                                 |
| domain       | halpreprod.archives-ouvertes.fr                                                |
| id           | 5                                                                              |
| password     | <redacted>                                                                     |
| last_login   | (null)                                                                         |
| is_superuser | f                                                                              |
| username     | hal-preprod                                                                    |
| first_name   |                                                                                |
| last_name    |                                                                                |
| email        |                                                                                |
| is_staff     | f                                                                              |
| is_active    | t                                                                              |
| date_joined  | 2019-05-13 11:53:32.301258+00                                                  |
+--------------+--------------------------------------------------------------------------------+
Jul 6 2022, 2:24 PM
ardumont updated the title for P1401 hal deposit user information in staging from hal deposit user information to hal deposit user information in staging.
Jul 6 2022, 2:24 PM
ardumont created P1401 hal deposit user information in staging.
Jul 6 2022, 2:11 PM
ardumont added inline comments to D8076: Make __iter__ actually return composite objids.
Jul 6 2022, 2:06 PM
ardumont added a comment to D7951: Deploy new origin intrinsic metadata journal client indexer.

You should be able to remove swh::deploy::worker::indexer_origin_intrinsic_metadata from swh-site/data/common/common.yaml, too

Jul 6 2022, 9:57 AM

Jul 5 2022

ardumont updated the task description for T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1.
Jul 5 2022, 4:55 PM · System administration, Indexer, Metadata workflow
ardumont added inline comments to D8067: Add a Storage.flavor property to the postgresql backend.
Jul 5 2022, 3:09 PM
ardumont accepted D8067: Add a Storage.flavor property to the postgresql backend.

lgtm (i've triggered back the build)

Jul 5 2022, 3:08 PM
ardumont added inline comments to D8066: Add the 'flavor' argument to db_utils.initialize_database_for_module().
Jul 5 2022, 2:20 PM
ardumont accepted D8066: Add the 'flavor' argument to db_utils.initialize_database_for_module().

one question on the last assertion test though.

Jul 5 2022, 2:07 PM
ardumont updated the task description for T4373: [cassandra] Test the new hardware.
Jul 5 2022, 10:04 AM · Storage manager, System administration

Jul 4 2022

ardumont added a comment to T4334: Test rollback of a repo migration .

Reopened as the need got clarified and the previous check used a repository which did not have any issues/merge-requests associated to it.
So let's check with a repo which does have those ^.

Jul 4 2022, 5:40 PM · System administration, GitLab migration, Roadmap 2020
ardumont accepted D8070: cff: Remove redundant @context setting.
Jul 4 2022, 5:35 PM
ardumont accepted D8076: Make __iter__ actually return composite objids.

lgtm

Jul 4 2022, 5:35 PM
ardumont accepted D8074: Remove ID-based filters.
Jul 4 2022, 5:31 PM
ardumont added a comment to D8074: Remove ID-based filters.

ok, one question inline.

Jul 4 2022, 5:31 PM
ardumont accepted D8075: Remove unused fallbacks for old msgpack version.

Workers which are running on buster [1] seems ok indeed so fine i guess.

Jul 4 2022, 5:23 PM
ardumont updated the task description for T4334: Test rollback of a repo migration .
Jul 4 2022, 4:01 PM · System administration, GitLab migration, Roadmap 2020
ardumont reopened T4334: Test rollback of a repo migration , a subtask of T4064: Test GitLab migration scripts, as Open.
Jul 4 2022, 4:00 PM · System administration, GitLab migration, Roadmap 2020
ardumont reopened T4334: Test rollback of a repo migration as "Open".
Jul 4 2022, 4:00 PM · System administration, GitLab migration, Roadmap 2020
ardumont added a comment to D8066: Add the 'flavor' argument to db_utils.initialize_database_for_module().

lgtm but it's missing the test case around it. It should not be too hard to add a

scenario which lifts the flavor parameter, shouldn't it? possibly, but I had it tested
from swh-storage (D8067)

Jul 4 2022, 11:43 AM
ardumont added a comment to T4135: staging: Deploy graphql service.

Heads up, current work to improve the jenkins tooling has been tested (docker-compose,
jenkins in docker, triggering pipeline with docker).

Jul 4 2022, 10:50 AM · System administration, GraphQL API
ardumont added a project to T4371: Deploy swh-scrubber on all storage instances: System administration.
Jul 4 2022, 10:47 AM · System administration, Archive integrity, Storage manager
ardumont added projects to T4371: Deploy swh-scrubber on all storage instances: Storage manager, Archive integrity.
Jul 4 2022, 10:47 AM · System administration, Archive integrity, Storage manager
ardumont updated the task description for T4371: Deploy swh-scrubber on all storage instances.
Jul 4 2022, 10:25 AM · System administration, Archive integrity, Storage manager
ardumont triaged T4371: Deploy swh-scrubber on all storage instances as Normal priority.
Jul 4 2022, 10:24 AM · System administration, Archive integrity, Storage manager
ardumont added a comment to D8066: Add the 'flavor' argument to db_utils.initialize_database_for_module().

lgtm but it's missing the test case around it. It should not be too hard to add a scenario which lifts the flavor parameter, shouldn't it?

Jul 4 2022, 9:55 AM

Jul 1 2022

ardumont committed rCJSWH0613c0c9d3df: README.md: Improve gogs setup instructions (authored by ardumont).
README.md: Improve gogs setup instructions
Jul 1 2022, 10:02 AM
ardumont committed rCJSWHa8a26a77f28a: Ignore docker-compose.override.yml (authored by ardumont).
Ignore docker-compose.override.yml
Jul 1 2022, 10:02 AM

Jun 30 2022

ardumont committed rCJSWHb0872b4ec80b: wip: Start swh-apps/update-dependencies job (authored by ardumont).
wip: Start swh-apps/update-dependencies job
Jun 30 2022, 5:56 PM
ardumont committed rCJSWH6ce55e57218b: wip: Start swh-apps/update-dependencies job (authored by ardumont).
wip: Start swh-apps/update-dependencies job
Jun 30 2022, 5:46 PM
ardumont committed rCJSWHdfac84b3435f: wip: Start swh-apps/update-dependencies job (authored by ardumont).
wip: Start swh-apps/update-dependencies job
Jun 30 2022, 5:43 PM
ardumont committed rCJSWHe41c157273e8: wip: Start swh-apps/update-dependencies job (authored by ardumont).
wip: Start swh-apps/update-dependencies job
Jun 30 2022, 5:41 PM