Page MenuHomeSoftware Heritage

ardumont (Antoine R. Dumont)
UserAdministrator

Projects (10)

User Details

User Since
Sep 7 2015, 3:42 PM (307 w, 4 d)
Roles
Administrator

Recent Activity

Yesterday

ardumont accepted D6047: Add stream_results_optional.
Fri, Jul 30, 4:41 PM
ardumont added a comment to D6047: Add stream_results_optional.

looks sound to me, i'd add the 2 missing tests (which ensure we don't call stream_result* with page_token) since we are touching that part.

Fri, Jul 30, 4:07 PM
ardumont added inline comments to D6047: Add stream_results_optional.
Fri, Jul 30, 4:02 PM
ardumont changed the status of T2345: Improve handling of recurrent loading tasks in scheduler from Open to Work in Progress.

(^ for a while ;)

Fri, Jul 30, 3:55 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont added a comment to T2345: Improve handling of recurrent loading tasks in scheduler.

Status on this, after the recent refactoring we did with @olasd to simplify the actual
implementation (backend and journal client). There remains to:

Fri, Jul 30, 3:54 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont committed rDSCH4fa29fe128c2: journal_client: Update get_last_status docstring (authored by ardumont).
journal_client: Update get_last_status docstring
Fri, Jul 30, 3:41 PM
ardumont committed rDSCH3b929d0bd9dc: journal_client: Refactor by inlining the update_position_offset (authored by ardumont).
journal_client: Refactor by inlining the update_position_offset
Fri, Jul 30, 3:41 PM
ardumont added inline comments to D6031: Add a quick start section in the documentation and simplify the configuration file loading mechanism in the cli.
Fri, Jul 30, 2:51 PM
ardumont added a comment to T3338: Load the archived bitbucket mercurial repositories.
13:40:06 softwareheritage@belvedere:5432=> select now(), count(distinct url) from origin o inner join origin_visit ov on o.id=ov.origin where o.url like 'https://bitbucket.org/%' and ov.type='hg';
+-------------------------------+--------+
|              now              | count  |
+-------------------------------+--------+
| 2021-07-30 11:39:37.122152+00 | 253848 |
+-------------------------------+--------+
(1 row)
Fri, Jul 30, 2:49 PM · System administration, Mercurial loader
ardumont claimed T3338: Load the archived bitbucket mercurial repositories.

(claiming i said ;)

Fri, Jul 30, 2:42 PM · System administration, Mercurial loader
ardumont placed T3338: Load the archived bitbucket mercurial repositories up for grabs.

(Claiming the task to find it back more easily through my activity view.)

Fri, Jul 30, 1:42 PM · System administration, Mercurial loader
ardumont moved T3338: Load the archived bitbucket mercurial repositories from in-progress to code-review/monitoring on the System administration board.
Fri, Jul 30, 1:33 PM · System administration, Mercurial loader
ardumont added a comment to T3338: Load the archived bitbucket mercurial repositories.

Started in the same tmux session as the sourceforge ingestion.

Fri, Jul 30, 1:33 PM · System administration, Mercurial loader
ardumont changed the status of T3338: Load the archived bitbucket mercurial repositories from Open to Work in Progress.
Fri, Jul 30, 1:04 PM · System administration, Mercurial loader
ardumont added a comment to T3338: Load the archived bitbucket mercurial repositories.

It would probably make sense to set up a new worker instance for this to avoid interfering with the regular loading.

Fri, Jul 30, 1:04 PM · System administration, Mercurial loader
ardumont closed T3337: Smoke test ingestion of bitbucket repositories with latest loader mercurial, a subtask of T3338: Load the archived bitbucket mercurial repositories, as Resolved.
Fri, Jul 30, 1:00 PM · System administration, Mercurial loader
ardumont closed T3337: Smoke test ingestion of bitbucket repositories with latest loader mercurial as Resolved.
Fri, Jul 30, 1:00 PM · System administration, Mercurial loader
ardumont added a comment to T3337: Smoke test ingestion of bitbucket repositories with latest loader mercurial.

Smoke test it with local bitbucket repositories

that's next.

Fri, Jul 30, 12:59 PM · System administration, Mercurial loader
ardumont created P1108 (production) running loader mercurial on local bitbucket repositories is fine as well (one run eventful -> another run uneventful).
Fri, Jul 30, 12:32 PM
ardumont claimed T3337: Smoke test ingestion of bitbucket repositories with latest loader mercurial.
Fri, Jul 30, 12:22 PM · System administration, Mercurial loader
ardumont changed the status of T3337: Smoke test ingestion of bitbucket repositories with latest loader mercurial from Open to Work in Progress.
Fri, Jul 30, 12:16 PM · System administration, Mercurial loader
ardumont changed the status of T3337: Smoke test ingestion of bitbucket repositories with latest loader mercurial, a subtask of T3338: Load the archived bitbucket mercurial repositories, from Open to Work in Progress.
Fri, Jul 30, 12:16 PM · System administration, Mercurial loader
ardumont added a comment to T3337: Smoke test ingestion of bitbucket repositories with latest loader mercurial.

Smoke test it with remote repositories.

Fri, Jul 30, 12:16 PM · System administration, Mercurial loader
ardumont created P1107 (production) running loader mercurial on remote origins is fine.
Fri, Jul 30, 12:15 PM
ardumont accepted D6045: converters: Preserve GPG signatures on releases.

lgtm

Fri, Jul 30, 11:31 AM
ardumont moved T3444: 26/07/2021: Unstuck infrastructure outage then post-mortem from in-progress to code-review/monitoring on the System administration board.
Fri, Jul 30, 11:22 AM · System administration
ardumont moved T3446: Restart scheduling regularly origins with relevant scheduling policies from in-progress to code-review/monitoring on the System administration board.
Fri, Jul 30, 11:22 AM · System administration
ardumont moved T3374: Ingest sourceforge repositories (origins of type git, svn, hg) from in-progress to code-review/monitoring on the System administration board.
Fri, Jul 30, 11:22 AM · System administration, Archive coverage, Origin-SourceForge
ardumont moved T3450: 404 error when visiting a successfully archived repository from in-progress to code-review/monitoring on the System administration board.
Fri, Jul 30, 11:22 AM · Storage manager, System administration
ardumont added a comment to T3450: 404 error when visiting a successfully archived repository.

Thanks for the heads up @ both of you.

Fri, Jul 30, 11:14 AM · Storage manager, System administration
ardumont triaged T3452: Replication lag between the dbs should raise icinga alerts as High priority.
Fri, Jul 30, 11:13 AM · Monitoring, System administration
ardumont changed the status of T3450: 404 error when visiting a successfully archived repository from Open to Work in Progress.
Fri, Jul 30, 10:40 AM · Storage manager, System administration
ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Fri, Jul 30, 10:25 AM · System administration, Archive coverage, Origin-SourceForge
ardumont added a comment to T3446: Restart scheduling regularly origins with relevant scheduling policies.

Heads up, this is running slightly different now.

Fri, Jul 30, 9:59 AM · System administration
ardumont committed rSPSITEa49123e9449c: worker17: Activate mercurial origins to ingest for oneshot loader (authored by ardumont).
worker17: Activate mercurial origins to ingest for oneshot loader
Fri, Jul 30, 9:44 AM

Thu, Jul 29

ardumont renamed T3337: Smoke test ingestion of bitbucket repositories with latest loader mercurial from Deploy swh.loader.mercurial 1.0 in production to Smoke test ingestion of bitbucket repositories with latest loader mercurial.
Thu, Jul 29, 6:36 PM · System administration, Mercurial loader
ardumont added a comment to T3336: Deploy swh.loader.mercurial 2.1 in staging.

For history purpose readabillty, this must bev2.1 git-patched version (not a release per say).
A more recent version release which is a tag v2.1.0 [1] has been done built with the work
solving the work @olasd started with versioned extid.

Thu, Jul 29, 6:33 PM · System administration, Mercurial loader
ardumont added a subtask for T3338: Load the archived bitbucket mercurial repositories: T3418: Decide a consistent policy on having multiple archived objects for the same extid.
Thu, Jul 29, 6:30 PM · System administration, Mercurial loader
ardumont added a parent task for T3418: Decide a consistent policy on having multiple archived objects for the same extid: T3338: Load the archived bitbucket mercurial repositories.
Thu, Jul 29, 6:30 PM · Storage manager, Mercurial loader
ardumont moved T3338: Load the archived bitbucket mercurial repositories from Backlog to Weekly backlog on the System administration board.

Latest mercurial loader v2.1 deployed [1] [2]
We should be able to continue with this now.

Thu, Jul 29, 6:29 PM · System administration, Mercurial loader
ardumont closed T3418: Decide a consistent policy on having multiple archived objects for the same extid as Resolved.
Thu, Jul 29, 6:21 PM · Storage manager, Mercurial loader
ardumont moved T3448: production: Deploy swh.loader.mercurial v2.1.0 from deployed/landed to done on the System administration board.
Thu, Jul 29, 6:21 PM · System administration, Storage manager, Mercurial loader
ardumont moved T3448: production: Deploy swh.loader.mercurial v2.1.0 from in-progress to deployed/landed on the System administration board.
Thu, Jul 29, 6:21 PM · System administration, Storage manager, Mercurial loader
ardumont added a comment to T3418: Decide a consistent policy on having multiple archived objects for the same extid.

Deployed \o/
Closing.

Thu, Jul 29, 6:21 PM · Storage manager, Mercurial loader
ardumont closed T3448: production: Deploy swh.loader.mercurial v2.1.0, a subtask of T3418: Decide a consistent policy on having multiple archived objects for the same extid, as Resolved.
Thu, Jul 29, 6:20 PM · Storage manager, Mercurial loader
ardumont closed T3448: production: Deploy swh.loader.mercurial v2.1.0 as Resolved.

It's working but the check does not pass green [1]. As far as i could tell, the unsuccessful
event [1] is seen as failure by the check.

Thu, Jul 29, 6:20 PM · System administration, Storage manager, Mercurial loader
ardumont closed T3443: Deploy lister v1.5.0 as Resolved.
Thu, Jul 29, 5:48 PM · System administration, Lister
ardumont moved T3443: Deploy lister v1.5.0 from in-progress to deployed/landed on the System administration board.
Thu, Jul 29, 5:48 PM · System administration, Lister
ardumont added a comment to T3443: Deploy lister v1.5.0.

Still ongoing.

17:31:19 softwareheritage-scheduler@belvedere:5432=> select now(), visit_type, count(*) from listed_origins lo inner join listers l on l.id=lo.lister_id where l.name='gitlab' and l.instance_name='gitlab' group by visit_type order by count(
*) desc;
+-------------------------------+------------+---------+
|              now              | visit_type |  count  |
+-------------------------------+------------+---------+
| 2021-07-29 15:46:55.979469+00 | git        | 1747880 |
+-------------------------------+------------+---------+
(1 row)
Thu, Jul 29, 5:47 PM · System administration, Lister
ardumont moved T3374: Ingest sourceforge repositories (origins of type git, svn, hg) from Backlog to in-progress on the System administration board.
Thu, Jul 29, 5:45 PM · System administration, Archive coverage, Origin-SourceForge
ardumont added a project to T3374: Ingest sourceforge repositories (origins of type git, svn, hg): System administration.
Thu, Jul 29, 5:45 PM · System administration, Archive coverage, Origin-SourceForge
ardumont renamed T3374: Ingest sourceforge repositories (origins of type git, svn, hg) from Ingest sourceforge repositories to Ingest sourceforge repositories (origins of type git, svn, hg).
Thu, Jul 29, 5:45 PM · System administration, Archive coverage, Origin-SourceForge
ardumont added a comment to T3374: Ingest sourceforge repositories (origins of type git, svn, hg).

The 'hg' ingestion started now that the latest loader mercurial got deployed.

Thu, Jul 29, 5:43 PM · System administration, Archive coverage, Origin-SourceForge
ardumont changed the status of T3448: production: Deploy swh.loader.mercurial v2.1.0, a subtask of T3418: Decide a consistent policy on having multiple archived objects for the same extid, from Open to Work in Progress.
Thu, Jul 29, 5:40 PM · Storage manager, Mercurial loader
ardumont changed the status of T3448: production: Deploy swh.loader.mercurial v2.1.0 from Open to Work in Progress.
Thu, Jul 29, 5:40 PM · System administration, Storage manager, Mercurial loader
ardumont added a comment to T3448: production: Deploy swh.loader.mercurial v2.1.0.

At the end of it all though, the final production check end-to-end for mercurial origin should go green.

Thu, Jul 29, 5:40 PM · System administration, Storage manager, Mercurial loader
ardumont updated the task description for T3448: production: Deploy swh.loader.mercurial v2.1.0.
Thu, Jul 29, 5:32 PM · System administration, Storage manager, Mercurial loader
ardumont accepted D6031: Add a quick start section in the documentation and simplify the configuration file loading mechanism in the cli.

Like i said yesterday, lgtm.

Thu, Jul 29, 1:25 PM
ardumont moved T3448: production: Deploy swh.loader.mercurial v2.1.0 from Backlog to Weekly backlog on the System administration board.
Thu, Jul 29, 1:24 PM · System administration, Storage manager, Mercurial loader
ardumont moved T3394: cassandra - origin url hashing encoding issue from Backlog to done on the System administration board.
Thu, Jul 29, 1:24 PM · System administration, Storage manager
ardumont moved T3397: Improve the documentation deployment to support multiple sphinx instances from Backlog to done on the System administration board.
Thu, Jul 29, 1:24 PM · System administration, Documentation
ardumont changed the status of T3402: Deploy swh-counters v0.8.0 and backfill origins, a subtask of T3127: Compute and display distribution of origins by forge, from Wontfix to Resolved.
Thu, Jul 29, 1:24 PM · Metrics/monitoring, Web app, Roadmap 2021, meta-task
ardumont changed the status of T3402: Deploy swh-counters v0.8.0 and backfill origins from Wontfix to Resolved.
Thu, Jul 29, 1:24 PM · Counters, System administration, Metrics/monitoring
ardumont changed the status of T3445: Monitor hypervisors as well from Invalid to Resolved.
Thu, Jul 29, 1:23 PM · System administration
ardumont moved T3086: Prepare disk replacement on granet from in-progress to done on the System administration board.
Thu, Jul 29, 1:23 PM · System administration
ardumont moved T3061: swh-search: Deploy visit_types indexation in production from in-progress to done on the System administration board.
Thu, Jul 29, 1:23 PM · System administration, Web app
ardumont moved T3159: Deploy swh-counters:v0.1.0 in staging from in-progress to done on the System administration board.
Thu, Jul 29, 1:23 PM · Staging environment, System administration, Monitoring
ardumont moved T3115: Upgrade zfs on all servers from in-progress to done on the System administration board.
Thu, Jul 29, 1:23 PM · System administration
ardumont moved T3060: Deploy swh-search v0.6.0 in **staging** from in-progress to done on the System administration board.
Thu, Jul 29, 1:23 PM · System administration, Archive search
ardumont moved T3242: Decommission ClearlyDefined resources from in-progress to done on the System administration board.
Thu, Jul 29, 1:23 PM · System administration
ardumont moved T3033: Replace first disk on storage1.staging from in-progress to done on the System administration board.
Thu, Jul 29, 1:23 PM · System administration
ardumont moved T3310: Deploy sourceforge lister on staging from in-progress to done on the System administration board.
Thu, Jul 29, 1:23 PM · System administration, Origin-SourceForge
ardumont moved T3041: [production] Provision enough space for the search ES cluster to ingest all intrinsic metadata from in-progress to done on the System administration board.
Thu, Jul 29, 1:23 PM · System administration, Archive search
ardumont moved T3350: Deploy sourceforge lister in production from in-progress to done on the System administration board.
Thu, Jul 29, 1:23 PM · Archive coverage, System administration, Origin-SourceForge
ardumont moved T2939: Replace out of order disks on db1.staging and storage1.staging from Weekly backlog to done on the System administration board.
Thu, Jul 29, 1:23 PM · System administration
ardumont moved T3222: Monitor daily indexes are present on the log cluster and logs are correctly ingested from deployed/landed to done on the System administration board.
Thu, Jul 29, 1:22 PM · System administration
ardumont moved T2877: Investigate spurious deposit logs from deployed/landed to done on the System administration board.
Thu, Jul 29, 1:22 PM · System administration, Staging environment, SWORD deposit
ardumont moved T2927: Unstuck jenkins from deployed/landed to done on the System administration board.
Thu, Jul 29, 1:22 PM · System administration
ardumont moved T2908: sentry does not log tracebacks in the swh-deposit server from deployed/landed to done on the System administration board.
Thu, Jul 29, 1:22 PM · System administration, SWORD deposit
ardumont moved T2910: Sentry: Increase disk space from deployed/landed to done on the System administration board.
Thu, Jul 29, 1:22 PM · Sentry, System administration
ardumont moved T2866: Integrate former Uffizi server to the proxmox cluster from deployed/landed to done on the System administration board.
Thu, Jul 29, 1:22 PM · System administration
ardumont moved T2774: Fix vault end-to-end check from deployed/landed to done on the System administration board.
Thu, Jul 29, 1:22 PM · Vault, System administration, Monitoring
ardumont moved T2888: Elasticsearch cluster failure during a rolling restart from deployed/landed to done on the System administration board.
Thu, Jul 29, 1:21 PM · System administration
ardumont moved T3203: docs: Document the firewall installation and procedures from deployed/landed to done on the System administration board.
Thu, Jul 29, 1:21 PM · Documentation, System administration
ardumont moved T3447: staging: Deploy swh.loader.mercurial v2.1.0 from deployed/landed to done on the System administration board.
Thu, Jul 29, 1:21 PM · System administration, Storage manager, Mercurial loader
ardumont moved T3447: staging: Deploy swh.loader.mercurial v2.1.0 from in-progress to deployed/landed on the System administration board.
Thu, Jul 29, 1:21 PM · System administration, Storage manager, Mercurial loader
ardumont claimed T3448: production: Deploy swh.loader.mercurial v2.1.0.
Thu, Jul 29, 1:21 PM · System administration, Storage manager, Mercurial loader
ardumont updated the task description for T3448: production: Deploy swh.loader.mercurial v2.1.0.
Thu, Jul 29, 1:21 PM · System administration, Storage manager, Mercurial loader
ardumont triaged T3448: production: Deploy swh.loader.mercurial v2.1.0 as Unbreak Now! priority.
Thu, Jul 29, 1:20 PM · System administration, Storage manager, Mercurial loader
ardumont closed T3447: staging: Deploy swh.loader.mercurial v2.1.0, a subtask of T3418: Decide a consistent policy on having multiple archived objects for the same extid, as Resolved.
Thu, Jul 29, 1:18 PM · Storage manager, Mercurial loader
ardumont closed T3447: staging: Deploy swh.loader.mercurial v2.1.0 as Resolved.
Thu, Jul 29, 1:18 PM · System administration, Storage manager, Mercurial loader
ardumont added a comment to T3447: staging: Deploy swh.loader.mercurial v2.1.0.

From the staging webapp, we identify a revision [1]

Thu, Jul 29, 1:17 PM · System administration, Storage manager, Mercurial loader
ardumont updated the task description for T3447: staging: Deploy swh.loader.mercurial v2.1.0.
Thu, Jul 29, 12:47 PM · System administration, Storage manager, Mercurial loader
ardumont changed the status of T3447: staging: Deploy swh.loader.mercurial v2.1.0, a subtask of T3418: Decide a consistent policy on having multiple archived objects for the same extid, from Open to Work in Progress.
Thu, Jul 29, 12:32 PM · Storage manager, Mercurial loader
ardumont changed the status of T3447: staging: Deploy swh.loader.mercurial v2.1.0 from Open to Work in Progress.
Thu, Jul 29, 12:32 PM · System administration, Storage manager, Mercurial loader
ardumont added a project to T3447: staging: Deploy swh.loader.mercurial v2.1.0: System administration.
Thu, Jul 29, 12:32 PM · System administration, Storage manager, Mercurial loader
ardumont changed the status of T3418: Decide a consistent policy on having multiple archived objects for the same extid from Open to Work in Progress.

by the way ^

Thu, Jul 29, 12:31 PM · Storage manager, Mercurial loader
ardumont triaged T3447: staging: Deploy swh.loader.mercurial v2.1.0 as Unbreak Now! priority.
Thu, Jul 29, 12:30 PM · System administration, Storage manager, Mercurial loader
ardumont added a comment to T3418: Decide a consistent policy on having multiple archived objects for the same extid.

Shipped the following modules to solve that problem:

  • swh.model v2.7.0
  • swh.storage v0.35.0
  • swh.loader.mercurial v2.1.0
Thu, Jul 29, 12:26 PM · Storage manager, Mercurial loader
ardumont closed D6036: Use versioned ExtIDs in loader mercurial implementation.
Thu, Jul 29, 11:53 AM