Page MenuHomeSoftware Heritage
Feed All Stories

Jul 29 2021

ardumont added a comment to T3418: Decide a consistent policy on having multiple archived objects for the same extid.

Deployed \o/
Closing.

Jul 29 2021, 6:21 PM · Storage manager, Mercurial loader
ardumont closed T3448: production: Deploy swh.loader.mercurial v2.1.0, a subtask of T3418: Decide a consistent policy on having multiple archived objects for the same extid, as Resolved.
Jul 29 2021, 6:20 PM · Storage manager, Mercurial loader
ardumont closed T3448: production: Deploy swh.loader.mercurial v2.1.0 as Resolved.

It's working but the check does not pass green [1]. As far as i could tell, the unsuccessful
event [1] is seen as failure by the check.

Jul 29 2021, 6:20 PM · System administration, Storage manager, Mercurial loader
ardumont closed T3443: Deploy lister v1.5.0 as Resolved.
Jul 29 2021, 5:48 PM · System administration, Lister
ardumont moved T3443: Deploy lister v1.5.0 from in-progress to deployed/landed/monitoring on the System administration board.
Jul 29 2021, 5:48 PM · System administration, Lister
ardumont added a comment to T3443: Deploy lister v1.5.0.

Still ongoing.

17:31:19 softwareheritage-scheduler@belvedere:5432=> select now(), visit_type, count(*) from listed_origins lo inner join listers l on l.id=lo.lister_id where l.name='gitlab' and l.instance_name='gitlab' group by visit_type order by count(
*) desc;
+-------------------------------+------------+---------+
|              now              | visit_type |  count  |
+-------------------------------+------------+---------+
| 2021-07-29 15:46:55.979469+00 | git        | 1747880 |
+-------------------------------+------------+---------+
(1 row)
Jul 29 2021, 5:47 PM · System administration, Lister
ardumont moved T3374: Ingest sourceforge repositories (origins of type git, svn, hg) from Backlog to in-progress on the System administration board.
Jul 29 2021, 5:45 PM · System administration, Archive coverage, Origin-SourceForge
ardumont added a project to T3374: Ingest sourceforge repositories (origins of type git, svn, hg): System administration.
Jul 29 2021, 5:45 PM · System administration, Archive coverage, Origin-SourceForge
ardumont renamed T3374: Ingest sourceforge repositories (origins of type git, svn, hg) from Ingest sourceforge repositories to Ingest sourceforge repositories (origins of type git, svn, hg).
Jul 29 2021, 5:45 PM · System administration, Archive coverage, Origin-SourceForge
ardumont added a comment to T3374: Ingest sourceforge repositories (origins of type git, svn, hg).

The 'hg' ingestion started now that the latest loader mercurial got deployed.

Jul 29 2021, 5:43 PM · System administration, Archive coverage, Origin-SourceForge
ardumont changed the status of T3448: production: Deploy swh.loader.mercurial v2.1.0, a subtask of T3418: Decide a consistent policy on having multiple archived objects for the same extid, from Open to Work in Progress.
Jul 29 2021, 5:40 PM · Storage manager, Mercurial loader
ardumont changed the status of T3448: production: Deploy swh.loader.mercurial v2.1.0 from Open to Work in Progress.
Jul 29 2021, 5:40 PM · System administration, Storage manager, Mercurial loader
ardumont added a comment to T3448: production: Deploy swh.loader.mercurial v2.1.0.

At the end of it all though, the final production check end-to-end for mercurial origin should go green.

Jul 29 2021, 5:40 PM · System administration, Storage manager, Mercurial loader
ardumont updated the task description for T3448: production: Deploy swh.loader.mercurial v2.1.0.
Jul 29 2021, 5:32 PM · System administration, Storage manager, Mercurial loader
vlorentz triaged T3449: pypi / package loader: handle dangling branches as Normal priority.
Jul 29 2021, 4:03 PM · Package Loader, Easy hack, PyPI loader
DanSeraf closed D6044: scanner: translate CoreSWHID instances before the request to the backend.
Jul 29 2021, 3:56 PM
DanSeraf committed rDTSCN98011a42261b: translate CoreSWHID instances before the request to the backend (authored by DanSeraf).
translate CoreSWHID instances before the request to the backend
Jul 29 2021, 3:56 PM
vlorentz moved T2431: Document how to export the graph edge dataset from sys-admin (docs/sysadm) to developers (docs/devel/) on the Documentation board.
Jul 29 2021, 3:54 PM · Documentation, Compressed graph service, Datasets
vlorentz moved T2431: Document how to export the graph edge dataset from archive-users (docs/user-guides/) to sys-admin (docs/sysadm) on the Documentation board.
Jul 29 2021, 3:54 PM · Documentation, Compressed graph service, Datasets
vlorentz added a comment to T2431: Document how to export the graph edge dataset.

It is now somewhat documented here: https://forge.softwareheritage.org/source/swh-environment/browse/master/docker/services/swh-graph/entrypoint.sh

Jul 29 2021, 3:54 PM · Documentation, Compressed graph service, Datasets
vlorentz closed T1936: integrate swh-graph into the docker environment as Resolved.
Jul 29 2021, 3:51 PM · Docker environment, Compressed graph service
vlorentz accepted D6044: scanner: translate CoreSWHID instances before the request to the backend.
Jul 29 2021, 3:50 PM
DanSeraf requested review of D6044: scanner: translate CoreSWHID instances before the request to the backend.
Jul 29 2021, 3:49 PM
DanSeraf closed D6027: swh-scanner: add 'auto' option as default policy.
Jul 29 2021, 2:22 PM
DanSeraf committed rDTSCNcd19bbbcee76: add 'auto' option as default policy (authored by DanSeraf).
add 'auto' option as default policy
Jul 29 2021, 2:22 PM
vlorentz added a comment to D6027: swh-scanner: add 'auto' option as default policy.

Thanks!

Jul 29 2021, 1:42 PM
swh-public-ci added a comment to D6027: swh-scanner: add 'auto' option as default policy.

Build is green

Jul 29 2021, 1:35 PM
DanSeraf updated the diff for D6027: swh-scanner: add 'auto' option as default policy.

policies description in CLI

Jul 29 2021, 1:32 PM
ardumont accepted D6031: Add a quick start section in the documentation and simplify the configuration file loading mechanism in the cli.

Like i said yesterday, lgtm.

Jul 29 2021, 1:25 PM
ardumont moved T3448: production: Deploy swh.loader.mercurial v2.1.0 from Backlog to Weekly backlog on the System administration board.
Jul 29 2021, 1:24 PM · System administration, Storage manager, Mercurial loader
ardumont moved T3394: cassandra - origin url hashing encoding issue from Backlog to done on the System administration board.
Jul 29 2021, 1:24 PM · System administration, Storage manager
ardumont moved T3397: Improve the documentation deployment to support multiple sphinx instances from Backlog to done on the System administration board.
Jul 29 2021, 1:24 PM · System administration, Documentation
ardumont changed the status of T3402: Deploy swh-counters v0.8.0 and backfill origins, a subtask of T3127: Compute and display distribution of origins by forge, from Wontfix to Resolved.
Jul 29 2021, 1:24 PM · Metrics/monitoring, Web app, Roadmap 2021, meta-task
ardumont changed the status of T3402: Deploy swh-counters v0.8.0 and backfill origins from Wontfix to Resolved.
Jul 29 2021, 1:24 PM · Counters, System administration, Metrics/monitoring
ardumont changed the status of T3445: Monitor hypervisors as well from Invalid to Resolved.
Jul 29 2021, 1:23 PM · System administration
ardumont moved T3086: Prepare disk replacement on granet from in-progress to done on the System administration board.
Jul 29 2021, 1:23 PM · System administration
ardumont moved T3061: swh-search: Deploy visit_types indexation in production from in-progress to done on the System administration board.
Jul 29 2021, 1:23 PM · System administration, Web app
ardumont moved T3159: Deploy swh-counters:v0.1.0 in staging from in-progress to done on the System administration board.
Jul 29 2021, 1:23 PM · Staging environment, System administration, Monitoring
ardumont moved T3115: Upgrade zfs on all servers from in-progress to done on the System administration board.
Jul 29 2021, 1:23 PM · System administration
ardumont moved T3060: Deploy swh-search v0.6.0 in **staging** from in-progress to done on the System administration board.
Jul 29 2021, 1:23 PM · System administration, Archive search
ardumont moved T3242: Decommission ClearlyDefined resources from in-progress to done on the System administration board.
Jul 29 2021, 1:23 PM · System administration
ardumont moved T3033: Replace first disk on storage1.staging from in-progress to done on the System administration board.
Jul 29 2021, 1:23 PM · System administration
ardumont moved T3310: Deploy sourceforge lister on staging from in-progress to done on the System administration board.
Jul 29 2021, 1:23 PM · System administration, Origin-SourceForge
ardumont moved T3041: [production] Provision enough space for the search ES cluster to ingest all intrinsic metadata from in-progress to done on the System administration board.
Jul 29 2021, 1:23 PM · System administration, Archive search
ardumont moved T3350: Deploy sourceforge lister in production from in-progress to done on the System administration board.
Jul 29 2021, 1:23 PM · Archive coverage, System administration, Origin-SourceForge
ardumont moved T2939: Replace out of order disks on db1.staging and storage1.staging from Weekly backlog to done on the System administration board.
Jul 29 2021, 1:23 PM · System administration
ardumont moved T3222: Monitor daily indexes are present on the log cluster and logs are correctly ingested from deployed/landed/monitoring to done on the System administration board.
Jul 29 2021, 1:22 PM · System administration
ardumont moved T2877: Investigate spurious deposit logs from deployed/landed/monitoring to done on the System administration board.
Jul 29 2021, 1:22 PM · System administration, Staging environment, SWORD deposit
ardumont moved T2927: Unstuck jenkins from deployed/landed/monitoring to done on the System administration board.
Jul 29 2021, 1:22 PM · System administration
ardumont moved T2908: sentry does not log tracebacks in the swh-deposit server from deployed/landed/monitoring to done on the System administration board.
Jul 29 2021, 1:22 PM · System administration, SWORD deposit
ardumont moved T2910: Sentry: Increase disk space from deployed/landed/monitoring to done on the System administration board.
Jul 29 2021, 1:22 PM · Sentry, System administration
ardumont moved T2866: Integrate former Uffizi server to the proxmox cluster from deployed/landed/monitoring to done on the System administration board.
Jul 29 2021, 1:22 PM · System administration
ardumont moved T2774: Fix vault end-to-end check from deployed/landed/monitoring to done on the System administration board.
Jul 29 2021, 1:22 PM · Vault, System administration, Monitoring
ardumont moved T2888: Elasticsearch cluster failure during a rolling restart from deployed/landed/monitoring to done on the System administration board.
Jul 29 2021, 1:21 PM · System administration
ardumont moved T3203: docs: Document the firewall installation and procedures from deployed/landed/monitoring to done on the System administration board.
Jul 29 2021, 1:21 PM · Documentation, System administration
ardumont moved T3447: staging: Deploy swh.loader.mercurial v2.1.0 from deployed/landed/monitoring to done on the System administration board.
Jul 29 2021, 1:21 PM · System administration, Storage manager, Mercurial loader
ardumont moved T3447: staging: Deploy swh.loader.mercurial v2.1.0 from in-progress to deployed/landed/monitoring on the System administration board.
Jul 29 2021, 1:21 PM · System administration, Storage manager, Mercurial loader
ardumont claimed T3448: production: Deploy swh.loader.mercurial v2.1.0.
Jul 29 2021, 1:21 PM · System administration, Storage manager, Mercurial loader
ardumont updated the task description for T3448: production: Deploy swh.loader.mercurial v2.1.0.
Jul 29 2021, 1:21 PM · System administration, Storage manager, Mercurial loader
ardumont triaged T3448: production: Deploy swh.loader.mercurial v2.1.0 as Unbreak Now! priority.
Jul 29 2021, 1:20 PM · System administration, Storage manager, Mercurial loader
ardumont closed T3447: staging: Deploy swh.loader.mercurial v2.1.0, a subtask of T3418: Decide a consistent policy on having multiple archived objects for the same extid, as Resolved.
Jul 29 2021, 1:18 PM · Storage manager, Mercurial loader
ardumont closed T3447: staging: Deploy swh.loader.mercurial v2.1.0 as Resolved.
Jul 29 2021, 1:18 PM · System administration, Storage manager, Mercurial loader
ardumont added a comment to T3447: staging: Deploy swh.loader.mercurial v2.1.0.

From the staging webapp, we identify a revision [1]

Jul 29 2021, 1:17 PM · System administration, Storage manager, Mercurial loader
vlorentz added inline comments to D6025: translator.py: Translate search query language to ES DSL.
Jul 29 2021, 12:49 PM
vlorentz added inline comments to D6025: translator.py: Translate search query language to ES DSL.
Jul 29 2021, 12:48 PM
ardumont updated the task description for T3447: staging: Deploy swh.loader.mercurial v2.1.0.
Jul 29 2021, 12:47 PM · System administration, Storage manager, Mercurial loader
ardumont changed the status of T3447: staging: Deploy swh.loader.mercurial v2.1.0, a subtask of T3418: Decide a consistent policy on having multiple archived objects for the same extid, from Open to Work in Progress.
Jul 29 2021, 12:32 PM · Storage manager, Mercurial loader
ardumont changed the status of T3447: staging: Deploy swh.loader.mercurial v2.1.0 from Open to Work in Progress.
Jul 29 2021, 12:32 PM · System administration, Storage manager, Mercurial loader
ardumont added a project to T3447: staging: Deploy swh.loader.mercurial v2.1.0: System administration.
Jul 29 2021, 12:32 PM · System administration, Storage manager, Mercurial loader
ardumont changed the status of T3418: Decide a consistent policy on having multiple archived objects for the same extid from Open to Work in Progress.

by the way ^

Jul 29 2021, 12:31 PM · Storage manager, Mercurial loader
ardumont triaged T3447: staging: Deploy swh.loader.mercurial v2.1.0 as Unbreak Now! priority.
Jul 29 2021, 12:30 PM · System administration, Storage manager, Mercurial loader
ardumont added a comment to T3418: Decide a consistent policy on having multiple archived objects for the same extid.

Shipped the following modules to solve that problem:

  • swh.model v2.7.0
  • swh.storage v0.35.0
  • swh.loader.mercurial v2.1.0
Jul 29 2021, 12:26 PM · Storage manager, Mercurial loader
ardumont closed D6036: Use versioned ExtIDs in loader mercurial implementation.
Jul 29 2021, 11:53 AM
ardumont committed rDLDHG9be124af2151: Use versioned ExtIDs in main loader mercurial implementation (authored by ardumont).
Use versioned ExtIDs in main loader mercurial implementation
Jul 29 2021, 11:53 AM
vlorentz closed D6030: server: Define make_app_from_configfile so it can be ran by gunicorn.
Jul 29 2021, 11:53 AM
vlorentz committed rDGRPH90d1109c5103: server: Define make_app_from_configfile so it can be ran by gunicorn (authored by vlorentz).
server: Define make_app_from_configfile so it can be ran by gunicorn
Jul 29 2021, 11:53 AM
ardumont accepted D6030: server: Define make_app_from_configfile so it can be ran by gunicorn.
Jul 29 2021, 11:52 AM
vlorentz accepted D6036: Use versioned ExtIDs in loader mercurial implementation.
Jul 29 2021, 11:26 AM
swh-public-ci added a comment to D6036: Use versioned ExtIDs in loader mercurial implementation.

Build is green

Jul 29 2021, 11:24 AM
ardumont updated the diff for D6036: Use versioned ExtIDs in loader mercurial implementation.

Improve testing

Jul 29 2021, 11:22 AM
vlorentz created P1106 hg extid version test working.
Jul 29 2021, 11:11 AM
swh-public-ci added a comment to D6025: translator.py: Translate search query language to ES DSL.

Build is green

Jul 29 2021, 11:00 AM
KShivendu updated the diff for D6025: translator.py: Translate search query language to ES DSL.

Rebase:

  • translator.py: Translate search query language to ES DSL
  • translator.py: Complete all the filters and add new tests
  • translator.py: Fix resource path for tox environment
  • translator: Changes suggested by vlorentz
  • translator.py: Update list of possible ql paths (suggested by vlorentz)
Jul 29 2021, 10:56 AM
sdruskat added a comment to T3078: Index CITATION.cff files.

Just to say I'm sorry I haven't made any progress here lately. Will get to it once CFF 1.2.0 is out the door. Has become a priority since GitHub and Zenodo now support it :).

Jul 29 2021, 10:53 AM · Intrinsic metadata, Easy hack
vlorentz added a comment to D6027: swh-scanner: add 'auto' option as default policy.

docstring for each policy

Jul 29 2021, 10:52 AM
vlorentz created P1105 hg extid version test.
Jul 29 2021, 10:48 AM
ardumont closed D6040: test: Make the child process wait longer so it gets actually killed.
Jul 29 2021, 10:43 AM
ardumont committed rDLDHGcae036cac707: test: Make the child process wait longer so it gets actually killed (authored by ardumont).
test: Make the child process wait longer so it gets actually killed
Jul 29 2021, 10:43 AM
vlorentz closed D6042: graph: don't run export+compression unless needed, and add documentation.
Jul 29 2021, 10:15 AM
vlorentz committed rDENV122eb0d250e6: graph: don't run export+compression unless needed, and add documentation (authored by vlorentz).
graph: don't run export+compression unless needed, and add documentation
Jul 29 2021, 10:15 AM
zack accepted D6027: swh-scanner: add 'auto' option as default policy.
Jul 29 2021, 10:01 AM
vlorentz committed rDSEAc34011b7c762: setup.py: Install swh_ql.so alongside .py files, so it can be seen by… (authored by vlorentz).
setup.py: Install swh_ql.so alongside .py files, so it can be seen by…
Jul 29 2021, 9:47 AM
vlorentz closed D6039: setup.py: Install swh_ql.so alongside .py files, so it can be seen by pkg_resources.
Jul 29 2021, 9:47 AM
KShivendu accepted D6039: setup.py: Install swh_ql.so alongside .py files, so it can be seen by pkg_resources.
Jul 29 2021, 9:44 AM

Jul 28 2021

ardumont added a comment to T3443: Deploy lister v1.5.0.

It's still ongoing:

Jul 28 2021, 6:27 PM · System administration, Lister
swh-public-ci added a comment to D6036: Use versioned ExtIDs in loader mercurial implementation.

Build is green

Jul 28 2021, 6:15 PM
ardumont updated the diff for D6036: Use versioned ExtIDs in loader mercurial implementation.

Ensure the written ExtIDs are written with version 1.

Jul 28 2021, 6:13 PM
ardumont added inline comments to D6040: test: Make the child process wait longer so it gets actually killed.
Jul 28 2021, 5:57 PM
vlorentz added inline comments to D6040: test: Make the child process wait longer so it gets actually killed.
Jul 28 2021, 5:56 PM
vlorentz added inline comments to D6040: test: Make the child process wait longer so it gets actually killed.
Jul 28 2021, 5:55 PM