Page MenuHomeSoftware Heritage
Feed All Stories

Jul 30 2021

vlorentz requested review of D6049: Do not exclude falsy git objects from being added..
Jul 30 2021, 6:14 PM
vlorentz closed D6048: Prevent swh_ql.so from being discovered by sphinx-autodoc.
Jul 30 2021, 6:12 PM
vlorentz committed rDSEA5093dd70db21: Prevent swh_ql.so from being discovered by sphinx-autodoc (authored by vlorentz).
Prevent swh_ql.so from being discovered by sphinx-autodoc
Jul 30 2021, 6:12 PM
vlorentz requested review of D6048: Prevent swh_ql.so from being discovered by sphinx-autodoc.
Jul 30 2021, 5:23 PM
vlorentz closed D6047: Add stream_results_optional.
Jul 30 2021, 4:44 PM
vlorentz committed rDCORE3337e79642e0: Add stream_results_optional (authored by vlorentz).
Add stream_results_optional
Jul 30 2021, 4:44 PM
ardumont accepted D6047: Add stream_results_optional.
Jul 30 2021, 4:41 PM
swh-public-ci added a comment to D6047: Add stream_results_optional.

Build is green

Jul 30 2021, 4:13 PM
vlorentz updated the diff for D6047: Add stream_results_optional.

oops I didn't mean to commit this yet

Jul 30 2021, 4:12 PM
swh-public-ci added a comment to D6047: Add stream_results_optional.

Build is green

Jul 30 2021, 4:11 PM
vlorentz updated the diff for D6047: Add stream_results_optional.

add test_stream_results_kwarg

Jul 30 2021, 4:09 PM
ardumont added a comment to D6047: Add stream_results_optional.

looks sound to me, i'd add the 2 missing tests (which ensure we don't call stream_result* with page_token) since we are touching that part.

Jul 30 2021, 4:07 PM
ardumont added inline comments to D6047: Add stream_results_optional.
Jul 30 2021, 4:02 PM
vlorentz requested review of D6047: Add stream_results_optional.
Jul 30 2021, 3:59 PM
ardumont changed the status of T2345: Improve handling of recurrent loading tasks in scheduler from Open to Work in Progress.

(^ for a while ;)

Jul 30 2021, 3:55 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
ardumont added a comment to T2345: Improve handling of recurrent loading tasks in scheduler.

Status on this, after the recent refactoring we did with @olasd to simplify the actual
implementation (backend and journal client). There remains to:

Jul 30 2021, 3:54 PM · Sprint 2021 01, Archive coverage, Scheduling utilities
douardda added a comment to P1110 bad stream_results_optional.

ok then

return itertools.chain([res], stream_results(f, page_token = res.page_token, **kwargs))
Jul 30 2021, 3:44 PM
ardumont committed rDSCH4fa29fe128c2: journal_client: Update get_last_status docstring (authored by ardumont).
journal_client: Update get_last_status docstring
Jul 30 2021, 3:41 PM
ardumont committed rDSCH3b929d0bd9dc: journal_client: Refactor by inlining the update_position_offset (authored by ardumont).
journal_client: Refactor by inlining the update_position_offset
Jul 30 2021, 3:41 PM
vlorentz added a comment to P1110 bad stream_results_optional.

Because, if there is a single yield expression in the function's AST, Python parses the function as a generator, so it can only return generator objects None.

Jul 30 2021, 3:39 PM
douardda added a comment to P1110 bad stream_results_optional.

why not something like:

Jul 30 2021, 3:36 PM
vlorentz edited P1110 bad stream_results_optional.
Jul 30 2021, 3:33 PM
vlorentz updated the language for P1110 bad stream_results_optional from autodetect to python.
Jul 30 2021, 3:31 PM
vlorentz created P1110 bad stream_results_optional.
Jul 30 2021, 3:31 PM
vlorentz created P1109 stream_results_optional.
Jul 30 2021, 3:21 PM
KShivendu requested review of D6046: elasticsearch.py: Integrate query langauge translator.
Jul 30 2021, 3:17 PM
KShivendu closed D6025: translator.py: Translate search query language to ES DSL.
Jul 30 2021, 3:03 PM
KShivendu committed rDSEA3428e71a042b: translator.py: Translate search query language to ES DSL (authored by KShivendu).
translator.py: Translate search query language to ES DSL
Jul 30 2021, 3:03 PM
swh-public-ci added a comment to D6025: translator.py: Translate search query language to ES DSL.

Build is green

Jul 30 2021, 3:03 PM
KShivendu updated the diff for D6025: translator.py: Translate search query language to ES DSL.
  • Changes suggested by vlorentz
    • Improve keyword filter test using raw string
    • Improve swh_ql.so not found error message
Jul 30 2021, 2:59 PM
ardumont added inline comments to D6031: Add a quick start section in the documentation and simplify the configuration file loading mechanism in the cli.
Jul 30 2021, 2:51 PM
ardumont added a comment to T3338: Load the archived bitbucket mercurial repositories.
13:40:06 softwareheritage@belvedere:5432=> select now(), count(distinct url) from origin o inner join origin_visit ov on o.id=ov.origin where o.url like 'https://bitbucket.org/%' and ov.type='hg';
+-------------------------------+--------+
|              now              | count  |
+-------------------------------+--------+
| 2021-07-30 11:39:37.122152+00 | 253848 |
+-------------------------------+--------+
(1 row)
Jul 30 2021, 2:49 PM · System administration, Mercurial loader
ardumont claimed T3338: Load the archived bitbucket mercurial repositories.

(claiming i said ;)

Jul 30 2021, 2:42 PM · System administration, Mercurial loader
douardda triaged T3453: Refactor the backend to make it scale better as High priority.
Jul 30 2021, 2:21 PM · Provenance database
ardumont placed T3338: Load the archived bitbucket mercurial repositories up for grabs.

(Claiming the task to find it back more easily through my activity view.)

Jul 30 2021, 1:42 PM · System administration, Mercurial loader
ardumont moved T3338: Load the archived bitbucket mercurial repositories from in-progress to code-review/await-feedback/pause on the System administration board.
Jul 30 2021, 1:33 PM · System administration, Mercurial loader
ardumont added a comment to T3338: Load the archived bitbucket mercurial repositories.

Started in the same tmux session [2] as the sourceforge ingestion [1]

Jul 30 2021, 1:33 PM · System administration, Mercurial loader
vlorentz added a project to T3452: Replication lag between the dbs should raise icinga alerts: Monitoring.
Jul 30 2021, 1:32 PM · Monitoring, System administration
ardumont changed the status of T3338: Load the archived bitbucket mercurial repositories from Open to Work in Progress.
Jul 30 2021, 1:04 PM · System administration, Mercurial loader
ardumont added a comment to T3338: Load the archived bitbucket mercurial repositories.

It would probably make sense to set up a new worker instance for this to avoid interfering with the regular loading.

Jul 30 2021, 1:04 PM · System administration, Mercurial loader
ardumont closed T3337: Smoke test ingestion of bitbucket repositories with latest loader mercurial, a subtask of T3338: Load the archived bitbucket mercurial repositories, as Resolved.
Jul 30 2021, 1:00 PM · System administration, Mercurial loader
ardumont closed T3337: Smoke test ingestion of bitbucket repositories with latest loader mercurial as Resolved.
Jul 30 2021, 1:00 PM · System administration, Mercurial loader
ardumont added a comment to T3337: Smoke test ingestion of bitbucket repositories with latest loader mercurial.

Smoke test it with local bitbucket repositories

that's next.

Jul 30 2021, 12:59 PM · System administration, Mercurial loader
ardumont created P1108 (production) running loader mercurial on local bitbucket repositories is fine as well (one run eventful -> another run uneventful).
Jul 30 2021, 12:32 PM
ardumont claimed T3337: Smoke test ingestion of bitbucket repositories with latest loader mercurial.
Jul 30 2021, 12:22 PM · System administration, Mercurial loader
ardumont changed the status of T3337: Smoke test ingestion of bitbucket repositories with latest loader mercurial from Open to Work in Progress.
Jul 30 2021, 12:16 PM · System administration, Mercurial loader
ardumont changed the status of T3337: Smoke test ingestion of bitbucket repositories with latest loader mercurial, a subtask of T3338: Load the archived bitbucket mercurial repositories, from Open to Work in Progress.
Jul 30 2021, 12:16 PM · System administration, Mercurial loader
ardumont added a comment to T3337: Smoke test ingestion of bitbucket repositories with latest loader mercurial.

Smoke test it with remote repositories.

Jul 30 2021, 12:16 PM · System administration, Mercurial loader
ardumont created P1107 (production) running loader mercurial on remote origins is fine.
Jul 30 2021, 12:15 PM
vlorentz closed D6045: converters: Preserve GPG signatures on releases.
Jul 30 2021, 12:09 PM
vlorentz committed rDLDGc67ab0269ef2: converters: Preserve GPG signatures on releases (authored by vlorentz).
converters: Preserve GPG signatures on releases
Jul 30 2021, 12:09 PM
vlorentz accepted D6025: translator.py: Translate search query language to ES DSL.
Jul 30 2021, 12:08 PM
vlorentz added inline comments to D6025: translator.py: Translate search query language to ES DSL.
Jul 30 2021, 12:08 PM
aeviso added inline comments to D6031: Add a quick start section in the documentation and simplify the configuration file loading mechanism in the cli.
Jul 30 2021, 11:49 AM
vlorentz added a comment to D6004: misc/coverage: Revamp and improve archive coverage widget.

Below is how it looks so far:

Jul 30 2021, 11:39 AM
ardumont accepted D6045: converters: Preserve GPG signatures on releases.

lgtm

Jul 30 2021, 11:31 AM
swh-public-ci added a comment to D6025: translator.py: Translate search query language to ES DSL.

Build is green

Jul 30 2021, 11:30 AM
aeviso accepted D6015: Use stored SQL functions for content_find_{all,one}() and merge Provenance*DB classes in a single ProvenanceDB.
Jul 30 2021, 11:30 AM
KShivendu updated the diff for D6025: translator.py: Translate search query language to ES DSL.

translator.py: Mention number of children while throwing exception in _traverse

Jul 30 2021, 11:27 AM
swh-public-ci added a comment to D6026: Add test for origin-revision layer.

Build is green

Jul 30 2021, 11:26 AM
ardumont moved T3444: 26/07/2021: Unstuck infrastructure outage then post-mortem from in-progress to code-review/await-feedback/pause on the System administration board.
Jul 30 2021, 11:22 AM · System administration
ardumont moved T3446: Restart scheduling regularly origins with relevant scheduling policies from in-progress to code-review/await-feedback/pause on the System administration board.
Jul 30 2021, 11:22 AM · System administration
ardumont moved T3374: Ingest sourceforge repositories (origins of type git, svn, hg) from in-progress to code-review/await-feedback/pause on the System administration board.
Jul 30 2021, 11:22 AM · System administration, Archive coverage, Origin-SourceForge
ardumont moved T3450: 404 error when visiting a successfully archived repository from in-progress to code-review/await-feedback/pause on the System administration board.
Jul 30 2021, 11:22 AM · Storage manager, System administration
aeviso updated the diff for D6026: Add test for origin-revision layer.

rebase

Jul 30 2021, 11:19 AM
ardumont added a comment to T3450: 404 error when visiting a successfully archived repository.

Thanks for the heads up @ both of you.

Jul 30 2021, 11:14 AM · Storage manager, System administration
ardumont triaged T3452: Replication lag between the dbs should raise icinga alerts as High priority.
Jul 30 2021, 11:13 AM · Monitoring, System administration
vlorentz added a project to T3450: 404 error when visiting a successfully archived repository: Storage manager.
Jul 30 2021, 11:07 AM · Storage manager, System administration
vlorentz added a comment to T3450: 404 error when visiting a successfully archived repository.

@ardumont noticed the replication was blocked, but our automated monitoring didn't alert us. He unblocked the replication, so your code should appear in the next hours.

Jul 30 2021, 11:07 AM · Storage manager, System administration
vlorentz removed a revision from T75: Check integrity of directories, revisions, and releases: D6045: converters: Preserve GPG signatures on releases.
Jul 30 2021, 10:59 AM · Archive content, Restricted Project
vlorentz edited tasks for D6045: converters: Preserve GPG signatures on releases, added: T3135: Improve integrity of ingested content; removed T75: Check integrity of directories, revisions, and releases.
Jul 30 2021, 10:59 AM
vlorentz added a revision to T3135: Improve integrity of ingested content: D6045: converters: Preserve GPG signatures on releases.
Jul 30 2021, 10:59 AM · Storage manager, Roadmap 2021, meta-task
vlorentz added a revision to T75: Check integrity of directories, revisions, and releases: D6045: converters: Preserve GPG signatures on releases.
Jul 30 2021, 10:58 AM · Archive content, Restricted Project
vlorentz added a task to D6045: converters: Preserve GPG signatures on releases: T75: Check integrity of directories, revisions, and releases.
Jul 30 2021, 10:58 AM
vlorentz added a comment to T75: Check integrity of directories, revisions, and releases.

Dulwich 0.19.10 (released in january 2019) changed the way they handle signatures on annotated tags, so we silently drop all signatures since we started using it (probably whenever we upgraded loaders to Buster).

Jul 30 2021, 10:58 AM · Archive content, Restricted Project
vlorentz requested review of D6045: converters: Preserve GPG signatures on releases.
Jul 30 2021, 10:57 AM
ardumont changed the status of T3450: 404 error when visiting a successfully archived repository from Open to Work in Progress.
Jul 30 2021, 10:40 AM · Storage manager, System administration
ardumont updated the task description for T3374: Ingest sourceforge repositories (origins of type git, svn, hg).
Jul 30 2021, 10:25 AM · System administration, Archive coverage, Origin-SourceForge
vlorentz triaged T3451: Convert the refresh-savecodenow-statuses cron to a systemd timer as Low priority.
Jul 30 2021, 10:25 AM · Web app, System administration
vlorentz added a project to T1510: Have a look at openAPI and decide whether we want to follow these specs: Web app.
Jul 30 2021, 10:16 AM · Web app
vlorentz added a project to T3282: Add support for "uninterpreted upstream object" in SWH model and storage: Data Model.
Jul 30 2021, 10:16 AM · Data Model
vlorentz added a project to T3112: Provenance index for the full archive: Provenance database.
Jul 30 2021, 10:15 AM · Roadmap 2022, Provenance database, Roadmap 2021, meta-task
vlorentz added a project to T2193: Add provenance feature: Provenance database.
Jul 30 2021, 10:15 AM · Provenance database, Roadmap 2020
vlorentz added a project to T3431: Implement a MongoDB backend for SWH-provenance : Provenance database.
Jul 30 2021, 10:15 AM · Provenance database
vlorentz created Provenance database.
Jul 30 2021, 10:14 AM
vlorentz triaged T3450: 404 error when visiting a successfully archived repository as High priority.

This may simply be replication lag between the loader's and the frontend's databases.

Jul 30 2021, 10:12 AM · Storage manager, System administration
ardumont added a comment to T3446: Restart scheduling regularly origins with relevant scheduling policies.

Heads up, this is running slightly different now.

Jul 30 2021, 9:59 AM · System administration
mdidas created T3450: 404 error when visiting a successfully archived repository.
Jul 30 2021, 9:59 AM · Storage manager, System administration
ardumont committed rSPSITEa49123e9449c: worker17: Activate mercurial origins to ingest for oneshot loader (authored by ardumont).
worker17: Activate mercurial origins to ingest for oneshot loader
Jul 30 2021, 9:44 AM

Jul 29 2021

vlorentz added inline comments to D6025: translator.py: Translate search query language to ES DSL.
Jul 29 2021, 9:12 PM
swh-public-ci added a comment to D6025: translator.py: Translate search query language to ES DSL.

Build is green

Jul 29 2021, 8:55 PM
KShivendu updated the diff for D6025: translator.py: Translate search query language to ES DSL.
  • translator.py: Mention filter category and name in exception
  • Squash
Jul 29 2021, 8:52 PM
ardumont renamed T3337: Smoke test ingestion of bitbucket repositories with latest loader mercurial from Deploy swh.loader.mercurial 1.0 in production to Smoke test ingestion of bitbucket repositories with latest loader mercurial.
Jul 29 2021, 6:36 PM · System administration, Mercurial loader
ardumont added a comment to T3336: Deploy swh.loader.mercurial 2.1 in staging.

For history purpose readabillty, this must bev2.1 git-patched version (not a release per say).
A more recent version release which is a tag v2.1.0 [1] has been done built with the work
solving the extid version inconsistency issue @olasd started.

Jul 29 2021, 6:33 PM · System administration, Mercurial loader
ardumont added a subtask for T3338: Load the archived bitbucket mercurial repositories: T3418: Decide a consistent policy on having multiple archived objects for the same extid.
Jul 29 2021, 6:30 PM · System administration, Mercurial loader
ardumont added a parent task for T3418: Decide a consistent policy on having multiple archived objects for the same extid: T3338: Load the archived bitbucket mercurial repositories.
Jul 29 2021, 6:30 PM · Storage manager, Mercurial loader
ardumont moved T3338: Load the archived bitbucket mercurial repositories from Backlog to Weekly backlog on the System administration board.

Latest mercurial loader v2.1 deployed [1] [2]
We should be able to continue with this now.

Jul 29 2021, 6:29 PM · System administration, Mercurial loader
ardumont closed T3418: Decide a consistent policy on having multiple archived objects for the same extid as Resolved.
Jul 29 2021, 6:21 PM · Storage manager, Mercurial loader
ardumont moved T3448: production: Deploy swh.loader.mercurial v2.1.0 from deployed/landed/monitoring to done on the System administration board.
Jul 29 2021, 6:21 PM · System administration, Storage manager, Mercurial loader
ardumont moved T3448: production: Deploy swh.loader.mercurial v2.1.0 from in-progress to deployed/landed/monitoring on the System administration board.
Jul 29 2021, 6:21 PM · System administration, Storage manager, Mercurial loader