Page MenuHomeSoftware Heritage
Feed All Stories

Nov 9 2021

vlorentz accepted D6590: add support for RCS keyword expansion over pserver protocol.
Nov 9 2021, 11:42 AM
vlorentz accepted D6571: Add support for a redis-based reporting for invalid mirrorred objects.

thx

Nov 9 2021, 11:41 AM
swh-public-ci added a comment to D6620: ra: Remove global variables..

Build is green

Nov 9 2021, 11:40 AM
vlorentz added 1 blocking reviewer(s) for D6620: ra: Remove global variables.: anlambert.

@anlambert Can you have a look too?

Nov 9 2021, 11:38 AM
vlorentz updated the diff for D6620: ra: Remove global variables..

add typeann + rebase

Nov 9 2021, 11:38 AM
swh-public-ci added a comment to D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.

Build is green

Nov 9 2021, 11:37 AM
ardumont updated the summary of D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.
Nov 9 2021, 11:36 AM
ardumont updated the diff for D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.

align docstring

Nov 9 2021, 11:36 AM
ardumont retitled D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted from SvnLoaderFromRemoteDump: Drop dump as soon as it's no longer required to SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.
Nov 9 2021, 11:35 AM
swh-public-ci added a comment to D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.

Build is green

Nov 9 2021, 11:35 AM
ardumont updated the diff for D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.

Fix tests (i forgot one adaptation in another test)

Nov 9 2021, 11:33 AM
vlorentz added inline comments to D6593: handle Attic-only RCS files over CVS pserver.
Nov 9 2021, 11:30 AM
Harbormaster failed remote builds in B24931: Diff 24056 for D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted!
Nov 9 2021, 11:24 AM
swh-public-ci added a comment to D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.

Build has FAILED

Nov 9 2021, 11:24 AM
ardumont updated the summary of D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.
Nov 9 2021, 11:23 AM
ardumont updated the diff for D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.

Improve implementations:

  • add tests
  • add types
Nov 9 2021, 11:22 AM
vsellier added a comment to T3705: Upgrade the ELK stack.

Everything looks good with logstash 1:7.15.1
The monitoring of the logstash errors is still working as previously:

root@logstash0:/usr/lib/nagios/plugins/swh# ./check_logstash_errors.sh 
OK - No errors detected

after closing the current system index:

root@logstash0:/usr/lib/nagios/plugins/swh# ./check_logstash_errors.sh 
CRITICAL - Logstash has detected some errors in outputs errors=9 non_retryable_errors=13
Nov 9 2021, 11:21 AM · System administration (Component upgrades)
ardumont updated the summary of D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.
Nov 9 2021, 11:21 AM
stsp added inline comments to D6593: handle Attic-only RCS files over CVS pserver.
Nov 9 2021, 11:19 AM
vsellier added a comment to T3705: Upgrade the ELK stack.

To upgrade kibana, upgrading the version looks enough. The migration is automatically done and all the configured elements are still available:

root@esnode1:~# curl -s http://10.168.100.61:9200/_cat/indices\?v=true\&s=index | grep kibana
health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana-event-log-7.15.1-000001 24Wb0rfUQuqab3Iody3Hrg   1   1          1            0     12.1kb            6kb  <-------- new index
green  open   .kibana-event-log-7.8.0-000001  6IjHICQVS2uX8qBekJLWsw   1   1          2            0     21.4kb         10.7kb
green  open   .kibana_2                       Oh9O6uB1R0-oNPbnhTM8kw   1   1       1928            3      1.5mb        788.4kb
green  open   .kibana_7.15.1_001              5fyk6NMUSE-3P6uhx-HSeg   1   1       1110           35      5.3mb          2.6mb  <-------- new index (automatically migrated from kibana_2)
green  open   .kibana_task_manager_1          vINZFVqCSJiDHHFMdYGwTA   1   1          5            0       32kb           16kb
green  open   .kibana_task_manager_7.15.1_001 pYeR_zFdTZO_jqxYS1DB9g   1   1         16          369      527kb        277.5kb  <-------- new index
root@esnode1:~# curl -s http://10.168.100.61:9200/_cat/aliases\?v=true\&s=index | grep kibana
alias                       index                           filter routing.index routing.search is_write_index
.kibana-event-log-7.15.1    .kibana-event-log-7.15.1-000001 -      -             -              true
.kibana-event-log-7.8.0     .kibana-event-log-7.8.0-000001  -      -             -              true
.kibana                     .kibana_7.15.1_001              -      -             -              -
.kibana_7.15.1              .kibana_7.15.1_001              -      -             -              -
.kibana_task_manager        .kibana_task_manager_7.15.1_001 -      -             -              -
.kibana_task_manager_7.15.1 .kibana_task_manager_7.15.1_001 -      -             -              -
Nov 9 2021, 11:03 AM · System administration (Component upgrades)
swh-public-ci added a comment to D6424: Perfect hashmap C implementation.

Build is green

Nov 9 2021, 10:49 AM
dachary updated the diff for D6424: Perfect hashmap C implementation.

Could you document the expected shard layout in RestructuredText format? This code is setting the (current version of the) layout "forever", so we should make sure to properly document it.

Nov 9 2021, 10:48 AM
ardumont requested review of D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.
Nov 9 2021, 10:42 AM
ardumont added a revision to T3719: loader svn fails to clean up properly disk when it crashes: D6622: SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted.
Nov 9 2021, 10:40 AM · SVN Loader
ardumont updated the task description for T3719: loader svn fails to clean up properly disk when it crashes.
Nov 9 2021, 10:38 AM · SVN Loader
ardumont renamed T3701: loading some svn origins while ignoring history sometimes raises from loading some svn origin while ignoring history sometimes raises to loading some svn origins while ignoring history sometimes raises.
Nov 9 2021, 10:21 AM · SVN Loader
ardumont closed T3701: loading some svn origins while ignoring history sometimes raises as Resolved.

Diffs landed, packaged and deployed.
Closing.

Nov 9 2021, 10:21 AM · SVN Loader
ardumont closed T3701: loading some svn origins while ignoring history sometimes raises, a subtask of T3694: Investigate svn loading failure, as Resolved.
Nov 9 2021, 10:21 AM · SVN Loader
ardumont committed rDLDSVN189dfd53009e: loader: Let log statement do the formatting (authored by ardumont).
loader: Let log statement do the formatting
Nov 9 2021, 10:08 AM
ardumont closed D6621: svn: Modify default loader to use the same as production task.
Nov 9 2021, 10:06 AM
ardumont committed rDLDSVNd1678c538293: svn: Modify default loader to use the same as production task (authored by ardumont).
svn: Modify default loader to use the same as production task
Nov 9 2021, 10:06 AM
ardumont added a comment to T1538: Add "forge" now.

Is the request for regular pulling?

Nov 9 2021, 10:00 AM · Add Forge Now , Roadmap 2022, meta-task, Roadmap 2021
ardumont closed T3694: Investigate svn loading failure as Resolved.
Nov 9 2021, 9:48 AM · SVN Loader
zack added a comment to T1538: Add "forge" now.

Is the request for regular pulling?
If yes, it should be added for visibility on the home page of the archive.
If no, it should be clear this is a one shot thing.

Nov 9 2021, 9:48 AM · Add Forge Now , Roadmap 2022, meta-task, Roadmap 2021
ardumont added a comment to T3694: Investigate svn loading failure.

@posenato Hello, i was gonna check this morning and ping you to let you know.

Nov 9 2021, 9:47 AM · SVN Loader

Nov 8 2021

moranegg added a comment to T3173: Provide access to deposit-clients to view dedicated moderation page.

Yannick hasn't answered yet. Will try again on the 15th.

Nov 8 2021, 10:39 PM · Monitoring, SWORD deposit, Web app
moranegg triaged T3720: [save request] submit openlb.net tarballs links to save code now as Normal priority.
Nov 8 2021, 10:39 PM · Easy hack, Ambassadors
moranegg added a comment to T3677: Separate origin-source-code and provenance-metadata in the deposit.

I'm not sure where we are at on this evolution.
Maybe we can discuss this next week.

Nov 8 2021, 10:36 PM · SWORD deposit
moranegg added a comment to T1538: Add "forge" now.

This is an example of a use-case in Open Science:
An institution has a forge and wishes to add the forge as a whole to SWH.

Nov 8 2021, 10:34 PM · Add Forge Now , Roadmap 2022, meta-task, Roadmap 2021
posenato added a comment to T3694: Investigate svn loading failure.

It has just finished saving the repository.
It worked perfectly.
Thank you a lot for your support.

Nov 8 2021, 7:03 PM · SVN Loader
ardumont created P1214 (An Untitled Masterwork).
Nov 8 2021, 6:57 PM
ardumont triaged T3719: loader svn fails to clean up properly disk when it crashes as Normal priority.
Nov 8 2021, 6:49 PM · SVN Loader
ardumont added a comment to T3694: Investigate svn loading failure.

All fixes have currently landed, got packaged in v0.7.3.
I've deployed it and triggered a run [1] for the origin (ongoing).

Nov 8 2021, 6:48 PM · SVN Loader
ardumont requested review of D6621: svn: Modify default loader to use the same as production task.
Nov 8 2021, 6:19 PM
ardumont added a revision to T3701: loading some svn origins while ignoring history sometimes raises: D6621: svn: Modify default loader to use the same as production task.
Nov 8 2021, 6:17 PM · SVN Loader
douardda added a watcher for Object storage: douardda.
Nov 8 2021, 5:51 PM
ardumont closed D6608: loader: Implement post_load to check for revision divergence.
Nov 8 2021, 5:26 PM
ardumont committed rDLDSVN9d93d8d0fa45: tests: Actually check the last visit is indeed a full visit (authored by ardumont).
tests: Actually check the last visit is indeed a full visit
Nov 8 2021, 5:26 PM
ardumont committed rDLDSVN64314742b54a: loader: Implement post_load to check for revision divergence (authored by anlambert).
loader: Implement post_load to check for revision divergence
Nov 8 2021, 5:26 PM
swh-public-ci added a comment to D6608: loader: Implement post_load to check for revision divergence.

Build is green

Nov 8 2021, 5:25 PM
ardumont updated the diff for D6608: loader: Implement post_load to check for revision divergence.
  • tests: Actually check the last visit is indeed a full visit
Nov 8 2021, 5:24 PM
swh-public-ci added a comment to D6608: loader: Implement post_load to check for revision divergence.

Build is green

Nov 8 2021, 5:19 PM
ardumont updated the diff for D6608: loader: Implement post_load to check for revision divergence.

Fix test no longer happy after the rebase introducing the post_load implementation

Nov 8 2021, 5:17 PM
vsellier added a comment to T3705: Upgrade the ELK stack.

The migration of ES can be performed with:

  1. elasticsearch migration
Nov 8 2021, 5:12 PM · System administration (Component upgrades)
ardumont added a comment to D6620: ra: Remove global variables..

There is one change in behavior: svn_special_path_non_link_data is reset at the same time as eol_style for a file. I assume it was an oversight not to do it before this diff

Nov 8 2021, 5:07 PM
ardumont accepted D6620: ra: Remove global variables..
Nov 8 2021, 4:57 PM
ardumont added a comment to D6620: ra: Remove global variables..

Can you please add types to the constructor parameter you updated?

Nov 8 2021, 4:57 PM
Harbormaster failed remote builds in B24921: Diff 24047 for D6608: loader: Implement post_load to check for revision divergence!
Nov 8 2021, 4:52 PM
swh-public-ci added a comment to D6608: loader: Implement post_load to check for revision divergence.

Build has FAILED

Nov 8 2021, 4:52 PM
ardumont updated the diff for D6608: loader: Implement post_load to check for revision divergence.

Rebase on latest master

Nov 8 2021, 4:50 PM
ardumont added a comment to T3338: Load the archived bitbucket mercurial repositories.

Remains 1 origin still ongoing... almost there...

Nov 8 2021, 4:44 PM · System administration, Mercurial loader
swh-public-ci added a comment to D6620: ra: Remove global variables..

Build is green

Nov 8 2021, 4:34 PM
vlorentz updated the summary of D6620: ra: Remove global variables..
Nov 8 2021, 4:33 PM
vlorentz updated the diff for D6620: ra: Remove global variables..

delete states recursively again

Nov 8 2021, 4:32 PM
vlorentz requested review of D6620: ra: Remove global variables..
Nov 8 2021, 4:23 PM
vsellier added a comment to T3705: Upgrade the ELK stack.

In order to validate the kibana upgrade, the kibana configuration can be copied locally with these commands:

Nov 8 2021, 4:21 PM · System administration (Component upgrades)
ardumont commandeered D6608: loader: Implement post_load to check for revision divergence.
Nov 8 2021, 4:18 PM
ardumont closed D6601: ra: Fix export of non link file with svn:special property set.
Nov 8 2021, 4:17 PM
ardumont committed rDLDSVN58a07c67d590: ra: Fix export of non link file with svn:special property set (authored by anlambert).
ra: Fix export of non link file with svn:special property set
Nov 8 2021, 4:17 PM
swh-public-ci added a comment to D6601: ra: Fix export of non link file with svn:special property set.

Build is green

Nov 8 2021, 4:16 PM
ardumont updated the diff for D6601: ra: Fix export of non link file with svn:special property set.

rebase

Nov 8 2021, 4:14 PM
ardumont commandeered D6601: ra: Fix export of non link file with svn:special property set.
Nov 8 2021, 4:14 PM
douardda closed D6538: Remove the RADOS backend.
Nov 8 2021, 4:07 PM
douardda committed rDOBJS646aa11297cc: Remove the RADOS backend (authored by douardda).
Remove the RADOS backend
Nov 8 2021, 4:07 PM
swh-public-ci added a comment to D6538: Remove the RADOS backend.

Build is green

Nov 8 2021, 4:06 PM
ardumont closed D6602: loader: Fix error when first revision number is not one.
Nov 8 2021, 4:06 PM
ardumont committed rDLDSVN51f4d54bdfd6: loader: Fix error when first revision number is not one (authored by anlambert).
loader: Fix error when first revision number is not one
Nov 8 2021, 4:06 PM
vlorentz added a comment to D6601: ra: Fix export of non link file with svn:special property set.

nevermind, I had an uncommitted change somewhere

Nov 8 2021, 4:05 PM
douardda updated the diff for D6538: Remove the RADOS backend.

rebase

Nov 8 2021, 4:04 PM
olasd added a comment to D6424: Perfect hashmap C implementation.

Some generic comments on the C side:

Nov 8 2021, 4:00 PM
swh-public-ci added a comment to D6571: Add support for a redis-based reporting for invalid mirrorred objects.

Build is green

Nov 8 2021, 3:52 PM
douardda updated the diff for D6571: Add support for a redis-based reporting for invalid mirrorred objects.

fix docstring rst syntax

Nov 8 2021, 3:44 PM
Harbormaster failed remote builds in B24912: Diff 24038 for D6571: Add support for a redis-based reporting for invalid mirrorred objects!
Nov 8 2021, 3:38 PM
swh-public-ci added a comment to D6571: Add support for a redis-based reporting for invalid mirrorred objects.

Build has FAILED

Nov 8 2021, 3:37 PM
swh-public-ci added a comment to D6570: Remove now useless revision date checker in fixer.

Build is green

Nov 8 2021, 3:35 PM
swh-public-ci added a comment to D6569: Add a --type option to 'swh storage replay'.

Build is green

Nov 8 2021, 3:35 PM
douardda updated the diff for D6571: Add support for a redis-based reporting for invalid mirrorred objects.

apply vlorentz' comments

Nov 8 2021, 3:29 PM
douardda added inline comments to D6571: Add support for a redis-based reporting for invalid mirrorred objects.
Nov 8 2021, 3:29 PM
douardda updated the diff for D6570: Remove now useless revision date checker in fixer.

rebase

Nov 8 2021, 3:27 PM
douardda updated the diff for D6569: Add a --type option to 'swh storage replay'.

Use a hardwritten list of accepted objects

Nov 8 2021, 3:26 PM
vsellier changed the status of T3705: Upgrade the ELK stack from Open to Work in Progress.

The preparation of the migration through the vagrant environment is in progress.

Nov 8 2021, 3:24 PM · System administration (Component upgrades)
vsellier added a comment to T3705: Upgrade the ELK stack.

Thanks for the info.
For the record, the entry point of the upgrade process: https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html

Nov 8 2021, 3:15 PM · System administration (Component upgrades)
swh-public-ci added a comment to D6602: loader: Fix error when first revision number is not one.

Build is green

Nov 8 2021, 3:02 PM
ardumont updated the diff for D6602: loader: Fix error when first revision number is not one.

Rebase

Nov 8 2021, 3:00 PM
ardumont commandeered D6602: loader: Fix error when first revision number is not one.
Nov 8 2021, 3:00 PM
zack updated the task description for T3717: Ingest opam instance https://coq.inria.fr/opam/released/.
Nov 8 2021, 2:41 PM · System administration, Archive coverage, Opam
zack added a project to T3717: Ingest opam instance https://coq.inria.fr/opam/released/: Archive coverage.
Nov 8 2021, 2:41 PM · System administration, Archive coverage, Opam
ardumont added a comment to D6601: ra: Fix export of non link file with svn:special property set.

I'm getting this when running the test on my machine, any idea why?
...

Nov 8 2021, 2:39 PM
vsellier triaged T3718: ingest pergamon apache logs as Normal priority.
Nov 8 2021, 2:35 PM · System administration
ardumont triaged T3717: Ingest opam instance https://coq.inria.fr/opam/released/ as Normal priority.
Nov 8 2021, 2:27 PM · System administration, Archive coverage, Opam