Page MenuHomeSoftware Heritage
Feed All Stories

Apr 15 2021

swh-public-ci added a comment to D5541: Add end to end save code now check.

Build is green

Apr 15 2021, 7:09 PM
ardumont updated the diff for D5541: Add end to end save code now check.

Fix typos and add a missing status check

Apr 15 2021, 7:08 PM
ardumont updated the test plan for D5541: Add end to end save code now check.
Apr 15 2021, 6:57 PM
ardumont updated the test plan for D5541: Add end to end save code now check.
Apr 15 2021, 6:56 PM
ardumont updated the test plan for D5541: Add end to end save code now check.
Apr 15 2021, 6:54 PM
ardumont requested review of D5541: Add end to end save code now check.
Apr 15 2021, 6:53 PM
ardumont added a revision to T2117: Save Code Now: End to End monitoring: D5541: Add end to end save code now check.
Apr 15 2021, 6:52 PM · System administration, Monitoring, Roadmap 2021
vsellier closed T2939: Replace out of order disks on db1.staging and storage1.staging as Resolved.

closing this issue as the quota of disks replacement will be over after T3243.
The disk on db1 looks stable for the moment. It will be removed from the zfs pool in case of problems.

Apr 15 2021, 6:14 PM · System administration
ardumont added a comment to P1007 can't read randomized origin visit status because... <- insert reason here ;).

Related to T2117

Apr 15 2021, 6:02 PM
ardumont claimed T2117: Save Code Now: End to End monitoring.
Apr 15 2021, 6:01 PM · System administration, Monitoring, Roadmap 2021
ardumont closed T3084: Fast track save code now requests, a subtask of T3082: Improve Save Code Now handling, as Resolved.
Apr 15 2021, 6:00 PM · Save Code Now, meta-task, Roadmap 2021, Web app
ardumont closed T3084: Fast track save code now requests as Resolved.
Apr 15 2021, 6:00 PM · System administration, Web app
vsellier closed D5539: Outboarding of tg1999.
Apr 15 2021, 5:57 PM
vsellier committed rSPSITE73f28d682007: Outboarding of tg1999 (authored by vsellier).
Outboarding of tg1999
Apr 15 2021, 5:57 PM
seirl added a reviewer for D5540: docs: Update for new schema: zack.
Apr 15 2021, 5:55 PM
seirl requested review of D5540: docs: Update for new schema.
Apr 15 2021, 5:55 PM
ardumont accepted D5539: Outboarding of tg1999.
Apr 15 2021, 5:54 PM
ardumont edited P1007 can't read randomized origin visit status because... <- insert reason here ;).
Apr 15 2021, 5:54 PM
ardumont edited P1007 can't read randomized origin visit status because... <- insert reason here ;).
Apr 15 2021, 5:52 PM
ardumont updated the title for P1007 can't read randomized origin visit status because... <- insert reason here ;) from Is this expected? to can't read randomized origin visit status because... <- insert reason here ;).
Apr 15 2021, 5:51 PM
vsellier added a comment to T3242: Decommission ClearlyDefined resources.

The compression and the transfer of the dump from clearly-defined server to saam is in progress.

ssh clearly-defined.internal.staging.swh.network "pbzip2 -c -9 /srv/softwareheritage/clearlydefined/clearcode-dump-2021-01-05/clearcode_backup-20210105-2111.dump" | pv | cat > clearcode_backup-20210105-2111.dump.bz2

When it will be done, and if the compression ratio is good, the compressed archive will be added to the public annex in a dataset/clearly-defined directory

Apr 15 2021, 5:49 PM · System administration
vsellier updated the task description for T3242: Decommission ClearlyDefined resources.
Apr 15 2021, 5:43 PM · System administration
vsellier added a comment to T3242: Decommission ClearlyDefined resources.

According to https://intranet.softwareheritage.org/wiki/Outboarding:

  • tg19999 unix account disabled

*TG1999 removed from interns members in the forge, no tasks were assigned to the user

  • He was not on the #swh-team channel
  • He was not a member of the swh-team ml
  • VPN certificated revoked
root@louvre:/etc/openvpn/keys# ./easyrsa revoke tg1999
Apr 15 2021, 5:42 PM · System administration
ardumont updated the title for P1007 can't read randomized origin visit status because... <- insert reason here ;) from narmol? to Is this expected?.
Apr 15 2021, 5:39 PM
ardumont created P1007 can't read randomized origin visit status because... <- insert reason here ;).
Apr 15 2021, 5:35 PM
vsellier removed a member for Interns: TG1999.
Apr 15 2021, 5:32 PM
vsellier added a revision to T3242: Decommission ClearlyDefined resources: D5539: Outboarding of tg1999.
Apr 15 2021, 5:26 PM · System administration
vsellier requested review of D5539: Outboarding of tg1999.
Apr 15 2021, 5:26 PM
swh-public-ci added a comment to D5527: swh_model_data: add parents to test revision.

Build is green

Apr 15 2021, 4:39 PM
seirl closed D5527: swh_model_data: add parents to test revision.
Apr 15 2021, 4:37 PM
seirl committed rDMOD1f6b3b9d5b41: swh_model_data: add parents to test revision (authored by seirl).
swh_model_data: add parents to test revision
Apr 15 2021, 4:37 PM
seirl updated the diff for D5527: swh_model_data: add parents to test revision.

rebase

Apr 15 2021, 4:37 PM
vlorentz closed D5538: cassandra: Use prepared statements in extid_index_*.
Apr 15 2021, 4:23 PM
vlorentz committed rDSTOa1fc5fb7143d: cassandra: Use prepared statements in extid_index_* (authored by vlorentz).
cassandra: Use prepared statements in extid_index_*
Apr 15 2021, 4:23 PM
vlorentz added a comment to T3251: Count authors from revisions and releases.

don't forget to count committers too

Apr 15 2021, 4:23 PM · Monitoring, Web app
anlambert accepted D5538: cassandra: Use prepared statements in extid_index_*.
Apr 15 2021, 4:11 PM
vlorentz requested review of D5538: cassandra: Use prepared statements in extid_index_*.
Apr 15 2021, 4:02 PM
anlambert closed D5532: developer-setup: Add nodejs 12 install instructions.
Apr 15 2021, 3:46 PM
anlambert committed rDDOCc67b75fbfc48: developer-setup: Add nodejs 12 install instructions (authored by anlambert).
developer-setup: Add nodejs 12 install instructions
Apr 15 2021, 3:46 PM
anlambert updated the diff for D5532: developer-setup: Add nodejs 12 install instructions.

Rebase

Apr 15 2021, 3:45 PM
ardumont added a comment to T2912: Next generation archive counters.

Let's go for it, then. May you take this over?

Apr 15 2021, 3:37 PM · Roadmap 2021, System administration, Monitoring, Web app
rdicosmo added a comment to T2912: Next generation archive counters.

This kind of journal client will be necessary in any case if we want to extend the usage of the counters for other perimeters (metadata count, origin per forge, ...)

Apr 15 2021, 3:35 PM · Roadmap 2021, System administration, Monitoring, Web app
ardumont accepted D5532: developer-setup: Add nodejs 12 install instructions.
Apr 15 2021, 3:35 PM
ardumont added a comment to T3084: Fast track save code now requests.

I saw a parmap origin which got scheduled (la la la ;)

Apr 15 2021, 3:33 PM · System administration, Web app
rdicosmo added a comment to T3084: Fast track save code now requests.

Pushed, packaged, deployed.

scheduler runner continues happily to schedule existing tasks and some new task with priority

Apr 15 13:12:51 saatchi swh[234257]: INFO:swh.scheduler.celery_backend.runner:Grabbed 2084 tasks load-git
Apr 15 13:12:54 saatchi swh[234257]: INFO:swh.scheduler.cli.admin.runner:Scheduled 4128 tasks
Apr 15 13:14:06 saatchi swh[234257]: INFO:swh.scheduler.celery_backend.runner:Grabbed 1 tasks load-pypi
Apr 15 13:14:06 saatchi swh[234257]: INFO:swh.scheduler.celery_backend.runner:Grabbed 1 tasks load-git (priority)
...

That task got done almost immediately...
So there you go ;)

Apr 15 2021, 3:30 PM · System administration, Web app
swh-public-ci added a comment to D4629: [POC] OpenAPI and Django REST Framework to specify / implement API v2.

Build is green

Apr 15 2021, 3:23 PM
ardumont moved T3084: Fast track save code now requests from in-progress to deployed/landed/monitoring on the System administration board.
Apr 15 2021, 3:18 PM · System administration, Web app
ardumont added a comment to T3084: Fast track save code now requests.

scheduler runner continues happily to schedule existing tasks and some new task with priority

Apr 15 13:12:51 saatchi swh[234257]: INFO:swh.scheduler.celery_backend.runner:Grabbed 2084 tasks load-git
Apr 15 13:12:54 saatchi swh[234257]: INFO:swh.scheduler.cli.admin.runner:Scheduled 4128 tasks
Apr 15 13:14:06 saatchi swh[234257]: INFO:swh.scheduler.celery_backend.runner:Grabbed 1 tasks load-pypi
Apr 15 13:14:06 saatchi swh[234257]: INFO:swh.scheduler.celery_backend.runner:Grabbed 1 tasks load-git (priority)
...
Apr 15 2021, 3:18 PM · System administration, Web app
vlorentz placed T3018: Allow querying raw_extrinsic_metadata by hash in swh-storage up for grabs.
Apr 15 2021, 3:17 PM · Storage manager, Extrinsic metadata
vlorentz added a subtask for T3090: Make loaders not rely on the 'metadata' column of the 'revision' table: T2564: migrate existing revisions metadata extra_headers to actual extra_headers field.
Apr 15 2021, 3:15 PM · Core Loader
vlorentz added a parent task for T2564: migrate existing revisions metadata extra_headers to actual extra_headers field: T3090: Make loaders not rely on the 'metadata' column of the 'revision' table.
Apr 15 2021, 3:15 PM · System administration, Storage manager
vlorentz closed T3090: Make loaders not rely on the 'metadata' column of the 'revision' table as Resolved.
Apr 15 2021, 3:15 PM · Core Loader
vlorentz closed T3090: Make loaders not rely on the 'metadata' column of the 'revision' table, a subtask of T3089: Remove the 'metadata' column of the 'revision' table, as Resolved.
Apr 15 2021, 3:15 PM · Storage manager, Archive content
vlorentz closed T3138: Make loaders stop writing in the 'metadata' column of the 'revision' table, a subtask of T3090: Make loaders not rely on the 'metadata' column of the 'revision' table, as Resolved.
Apr 15 2021, 3:15 PM · Core Loader
vlorentz closed T3138: Make loaders stop writing in the 'metadata' column of the 'revision' table as Resolved.
Apr 15 2021, 3:15 PM · Core Loader
vlorentz closed T3139: Make loaders stop reading from the 'metadata' column of the 'revision' table, a subtask of T3138: Make loaders stop writing in the 'metadata' column of the 'revision' table, as Resolved.
Apr 15 2021, 3:15 PM · Core Loader
vlorentz closed T3139: Make loaders stop reading from the 'metadata' column of the 'revision' table as Resolved.
Apr 15 2021, 3:15 PM · Core Loader
vlorentz closed T3142: Make loaders write to the ExtId storage, a subtask of T3143: Migrate revision metadata to extid in the storage, as Resolved.
Apr 15 2021, 3:15 PM · System administration, Storage manager, Core Loader
vlorentz closed T3142: Make loaders write to the ExtId storage as Resolved.
Apr 15 2021, 3:15 PM · Core Loader
anlambert updated the diff for D4629: [POC] OpenAPI and Django REST Framework to specify / implement API v2.

Rebase

Apr 15 2021, 3:14 PM
vlorentz closed T3158: Document how deposits are loaded as Resolved.
Apr 15 2021, 3:11 PM · Documentation, SWORD deposit
vlorentz closed T3226: swh identify with type=snapshot shows dependency not installed error as Resolved.
Apr 15 2021, 3:11 PM · Data Model, SWH command line interface
vsellier changed the status of T3242: Decommission ClearlyDefined resources from Open to Work in Progress.

thanks, I will move the dump in a secure place where it will be backuped.

Apr 15 2021, 3:05 PM · System administration
ardumont closed D5535: tests: Complete checks on message with priority consumption.
Apr 15 2021, 3:04 PM
ardumont committed rDSCH974c0c2e0512: tests: Complete checks on message with priority consumption (authored by ardumont).
tests: Complete checks on message with priority consumption
Apr 15 2021, 3:04 PM
vlorentz closed D5536: Enable sphinx-build --color by default.

357a110ab80cea6c200e2aced24ff0d0339a445f

Apr 15 2021, 3:03 PM
vlorentz committed rDDOC357a110ab80c: Enable sphinx-build --color by default (authored by vlorentz).
Enable sphinx-build --color by default
Apr 15 2021, 3:03 PM
vsellier added a comment to T3243: Replace /dev/sdb and /dev/sdc on storage1.staging.

Email sent to the dsi to launch the replacement.

Apr 15 2021, 3:03 PM · System administration, Staging environment
vlorentz closed D5537: Disable colors when running sphinx-build.
Apr 15 2021, 3:02 PM
vlorentz committed rCJSWHa97d3fa87bca: Disable colors when running sphinx-build (authored by vlorentz).
Disable colors when running sphinx-build
Apr 15 2021, 3:02 PM
anlambert accepted D5535: tests: Complete checks on message with priority consumption.

Looks good to me.

Apr 15 2021, 3:00 PM
swh-public-ci added a comment to D5535: tests: Complete checks on message with priority consumption.

Build is green

Apr 15 2021, 3:00 PM
ardumont added a comment to D5535: tests: Complete checks on message with priority consumption.

Thanks for the typos ping ;)
fixed.

Apr 15 2021, 2:58 PM
ardumont updated the diff for D5535: tests: Complete checks on message with priority consumption.

Fix sentence typos

Apr 15 2021, 2:57 PM
swh-public-ci added a comment to D5535: tests: Complete checks on message with priority consumption.

Build is green

Apr 15 2021, 2:57 PM
ardumont updated the diff for D5535: tests: Complete checks on message with priority consumption.

Adapt according to suggestion

Apr 15 2021, 2:55 PM
ardumont added a comment to D5535: tests: Complete checks on message with priority consumption.

lgtm, but I don't understand what this does 🤷

Apr 15 2021, 2:49 PM
vsellier added a comment to T3243: Replace /dev/sdb and /dev/sdc on storage1.staging.

In preparation of the disk replacement, their leds must be activated to make the emplacement identifiable:

  • Ensure all the led are off
root@storage1:~# ls /dev/sd* | grep -e "[a-z]$" | xargs -n1 -t -i{} ledctl normal={} 
ledctl normal=/dev/sda 
ledctl normal=/dev/sdb 
ledctl normal=/dev/sdc 
ledctl normal=/dev/sdd 
ledctl normal=/dev/sde 
ledctl normal=/dev/sdf 
ledctl normal=/dev/sdg 
ledctl normal=/dev/sdh 
ledctl normal=/dev/sdi 
ledctl normal=/dev/sdj 
ledctl normal=/dev/sdk 
ledctl normal=/dev/sdl 
ledctl normal=/dev/sdm 
ledctl normal=/dev/sdn
  • light on
root@storage1:~# ledctl locate=/dev/sdb
root@storage1:~# ledctl locate=/dev/sdc
Apr 15 2021, 2:31 PM · System administration, Staging environment
anlambert closed D5533: developer-setup: Ensure lsb_release for dependencies installation.
Apr 15 2021, 2:25 PM
anlambert committed rDDOC4b153225b2b3: developer-setup: Ensure lsb_release for dependencies installation (authored by anlambert).
developer-setup: Ensure lsb_release for dependencies installation
Apr 15 2021, 2:25 PM
anlambert requested changes to D5535: tests: Complete checks on message with priority consumption.
Apr 15 2021, 2:21 PM
vlorentz added a comment to D5535: tests: Complete checks on message with priority consumption.

lgtm, but I don't understand what this does 🤷

Apr 15 2021, 2:20 PM
vlorentz accepted D5533: developer-setup: Ensure lsb_release for dependencies installation.
Apr 15 2021, 2:16 PM
vlorentz added a comment to D5529: Add test to ensure that an exception is raised if unique_key isn't implemented.

oh and actually, there is no reason for this test to be specific to the in-memory implementation. Add it to the regular test suite

Apr 15 2021, 2:09 PM
vlorentz requested changes to D5529: Add test to ensure that an exception is raised if unique_key isn't implemented.
Apr 15 2021, 2:09 PM
anlambert closed D5534: cypress/origin-save: Fix a flaky test.
Apr 15 2021, 1:49 PM
anlambert committed rDWAPPSb095564e54d4: cypress/origin-save: Fix a flaky test (authored by anlambert).
cypress/origin-save: Fix a flaky test
Apr 15 2021, 1:49 PM
anlambert accepted D5537: Disable colors when running sphinx-build.
Apr 15 2021, 1:43 PM
anlambert accepted D5536: Enable sphinx-build --color by default.
Apr 15 2021, 1:42 PM
vlorentz added reviewers for D5537: Disable colors when running sphinx-build: olasd, Reviewers.
Apr 15 2021, 1:39 PM
vlorentz requested review of D5537: Disable colors when running sphinx-build.
Apr 15 2021, 1:39 PM
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSCHe4b9ac5aaa8d: Merge tag 'debian/0.12.0-1_swh1' into debian/buster-swh (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Merge tag 'debian/0.12.0-1_swh1' into debian/buster-swh
Apr 15 2021, 1:38 PM
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSCH8a3a8f940b55: Updated backport on buster-swh from debian/0.12.0-1_swh1 (unstable-swh) (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Updated backport on buster-swh from debian/0.12.0-1_swh1 (unstable-swh)
Apr 15 2021, 1:38 PM
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSCH49541352d937: pristine-tar data for swh-scheduler_0.12.0.orig.tar.gz (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
pristine-tar data for swh-scheduler_0.12.0.orig.tar.gz
Apr 15 2021, 1:36 PM
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSCH38ef60a72a1d: Updated debian changelog for version 0.12.0 (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Updated debian changelog for version 0.12.0
Apr 15 2021, 1:36 PM
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSCH9db86c1d5478: Update upstream source from tag 'debian/upstream/0.12.0' (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Update upstream source from tag 'debian/upstream/0.12.0'
Apr 15 2021, 1:36 PM
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSCHa8b8fde29662: New upstream version 0.12.0 (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
New upstream version 0.12.0
Apr 15 2021, 1:36 PM
vlorentz requested review of D5536: Enable sphinx-build --color by default.
Apr 15 2021, 1:35 PM
ardumont closed D5526: conf/loader: Declare save code now queues to consume from.
Apr 15 2021, 1:32 PM
ardumont committed rDENV2f08ed2d4f4e: conf/loader: Declare save code now queues to consume from (authored by ardumont).
conf/loader: Declare save code now queues to consume from
Apr 15 2021, 1:32 PM