Query: Advanced Search

wip: Start swh-apps/update-dependencies job

README.md: Adapt docker instructions

	Include stories about projects I am a member of.

fix typo for fun and profit! ;)

couple of questions inline.

Fixes included in the diff landed [1].

Install `swh provenance origin from-journal` cli and tests

14:07:39 swh-scrubber@belvedere:5432=> select * from corrupt_object where id='swh:1:rel:a03d79c816c7ffebe3e8d90d7d395482075b086f';
+-[ RECORD 1 ]-----+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id               | swh:1:rel:a03d79c816c7ffebe3e8d90d7d395482075b086f                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| datastore        | 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| object           | \x88a46e616d65c41864656269616e2f302e32362e312d307562756e7475322e32a76d657373616765c44574616767696e67207061636b61676520707974686f6e2d676c616e63652d73746f72652076657273696f6e2064656269616e2f302e32362e312d307562756e7475322e320aa6746172676574c414e4c28d0c01ea1274e8e9b28f113572c36211d538ab7461726765745f74797065a87265766973696f6ea973796e746865746963c2a6617574686f7283a866756c6c6e616d65c429436f72657920427279616e74203c636f7265792e627279616e744063616e6f6e6963616c2e636f6d3ea46e616d65c40c436f72657920427279616e74a5656d61696cc41a636f7265792e627279616e744063616e6f6e6963616c2e636f6da46461746582a974696d657374616d7082a77365636f6e6473ce5cfe6f7bac6d6963726f7365636f6e647300ac6f66667365745f6279746573c4052d30343030a26964c414a03d79c816c7ffebe3e8d90d7d395482075b086f |
| first_occurrence | 2022-06-27 15:43:47.263775+00                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Looks good enough to start having a local jenkins, that will ease tampering with whatever we need to develop on it.
This without having to tamper directly with the main swh ci instance.
So thanks.

thx (and i guess that fixes the master build as well).

Drop self.conn shenanigans (tests no longer stuck but journal client tests are not happy)

I'm of a mind of just push it and check whether that's not some discrepancy with the master build somehow.
Master build is fine and local build as well.

Revert to original commit

Tentatively work around the rabbitmq tests build failures 5/n

Tentatively work around the rabbitmq tests build failures 4/n

Tentatively work around the rabbitmq initialization 3/n

Tentatively work around the rabbitmq initialization.

What the next steps here?

well, i'll trust the copyright date are ok according to the existing code ;)

scrubber: Set objstorage key to noop unused by scrubber services

including the right amount of commits

In T4361#87419, @ardumont wrote:

wip tryout [1] without those tests to ensure whether it's the rabbitmq fixture which hangs or not.

[1] D8023#209122

wip tryout [1] without those tests to ensure whether it's the rabbitmq fixture which hangs or not.

wip tryout (drop rabbitmq from fixture) to check if that still hangs or not

That also seem to incur a heavy load on the jenkins node thyssen [1] as lots of processes seem to run for that job [2].

11:28 <+swhbot> icinga PROBLEM: service load on thyssen.internal.softwareheritage.org is WARNING: WARNING - load average: 30.18, 22.54, 11.58

It's been done for a while now.
And there is a mechanism in place in swh.core.tarball to extend it further if need be.
Closing.

So taking a bit more look into this possible new lister, we'd end up with the following
possible outputs:

artifacts url which are mostly tarballs [1] and sometimes files [2]
dvcs repositories delegated to dedicated loader to ingestion: svn [3], hg [4], git [5] (out of guix manifest)
Other stuff can be ignored as we don't have anything relevant to ingest [6]

provenance: Increase build timeout

You should set that in the scrubber config in common.yaml rather than using a per-host override

yes, totally, i'll attend to that later in the day.

You should set that in the scrubber config in common.yaml rather than using a per-host override

Seems the job is well on its way:

07:59:35 swh-scrubber@belvedere:5432=> select now(), count(*) from corrupt_object ;
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2022-06-28 05:59:39.490737+00 | 76511 |
+-------------------------------+-------+
(1 row)

scrubber: Deactivate the unneeded objstorage configuration part

Amend commit message

Use proper repository to update diff

After discussing with olasd, turns out that the part declaring the objstorage is not currently needed so i'll try to do better.

lgtm, especially better if you attend to val's good suggestions ;)

Provision scrubber1 for checker services

Advanced Search
Use Results
Edit Query
Hide Query

Jun 30 2022

Jun 29 2022

Jun 28 2022

Jun 27 2022

Advanced SearchUse ResultsEdit QueryHide Query

Jun 30 2022

Jun 29 2022

Jun 28 2022

Jun 27 2022

Advanced Search
Use Results
Edit Query
Hide Query