Handle @ardumont comments
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Feed All Stories
All Stories
All Stories
Sep 2 2021
Sep 2 2021
anlambert updated the diff for D6170: origin_save: Filter out visit type with no scheduler load-* task type.
anlambert added a comment to D6170: origin_save: Filter out visit type with no scheduler load-* task type.
In D6170#159685, @ardumont wrote:Thanks.
I like how you were able to scrub out the scheduler mocking part as well, nice work!
That had become a mess to maintain so kudos.
I like how you were able to scrub out the scheduler mocking part as well, nice work!
That had become a mess to maintain so kudos.
anlambert updated subscribers of D6170: origin_save: Filter out visit type with no scheduler load-* task type.
In D6170#159680, @vlorentz wrote:Wouldn't a config option be simpler?
vlorentz added a comment to D6170: origin_save: Filter out visit type with no scheduler load-* task type.
Wouldn't a config option be simpler?
anlambert requested review of D6170: origin_save: Filter out visit type with no scheduler load-* task type.
ardumont committed rSPSITEa2706d97972d: common: Activate graph options for production cookers (authored by ardumont).
common: Activate graph options for production cookers
ardumont committed rSPSITE9d72822b1fc6: Enable vault cookers to access swh-graph if configuration requires it (authored by ardumont).
Enable vault cookers to access swh-graph if configuration requires it
looks good
I updated the task with a breakdown of the cost of getting each info.
vlorentz updated the task description for T3542: Decide what metadata we want to / can collect from GitHub.
vlorentz updated the task description for T3542: Decide what metadata we want to / can collect from GitHub.
vlorentz updated the task description for T3542: Decide what metadata we want to / can collect from GitHub.
In D5992#159578, @ardumont wrote:What do you think ?
yes, good idea. That'd simplify the setup both in docker and prod (as in nothing to do ;)
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSCHfa9d3286818b: Updated backport on buster-swh from debian/0.18.0-1_swh1 (unstable-swh) (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Updated backport on buster-swh from debian/0.18.0-1_swh1 (unstable-swh)
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSCHdb6bff1747ce: Merge tag 'debian/0.18.0-1_swh1' into debian/buster-swh (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Merge tag 'debian/0.18.0-1_swh1' into debian/buster-swh
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSCHb364c7ea44df: pristine-tar data for swh-scheduler_0.18.0.orig.tar.gz (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
pristine-tar data for swh-scheduler_0.18.0.orig.tar.gz
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSCH1a70fd612b29: Updated debian changelog for version 0.18.0 (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Updated debian changelog for version 0.18.0
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSCH66bf492bbb13: New upstream version 0.18.0 (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
New upstream version 0.18.0
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSCH22baf3f0201c: Update upstream source from tag 'debian/upstream/0.18.0' (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Update upstream source from tag 'debian/upstream/0.18.0'
ardumont committed rDSCHecc14007aa25: runner: Improve help message on the task types flag. (authored by ardumont).
runner: Improve help message on the task types flag.
ardumont committed rDSCH63fdda00f5f9: send-to-celery: Add more options to allow scheduling of edge cases (authored by ardumont).
send-to-celery: Add more options to allow scheduling of edge cases
swh-public-ci added a comment to D5818: send-to-celery: Add more options to allow scheduling of edge case origins.
Build is green
ardumont added a comment to D5818: send-to-celery: Add more options to allow scheduling of edge case origins.
There's a weird set of changes that seems to be mixed in to the new send-to-celery options, I'm not sure that was intended?
ardumont updated the diff for D5818: send-to-celery: Add more options to allow scheduling of edge case origins.
Only target the one commit for the diff
(marking this diff as "Changes Requested" so I get a notifications when you update it)
Sep 1 2021
Sep 1 2021
In T3544#69746, @olasd wrote:I can see a few alternatives to using git:// over tcp:
- Give our swh bot accounts SSH keys, and use that to clone from GitHub over ssh.
The dulwich HTTP(s) support is implemented on top of urllib(3?).
The scheduler is getting there.
We are now able to trigger a runner for that part:
- package python3-swh.search upgraded to version 0.11.4-2, the problem is fixed
- the new index is well created:
root@search0:/# curl -s http://search-esnode0:9200/_cat/indices\?v health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open origin-v0.11 HljzsdD9SmKI7-8ekB_q3Q 80 0 0 0 4.2kb 4.2kb green close origin HthJj42xT5uO7w3Aoxzppw 80 0 green close origin-v0.9.0 o7FiYJWnTkOViKiAdCXCuA 80 0 green open origin-v0.10.0 -fvf4hK9QDeN8qYTJBBlxQ 80 0 1981623 559384 2.3gb 2.3gb green close origin-backup-20210209-1736 P1CKjXW0QiWM5zlzX46-fg 80 0 green close origin-v0.5.0 SGplSaqPR_O9cPYU4ZsmdQ 80 0
- journal clients enabled and restarted
- the journal clients lags should recover in less than 12h
- waiting some time to estimate the duration with only one journal client per type
The send-to-celery part LGTM, thanks.
ardumont moved T3518: Enable vault cookers to access swh-graph from in-progress to code-review/await-feedback/pause on the System administration board.
The problem was fixed by rDSEA68347a5604c74150197f691593cbb05bdd34396f
thanks @olasd
Deployment of version v0.11.4 in staging:
On search0:
- puppet stopped
- stop and disable the journal clients and search backend
- update the swh-search configuration to use a origin-v0.11 index
root@search0:/etc/softwareheritage/search# diff -U2 /tmp/server.yml server.yml --- /tmp/server.yml 2021-09-01 13:42:29.347951302 +0000 +++ server.yml 2021-09-01 13:42:35.739953523 +0000 @@ -7,5 +7,5 @@ indexes: origin: - index: origin-v0.10.0 + index: origin-v0.11 read_alias: origin-read write_alias: origin-write
- update the journal-clients to use a group id swh.search.journal_client.[indexed|object]-v0.11
root@search0:/etc/softwareheritage/search# diff -U3 /tmp/journal_client_objects.yml journal_client_objects.yml --- /tmp/journal_client_objects.yml 2021-09-01 13:44:49.843999978 +0000 +++ journal_client_objects.yml 2021-09-01 13:45:03.972004852 +0000 @@ -5,7 +5,7 @@ journal: brokers: - journal0.internal.staging.swh.network - group_id: swh.search.journal_client-v0.10.0 + group_id: swh.search.journal_client-v0.11 prefix: swh.journal.objects object_types: - origin root@search0:/etc/softwareheritage/search# diff -U3 /tmp/journal_client_indexed.yml journal_client_indexed.yml --- /tmp/journal_client_indexed.yml 2021-09-01 13:44:44.847998252 +0000 +++ journal_client_indexed.yml 2021-09-01 13:44:57.020002454 +0000 @@ -5,7 +5,7 @@ journal: brokers: - journal0.internal.staging.swh.network - group_id: swh.search.journal_client.indexed-v0.10.0 + group_id: swh.search.journal_client.indexed-v0.11 prefix: swh.journal.indexed object_types: - origin_intrinsic_metadata
- perform a system upgrade, a reboot was not required
- enable and start swh-search backend
- An error occurs after the restart:
Sep 01 14:19:12 search0 python3[4066688]: 2021-09-01 14:19:12 [4066688] root:ERROR command 'cc' failed with exit status 1
Traceback (most recent call last):
File "/usr/lib/python3.7/distutils/unixccompiler.py", line 118, in _compile
extra_postargs)
File "/usr/lib/python3.7/distutils/ccompiler.py", line 909, in spawn
spawn(cmd, dry_run=self.dry_run)
File "/usr/lib/python3.7/distutils/spawn.py", line 36, in spawn
_spawn_posix(cmd, search_path, dry_run=dry_run)
File "/usr/lib/python3.7/distutils/spawn.py", line 159, in _spawn_posix
% (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'cc' failed with exit status 1Landed
I think it'd be best to have those tested during packaging.
Testing is always a good idea, looks good to me.
I don't think this is something that we should do by default, because the semantics of
a URI with no scheme are ambiguous in general. If the semantics of the nix
sources.json is that download URIs with no protocol have an implicit http scheme
added, then such url mangling should be implemented in the nix loader directly.
I think it'd be best to have those tested during packaging.
Dropping the subtask about the full packaging.
ardumont changed the status of T3518: Enable vault cookers to access swh-graph, a subtask of T3505: Make the git-bare cooker available to the staff and beta-testers in the production webapp, from Open to Work in Progress.
ardumont changed the status of T3518: Enable vault cookers to access swh-graph from Open to Work in Progress.
ardumont closed T3543: Debian package python3-swh.graph.client, a subtask of T3518: Enable vault cookers to access swh-graph, as Resolved.
Packaging ok:
Package python3-swh.graph.client built [1]
vlorentz updated the task description for T3542: Decide what metadata we want to / can collect from GitHub.
vlorentz updated the task description for T3542: Decide what metadata we want to / can collect from GitHub.
no and yes, respectively
buster:
swh-debian-build-stable ++ git branch ++ grep '*' ++ cut '-d ' -f2 + CURRENT_BRANCH=debian/buster-swh + '[' debian/buster-swh = debian/buster-swh ']' + gbp buildpackage --git-builder=sbuild --nolog --batch --no-clean-source --no-run-lintian --arch-all --source --force-orig-source --build-dep-resolver=aptitude '--extra-repository=deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main' '--extra-repository=deb http://deb.debian.org/debian/ buster-backports main' --build-failed-commands %SBUILD_SHELL gbp:info: Tarballs 'swh.graph_0.5.0.orig.tar.gz' not found at '/home/tony/debian/tarballs/' gbp:info: Exporting 'HEAD' to '/home/tony/debian/build-area/swh.graph-tmp' gbp:info: Moving '/home/tony/debian/build-area/swh.graph-tmp' to '/home/tony/debian/build-area/swh.graph-0.5.0' gbp:info: Performing the build dpkg-source: info: using source format '3.0 (quilt)' dpkg-source: info: building swh.graph using existing ./swh.graph_0.5.0.orig.tar.gz dpkg-source: warning: ignoring deletion of directory java/target dpkg-source: warning: ignoring deletion of file java/target/swh-graph-0.5.0.jar, use --include-removal to override dpkg-source: info: building swh.graph in swh.graph_0.5.0-1~swh1~bpo10+1.debian.tar.xz dpkg-source: info: building swh.graph in swh.graph_0.5.0-1~swh1~bpo10+1.dsc sbuild (Debian sbuild) 0.78.1 (09 February 2019) on localhost
- Add 'has debian packaging branch' tag to the repository
- Install hook to cascade the debian packaging build [1]
- Ensure necessary ci jobs is installed to build package (already there)
vsellier closed T3484: Fix the release builds for swh-search, a subtask of T3433: Deploy swh.search v0.10/v0.11, as Resolved.
The build is now fixed and the v0.11.4 version is ready to be deployed on the environments
Local build successful [1]
ardumont changed the status of T3543: Debian package python3-swh.graph.client from Open to Work in Progress.
ardumont changed the status of T3543: Debian package python3-swh.graph.client, a subtask of T3518: Enable vault cookers to access swh-graph, from Open to Work in Progress.
I don't think this is something that we should do by default, because the semantics of a URI with no scheme are ambiguous in general.
Please use less single-letter variable names in lister.py
do we need the "list of forks" if we keep the "fork of what"? I mean these are the 2 ends of the fork relation, right?
It means one of the members is different, but it's impossible to tell which.
jayeshv committed rDGQL3b7f93144be2: Initial commit. First version of the GraphQL API service (authored by jayeshv).
Initial commit. First version of the GraphQL API service
vsellier added a comment to D6139: cassandra: Add option to select (hopefully) more efficient batch insertion algos.
Test with 10 replayers with the 3 kind of algorithm:
- first interval: one-by-one
- second interval: concurremt
- third interval: batch:
Build is green
- Remove old debug logging and improve other's messages
- Rework ProvenanceStorageRabbitMQWorker to handle connection loss
- Remove old client/server storage based on swh.core.api.RPCClient
What do you think ?
ardumont added inline comments to D6138: package/utils: Handle downloads for urls with missing schema.
I've triggered back the build and now it's in a happy place.
In D5992#159553, @ardumont wrote:Also I just realized, as it's not completely ready afaik, don't land it yet though.
@anlambert What do you think of this?
Do we want for example to have a configuration (file) option to hide it and we toggle it later, when it's ready?
That'd avoid to let this diff hang.TIA
vsellier committed rDSNIP30b06ccd0294: grid5000/cassandra: fix statsd configuration of gunicorn services (authored by vsellier).
grid5000/cassandra: fix statsd configuration of gunicorn services
The monitoring icinga alerts have been deployed:
ardumont committed rSPSITEdb047bc1d886: swh-scheduler-journal-client: Delay the restart of failing service (authored by ardumont).
swh-scheduler-journal-client: Delay the restart of failing service
ardumont committed rSPSITE2434b83e98e9: Ensure icinga alerts are raised if scheduler journal client service is down (authored by ardumont).
Ensure icinga alerts are raised if scheduler journal client service is down
LGTM