Everything is released correctly and deployed on staging
Feed Advanced Search
Apr 9 2021
Apr 9 2021
vsellier closed T3165: Generate historical data from the new counters series, a subtask of T2912: Next generation archive counters, as Resolved.
vsellier closed T3215: Deploy the new counters in staging, a subtask of T2912: Next generation archive counters, as Resolved.
I finally found why the graphs looks weird : https://forge.softwareheritage.org/source/swh-web/browse/master/swh/web/misc/urls.py$31
With a dirty patch on the server, it's way better:
vlorentz added a comment to T3215: Deploy the new counters in staging.
ardumont added a comment to T3215: Deploy the new counters in staging.
\o/
vsellier added a comment to T3215: Deploy the new counters in staging.
The pipeline is deployed in staging.
It's working but it seems the graphs need some initial values in staging to make the rendering correctly:
Apr 8 2021
Apr 8 2021
vlorentz closed T2540: support the loading of metadata-only deposits in the metadata storage, a subtask of T3128: Improve deposit integration, management and display, as Resolved.
Apr 7 2021
Apr 7 2021
vsellier reopened T3165: Generate historical data from the new counters series, a subtask of T2912: Next generation archive counters, as Work in Progress.
vsellier reopened T3165: Generate historical data from the new counters series as "Work in Progress".
Reopening as the release is not working on the stable branch
vsellier changed the status of T3215: Deploy the new counters in staging from Open to Work in Progress.
vsellier closed T3165: Generate historical data from the new counters series, a subtask of T2912: Next generation archive counters, as Resolved.
Apr 6 2021
Apr 6 2021
Apr 5 2021
Apr 5 2021
rdicosmo updated subscribers of T3128: Improve deposit integration, management and display.
rdicosmo renamed T3128: Improve deposit integration, management and display from Improve deposit management and display to Improve deposit integration, management and display.
Apr 1 2021
Apr 1 2021
vsellier closed T3190: counters: Error during directory topic ingestion, a subtask of T2912: Next generation archive counters, as Resolved.
vsellier added a comment to T3190: counters: Error during directory topic ingestion.
vlorentz added a comment to T3190: counters: Error during directory topic ingestion.
@ardumont the linked code is for producer, the bug is in consumers
ardumont added a comment to T3190: counters: Error during directory topic ingestion.
An improvment of the journal client is necessary to add the support of this configuration like for the producer:
vsellier added a comment to T3190: counters: Error during directory topic ingestion.
It seems the problem is not present anymore with a higher max message size ('500 * 1024 * 1024').
vlorentz added a comment to T3190: counters: Error during directory topic ingestion.
What version of zstd is installed on the system? that bug was introduced in v1.4.5 (https://github.com/facebook/zstd/commit/718f00ff6fe42db7e6ba09a7f7992b3e85283f77), and fixed in v1.4.7 (https://github.com/facebook/zstd/pull/2272/commits/1302f8d67691356b2f0aec8c62c1a7af2886a7cc)
vlorentz added a comment to T3190: counters: Error during directory topic ingestion.
This looks like https://github.com/edenhill/librdkafka/issues/2672 , which is caused by a bug in zstd: https://github.com/facebook/zstd/issues/2222
vsellier added a comment to T3190: counters: Error during directory topic ingestion.
for the record, increasing the property message.max.bytes to 100 * 1024 * 1024 in the consumer configuration is not solving the problem
vsellier added a comment to T3190: counters: Error during directory topic ingestion.
The same problem occured during the poc, theses messages were ignored by using this consumer configuration "errors.tolerance": 'all' [1].
I will try to find if there is a more elegant way to deal with this issue ;)
vsellier updated the task description for T3190: counters: Error during directory topic ingestion.
vsellier changed the status of T3190: counters: Error during directory topic ingestion from Open to Work in Progress.
Mar 31 2021
Mar 31 2021
Mar 26 2021
Mar 26 2021
vsellier added a comment to T3165: Generate historical data from the new counters series.
The final counters architecture looks like this with this improvment:
ardumont added a comment to T3165: Generate historical data from the new counters series.
Great idea.
vsellier added a comment to T3165: Generate historical data from the new counters series.
An improvment idea came to me during the refactoring, the script can be splitted and integrated in the 'swh-counters' codebase.
Mar 25 2021
Mar 25 2021
node counters1.internal.softwareheritage.org deployed by terraform. The inventory section is created accordingly[1].
The journal_client is running.
vsellier closed T3175: Prepare production environment, a subtask of T2912: Next generation archive counters, as Resolved.
vsellier added a revision to T3175: Prepare production environment: D5338: counters: Declare production node.
vsellier changed the status of T3165: Generate historical data from the new counters series, a subtask of T2912: Next generation archive counters, from Open to Work in Progress.
vsellier changed the status of T3165: Generate historical data from the new counters series from Open to Work in Progress.
vsellier closed T3164: Expose counters in prometheus format, a subtask of T2912: Next generation archive counters, as Resolved.
moranegg triaged T3174: Filter deposit-admin view by deposit client on admin (moderation) page as Normal priority.
moranegg triaged T3173: Provide access to deposit-clients to view dedicated moderation page as Normal priority.
vsellier added a comment to T3164: Expose counters in prometheus format.
The counters are now exposed throught a /metrics enpoint and ingested by prometheus.
They are well tagged per environment so we will be able to isolate the counters for each one:
Mar 24 2021
Mar 24 2021
vsellier added a comment to T3164: Expose counters in prometheus format.
The current serie were the counters are stored is named sql_swh_archive_object_count, the serie for swh-counters could be swh_archive_object_count
vsellier changed the status of T3164: Expose counters in prometheus format, a subtask of T2912: Next generation archive counters, from Open to Work in Progress.
vsellier changed the status of T3164: Expose counters in prometheus format from Open to Work in Progress.
Mar 22 2021
Mar 22 2021
vsellier removed a project from T3165: Generate historical data from the new counters series: Web app.
vsellier closed T3159: Deploy swh-counters:v0.1.0 in staging, a subtask of T2912: Next generation archive counters, as Resolved.
A new vm counters0.internal.staging.swh.network is deployed and hosting redis, swh-counters and its journal-client.
The lag in staging will be recovered in a couple of hours.
vsellier added a revision to T3159: Deploy swh-counters:v0.1.0 in staging: D5297: staging: Add counters0 vm.
Mar 19 2021
Mar 19 2021
vsellier moved T3159: Deploy swh-counters:v0.1.0 in staging from Backlog to in-progress on the System administration board.
vsellier changed the status of T3159: Deploy swh-counters:v0.1.0 in staging from Open to Work in Progress.
Mar 17 2021
Mar 17 2021
vsellier closed T3147: Package swh-counters module as a debian package, a subtask of T2912: Next generation archive counters, as Resolved.
vsellier changed the status of T3147: Package swh-counters module as a debian package, a subtask of T2912: Next generation archive counters, from Open to Work in Progress.
vsellier closed T3146: Add pytest-redis package on the swh repository, a subtask of T2912: Next generation archive counters, as Resolved.
Mar 15 2021
Mar 15 2021
rdicosmo moved T2912: Next generation archive counters from Backlog to Work in progress on the Roadmap 2021 board.
vsellier added a revision to T2912: Next generation archive counters: D5253: Implement remote service.
vlorentz triaged T3129: Reliable monitoring of services: for users and for admins as Normal priority.
Mar 14 2021
Mar 14 2021
Mar 12 2021
Mar 12 2021
vsellier added a revision to T2912: Next generation archive counters: D5236: Add redis to the base build image.
vsellier added a revision to T2912: Next generation archive counters: D5232: Implement counters pipeline.
Mar 11 2021
Mar 11 2021
Mar 10 2021
Mar 10 2021
vsellier added a revision to T2912: Next generation archive counters: D5229: swh-counters: Implement the cli skeleton.
Mar 5 2021
Mar 5 2021
vsellier added a comment to T2912: Next generation archive counters.
- Repository created : https://forge.softwareheritage.org/source/swh-counters/
- Jenkins jobs configured : https://jenkins.softwareheritage.org/job/DCNT/
vsellier changed the status of T2912: Next generation archive counters from Open to Work in Progress.
Let's start the subject ;)
Feb 18 2021
Feb 18 2021
vsellier moved T2774: Fix vault end-to-end check from in-progress to deployed/landed/monitoring on the System administration board.
vsellier added a comment to T2774: Fix vault end-to-end check.
Thanks @anlambert, the monitoring comes back to green
Feb 17 2021
Feb 17 2021
ardumont updated subscribers of T2774: Fix vault end-to-end check.
@anlambert agrees with the previous hypothesis ^ and is working on a fix
vsellier added a comment to T2774: Fix vault end-to-end check.
With tcpdump, it seems swh-web don't add the headers to don't cache the response in case of a 404:
GET /api/1/vault/directory/a317baff051f68e83557d51e59539dac2ff55b34/ HTTP/1.1 Host: archive.softwareheritage.org User-Agent: python-requests/2.21.0 Accept: */* X-Forwarded-For: 128.93.166.14 X-Forwarded-Proto: https Accept-Encoding: gzip X-Varnish: 230399
vsellier added a comment to T2774: Fix vault end-to-end check.
after digging, it seems the request with a 404 return code are cached by varnish.
When the test is launched, a first request is done which returns a 404 and the post is issued. When the check try to get the status of the cooking, the initial 404 is returned by varnish
vsellier added a comment to T2774: Fix vault end-to-end check.
It seems the scheduler has missed some updates. After an upgrade of the python3-swh-.* packages, the error is again the initial one.
vsellier added a comment to T2774: Fix vault end-to-end check.
After an upgrade of the packages on pergamon and vangogh, the error is now :
Feb 17 10:49:38 vangogh python3[1990225]: 2021-02-17 10:49:38 [1990225] root:ERROR <RemoteException 500 InvalidDatetimeFormat: ['invalid input syntax for type timestamp with time zone: "Timestamp(seconds=1613558977, nanoseconds=999614000)"\nCONTEXT: COPY tmp_task, line 1, column next_run: "Timestamp(seconds=1613558977, nanoseconds=999614000)"\n']>
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/swh/core/api/asynchronous.py", line 71, in middleware_handler
return await handler(request)
File "/usr/lib/python3/dist-packages/swh/core/api/asynchronous.py", line 178, in decorated_meth
result = obj_meth(**kw)
File "/usr/lib/python3/dist-packages/swh/core/db/common.py", line 62, in _meth
return meth(self, *args, db=db, cur=cur, **kwargs)
File "/usr/lib/python3/dist-packages/swh/vault/backend.py", line 220, in cook
self.create_task(obj_type, obj_id, sticky)
File "/usr/lib/python3/dist-packages/swh/core/db/common.py", line 62, in _meth
return meth(self, *args, db=db, cur=cur, **kwargs)
File "/usr/lib/python3/dist-packages/swh/vault/backend.py", line 163, in create_task
task_id = self._send_task(obj_type, hex_id)
File "/usr/lib/python3/dist-packages/swh/vault/backend.py", line 139, in _send_task
added_tasks = self.scheduler.create_tasks([task])
File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 181, in meth_
return self.post(meth._endpoint_path, post_data)
File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 278, in post
return self._decode_response(response)
File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 354, in _decode_response
self.raise_for_status(response)
File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 344, in raise_for_status
raise exception from None
swh.core.api.RemoteException: <RemoteException 500 InvalidDatetimeFormat: ['invalid input syntax for type timestamp with time zone: "Timestamp(seconds=1613558977, nanoseconds=999614000)"\nCONTEXT: COPY tmp_task, line 1, column next_run: "Timestamp(seconds=1613558977, nanoseconds=999614000)"\n']>Feb 16 2021
Feb 16 2021
vsellier added a comment to T2912: Next generation archive counters.
I wrote a proposal for the next steps [1] so we could start the work on these counters. All comments/contributions are welcome.
Feb 5 2021
Feb 5 2021
douardda added a comment to T2912: Next generation archive counters.
@vsellier nice. Note that if we draw these with a y-axis starting from 0, the step shape will be really negligible, so IMHO it's really not a problem.
vsellier added a comment to T2912: Next generation archive counters.
Nice, thanks for confirming this at the source.