done, and released as swh-vault v1.0.0 :)
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Feed Advanced Search
Advanced Search
Advanced Search
Aug 27 2021
Aug 27 2021
ardumont closed T3503: staging: vault: Deploy v1.0.0, a subtask of T3507: prod: vault: Deploy v1.0.0, as Resolved.
ardumont changed the status of T3507: prod: vault: Deploy v1.0.0, a subtask of T3506: Get rid of the concept of vault "object_type", from Open to Work in Progress.
Aug 26 2021
Aug 26 2021
vlorentz changed the status of T3505: Make the git-bare cooker available to the staff and beta-testers in the production webapp from Open to Work in Progress.
vlorentz changed the status of T3505: Make the git-bare cooker available to the staff and beta-testers in the production webapp, a subtask of T3504: Make the git-bare cooker publicly available, from Open to Work in Progress.
vlorentz removed a parent task for T843: Vault: Add a "git bare" tarball cooker: T3504: Make the git-bare cooker publicly available.
vlorentz removed a subtask for T3504: Make the git-bare cooker publicly available: T843: Vault: Add a "git bare" tarball cooker.
vlorentz added a parent task for T843: Vault: Add a "git bare" tarball cooker: T3504: Make the git-bare cooker publicly available.
vlorentz closed T843: Vault: Add a "git bare" tarball cooker, a subtask of T3096: Efficient and reliable download via the Vault, as Resolved.
ardumont moved T3507: prod: vault: Deploy v1.0.0 from Backlog to Weekly backlog on the System administration board.
vlorentz added a subtask for T3507: prod: vault: Deploy v1.0.0: T3503: staging: vault: Deploy v1.0.0.
vlorentz added revisions to T3506: Get rid of the concept of vault "object_type": D6141: Bump minimum swh-vault version, D6126: vault.spec.js: Add test checking the LocalStorage is used to show the task list, D6113: vault API: Rename bundle types and use SWHIDs to identify objects, D6112: Rename bundle types and use SWHIDs everywhere instead of raw sha1_git.
vlorentz changed the status of T3506: Get rid of the concept of vault "object_type" from Open to Work in Progress.
vlorentz changed the status of T3506: Get rid of the concept of vault "object_type", a subtask of T3505: Make the git-bare cooker available to the staff and beta-testers in the production webapp, from Open to Work in Progress.
vlorentz triaged T3505: Make the git-bare cooker available to the staff and beta-testers in the production webapp as Normal priority.
Aug 23 2021
Aug 23 2021
vlorentz added revisions to T843: Vault: Add a "git bare" tarball cooker: D6056: Don't hide git-fsck error messages, even when none is expected, D6057: Add typing, and fix potential issues found by mypy., D6096: git_bare: Fix crash on dangling branches, D6097: git_bare: Deduplicate loading of release subgraphs, D6098: git_bare: Skip loading empty lists, D6099: git_bare: Add support for directories as snapshot branches, D6100: git_bare: add some modularity to directory creation, D6101: Rename obj_type to bundle_type, D6102: git_bare: Use an enum to represent the root object type, D6103: git_bare: Add support for directories as snapshot branches, D6104: git_bare: Add support for contents pointed by snapshots, D6105: git_bare: Statically check for exhaustiveness of enum matching, D6111: Add support for releases pointing to other releases or contents..
Aug 2 2021
Aug 2 2021
vlorentz added revisions to T843: Vault: Add a "git bare" tarball cooker: D5795: git_bare: Add support for skipped/missing/absent/hidden contents, D5935: Make swh.graph dependency optional, D5938: Make swh.graph dependency optional 2/2, D5940: git_bare: Add support for filtered content with Git >= 2.21, D5999: git_bare: Add partial support for snapshots (no release or swh-graph support yet), D6000: git_bare: Add support for annotated tags pointing to commits, D6002: git_bare: Add support for swh-graph when loading a snapshot.
Jul 29 2021
Jul 29 2021
ardumont moved T2774: Fix vault end-to-end check from deployed/landed/monitoring to done on the System administration board.
Jun 29 2021
Jun 29 2021
vlorentz renamed T3412: Deploy vault "git bare" tarball cooker (swh-vault v0.6) from Deploy new vault "git bare" tarball cooker to Deploy vault "git bare" tarball cooker (swh-vault v0.6).
ardumont closed T3412: Deploy vault "git bare" tarball cooker (swh-vault v0.6), a subtask of T843: Vault: Add a "git bare" tarball cooker, as Resolved.
Deployed v0.6.4.
ardumont updated the task description for T3412: Deploy vault "git bare" tarball cooker (swh-vault v0.6).
ardumont moved T3412: Deploy vault "git bare" tarball cooker (swh-vault v0.6) from Backlog to deployed/landed/monitoring on the System administration board.
ardumont added a project to T3412: Deploy vault "git bare" tarball cooker (swh-vault v0.6): System administration.
status:
- unstable build is fine.
- stable build failed [1] [2]
The build is still breaking in a different way [1]
Debian build is broken [1], currently investigating to reproduce/fix it.
Jun 28 2021
Jun 28 2021
ardumont updated the task description for T3412: Deploy vault "git bare" tarball cooker (swh-vault v0.6).
Debian build is broken [1], currently investigating to reproduce/fix it.
ardumont updated the task description for T3412: Deploy vault "git bare" tarball cooker (swh-vault v0.6).
ardumont updated the task description for T3412: Deploy vault "git bare" tarball cooker (swh-vault v0.6).
May 11 2021
May 11 2021
vlorentz added revisions to T843: Vault: Add a "git bare" tarball cooker: D5734: git_bare: Use directory_get_entries instead of directory_ls, it should be faster, D5733: git_bare: Refactor the graph descent using explicit stacks instead of the call stack., D5731: git_bare: Optionally access the objstorage directly, D5730: git_bare: Use batched content_get() instead of content_find(), D5729: content_get: Add support for queries by sha1_git.
May 10 2021
May 10 2021
vlorentz changed the status of T843: Vault: Add a "git bare" tarball cooker, a subtask of T3096: Efficient and reliable download via the Vault, from Open to Work in Progress.
vlorentz changed the status of T843: Vault: Add a "git bare" tarball cooker from Open to Work in Progress.
vlorentz added a parent task for T843: Vault: Add a "git bare" tarball cooker: T3096: Efficient and reliable download via the Vault.
vlorentz added revisions to T843: Vault: Add a "git bare" tarball cooker: D5708: git_bare: When possible, use swh-graph instead of swh-storage to query revision history, D5658: git_bare: Deduplicate object downloads and writes, D5656: Add a naive git bare cooker, D5652: identifiers: Expose git_object instead of manifest.
vlorentz changed the status of T3096: Efficient and reliable download via the Vault from Open to Work in Progress.
vlorentz moved T3096: Efficient and reliable download via the Vault from Backlog to Work in progress on the Roadmap 2021 board.
May 4 2021
May 4 2021
I'm going to close this, because we decided to use a different approach for the git bare cooker.
Apr 30 2021
Apr 30 2021
Apr 29 2021
Apr 29 2021
Apr 27 2021
Apr 27 2021
Apr 16 2021
Apr 16 2021
vsellier placed T2885: investigate the persistent seemingly benign errors on cooking progress up for grabs.
Apr 8 2021
Apr 8 2021
anlambert closed T3181: docker-dev: fix vault test which fails the build as Resolved by committing rDENV80e43a5fbe72: docker/tests: Fix symlink handling in test_vault_directory.
So either the vault is fine and the test needs to be improved.
Or the vault builds the tarball without taking into account the symlink case.
Apr 2 2021
Apr 2 2021
Mar 29 2021
Mar 29 2021
Mar 9 2021
Mar 9 2021
Mar 8 2021
Mar 8 2021
rdicosmo added a subtask for T3096: Efficient and reliable download via the Vault: T2220: swh-graph in production.
Feb 18 2021
Feb 18 2021
vsellier moved T2774: Fix vault end-to-end check from in-progress to deployed/landed/monitoring on the System administration board.
Thanks @anlambert, the monitoring comes back to green
Feb 17 2021
Feb 17 2021
@anlambert agrees with the previous hypothesis ^ and is working on a fix
With tcpdump, it seems swh-web don't add the headers to don't cache the response in case of a 404:
GET /api/1/vault/directory/a317baff051f68e83557d51e59539dac2ff55b34/ HTTP/1.1 Host: archive.softwareheritage.org User-Agent: python-requests/2.21.0 Accept: */* X-Forwarded-For: 128.93.166.14 X-Forwarded-Proto: https Accept-Encoding: gzip X-Varnish: 230399
after digging, it seems the request with a 404 return code are cached by varnish.
When the test is launched, a first request is done which returns a 404 and the post is issued. When the check try to get the status of the cooking, the initial 404 is returned by varnish
It seems the scheduler has missed some updates. After an upgrade of the python3-swh-.* packages, the error is again the initial one.
After an upgrade of the packages on pergamon and vangogh, the error is now :
Feb 17 10:49:38 vangogh python3[1990225]: 2021-02-17 10:49:38 [1990225] root:ERROR <RemoteException 500 InvalidDatetimeFormat: ['invalid input syntax for type timestamp with time zone: "Timestamp(seconds=1613558977, nanoseconds=999614000)"\nCONTEXT: COPY tmp_task, line 1, column next_run: "Timestamp(seconds=1613558977, nanoseconds=999614000)"\n']>
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/swh/core/api/asynchronous.py", line 71, in middleware_handler
return await handler(request)
File "/usr/lib/python3/dist-packages/swh/core/api/asynchronous.py", line 178, in decorated_meth
result = obj_meth(**kw)
File "/usr/lib/python3/dist-packages/swh/core/db/common.py", line 62, in _meth
return meth(self, *args, db=db, cur=cur, **kwargs)
File "/usr/lib/python3/dist-packages/swh/vault/backend.py", line 220, in cook
self.create_task(obj_type, obj_id, sticky)
File "/usr/lib/python3/dist-packages/swh/core/db/common.py", line 62, in _meth
return meth(self, *args, db=db, cur=cur, **kwargs)
File "/usr/lib/python3/dist-packages/swh/vault/backend.py", line 163, in create_task
task_id = self._send_task(obj_type, hex_id)
File "/usr/lib/python3/dist-packages/swh/vault/backend.py", line 139, in _send_task
added_tasks = self.scheduler.create_tasks([task])
File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 181, in meth_
return self.post(meth._endpoint_path, post_data)
File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 278, in post
return self._decode_response(response)
File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 354, in _decode_response
self.raise_for_status(response)
File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 344, in raise_for_status
raise exception from None
swh.core.api.RemoteException: <RemoteException 500 InvalidDatetimeFormat: ['invalid input syntax for type timestamp with time zone: "Timestamp(seconds=1613558977, nanoseconds=999614000)"\nCONTEXT: COPY tmp_task, line 1, column next_run: "Timestamp(seconds=1613558977, nanoseconds=999614000)"\n']>Feb 3 2021
Feb 3 2021
tenma retitled D4766: [WIP] git bare revision cooker from WIP git bare revision cooker to [WIP] git bare revision cooker.
Jan 25 2021
Jan 25 2021
Deployed configuration change in staging without issues.
Deployed configuration change in produciton as well.
Rebase
Jan 4 2021
Jan 4 2021
Dec 21 2020
Dec 21 2020
tenma added a revision to T843: Vault: Add a "git bare" tarball cooker: D4766: [WIP] git bare revision cooker.
Dec 15 2020
Dec 15 2020
ardumont added a comment to T2885: investigate the persistent seemingly benign errors on cooking progress.
Yes, i think it's just the pattern the webapp uses.