My recent work on fixing the subversion loader issues in production based on sentry reports made me think again about how we could resolve that task.
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Advanced Search
Dec 14 2021
Dec 13 2021
Explicitly setting the LIBFFI_TMPDIR environment variable indeed fixes the hang: D6828.
Dec 10 2021
Dec 9 2021
Dec 8 2021
Looks good to me.
I think the proper fix is to rather include the swh-perfecthash documentation index in swh-docs/docs/index.rst and swh-docs/docs/api-reference.rst.
Looks good to me.
Turns out it has already been handled in rDOBJSRPL8098798820bb2025f9bfc7c6b2b46e08a818b797, my local repo was not up to date.
Rebase
Looks good to me, thanks !
Dec 7 2021
It is possible that more key cryptographic software will include these files.
SWHID v2 should also prevent the hash collision issues we are currently facing (some related to the SHAttered attack on SHA1), see T3775 and SWH-LOADER-GIT-JT.
concurrency issue between the swh-web and swh-web-cron startup, probably not blocker
Loading the repository in docker environment gives me the following traceback:
swh-loader_1 | [2021-12-07 12:08:06,876: INFO/MainProcess] Task swh.loader.git.tasks.UpdateGitRepository[a1aa28c0-1cb0-4e2a-8ae2-720ba6ca439e] received swh-loader_1 | [2021-12-07 12:08:06,877: INFO/MainProcess] loader@b11bfd448510 ready. swh-loader_1 | [2021-12-07 12:08:06,957: DEBUG/ForkPoolWorker-1] Loading config file /loader.yml swh-loader_1 | [2021-12-07 12:08:09,904: INFO/ForkPoolWorker-1] Load origin 'https://gitlab.com/sequoia-pgp/sequoia' with type 'git' swh-loader_1 | [2021-12-07 12:08:09,908: DEBUG/ForkPoolWorker-1] Transport url to communicate with server: https://gitlab.com/sequoia-pgp/sequoia swh-loader_1 | [2021-12-07 12:08:09,909: DEBUG/ForkPoolWorker-1] Client Urllib3HttpGitClient('https://gitlab.com/sequoia-pgp/sequoia/', dumb=None) to fetch pack at /sequoia-pgp/sequoia swh-loader_1 | [2021-12-07 12:08:10,422: DEBUG/ForkPoolWorker-1] local_heads_count=0 swh-loader_1 | [2021-12-07 12:08:10,422: DEBUG/ForkPoolWorker-1] remote_heads_count=1821 swh-loader_1 | [2021-12-07 12:08:10,422: DEBUG/ForkPoolWorker-1] wanted_refs_count=1821 swh-loader_1 | [2021-12-07 12:09:17,112: ERROR/ForkPoolWorker-1] Loading failure, updating to `failed` status swh-loader_1 | Traceback (most recent call last): swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/client.py", line 29, in raise_for_status swh-loader_1 | super().raise_for_status(response) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 344, in raise_for_status swh-loader_1 | raise exception from None swh-loader_1 | swh.core.api.RemoteException: <RemoteException 500 HashCollision: ['sha1', '38762cf7f55934b34d179ae6a4c80cadccbb7f0a', [{'blake2s256': '30e4bd16c3f98e74429d237c19ca9def702e5720cb124cb4b92e74f989aaf116', 'sha1': '38762cf7f55934b34d179ae6a4c80cadccbb7f0a', 'sha1_git': 'b621eeccd5c7edac9b7dcba35a8d5afd075e24f2', 'sha256': 'd4488775d29bdef7993367d541064dbdda50d383f89f0aa13a6ff2e0894ba5ff'}, {'blake2s256': '8f677e3214ca8b2acad91884a1571ef3f12b786501f9a6bedfd6239d82095dd2', 'sha1': '38762cf7f55934b34d179ae6a4c80cadccbb7f0a', 'sha1_git': 'ba9aaa145ccd24ef760cf31c74d8f7ca1a2e47b0', 'sha256': '2bb787a73e37352f92383abe7e2902936d1059ad9f1ba6daaa9c1e58ee6970d0'}]]> swh-loader_1 | swh-loader_1 | During handling of the above exception, another exception occurred: swh-loader_1 | swh-loader_1 | Traceback (most recent call last): swh-loader_1 | File "/src/swh-loader-core/swh/loader/core/loader.py", line 339, in load swh-loader_1 | self.store_data() swh-loader_1 | File "/src/swh-loader-core/swh/loader/core/loader.py", line 458, in store_data swh-loader_1 | self.storage.directory_add([directory]) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/proxies/buffer.py", line 171, in directory_add swh-loader_1 | stats = self.object_add(directories, object_type="directory", keys=["id"]) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/proxies/buffer.py", line 224, in object_add swh-loader_1 | return self.flush() swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/proxies/buffer.py", line 286, in flush swh-loader_1 | stats = add_fn(list(batch)) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/proxies/filter.py", line 58, in content_add swh-loader_1 | [x for x in content if x.sha256 in contents_to_add] swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/client.py", line 45, in content_add swh-loader_1 | return self.post("content/add", {"content": content}) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 278, in post swh-loader_1 | return self._decode_response(response) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 354, in _decode_response swh-loader_1 | self.raise_for_status(response) swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/client.py", line 39, in raise_for_status swh-loader_1 | raise HashCollision(*e.args[0]["args"]) swh-loader_1 | swh.storage.exc.HashCollision: ('sha1', '38762cf7f55934b34d179ae6a4c80cadccbb7f0a', [{'sha256': 'd4488775d29bdef7993367d541064dbdda50d383f89f0aa13a6ff2e0894ba5ff', 'sha1': '38762cf7f55934b34d179ae6a4c80cadccbb7f0a', 'sha1_git': 'b621eeccd5c7edac9b7dcba35a8d5afd075e24f2', 'blake2s256': '30e4bd16c3f98e74429d237c19ca9def702e5720cb124cb4b92e74f989aaf116'}, {'sha256': '2bb787a73e37352f92383abe7e2902936d1059ad9f1ba6daaa9c1e58ee6970d0', 'sha1': '38762cf7f55934b34d179ae6a4c80cadccbb7f0a', 'sha1_git': 'ba9aaa145ccd24ef760cf31c74d8f7ca1a2e47b0', 'blake2s256': '8f677e3214ca8b2acad91884a1571ef3f12b786501f9a6bedfd6239d82095dd2'}])
Fix typo in commit message
Dec 6 2021
Rebase and update diff after swh-model 3.1.0 release
Add an autouse fixture to reset the blake2 cache
Rebase
Looks good to me.
Looks good to me.
I also tested the listing of Ubuntu historical releases and it went fine.
Use enumerate in test