Page MenuHomeSoftware Heritage
Feed Advanced Search

Dec 14 2021

anlambert added a comment to T611: support for external definitions in the svn/subversion loader.

My recent work on fixing the subversion loader issues in production based on sentry reports made me think again about how we could resolve that task.

Dec 14 2021, 3:51 PM · SVN Loader

Dec 13 2021

anlambert closed T3803: swh-search tests are hanging since elasticsearch 7.16 release as Resolved by committing rDSEA933c78830448: conftest: Fix tests hang since elasticsearch 7.16 release.
Dec 13 2021, 4:13 PM · Archive search
anlambert closed D6828: conftest: Fix tests hang since elasticsearch 7.16 release.
Dec 13 2021, 4:13 PM
anlambert committed rDSEA933c78830448: conftest: Fix tests hang since elasticsearch 7.16 release (authored by anlambert).
conftest: Fix tests hang since elasticsearch 7.16 release
Dec 13 2021, 4:13 PM
anlambert requested review of D6828: conftest: Fix tests hang since elasticsearch 7.16 release.
Dec 13 2021, 4:07 PM
anlambert added a comment to T3803: swh-search tests are hanging since elasticsearch 7.16 release.

Explicitly setting the LIBFFI_TMPDIR environment variable indeed fixes the hang: D6828.

Dec 13 2021, 4:06 PM · Archive search
anlambert added a revision to T3803: swh-search tests are hanging since elasticsearch 7.16 release: D6828: conftest: Fix tests hang since elasticsearch 7.16 release.
Dec 13 2021, 4:03 PM · Archive search
anlambert triaged T3803: swh-search tests are hanging since elasticsearch 7.16 release as Normal priority.
Dec 13 2021, 2:42 PM · Archive search
anlambert committed rDWAPPS4f5570cf15f3: yarn.lock: Deduplicate dependencies (authored by anlambert).
yarn.lock: Deduplicate dependencies
Dec 13 2021, 2:19 PM
anlambert committed rDWAPPSb0c09bbb3d42: package.json: Upgrade dependencies (authored by anlambert).
package.json: Upgrade dependencies
Dec 13 2021, 2:19 PM
anlambert closed D6826: misc/coverage: Do not put coverage page in cache.
Dec 13 2021, 2:19 PM
anlambert committed rDWAPPSc9b40186e88a: misc/coverage: Do not put coverage page in cache (authored by anlambert).
misc/coverage: Do not put coverage page in cache
Dec 13 2021, 2:19 PM
anlambert requested review of D6826: misc/coverage: Do not put coverage page in cache.
Dec 13 2021, 1:52 PM

Dec 10 2021

anlambert closed D6820: tests/test_loader: Wrap empty svn repo creation in a fixture.
Dec 10 2021, 2:59 PM
anlambert committed rDLDSVN0ae91422d8cb: tests/test_loader: Wrap empty svn repo creation in a fixture (authored by anlambert).
tests/test_loader: Wrap empty svn repo creation in a fixture
Dec 10 2021, 2:59 PM

Dec 9 2021

anlambert requested review of D6820: tests/test_loader: Wrap empty svn repo creation in a fixture.
Dec 9 2021, 5:14 PM

Dec 8 2021

anlambert accepted D6799: model: Add a check() method to model objects.

Looks good to me.

Dec 8 2021, 5:27 PM
anlambert accepted D6797: model: Add support for None to the type checker.
Dec 8 2021, 5:26 PM
anlambert accepted D6798: model: Deduplicate calls to hashlib..

Agreed.

Dec 8 2021, 5:25 PM
anlambert added a comment to D6793: docs: Fix docs build.

I think the proper fix is to rather include the swh-perfecthash documentation index in swh-docs/docs/index.rst and swh-docs/docs/api-reference.rst.

Dec 8 2021, 5:09 PM
anlambert accepted D6792: moma: Set the memory usage to the currently defined value.

Looks good to me.

Dec 8 2021, 4:29 PM
anlambert closed D6785: proxies/retry: Remove no longer needed tenacity workarounds.
Dec 8 2021, 4:01 PM
anlambert committed rDSTO7cb4128e40f5: proxies/retry: Remove no longer needed tenacity workarounds (authored by anlambert).
proxies/retry: Remove no longer needed tenacity workarounds
Dec 8 2021, 4:01 PM
anlambert requested review of D6785: proxies/retry: Remove no longer needed tenacity workarounds.
Dec 8 2021, 3:38 PM
anlambert closed D6789: from_disk: Implement Directory.__contains__.
Dec 8 2021, 3:29 PM
anlambert committed rDMODf6e0a289ef6e: from_disk: Implement Directory.__contains__ (authored by anlambert).
from_disk: Implement Directory.__contains__
Dec 8 2021, 3:29 PM
anlambert triaged T3779: Memcached seems to be automatically cleared pretty often on moma as High priority.
Dec 8 2021, 3:25 PM · Web app, System administration
anlambert accepted D6788: hypothesis_strategies: Ensure to generate valid directory entry name (again).
Dec 8 2021, 3:16 PM
anlambert requested changes to D6788: hypothesis_strategies: Ensure to generate valid directory entry name (again).
Dec 8 2021, 3:16 PM
anlambert accepted D6787: Revert "hypothesis_strategies: Ensure to generate valid directory entry name".
Dec 8 2021, 3:15 PM
anlambert created P1241 (An Untitled Masterwork).
Dec 8 2021, 3:03 PM
anlambert requested review of D6789: from_disk: Implement Directory.__contains__.
Dec 8 2021, 2:44 PM
anlambert updated the title for P1240 subversion loader svn:exernals support (quick and dirty POC) from subversion loader sv:exernals support (quick and dirty POC) to subversion loader svn:exernals support (quick and dirty POC).
Dec 8 2021, 2:10 PM
anlambert edited P1240 subversion loader svn:exernals support (quick and dirty POC).
Dec 8 2021, 2:10 PM
anlambert created P1240 subversion loader svn:exernals support (quick and dirty POC).
Dec 8 2021, 1:42 PM
anlambert closed D6783: Remove no longer needed tenacity workarounds.
Dec 8 2021, 1:32 PM
anlambert committed rDLS445d539b3fb1: Remove no longer needed tenacity workarounds (authored by anlambert).
Remove no longer needed tenacity workarounds
Dec 8 2021, 1:32 PM
anlambert abandoned D6786: replayer/replay: Remove no longer needed tenacity workarounds.

Turns out it has already been handled in rDOBJSRPL8098798820bb2025f9bfc7c6b2b46e08a818b797, my local repo was not up to date.

Dec 8 2021, 1:31 PM
anlambert updated the diff for D6783: Remove no longer needed tenacity workarounds.

Rebase

Dec 8 2021, 1:28 PM
anlambert requested review of D6786: replayer/replay: Remove no longer needed tenacity workarounds.
Dec 8 2021, 12:12 PM
anlambert requested review of D6783: Remove no longer needed tenacity workarounds.
Dec 8 2021, 11:53 AM
anlambert accepted D6782: Increase the swh-web timeout for swh-storage requests.

Looks good to me, thanks !

Dec 8 2021, 10:54 AM

Dec 7 2021

anlambert closed D6778: loader: Do not raise error when reloading a repo not fully ingested.
Dec 7 2021, 7:24 PM
anlambert committed rDLDSVNfffd2c45ec4f: loader: Do not raise error when reloading a repo not fully ingested (authored by anlambert).
loader: Do not raise error when reloading a repo not fully ingested
Dec 7 2021, 7:24 PM
anlambert requested review of D6778: loader: Do not raise error when reloading a repo not fully ingested.
Dec 7 2021, 6:02 PM
anlambert added a comment to T3775: Dealing with repositories with contents that produces hash conflicts (example included from GitLab).

It is possible that more key cryptographic software will include these files.

Dec 7 2021, 4:56 PM · meta-task, Roadmap 2022, Save Code Now, Origin-GitLab
anlambert added a comment to T3134: SWHID v2.

SWHID v2 should also prevent the hash collision issues we are currently facing (some related to the SHAttered attack on SHA1), see T3775 and SWH-LOADER-GIT-JT.

Dec 7 2021, 4:55 PM · Roadmap 2022, Roadmap 2020, Data Model, Web app, meta-task, Roadmap 2021
anlambert committed rDWAPPS61647a4bfd19: .gitignore: Remove debian/ entry (authored by anlambert).
.gitignore: Remove debian/ entry
Dec 7 2021, 3:49 PM
anlambert added a comment to T3778: The docker-dev build is often failing.

concurrency issue between the swh-web and swh-web-cron startup, probably not blocker

Dec 7 2021, 3:02 PM · System administration
anlambert committed rDWAPPSd9b6681d4e92: templates/fundraising-banner: Update banner image (authored by anlambert).
templates/fundraising-banner: Update banner image
Dec 7 2021, 2:54 PM
anlambert committed rDWAPPS92973b58f716: package.json: Upgrade dependencies (authored by anlambert).
package.json: Upgrade dependencies
Dec 7 2021, 2:54 PM
anlambert committed rDWAPPSbf9ca973cfbb: settings/tests: Ensure scheduler not available when running cypress (authored by anlambert).
settings/tests: Ensure scheduler not available when running cypress
Dec 7 2021, 2:54 PM
anlambert closed D6763: browse/releases: Fix URLs to browse release targets.
Dec 7 2021, 2:54 PM
anlambert committed rDWAPPS144260c06e26: browse/releases: Fix URLs to browse release targets (authored by anlambert).
browse/releases: Fix URLs to browse release targets
Dec 7 2021, 2:54 PM
anlambert closed T3776: cassandra tests are failing in the swh-environment build as Resolved by committing rDSTO615fb99eb708: test_cassandra: Fix failing tests since swh-model update.
Dec 7 2021, 1:56 PM · Storage manager
anlambert closed D6768: test_cassandra: Fix failing tests since swh-model update.
Dec 7 2021, 1:56 PM
anlambert committed rDSTO615fb99eb708: test_cassandra: Fix failing tests since swh-model update (authored by anlambert).
test_cassandra: Fix failing tests since swh-model update
Dec 7 2021, 1:56 PM
anlambert added inline comments to D6768: test_cassandra: Fix failing tests since swh-model update.
Dec 7 2021, 1:56 PM
anlambert requested review of D6768: test_cassandra: Fix failing tests since swh-model update.
Dec 7 2021, 1:46 PM
anlambert added a revision to T3776: cassandra tests are failing in the swh-environment build: D6768: test_cassandra: Fix failing tests since swh-model update.
Dec 7 2021, 1:38 PM · Storage manager
anlambert added a comment to T3775: Dealing with repositories with contents that produces hash conflicts (example included from GitLab).

Loading the repository in docker environment gives me the following traceback:

swh-loader_1                        | [2021-12-07 12:08:06,876: INFO/MainProcess] Task swh.loader.git.tasks.UpdateGitRepository[a1aa28c0-1cb0-4e2a-8ae2-720ba6ca439e] received
swh-loader_1                        | [2021-12-07 12:08:06,877: INFO/MainProcess] loader@b11bfd448510 ready.
swh-loader_1                        | [2021-12-07 12:08:06,957: DEBUG/ForkPoolWorker-1] Loading config file /loader.yml
swh-loader_1                        | [2021-12-07 12:08:09,904: INFO/ForkPoolWorker-1] Load origin 'https://gitlab.com/sequoia-pgp/sequoia' with type 'git'
swh-loader_1                        | [2021-12-07 12:08:09,908: DEBUG/ForkPoolWorker-1] Transport url to communicate with server: https://gitlab.com/sequoia-pgp/sequoia
swh-loader_1                        | [2021-12-07 12:08:09,909: DEBUG/ForkPoolWorker-1] Client Urllib3HttpGitClient('https://gitlab.com/sequoia-pgp/sequoia/', dumb=None) to fetch pack at /sequoia-pgp/sequoia
swh-loader_1                        | [2021-12-07 12:08:10,422: DEBUG/ForkPoolWorker-1] local_heads_count=0
swh-loader_1                        | [2021-12-07 12:08:10,422: DEBUG/ForkPoolWorker-1] remote_heads_count=1821
swh-loader_1                        | [2021-12-07 12:08:10,422: DEBUG/ForkPoolWorker-1] wanted_refs_count=1821
swh-loader_1                        | [2021-12-07 12:09:17,112: ERROR/ForkPoolWorker-1] Loading failure, updating to `failed` status
swh-loader_1                        | Traceback (most recent call last):
swh-loader_1                        |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/client.py", line 29, in raise_for_status
swh-loader_1                        |     super().raise_for_status(response)
swh-loader_1                        |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 344, in raise_for_status
swh-loader_1                        |     raise exception from None
swh-loader_1                        | swh.core.api.RemoteException: <RemoteException 500 HashCollision: ['sha1', '38762cf7f55934b34d179ae6a4c80cadccbb7f0a', [{'blake2s256': '30e4bd16c3f98e74429d237c19ca9def702e5720cb124cb4b92e74f989aaf116', 'sha1': '38762cf7f55934b34d179ae6a4c80cadccbb7f0a', 'sha1_git': 'b621eeccd5c7edac9b7dcba35a8d5afd075e24f2', 'sha256': 'd4488775d29bdef7993367d541064dbdda50d383f89f0aa13a6ff2e0894ba5ff'}, {'blake2s256': '8f677e3214ca8b2acad91884a1571ef3f12b786501f9a6bedfd6239d82095dd2', 'sha1': '38762cf7f55934b34d179ae6a4c80cadccbb7f0a', 'sha1_git': 'ba9aaa145ccd24ef760cf31c74d8f7ca1a2e47b0', 'sha256': '2bb787a73e37352f92383abe7e2902936d1059ad9f1ba6daaa9c1e58ee6970d0'}]]>
swh-loader_1                        | 
swh-loader_1                        | During handling of the above exception, another exception occurred:
swh-loader_1                        | 
swh-loader_1                        | Traceback (most recent call last):
swh-loader_1                        |   File "/src/swh-loader-core/swh/loader/core/loader.py", line 339, in load
swh-loader_1                        |     self.store_data()
swh-loader_1                        |   File "/src/swh-loader-core/swh/loader/core/loader.py", line 458, in store_data
swh-loader_1                        |     self.storage.directory_add([directory])
swh-loader_1                        |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/proxies/buffer.py", line 171, in directory_add
swh-loader_1                        |     stats = self.object_add(directories, object_type="directory", keys=["id"])
swh-loader_1                        |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/proxies/buffer.py", line 224, in object_add
swh-loader_1                        |     return self.flush()
swh-loader_1                        |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/proxies/buffer.py", line 286, in flush
swh-loader_1                        |     stats = add_fn(list(batch))
swh-loader_1                        |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/proxies/filter.py", line 58, in content_add
swh-loader_1                        |     [x for x in content if x.sha256 in contents_to_add]
swh-loader_1                        |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/client.py", line 45, in content_add
swh-loader_1                        |     return self.post("content/add", {"content": content})
swh-loader_1                        |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 278, in post
swh-loader_1                        |     return self._decode_response(response)
swh-loader_1                        |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 354, in _decode_response
swh-loader_1                        |     self.raise_for_status(response)
swh-loader_1                        |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/client.py", line 39, in raise_for_status
swh-loader_1                        |     raise HashCollision(*e.args[0]["args"])
swh-loader_1                        | swh.storage.exc.HashCollision: ('sha1', '38762cf7f55934b34d179ae6a4c80cadccbb7f0a', [{'sha256': 'd4488775d29bdef7993367d541064dbdda50d383f89f0aa13a6ff2e0894ba5ff', 'sha1': '38762cf7f55934b34d179ae6a4c80cadccbb7f0a', 'sha1_git': 'b621eeccd5c7edac9b7dcba35a8d5afd075e24f2', 'blake2s256': '30e4bd16c3f98e74429d237c19ca9def702e5720cb124cb4b92e74f989aaf116'}, {'sha256': '2bb787a73e37352f92383abe7e2902936d1059ad9f1ba6daaa9c1e58ee6970d0', 'sha1': '38762cf7f55934b34d179ae6a4c80cadccbb7f0a', 'sha1_git': 'ba9aaa145ccd24ef760cf31c74d8f7ca1a2e47b0', 'blake2s256': '8f677e3214ca8b2acad91884a1571ef3f12b786501f9a6bedfd6239d82095dd2'}])
Dec 7 2021, 1:23 PM · meta-task, Roadmap 2022, Save Code Now, Origin-GitLab
anlambert requested review of D6763: browse/releases: Fix URLs to browse release targets.
Dec 7 2021, 12:29 PM
anlambert closed D6760: hypothesis_strategies: Ensure to generate valid directory entry name.
Dec 7 2021, 11:45 AM
anlambert committed rDMODc525484e4737: hypothesis_strategies: Ensure to generate valid directory entry name (authored by anlambert).
hypothesis_strategies: Ensure to generate valid directory entry name
Dec 7 2021, 11:45 AM
anlambert updated the summary of D6760: hypothesis_strategies: Ensure to generate valid directory entry name.
Dec 7 2021, 11:35 AM
anlambert updated the diff for D6760: hypothesis_strategies: Ensure to generate valid directory entry name.

Fix typo in commit message

Dec 7 2021, 11:35 AM
anlambert requested review of D6760: hypothesis_strategies: Ensure to generate valid directory entry name.
Dec 7 2021, 11:34 AM
anlambert closed D6750: debian: Add md5 sum fallback when sha* checksum is missing in metadata.
Dec 7 2021, 10:55 AM
anlambert committed rDLDBASE2d9e93a2f246: debian: Add md5 sum fallback when sha* checksum is missing in metadata (authored by anlambert).
debian: Add md5 sum fallback when sha* checksum is missing in metadata
Dec 7 2021, 10:55 AM

Dec 6 2021

anlambert added inline comments to D6750: debian: Add md5 sum fallback when sha* checksum is missing in metadata.
Dec 6 2021, 8:03 PM
anlambert updated the diff for D6750: debian: Add md5 sum fallback when sha* checksum is missing in metadata.

Rebase and update diff after swh-model 3.1.0 release

Dec 6 2021, 7:56 PM
anlambert closed D6755: hashutil: Add support for md5 sum.
Dec 6 2021, 7:36 PM
anlambert committed rDMOD37364c24b509: hashutil: Add support for md5 sum (authored by anlambert).
hashutil: Add support for md5 sum
Dec 6 2021, 7:36 PM
anlambert closed D6752: test_hashutil: Port tests from unittest to pytest.
Dec 6 2021, 7:36 PM
anlambert committed rDMOD243520daa827: test_hashutil: Port tests from unittest to pytest (authored by anlambert).
test_hashutil: Port tests from unittest to pytest
Dec 6 2021, 7:36 PM
anlambert updated the diff for D6755: hashutil: Add support for md5 sum.

Rebase

Dec 6 2021, 7:29 PM
anlambert updated the diff for D6752: test_hashutil: Port tests from unittest to pytest.

Add an autouse fixture to reset the blake2 cache

Dec 6 2021, 7:28 PM
anlambert added inline comments to D6752: test_hashutil: Port tests from unittest to pytest.
Dec 6 2021, 7:06 PM
anlambert requested review of D6755: hashutil: Add support for md5 sum.
Dec 6 2021, 5:46 PM
anlambert added inline comments to D6750: debian: Add md5 sum fallback when sha* checksum is missing in metadata.
Dec 6 2021, 5:44 PM
anlambert added a revision to T2400: Ingest current and historical Ubuntu releases: D6755: hashutil: Add support for md5 sum.
Dec 6 2021, 5:43 PM · System administration, Debian loader, Package Loader, Archive coverage
anlambert updated the diff for D6752: test_hashutil: Port tests from unittest to pytest.

Rebase

Dec 6 2021, 5:42 PM
anlambert requested review of D6752: test_hashutil: Port tests from unittest to pytest.
Dec 6 2021, 5:26 PM
anlambert added inline comments to D6750: debian: Add md5 sum fallback when sha* checksum is missing in metadata.
Dec 6 2021, 2:50 PM
anlambert accepted D6749: debian: Remove the extrinsic version from release names.

Looks good to me.

Dec 6 2021, 2:44 PM
anlambert accepted D6748: debian: Fix confusion between the two versions.

Looks good to me.

Dec 6 2021, 2:41 PM
anlambert requested review of D6750: debian: Add md5 sum fallback when sha* checksum is missing in metadata.
Dec 6 2021, 2:38 PM
anlambert added a revision to T2400: Ingest current and historical Ubuntu releases: D6750: debian: Add md5 sum fallback when sha* checksum is missing in metadata.
Dec 6 2021, 2:36 PM · System administration, Debian loader, Package Loader, Archive coverage
anlambert added a comment to T2400: Ingest current and historical Ubuntu releases.

I also tested the listing of Ubuntu historical releases and it went fine.

Dec 6 2021, 1:06 PM · System administration, Debian loader, Package Loader, Archive coverage
anlambert closed D6744: debian: Update last_update for a package when required.
Dec 6 2021, 10:47 AM
anlambert committed rDLS15fa84cf7e35: debian: Update last_update for a package when required (authored by anlambert).
debian: Update last_update for a package when required
Dec 6 2021, 10:47 AM
anlambert updated the diff for D6744: debian: Update last_update for a package when required.

Use enumerate in test

Dec 6 2021, 10:43 AM

Dec 3 2021

anlambert requested review of D6744: debian: Update last_update for a package when required.
Dec 3 2021, 5:53 PM
anlambert added a revision to T2400: Ingest current and historical Ubuntu releases: D6744: debian: Update last_update for a package when required.
Dec 3 2021, 5:50 PM · System administration, Debian loader, Package Loader, Archive coverage
anlambert closed D6743: debian: Provide last_update to produced ListedOrigin models.
Dec 3 2021, 4:18 PM
anlambert committed rDLS93f17d4d9cb1: debian: Provide last_update to produced ListedOrigin models (authored by anlambert).
debian: Provide last_update to produced ListedOrigin models
Dec 3 2021, 4:18 PM
anlambert added a revision to T2400: Ingest current and historical Ubuntu releases: D6743: debian: Provide last_update to produced ListedOrigin models.
Dec 3 2021, 4:14 PM · System administration, Debian loader, Package Loader, Archive coverage
anlambert created P1233 debian lister with last_update.
Dec 3 2021, 4:02 PM
anlambert closed D6741: debian: Do not raise when a component cannot be found for a suite.
Dec 3 2021, 2:45 PM
anlambert committed rDLS605b13a676c4: debian: Do not raise when a component cannot be found for a suite (authored by anlambert).
debian: Do not raise when a component cannot be found for a suite
Dec 3 2021, 2:45 PM