Page MenuHomeSoftware Heritage
Create Task
Maniphest T3595

Support disordered directory entries in git
Closed, MigratedEdits Locked
Actions

Description

Some git directories don't have their entries in the right order, which is an issue for us because:

  • swh-model reorders them before checksuming
  • the postgresql storage does not preserve the relative order of file and dir entries
  • the cassandra storage does not preserve order at all (they are sorted by null-padded name)

Relevant issue to consider to regarding Cassandra: T3582

Sentry: SWH-LOADER-GIT-QF

Related Objects
Search...

StatusAssignedTask
 Migratedgitlab-migrationT3096 Efficient and reliable download via the Vault
 Migratedgitlab-migrationT887 Vault: "snapshot" cooker
 Migratedgitlab-migrationT3504 Make the git-bare cooker publicly available
 Migratedgitlab-migrationT3505 Make the git-bare cooker available to the staff and beta-testers in the production webapp
 Migratedgitlab-migrationT3506 Get rid of the concept of vault "object_type"
 Migratedgitlab-migrationT3507 prod: vault: Deploy v1.0.0
 Migratedgitlab-migrationT3503 staging: vault: Deploy v1.0.0
 Migratedgitlab-migrationT843 Vault: Add a "git bare" tarball cooker
 Migratedgitlab-migrationT3412 Deploy vault "git bare" tarball cooker (swh-vault v0.6)
 Migratedgitlab-migrationT3518 Enable vault cookers to access swh-graph
 Migratedgitlab-migrationT3543 Debian package python3-swh.graph.client
 Migratedgitlab-migrationT3565 Add an option in swh-web to allow git-bare cooking via the UI for beta-testers
 Migratedgitlab-migrationT3551 Fix git-fsck errors in the git-bare cooker
 Migratedgitlab-migrationT3552 Fix corrupted releases, revisions, and directories in the storage
 Migratedgitlab-migrationT3566 git-fsck errors in the git-bare cooker are not properly reported
 Migratedgitlab-migrationT3731 Try parallelizing directory_get_entries with an async swh-storage client
 Migratedgitlab-migrationT3924 Write mailmaps after cooking git-bare archives with display names?
 Migratedgitlab-migrationT3135 Improve integrity of ingested content
 Migratedgitlab-migrationT3594 Faithfully store weird git objects
 Migratedgitlab-migrationT3595 Support disordered directory entries in git

Event Timeline

vlorentz triaged this task as Normal priority.Sep 22 2021, 1:34 PM
vlorentz created this task.
vlorentz updated the task description. (Show Details)
vlorentz added a comment.Sep 22 2021, 1:37 PM

Possible solution: store a rank along with each directory entry, but ignore it unless we are reconstructing a git object or computing a SWHID (v1?)

vlorentz mentioned this in T3586: Figure out what to do with 'misordered' directories in Cassandra.Sep 22 2021, 1:44 PM
vlorentz added a comment.EditedSep 22 2021, 2:51 PM

Complete proposal to implement the above solution:

  1. add this to swh.model.model.DirectoryEntry (Directory.from_dict would set the rank if it's missing):
rank = attr.ib(type=int, validators=type_validator())
"""Zero-based index of this entry in a directory."""
  1. In postgresql, change the type of dir_entries/file_entries/rev_entries from bigint[] to bigint[2][]: pairs of (directory_entry object id, rank). The migration would be: 1. duplicate columns and make python write to both but read only the old 2. fill the new columns (looooong) 3. drop the old columns, rename and make python use the new ones
  1. in cassandra, add column rank to directory_entry table. We can initialize them at 0 and make the Python code fill it when reading. (we can also have a script fill it, but it's not mandatory)
vsellier added a subscriber: vsellier.Sep 27 2021, 11:38 AM
vlorentz updated the task description. (Show Details)Oct 11 2021, 2:49 PM
ardumont added a subscriber: ardumont.Oct 22 2021, 1:54 PM

I came across a rather small repository [1] which i believe raise the same issue.
So it may help to keep its reference to ease the testing of the improvment discussed here.
Feel free to dismiss if not that useful.

swh-loader_1                     | [2021-10-22 11:47:39,586: INFO/MainProcess] Task swh.loader.git.tasks.UpdateGitRepository[3b8b9037-f344-44e5-ab97-a3a310d9214f] received
swh-loader_1                     | [2021-10-22 11:47:39,632: INFO/ForkPoolWorker-1] Load origin 'https://github.com/technoweenie/attachment_fu' with type 'git'
swh-loader_1                     | Enumerating objects: 3549, done.
swh-loader_1                     | Total 3549 (delta 0), reused 0 (delta 0), pack-reused 3549
swh-loader_1                     | [2021-10-22 11:47:41,486: INFO/ForkPoolWorker-1] Listed 23 refs for repo https://github.com/technoweenie/attachment_fu
swh-loader_1                     | [2021-10-22 11:47:42,063: ERROR/ForkPoolWorker-1] Loading failure, updating to `failed` status
swh-loader_1                     | Traceback (most recent call last):
swh-loader_1                     |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/core/loader.py", line 339, in load
swh-loader_1                     |     self.store_data()
swh-loader_1                     |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/core/loader.py", line 457, in store_data
swh-loader_1                     |     for directory in self.get_directories():
swh-loader_1                     |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/git/loader.py", line 376, in get_directories
swh-loader_1                     |     yield converters.dulwich_tree_to_directory(raw_obj)
swh-loader_1                     |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/git/converters.py", line 104, in dulwich_tree_to_directory
swh-loader_1                     |     check_id(dir_)
swh-loader_1                     |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/git/converters.py", line 39, in check_id
swh-loader_1                     |     f"Expected {type(obj).__name__} hash to be {obj.id.hex()}, "
swh-loader_1                     | swh.loader.git.converters.HashMismatch: Expected Directory hash to be 6ac72a0858a5d5028d7f502de8777fbd5bdb8cae, got e23127f28dd0e1cf6e92a7b81cb9dbc53b44aa37
$ time git clone https://github.com/technoweenie/attachment_fu
Cloning into 'attachment_fu'...
remote: Enumerating objects: 1737, done.
remote: Total 1737 (delta 0), reused 0 (delta 0), pack-reused 1737
Receiving objects: 100% (1737/1737), 377.44 KiB | 3.85 MiB/s, done.
Resolving deltas: 100% (740/740), done.
git clone https://github.com/technoweenie/attachment_fu  0.05s user 0.02s system 8% cpu 0.837 total
$ du -sh attachment_fu
964K    attachment_fu
vlorentz closed this task as Wontfix.Nov 26 2021, 4:41 PM
vlorentz claimed this task.

We decided to store manifests instead. T3594#74385

gitlab-migration changed the task status from Wontfix to Migrated.Jan 8 2023, 10:02 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal