Page MenuHomeSoftware Heritage
Create Task
Maniphest T3584

loader mercurial edge case about missing mapping from revision to hgnode-id
Closed, MigratedEdits Locked
Actions

Description

Some missing mapping somehow happens which prevent the loader from finishing [2]

I recall seeing that this part is not covered in the loader so we are missing the right test around this to fix it.

ERROR:swh.loader.mercurial.LoaderFromDisk:Loading failure, updating to `failed` status
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 339, in load
    self.store_data()
  File "/usr/lib/python3/dist-packages/swh/loader/mercurial/from_disk.py", line 452, in store_data
    target = self.get_revision_id_from_hg_nodeid(node_id)
  File "/usr/lib/python3/dist-packages/swh/loader/mercurial/from_disk.py", line 526, in get_revision_id_from_hg_nodeid
    assert len(from_storage) == 1, msg % (hg_nodeid.hex(), len(from_storage))
AssertionError: Expected 1 match from storage for hg node '64353138623066626536626164656565336638303062663664343638313639343833643838626338', got 0
{'status': 'failed'}

[1] P1169

[2] https://sentry.softwareheritage.org/share/issue/342aa3bb1aed46b7bf99efa205f7e081/

Revisions and Commits

rDLDHG Mercurial loader
D6329rDLDHG395ea80518cd Fix branch bookmark id format so ingestion can finish
D6300rDLDHGef502bcdf3a2 Capture missing revision <-> hgnode-id scenario in a xfail test

Related Objects

Event Timeline

ardumont renamed this task from loader mercurial edge case about missing to loader mercurial edge case about missing mapping from revision to hgnode-id.Sep 17 2021, 11:21 AM
ardumont triaged this task as Normal priority.
ardumont created this task.
ardumont added a comment.Sep 17 2021, 11:32 AM

This one seems small enough to use as data for analyzing and possibly fixing the test:

Sep 14 04:09:01 worker17 python3[354854]: [2021-09-14 04:09:01,867: INFO/ForkPoolWorker-292] Load origin 'https://bitbucket.org/avnomad/d' with type 'hg'
Sep 14 04:09:02 worker17 python3[354854]: [2021-09-14 04:09:02,826: ERROR/ForkPoolWorker-292] Loading failure, updating to `failed` status
                                          Traceback (most recent call last):
                                            File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 339, in load
                                              self.store_data()
                                            File "/usr/lib/python3/dist-packages/swh/loader/mercurial/from_disk.py", line 397, in store_data
                                              target = self.get_revision_id_from_hg_nodeid(node_id)
                                            File "/usr/lib/python3/dist-packages/swh/loader/mercurial/from_disk.py", line 471, in get_revision_id_from_hg_nodeid
                                              assert len(from_storage) == 1, msg % (hg_nodeid.hex(), len(from_storage))
                                          AssertionError: Expected 1 match from storage for hg node '38663535643238346139643464376432313166303463626336373865396632313562333034343034', got 0
Sep 14 04:09:03 worker17 python3[354854]: [2021-09-14 04:09:03,137: INFO/ForkPoolWorker-292] Task swh.loader.mercurial.tasks.LoadMercurial[fd077098-7779-4633-805b-f0cb0ad85b2a] succeeded in 1.9395398199558258s: {'status': 'failed'}

That repository is stored in our backup in saam, made a copy of it [1]:

swhworker@worker17:~$ grep https://bitbucket.org/avnomad/d /srv/storage/space/mirrors/boatbucket/mapping-to-repos.txt
...
refugees/data/57/57b64b0b-d575-4c4c-aa73-1ddb8c9a788f/main-repo.mercurial https://bitbucket.org/avnomad/d
..

[1]

anomad-d.tar.gz2 MBDownload

ardumont added a project: Mercurial loader.Sep 17 2021, 12:18 PM
ardumont added a revision: D6300: Capture missing revision <-> hgnode-id scenario in a xfail test.Sep 17 2021, 5:24 PM
ardumont updated the task description. (Show Details)Sep 22 2021, 10:39 AM
ardumont added a comment.EditedSep 23 2021, 9:50 AM

The test helped, it's a mismatch format problem.
Uncomment the test, place the right pdb stanza in the code and behold:

The hg_nodeid received is a byte string with an hexa id. [1]
The cache code is actually id in bytes.
So the mismatch makes it not found.

Transforming the stuff correctly, everything gets put back together [2]

[1]

(Pdb++) hg_nodeid
b'8f55d284a9d4d7d211f04cbc678e9f215b304404'
(Pdb++) self._revision_nodeid_to_sha1git.get(hg_nodeid)
(Pdb++) self._revision_nodeid_to_sha1git.keys()
dict_keys([b'\xde\x1f\x19\xdc\xb0\x0f\xe2\xf7\xaa]t%\xee\xe5\x02\x82\xd8\xdd\xbe\xcd', b'\xa7\xf4\x0c\xd2`\x9f\xa8\x1d\xf8\xf0\xf4,.\xd6\x9d%\x0c-\xd9#', b"\x9f\xc5\x8c'\xe7m\x9as\r\x7fpf|0\xbee'\xf3\xda\xe2", b'K\x04\xa4\xa6\x1d\xe3\xf4\xd8\x07|\x10\xb3\xb3\xbe\xfe\xe0\xe2\xffT\x0f', b'\x9f\xd4\xadw\xc5\xf5\r\xf0\x14\xf7\x07q\\\x81\xbb\x13/\xcccW', b'\xb59\xb3\x1f"\'\xcf\xa1\xaa\xb5\x13\xd3\xf9\xa5\xfcd\xad\xbf\xfc\x13', b'\xe5q\xf4\xa0\x8d\xd7^.Wv\x048\xb0{{Z\xeaVJ\x8e', b'\x8fU\xd2\x84\xa9\xd4\xd7\xd2\x11\xf0L\xbcg\x8e\x9f![0D\x04'])

[2]

(Pdb++) from swh.model import hashutil
(Pdb++) hashutil.hash_to_bytes(hg_nodeid)  
b'8f55d284a9d4d7d211f04cbc678e9f215b304404'  --------->  does not help
(Pdb++) hashutil.hash_to_bytes(hg_nodeid.decode())
b'\x8fU\xd2\x84\xa9\xd4\xd7\xd2\x11\xf0L\xbcg\x8e\x9f![0D\x04'
(Pdb++) test = hashutil.hash_to_bytes(hg_nodeid.decode())   -------------> make a little more effort
(Pdb++) self._revision_nodeid_to_sha1git[test]
b'\x03/s*\xe5yhp\x00\x1e5\xa5\xc0r*\xe0#\xf3\xa3\xf3'    -------------> ok then! So there is something amiss somewhere
ardumont added a comment.Sep 23 2021, 10:03 AM

Ok, found where the wrong format is found somehow, the branching_info.bookmarks is not in the right format.

(Pdb++) branching_info.tips
{b'default': b'\x8fU\xd2\x84\xa9\xd4\xd7\xd2\x11\xf0L\xbcg\x8e\x9f![0D\x04'}
(Pdb++) branching_info.bookmarks
{b'master': b'8f55d284a9d4d7d211f04cbc678e9f215b304404'}
ardumont added a subscriber: Alphare.Sep 23 2021, 10:13 AM

@Alphare any clues as to why the format here is not in sync? ^

{b'master': b'8f55d284a9d4d7d211f04cbc678e9f215b304404'}

shoud be more like this one:

{b'default': b'\x8fU\xd2\x84\xa9\xd4\xd7\xd2\x11\xf0L\xbcg\x8e\x9f![0D\x04'}

So the mapping hgnode-id <-> revision is correctly mapped.

ardumont added a revision: D6329: Fix branch bookmark id format so ingestion can finish.Sep 23 2021, 10:14 AM
ardumont added a commit: rDLDHGef502bcdf3a2: Capture missing revision <-> hgnode-id scenario in a xfail test.Sep 23 2021, 10:54 AM
ardumont added a commit: rDLDHG395ea80518cd: Fix branch bookmark id format so ingestion can finish.
ardumont closed this task as Resolved.Sep 23 2021, 11:07 AM
ardumont claimed this task.

Deployed v2.3.1 with that fix.

gitlab-migration changed the task status from Resolved to Migrated.Jan 8 2023, 4:35 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal