Page MenuHomeSoftware Heritage

Gitorious import: ObjectFormatException raised when badly formatted tag object exists in the repository
Closed, ResolvedPublic

Description

On some disk repository, errors occur on badly formatted object.

Steps to reproduce on a local storage with latest swh-loader-git

Use /srv/storage/space/mirrors/gitorious.org/mnt/repositories/hylafax/hylafax.git:

repo = 'hylafax.git'
origin_url = 'http://foo/bar/git'

import logging
logging.basicConfig(level=logging.DEBUG)

from swh.loader.git.tasks import LoadDiskGitRepository

t = LoadDiskGitRepository()
t.run(origin_url=origin_url, directory=repo, date='2016-05-03T15:16:32+00:00')

output:

DEBUG:swh.scheduler.task.LoadDiskGitRepository:Creating git origin for http://hylafax.git
DEBUG:swh.scheduler.task.LoadDiskGitRepository:Done creating git origin for http://hylafax.git
Traceback (most recent call last):
  File "./load-git-disk.py", line 26, in <module>
    main()
  File "/usr/lib/python3/dist-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "./load-git-disk.py", line 22, in main
    date='2016-05-03T15:16:32+00:00')
  File "/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/swh/scheduler/task.py", line 35, in run
    raise e from None
  File "/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/swh/scheduler/task.py", line 32, in run
    result = self.run_task(*args, **kwargs)
  File "/home/tony/work/inria/repo/swh/swh-environment/swh-loader-git/swh/loader/git/tasks.py", line 39, in run_task
    return loader.load(origin_url, directory, dateutil.parser.parse(date))
  File "/home/tony/work/inria/repo/swh/swh-environment/swh-loader-git/swh/loader/git/base.py", line 422, in load
    self.fetch_data()
  File "/home/tony/work/inria/repo/swh/swh-environment/swh-loader-git/swh/loader/git/loader.py", line 48, in fetch_data
    type_name = self.repo[oid].type_name
  File "/usr/lib/python3/dist-packages/dulwich/repo.py", line 470, in __getitem__
    return self.object_store[name]
  File "/usr/lib/python3/dist-packages/dulwich/object_store.py", line 119, in __getitem__
    return ShaFile.from_raw_string(type_num, uncomp, sha=sha)
  File "/usr/lib/python3/dist-packages/dulwich/objects.py", line 391, in from_raw_string
    obj.set_raw_string(string, sha)
  File "/usr/lib/python3/dist-packages/dulwich/objects.py", line 303, in set_raw_string
    self.set_raw_chunks([text], sha)
  File "/usr/lib/python3/dist-packages/dulwich/objects.py", line 308, in set_raw_chunks
    self._deserialize(chunks)
  File "/usr/lib/python3/dist-packages/dulwich/objects.py", line 747, in _deserialize
    raise ObjectFormatException("Unknown field %s" % field)
dulwich.errors.ObjectFormatException: Unknown field b'Thu'

Note: load-git-disk.py is a wrapper around the scenario described (cf. P185)

Event Timeline

ardumont renamed this task from Gitorious import: to Gitorious import: ObjectFormatException on what looks like a date field.Oct 26 2017, 3:48 PM
ardumont updated the task description. (Show Details)

Patching the version to print the identifier in error, i retrieve the following object ae51106031a0bb39a8def57a8592f70116487eab (which is amongst the badly formatted tags listed by git fsck below).

Checking that repository again with git fsck yields errors relative to date as well:

...
error in tag ae51106031a0bb39a8def57a8592f70116487eab: missingSpaceBeforeDate: invalid author/committer line - missing space before date
...

full output:

Checking object directories: 100% (256/256), done.
error in tag 42fb8f85e940bf34ca0d6953cde3cea24d30aeec: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 18047fb371daea89f5a7019a3cc6fcc9e8b7b810: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag e933d75f50ba4cb7e0a57797a06b661592664658: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 9f013e6a42be1ae853c17129ad87977574b50750: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag fc1291499d5fe2e0c70fd08b76fabb07bddb133e: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 43155d1dbe18f47eac2a5d05ff87fe1e54f328b8: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag dfc9daf8fb126e3bfe3c42e7ff0e6086771d34a9: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 8980cf11bf5b1a87f099763aa4bfe02dc0656d57: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 5c1fb1d5a7b6a06c7d327c595a350ad47f537c65: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 9687aab16e42fec40152b86ffadfd39bce098edb: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag de9008c8c3d157419cc62eb3b02dfa9134fe1540: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 12078c51eb6a710373caa99c4f61b7f16c799115: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 8907d5129b897fbc94afcd77eb6d6e43b2800031: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 1455537089bbce154d9810f8f5c9e446d24b2bdb: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 037dbbdba40e367a092ab286f5250bbec08dfdfe: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 165be8aaedd29accb5a43dacae3d99d2e4180cfa: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag fb31e111ecc29efa09fc066387c10c1d2bac2091: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 1debe514dad7cc474092716f6eb5eb19e541d4b5: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 1bec4dc9b398907fe1f240d62e2b5a730b94e783: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag f9b5f56ee5ce763548de8f42e5dc8f9e8567fedf: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag d4c3630fdc1fe981c71d5ab57e261f5d4389b4d8: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 44208693a1155b92a2b9c386e145809b6d16b3e4: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag ddb41113b1133e809d89d3eb483f413f85ae239d: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 3dc07482c3c88f236a5fba6970d9bff6ce7f6d18: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 5e89ec5a119ca112dc0c354a8b67ce915168a531: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag a721c432c8a17b97814550b47ed61689a252db09: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 13de33bd4a8e7bff87402a576e315e4b4f4d7e1f: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag eb215a54d6ef57986659481fe754e4f0137d4e31: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 0e8f774c7fcd02dacb504548a8d702f92ae520c2: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 1711f4f39244d73ca37b23ff800d93298258fc9f: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag dc9f71c747a0051563167ee7e99d67b3286da902: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 48d0e878f6ba7d56a7df4696b883ab9f455faeb1: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 1fca6178a056c77766b1a75e3a83fd89b08c00ed: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 68edb648841c04e784bf861788e4500b5604609c: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 5ff635022b54a8cb343665f6c845d082bb4ff87a: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 1b6115c57de3f678bd467dca13b680a4c29a0b99: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 4e6557b4f044e9ef035eb5529071943d2952611b: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag d899bf7d61d22c84a7f050ef09dae0f60f06090a: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 29f267ed94c0c36e40636ec162fa1f3dda46013c: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag ed604b278b11c10a87420855a43505e5d3366ea3: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag fa3eda7e2c22180ef4715a11fe995d59b33cba8f: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag fe8e893829afe37c55b7bf78c1ac490c74dc3d46: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 5dd359b34f12bd00338485fb37caba1411a3a02b: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 6954aa47b75a12d73cb02cadaacc574ca5e0c3b4: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 04d00a6127d493b04931742afcc95196a6f5d174: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 70cbc5c24c0c050a45dd2a8b90462a35189964e8: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag a010e5dc7bed4c40c6502075af4c0750cdee29b3: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 961b4f4363d53983217c801f6bf83ba76855ac4b: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 4cfc0e53943826bddd99dd5917d1869350bb0089: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 1ce0934d125f38bfa1fdf39b2ea4a195c6d5d2f7: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 619164ee851693697e1f4a4d911dcbc13e52515f: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 62b19354bcbdb731511e65dad433373934dca79a: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 6bf0c3d9e025fbfaaa2c5e947e6b703dbd6b4c45: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag f27ebf67b6f542acfd9f8cbd18587f2e9771851e: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 1633d9f0f64d20ea57e68559263dee2277175685: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 22bdd6e3b9edd341deaeac7571b90afb501176d8: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag c1d79c5faf2bb8b46411f977cd1ef92df140efa2: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag a3371c02ac951a1e29d82fffe28e5d548804ed21: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag dcae980b3bb15c625c701f7846dba660f0e96ec6: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 95d2c078318b2cde8c04ea6c24f72ebef3a59da9: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 31a8bcbfde568d92d0f10e2bb1c039b78912bc02: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 04f9bc021cfe7214aeaadea7b692f3abf14e87ed: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 5f42dacb1ddebb3bf547f8a1f05992c81d663155: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag d91e435af77309e0879e63f0c61a4dff96444771: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag f514b306fa94ee0161a97087cd5f6f5bea060e2d: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 780bd2af5ce4cc3cb5328d67c5fef282c00982e8: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag b61729995904f3ff2ec33041a8c1959bc995f952: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 63eae36e124768e89a7c8cd46d7e7e2254a05668: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 73a6526efb452dbe49d079421d989fa400334da9: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 18c8325e76284d5fa72e5736bd4263456bb17e32: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 554998aad2d4b2bc94c16ad10f9e8f3bd539af51: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 16d91fc6f0e5b53e28ccb13ea229da2fa9f24eeb: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 6bdd7296e09e2469a7a52cdcaf22a10ea5a79e37: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 21e48ff79b6d1f9634bf8feba35e602d05f0cc19: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 6a616632aca8916c626911315be3991253540228: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag f58623a822a651056f278002d59792c3f2ef3820: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 8f2fda936a5470168f56fadc1b22a9577f93836e: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag ee0f8dcc678cc0a706a1a6d393b71b761901c2b3: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 09f55938f55c2888ded6ae46866e386d6f933fe6: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 8ac7ba1e633047e2fba52cbc466480a57229afba: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 1c685d94637318a15ce5d88bb3e427c84c7b1534: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag e551a54519f418cb709fdf1d6854f14fb7abf2e2: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag e14cbfd8d5894b0050141b069551f1e29edc9274: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag d966dbd22f6eb27baed6cf810a9d9d14210eb589: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag f91c7a52cdf20279eaafb0c536efcebfc8c7cfd4: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 9864248bfeb4737448bab2fe368eef4798809578: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag ee6406a3bd659517089987b591e5438979cb42a3: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 69d5843e81901a1a2838267adff75e0a56d33786: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag e4128d8da0aaf886a5f095ffeb992f320f02bd88: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 20f03614ec5b38e1f3845897fb06b7cc6375bf8f: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 36dbbf3912cc2cf83022dbfff1ee081733c4b175: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag ae51106031a0bb39a8def57a8592f70116487eab: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag ca8a83070009d21b31830d8ed4d229684a5bce7d: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 059bc1a937b3ed222e46811bde608c29b9eef510: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 0c8a89f3c165324567bbbf63752a390f71f6776e: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 44b2d20c1ce81344afce8a67164e3ab926fa6743: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag c57f007be567b5f0a8521fda96b2180d602f195a: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 046204f51910b98c31381b034435ee027a56d8d1: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag b5919249d73993b2562ef1562eef01d23a8b2ebc: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag c1b873cc8ec0aabc39b0c2467e6fe96f3f96836f: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 6d386e8cc721052b852b11b5650582e3d3dab2c2: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag da3e3f24a23f3c98cf68766785d176bd53365b9b: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 62708e43733193c3eade2f0eb13d1bc0b9d8af80: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag f4447248d2d0e187090eaa66efde9705327e0c54: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag a28165de4090b8affc287b144a5d9c96e7f724e4: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag f49d58c87fa67a19e69703c8bf47d8433dc7dc96: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 75d2b79da8c458de962438f5c254cd4d73fb291b: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag b2b424bc58527e5faf4f4fd74a1efd09a9d41f9e: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 49aebc1b56900007a400d4563896c931223a74c4: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag f2f4c442a694a0c0af4d3931e96b67c70515cbf5: missingSpaceBeforeDate: invalid author/committer line - missing space before date
error in tag 56daffb5be0fa5c4b748ff1501dae3acbaf61784: missingSpaceBeforeDate: invalid author/committer line - missing space before date
Checking objects: 100% (16095/16095), done.
dangling tag 63e1993c2c2af7714181c7c4ca05e6a292347bfb
ardumont renamed this task from Gitorious import: ObjectFormatException on what looks like a date field to Gitorious import: ObjectFormatException raised when badly formatted tag object exists in the repository.Oct 26 2017, 4:07 PM