Page MenuHomeSoftware Heritage

googlecode import: Analyze and fix errors
Closed, ResolvedPublic

Description

Almost every mercurial origins is in error since the last googlecode import on mercurial.
This is for referencing, analyzing and fixing those issues.

Here is the incomplete output:

$ ./kibana_fetch_logs.py > hg-loader-error-logs-from-kibana-20180215-20180216.txt
$ cat hg-loader-error-logs-from-kibana-20180215-20180216.txt| ./group_by_exception.py --loader-type hg | jq .
{
 ...
  "googlecode": {
    "total": 123129,
    "errors": {
      "FileNotFoundError(2, \"No usable temporary director": 123066,  # that's T694
      "OSError(28, 'No space left on device')": 27,                                         # that's T694 in another part of the loading
      "PatoolError('error extracting /srv/storage/space/m": 19,               
      "CommandError(b'bundle', b'-t', b'none-v2', b'-a', ": 17,                      
    }
  },
...
}

Event Timeline

ardumont created this task.Feb 16 2018, 2:13 PM
"PatoolError('error extracting /srv/storage/space/m": 19,

Those errors are also an instance of T694 (worker 13 out of disk -> consumes and fail lightingly fast the remaining tasks)

$ grep -c PatoolError hg-loader-error-logs-from-kibana-20180215-20180216.txt
19
$ grep  PatoolError hg-loader-error-logs-from-kibana-20180215-20180216.txt | grep -ci 'no space left on device'
19

"CommandError(b'bundle', b'-t', b'none-v2', b'-a', ": 17,

I have checked injection of some of the repositories listed in there and they are fine.
My guess is that it's also T694.

List of those paths:

$for f in $(grep CommandError hg-loader-error-logs-from-kibana-20180215-20180216.txt | ./dump_archive_path.py); do echo $(stat -c %s $f) $f; done                                                     
535662129 /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/a/a-smop4/a-smop4-source-archive.zip
328021882 /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/a/a384088-sl/a384088-sl-source-archive.zip
60063915 /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/a/abishekahluwalia-testgs/abishekahluwalia-testgs-source-archive.zip
1463999 /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/a/actron-v4-006/actron-v4-006-source-archive.zip
68614328 /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/a/aexyno-tracks/aexyno-tracks-source-archive.zip
84421363 /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/a/agilefaizan-testingapp/agilefaizan-testingapp-source-archive.zip
40626654 /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/a/agungwinarto779-testing/agungwinarto779-testing-source-archive.zip
66257177 /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/a/alanlvee-drive-sample/alanlvee-drive-sample-source-archive.zip
34151244 /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/a/alclsdlddj-asd/alclsdlddj-asd-source-archive.zip
5759479 /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/a/aleksblacktag-clone/aleksblacktag-clone-source-archive.zip
54294940 /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/j/jigardesai12-jig12/jigardesai12-jig12-source-archive.zip
54287376 /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/j/jigardesai12-jigar/jigardesai12-jigar-source-archive.zip
54291860 /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/j/jigardesai12-jigar1/jigardesai12-jigar1-source-archive.zip
23577949 /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/j/jigneshpabarilinux-testgraph/jigneshpabarilinux-testgraph-source-archive.zip
23642768 /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/j/johannchien-sss/johannchien-sss-source-archive.zip
32672190 /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/j/johannessaam-trunk/johannessaam-trunk-source-archive.zip
2490217 /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/x/xiaowei8868-asmack/xiaowei8868-asmack-source-archive.zip

I have a problem in the log fetcher (that's why it's said to be incomplete in the task description).

I have another issue, the tags can contain empty space and it's not correctly dealt with on some origins.
(e.g. /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/0/0nima0-f/0nima0-f-source-archive.zip)

[2018-02-15 17:19:51,316: ERROR/Worker-1] Loading failure, updating to `partial` status
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 861, in load
    self.store_data()
  File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 974, in store_data
    self.send_batch_releases(self.get_releases())
  File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 660, in send_batch_releases
    send_in_packets(releases, self.send_releases, packet_size)
  File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets
    for obj in objects:
  File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 409, in get_releases
    node, name = t.split(b' ')
ValueError: too many values to unpack (expected 2)
ardumont added a comment.EditedFeb 16 2018, 3:11 PM

I have a problem in the log fetcher (that's why it's said to be incomplete in the task description).

Well, somehow related to the issue. Stracktrace exceptions, instead of being fully stored in one log message, are splitted into multiple log messages...
It's the way the messages are stored though as i found those in the kibana instance as well.
All those logs are exclusively coming from the worker13 (the one out of disk in T694 so that might be related).

Last relevant error found in crossing multiple streams... (noooooo).

archive: /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/j/johnbrugge-pipeline-modules/johnbrugge-pipeline-modules-source-archive.zip

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 861, in load
self.store_data()
File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 974, in store_data
self.send_batch_releases(self.get_releases())
File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 660, in send_batch_releases
send_in_packets(releases, self.send_releases, packet_size)
File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets
for obj in objects:
File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 412, in get_releases
'target': hashutil.hash_to_bytes(node.decode()),
File "/usr/lib/python3/dist-packages/swh/model/hashutil.py", line 261, in hash_to_bytes
return bytes.fromhex(hash)
ValueError: non-hexadecimal number found in fromhex() arg at position 0

Well, sure, if the .hgtags is corrupted...

<<<<<<< local
bc59558bb7053b68c57a16bcfde98fe1e6d403a1 bks-epub3-beta
4571d156ffd8246314e51269fc02ca60ded15b49 bks-epub3-beta2
bfc11100e4a48073cb2571b42a8a415daa8da163 bks-epub3-1.3
=======
ca2d0d49533e7b3a6309f62417fd042e060fbfa0 1.3
>>>>>>> other
99bca97d1bcc9b2c243d2d0c555519fe1f6f2f12 bks-epub3-1.3.1
ardumont added a comment.EditedFeb 20 2018, 4:49 PM

Latest version with fixes deployed.
Still remains actions on T964 and clean up on T976 before scheduling another run.

ardumont closed this task as Resolved.Feb 20 2018, 4:49 PM