Symmetrically to other loader reports, here is one on the current redundant failures we have on our git origins.
The period covered is from August to October 2018 (in progress):
```
$ cat git-output-august-october-2018.txt | python3 -m group_by_exception --loader-type git --no-aggregate | jq .
{
"total": 821971167689,
"errors": {
"dulwich.errors.GitProtocolError: unexpected http resp 401 for": 53481,FileNotFoundError(2, 'No such file or directory')": 648242, // <- worker03 had a missing mount point, had the issue with mercurial loader
"gai"dulwich.error(-3, 'Temporary failure in names.GitProtocolError: unexpected http resolution')": 17107p 401 for": 403090,
"StorageAPIError(ConnectionError(MaxRetry"psycopg2.extensions.QueryCanceledError('None: Max retries exceeded with url: : canceling statement due to statement timeout": 33056628,
"psycopg2.extensions.QueryCanceledError: canceling statement due to statement timeout": 2125"502 Bad Gateway": 34300,
"502 Bad Gateway"gaierror(-3, 'Temporary failure in name resolution')": 19237493,
"swh.objstorage.exc.ObjStorageAPIError: An unexpected error occurred in the api backend:": 1302"consumer: Cannot connect to amqp": 6649,
"socket.gaierror: [Errno -3] Temporary failure in name resolution": 557"pgbouncer cannot connect to server": 4811,
"consumer: Cannot connect to amqp": 482"StorageAPIError(ConnectionError(MaxRetryError('None: Max retries exceeded with url: ": 3307,
"ValueError: invalid literal for int() with base 16: b''": 426"Worker exited prematurely: signal 9": 2749,
"OSError: Pack file too big for repository"MemoryError": 260164,
"MemoryError": 232"ValueError: invalid literal for int() with base 16: b''": 1898,
"Worker exi"swh.objstorage.exc.ObjStorageAPIError: An unexpected prematurely: signal 9": 225error occurred in the api backend:": 1309,
"dulwich.e"OSErrors.GitProtocolError: unexpected http resp 503": 166: Pack file too big for repository": 978,
"ConnectionResetError: [Errno 104] Connection reset by peer": 158"TypeError: snapshot_add() got an unexpected keyword argument 'back_compat'": 890,
"http.client.IncompleteRead:": 137"ConnectionResetError: [Errno 104] Connection reset by peer": 597,
"dulwich."socket.gaierrors.NotGitRepository": 70: [Errno -3] Temporary failure in name resolution": 563,
"504 Gateway Time-out": 61"http.client.IncompleteRead:": 509,
"ConnectionResetE"dulwich.error(104, 'Connection reset by peer')": 34s.NotGitRepository": 231,
"BrokenPipeError: [Errno 32] Broken pipe": 33208,
"dulwich.errors.ObjectFormatException: invalid literal for int() with base 10:": 30GitProtocolError: unexpected http resp 503": 168,
"psycopg2.IntegrityError: duplicate key value violates unique constraint "504 Gateway Time-out": 1727,
"psycopg2.extensions.TransactionRollbackError: deadlock detected": 15"dulwich.errors.ObjectFormatException: invalid literal for int() with base 10:": 115,
"SSL connec"StorageDBError(Operation has been closed unexpectedly": 8alError('could not connect to server: Connection refused": 113,
"Value"psycopg2.IntegrityError: year is out of range": 7duplicate key value violates unique constraint ": 102,
"psycopg2.Integrity"ConnectionResetError: null value in column ": 7(104, 'Connection reset by peer')": 89,
"dulwich.e"psycopg2.InternalErrors.GitProtocolError: unexpected http resp 500": 6: current transaction is aborted, commands ignored until end of transaction block": 53,
"OSError(timeout('timed out',),)": 4"psycopg2.extensions.TransactionRollbackError: deadlock detected": 39,
"OSError(ConnectionRefused"psycopg2.IntegrityError(111, 'Connection refused')": 4: null value in column ": 38,
"psycopg2.InternalError: current transa"SSL connection is aborted, commands ignorhas been closed until end of transaction block": 3expectedly": 29,
"pgbouncer cannot connect to server": 2socket.gaierror: [Errno -5] No address associated with hostname": 25,
"psycopg2.DatabaseError: server conn crashed?": 22,
"ValueError: invalid literal for int() with base 16:": 20,
"OSError(ConnectionRefusedError(111, 'Connection refused')": 15,
"ValueError: year is out of range": 14,
"KeyError": 14,
"OSError(timeout('timed out',),)": 12,
"OSError(113, 'No route to host')": 12,
"dulwich.errors.GitProtocolError: unexpected http resp 500": 11,
"TimeoutError: [Errno 110] Connection timed out": 8,
"Worker exited prematurely: signal 15 (SIGTERM).',)": 6,
"socket.gaierror: [Errno -2] Name or service not known": 5,
"dulwich.errors.GitProtocolError: unexpected http resp 502 for": 4,
"dulwich.errors.HangupException: The remote server unexpectedly closed the connection.": 4,
"TimeoutError(110, 'Connection timed out')": 4,
"OSError: [Errno 113] No route to host": 3,
"ValueError: Wrong content type `text/html` for API response": 3,
"psycopg2.OperationalError: terminating connection due to administrator command": 2,
"dulwich.errors.GitProtocolError: unexpected http resp 502 forOverflowError: timestamp out of range for platform time_t": 2,
"amqp.exceptions.NotFound: Basic.publish: (404) NOT_FOUND": 2,
"CONNECTION_FORCED - broker forced connection closure with reason": 2,
"OSError(OSError(113, 'No route to host'),)": 2,
"Timed out waiting for UP message from": 1,
"psycopg2.OperationalError: index row size": 1,
"OverflowE"dulwich.error: timestamp out of range for platform time_ts.ObjectFormatException: Unknown field": 1,
"Key"dulwich.errors.GitProtocolErrorr: unexpected http resp 429": 1,
"dulwich.errors.ObjectFormatException: Unknown fieldgaierror(-2, 'Name or service not known')": 1,
"OSError(5, 'Input/output error')": 1,
"msgpack.exceptions.UnpackValueError: Unpack failed: error = 0": 1,
"dulwich.e"ValueErrors.GitProtocolError: unexpected http resp 429: not enough values to unpack": 1
}
}
```
As usual, some are issues, some are not (e.g 401)
Associated to the error is the number of occurrences.
That will help to determine the next issue to fix.
This is a meta-task to open and track the associated issue if it is deemed one.
dashboard: http://kibana0.internal.softwareheritage.org:5601/app/kibana#/dashboard/22195930-d36e-11e8-913b-077937c6a5ef
Related T1246#23774
Related T1159#23772