Page MenuHomeSoftware Heritage
Paste P437

wikimedia phabricator lister: ChunkedEncodingError
ActivePublic

Authored by ardumont on Jun 18 2019, 1:34 PM.
Jun 18 11:24:42 worker01 python3[12919]: [2019-06-18 11:24:42,238: INFO/MainProcess] Received task: swh.lister.phabricator.tasks.FullPhabricatorLister[551ff6d5-473b-4e6f-8e70-9d8041ab5fce]
Jun 18 11:27:32 worker01 python3[12934]: [2019-06-18 11:27:32,544: ERROR/ForkPoolWorker-5] Task swh.lister.phabricator.tasks.FullPhabricatorLister[551ff6d5-473b-4e6f-8e70-9d8041ab5fce] raised unexpected: ChunkedEncodingError(ProtocolError('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read)),)
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/response.py", line 601, in _update_chunk_length
self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/response.py", line 360, in _error_catcher
yield
File "/usr/lib/python3/dist-packages/urllib3/response.py", line 666, in read_chunked
self._update_chunk_length()
File "/usr/lib/python3/dist-packages/urllib3/response.py", line 605, in _update_chunk_length
raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/models.py", line 750, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/usr/lib/python3/dist-packages/urllib3/response.py", line 490, in stream
for line in self.read_chunked(amt, decode_content=decode_content):
File "/usr/lib/python3/dist-packages/urllib3/response.py", line 694, in read_chunked
self._original_response.close()
File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python3/dist-packages/urllib3/response.py", line 378, in _error_catcher
raise ProtocolError('Connection broken: %r' % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 382, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/lib/python3/dist-packages/swh/scheduler/task.py", line 45, in __call__
return super().__call__(*args, **kwargs)
File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 641, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/lib/python3/dist-packages/swh/lister/phabricator/tasks.py", line 24, in full_phabricator_lister
lister.run()
File "/usr/lib/python3/dist-packages/swh/lister/phabricator/lister.py", line 152, in run
super().run(min_bound, max_bound)
File "/usr/lib/python3/dist-packages/swh/lister/core/indexing_lister.py", line 227, in run
for i in ingest_indexes():
File "/usr/lib/python3/dist-packages/swh/lister/core/indexing_lister.py", line 209, in ingest_indexes
response, injected_repos = self.ingest_data(index)
File "/usr/lib/python3/dist-packages/swh/lister/core/lister_base.py", line 482, in ingest_data
response = self.safely_issue_request(identifier)
File "/usr/lib/python3/dist-packages/swh/lister/core/lister_base.py", line 291, in safely_issue_request
r = self.transport_request(identifier)
File "/usr/lib/python3/dist-packages/swh/lister/core/lister_transports.py", line 182, in transport_request
return self._transport_action(identifier)
File "/usr/lib/python3/dist-packages/swh/lister/core/lister_transports.py", line 161, in _transport_action
response = self.session.get(path, **params)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 546, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 686, in send
r.content
File "/usr/lib/python3/dist-packages/requests/models.py", line 828, in content
self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
File "/usr/lib/python3/dist-packages/requests/models.py", line 753, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

Event Timeline

docker-run tryout for that same forge was fine.

$ doco exec swh-lister bash -c 'psql swh-listers'
psql (9.6.13, server 11.2 (Debian 11.2-1.pgdg90+1))
WARNING: psql major version 9.6, server major version 11.
         Some psql features might not work.
Type "help" for help.

swh-listers=# \conninfo
You are connected to database "swh-listers" as user "postgres" on host "swh-listers-db" at port "5432".
swh-listers=# select instance, count(*) from phabricator_repo group by instance;
 instance  | count
-----------+-------
 blender   |    44
 llvm      |    14
 wikimedia |  2224
(3 rows)

Related T1801

ardumont changed the title of this paste from wikimedia lister: ChunkedEncodingError to wikimedia phabricator lister: ChunkedEncodingError.Jun 18 2019, 1:37 PM

Might be a flaky thing somewhere in our stack.
Rescheduling that task and it worked.

13:36:40 softwareheritage-scheduler@belvedere:5432=> select * from task where type in ('list-phabricator-full', 'list-phabricator-incremental');
┌───────────┬──────────────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────┬───────────────────────────────┬──────────────────┬────────────────────┬───────────┬──────────────┬──────────┐
│    id     │             type             │                                              arguments                                              │           next_run            │ current_interval │       status       │  policy   │ retries_left │ priority │
├───────────┼──────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────┼──────────────────┼────────────────────┼───────────┼──────────────┼──────────┤
│ 167877143 │ list-phabricator-full        │ {"args": [], "kwargs": {"instance": "swh", "forge_url": "https://forge.softwareheritage.org"}}      │ 2019-06-17 12:52:32.171833+00 │ 90 days          │ next_run_scheduled │ recurring │            0 │ ¤        │
│ 167877665 │ list-phabricator-full        │ {"args": [], "kwargs": {"instance": "wikimedia", "forge_url": "https://phabricator.wikimedia.org"}} │ 2019-06-18 11:24:15.622736+00 │ 90 days          │ next_run_scheduled │ recurring │            0 │ ¤        │
│ 167877620 │ list-phabricator-full        │ {"args": [], "kwargs": {"instance": "blender", "forge_url": "https://developer.blender.org"}}       │ 2019-06-18 11:17:02.302068+00 │ 90 days          │ next_run_scheduled │ recurring │            0 │ ¤        │
│ 167877509 │ list-phabricator-full        │ {"args": [], "kwargs": {"instance": "llvm", "forge_url": "https://reviews.llvm.org"}}               │ 2019-06-18 08:59:16.620607+00 │ 90 days          │ next_run_scheduled │ recurring │            0 │ ¤        │
│ 167877295 │ list-phabricator-incremental │ {"args": [], "kwargs": {"instance": "swh", "forge_url": "https://forge.softwareheritage.org"}}      │ 2019-06-17 16:42:58.010865+00 │ 7 days           │ next_run_scheduled │ recurring │            0 │ ¤        │
└───────────┴──────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────────────────────────┴──────────────────┴────────────────────┴───────────┴──────────────┴──────────┘
(5 rows)

Time: 4.337 ms
13:36:41 softwareheritage-scheduler@belvedere:5432=> \q
$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ task respawn 167877665
Respawn tasks ('167877665',)

Let's just see if that occurs again later.