Page MenuHomeSoftware Heritage

package.loader: catch the EOFError exception in uncompress function
ClosedPublic

Authored by lewo on Fri, Mar 20, 11:29 AM.

Details

Summary

This exception has been seen on staging but we were unfortunately not
able to reproduce it. It could be related to underlay storage
issues. Here is the traceback:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/swh/loader/package/loader.py", line 291, in load
  self._load_revision(p_info, origin)
File "/usr/lib/python3/dist-packages/swh/loader/package/loader.py", line 368, in _load_revision
  uncompressed_path = self.uncompress(dl_artifacts, dest=tmpdir)
File "/usr/lib/python3/dist-packages/swh/loader/package/loader.py", line 211, in uncompress
  uncompress(a_path, dest=uncompressed_path)
File "/usr/lib/python3/dist-packages/swh/core/tarball.py", line 72, in uncompress
  shutil.unpack_archive(tarpath, extract_dir=dest)
File "/usr/lib/python3.7/shutil.py", line 999, in unpack_archive
  func(filename, extract_dir, **kwargs)
File "/usr/lib/python3.7/shutil.py", line 934, in _unpack_tarfile
  tarobj.extractall(extract_dir)
File "/usr/lib/python3.7/tarfile.py", line 2002, in extractall
  numeric_owner=numeric_owner)
File "/usr/lib/python3.7/tarfile.py", line 2044, in extract
  numeric_owner=numeric_owner)
File "/usr/lib/python3.7/tarfile.py", line 2114, in _extract_member
  self.makefile(tarinfo, targetpath)
File "/usr/lib/python3.7/tarfile.py", line 2163, in makefile
  copyfileobj(source, target, tarinfo.size, ReadError, bufsize)
File "/usr/lib/python3.7/tarfile.py", line 247, in copyfileobj
  buf = src.read(bufsize)
File "/usr/lib/python3.7/bz2.py", line 178, in read
  return self._buffer.read(size)
File "/usr/lib/python3.7/_compression.py", line 68, in readinto
  data = self.read(len(byte_view))
File "/usr/lib/python3.7/_compression.py", line 99, in read
  raise EOFError("Compressed file ended before the "
EOFError: Compressed file ended before the end-of-stream marker was reached

This happened when decompressing the file
https://downloads.sourceforge.net//asio/asio-1.12.1.tar.bz2 but the
loader executed locally can decompress it without any issue.

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

lewo created this revision.Fri, Mar 20, 11:29 AM
lewo added a subscriber: ardumont.Fri, Mar 20, 11:29 AM

Did you try truncating a .tar file to reproduce the issue?

Or rather, a .tar.bz2

ardumont added inline comments.Fri, Mar 20, 12:45 PM
swh/loader/package/loader.py
381 ↗(On Diff #10173)

you can add a test on functional loader which patches the uncompress method to raise this.

lewo added a comment.Fri, Mar 20, 3:21 PM

Did you try truncating a .tar file to reproduce the issue?

No, and that's a good idea!
I just tried but unfortunately, I didn't reproduce;(

lewo added a comment.Fri, Mar 20, 3:27 PM

I actually succeeded with a bigger truncated file actually: 70KB

lewo updated this revision to Diff 10186.Fri, Mar 20, 4:13 PM

I added a test for the EOFError.

lewo updated this revision to Diff 10187.Fri, Mar 20, 4:15 PM

Fix a typo

D2860 and especially are going your way D2862 ;)

lewo added a comment.Mon, Mar 23, 10:44 AM

What about this diff? I think it's still valuable since it catch more precisely errors and add a test case;)
WDYT?

Yes, i think we can land it after D2860 and D2862 ;)

D2862 will removes the try: except in the _load_revision and captures all in the main loop.
So in effect, that will make that diff add a scenario and not touch the code.

ardumont requested changes to this revision.Mon, Mar 23, 1:03 PM

Needs a rebase now ;)

And then i'll accept it.

This revision now requires changes to proceed.Mon, Mar 23, 1:03 PM
lewo updated this revision to Diff 10207.Mon, Mar 23, 2:21 PM

Rebased

lewo updated this revision to Diff 10208.Mon, Mar 23, 2:22 PM

UPdate cmomit message to only mention the test

ardumont accepted this revision.Mon, Mar 23, 2:35 PM

Thanks!

This revision is now accepted and ready to land.Mon, Mar 23, 2:35 PM
This revision was automatically updated to reflect the committed changes.