save code now: Failure to ingest new 'archives' type when head response is incomplete
Closed, MigratedEdits Locked
Actions

Assigned To

Authored By

	ardumont
	Jun 7 2021, 12:29 PM

Description

The following task got submitted but it's missing some mandatory information [1]
Which is missing a timestamp (null in db) [2]

[1]

Jun 07 10:21:27 worker10 python3[2047318]: [2021-06-07 10:21:27,717: INFO/MainProcess] Received task: swh.loader.package.archive.tasks.LoadArchive[f8644e5c-9149-4e60-a112-9e0c23c1f824]
Jun 07 10:21:32 worker10 python3[2047323]: [2021-06-07 10:21:32,126: ERROR/ForkPoolWorker-1] Failed loading branch releases/eqp-09d for https://www.cs.unm.edu/~mccune/old-ftp/
                                           Traceback (most recent call last):
                                             File "/usr/lib/python3/dist-packages/swh/loader/package/loader.py", line 576, in load
                                               res = self._load_revision(p_info, origin)
                                             File "/usr/lib/python3/dist-packages/swh/loader/package/loader.py", line 719, in _load_revision
                                               p_info, uncompressed_path, directory=directory.hash
                                             File "/usr/lib/python3/dist-packages/swh/loader/package/archive/loader.py", line 159, in build_revision
                                               normalized_time = TimestampWithTimezone.from_datetime(parsed_time)
                                             File "/usr/lib/python3/dist-packages/swh/model/model.py", line 258, in from_datetime
                                               return cls.from_dict(dt)
                                             File "/usr/lib/python3/dist-packages/swh/model/model.py", line 251, in from_dict
                                               timestamp=Timestamp.from_dict(d["timestamp"]),
                                           TypeError: 'NoneType' object is not subscriptable
Jun 07 10:21:34 worker10 python3[2047323]: [2021-06-07 10:21:34,227: ERROR/ForkPoolWorker-1] Failed loading branch releases/eqp-09e for https://www.cs.unm.edu/~mccune/old-ftp/
                                           Traceback (most recent call last):
                                             File "/usr/lib/python3/dist-packages/swh/loader/package/loader.py", line 576, in load
                                               res = self._load_revision(p_info, origin)
                                             File "/usr/lib/python3/dist-packages/swh/loader/package/loader.py", line 719, in _load_revision
                                               p_info, uncompressed_path, directory=directory.hash
                                             File "/usr/lib/python3/dist-packages/swh/loader/package/archive/loader.py", line 159, in build_revision
                                               normalized_time = TimestampWithTimezone.from_datetime(parsed_time)
                                             File "/usr/lib/python3/dist-packages/swh/model/model.py", line 258, in from_datetime
                                               return cls.from_dict(dt)
                                             File "/usr/lib/python3/dist-packages/swh/model/model.py", line 251, in from_dict
                                               timestamp=Timestamp.from_dict(d["timestamp"]),
                                           TypeError: 'NoneType' object is not subscriptable
Jun 07 10:21:34 worker10 python3[2047323]: [2021-06-07 10:21:34,273: WARNING/ForkPoolWorker-1] 2 failed branches
Jun 07 10:21:34 worker10 python3[2047323]: [2021-06-07 10:21:34,273: WARNING/ForkPoolWorker-1] Failed branches: releases/eqp-09d
Jun 07 10:21:34 worker10 python3[2047323]: [2021-06-07 10:21:34,273: WARNING/ForkPoolWorker-1] releases/eqp-09e

[2]

softwareheritage-scheduler=> select * from task where type like 'load-archive-files' and arguments#>>'{kwargs,url}' = 'https://www.cs.unm.edu/~mccune/old-ftp/';
+-[ RECORD 1 ]-----+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id               | 381572629                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| type             | load-archive-files                                                                                                                                                                                                                                                                                                                                                                                                                             |
| arguments        | {"args": [], "kwargs": {"url": "https://www.cs.unm.edu/~mccune/old-ftp/", "artifacts": [{"url": "https://web.archive.org/web/20190318010151/https://www.cs.unm.edu/~mccune/old-ftp/eqp-09d.tar.gz", "time": null, "length": 0, "version": "eqp-09d"}, {"url": "https://web.archive.org/web/20190318010151/https://www.cs.unm.edu/~mccune/old-ftp/eqp-09e.tar.gz", "time": null, "length": 0, "version": "eqp-09e"}], "snapshot_append": true}} |
| next_run         | 2021-06-07 10:21:19.069381+00                                                                                                                                                                                                                                                                                                                                                                                                                  |
| current_interval | 1 day                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| status           | completed                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| policy           | oneshot                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| retries_left     | 2                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| priority         | high                                                                                                                                                                                                                                                                                                                                                                                                                                           |
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Revisions and Commits

rDWAPPS Web applications
	D5859	rDWAPPSa0db251b3280 common/origin_save/origin_exists: Handle Internet Archive artifact URLs

Related Objects

Mentioned In: T3213: Enable save code now of software source code archives for specific users
Mentioned Here: D5859: common/origin_save/origin_exists: Handle Internet Archive artifact URLs

Event Timeline

ardumont triaged this task as High priority.Jun 7 2021, 12:29 PM

ardumont created this task.

ardumont mentioned this in T3213: Enable save code now of software source code archives for specific users.Jun 7 2021, 12:31 PM

ah. It's an edge case from the start, nice!

[1] and [2] from [3] are both 404...
Thus the null timestamp which could not have been fetched through the head request...
I shall investigate why the origins got scheduled even though the head requests should
have raised or something...

[1] https://www.cs.unm.edu/~mccune/old-ftp/eqp-09d.tar.gz
[2] https://www.cs.unm.edu/~mccune/old-ftp/eqp-09e.tar.gz
[3] https://www.cs.unm.edu/~mccune/eqp/

ardumont updated the task description. (Show Details)Jun 7 2021, 2:40 PM

I did not realize immediately that the archive urls used for the ingestion (submitted by
the users) are the ones from the internet archive! Neat trick!

I was wondering how it ended up enqueued... with a length of 0...

The internet archives provides some of the required information (including the archive)
but not the Last-Modified [2] entry expected by the save code now ui... Thus the
"time" null entry which makes the loader fail so abruptely...

[1]

http --head https://web.archive.org/web/20190318010151/https://www.cs.unm.edu/~mccune/old-ftp/eqp-09d.tar.gz
HTTP/1.1 302 FOUND
Connection: keep-alive
Content-Length: 0
Content-Type: text/plain; charset=utf-8
Date: Mon, 07 Jun 2021 12:43:52 GMT
Location: https://web.archive.org/web/20161203222044/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09d.tar.gz
Permissions-Policy: interest-cohort=()
Referrer-Policy: no-referrer-when-downgrade
Server: nginx/1.15.8
Server-Timing: LoadShardBlock;dur=360.031203, PetaboxLoader3.resolve;dur=212.834143, esindex;dur=0.011231, PetaboxLoader3.datanode;dur=69.681205, exclusion.robots.policy;dur=0.246072, captures_list;dur=386.842585, exclusion.robots;dur=0.257730, RedisCDXSource;dur=1.673600, CDXLines.iter;dur=21.708820
X-App-Server: wwwb-app13
X-Archive-Redirect-Reason: found capture at 20161203222044
X-Cache-Key: httpsweb.archive.org/web/20190318010151/https://www.cs.unm.edu/~mccune/old-ftp/eqp-09d.tar.gzFR
X-NA: 0
X-NID: -
X-Page-Cache: HIT
X-RL: 0
X-Tr: 394
X-location: All
X-ts: 302

[2] https://forge.softwareheritage.org/source/swh-web/browse/master/swh/web/common/origin_save.py$213

[3] https://forge.softwareheritage.org/source/swh-loader-core/browse/master/swh/loader/package/archive/loader.py$0-154

ardumont renamed this task from save code now: Failure to ingest new 'archives' type to save code now: Failure to ingest new 'archives' type when head response is incomplete.Jun 7 2021, 5:06 PM

Thanks @ardumont for investigating this. The fact that the IA does not provide the LastModified information may make sense for their specific case (it is possible that they do not have kept the LastModified info from the original location).

What could be a decent heuristic to handle this kind of situation?

What could be a decent heuristic to handle this kind of situation?

From the point of view that archiving the origin and its tarballs is of the utmost
important, the following should be acceptable.

When the case described here arise, we could evolve slightly the webapp and the loader
so the former schedules a task with some more arguments [1] and the latter accepts null
revision commit/author time.

Details:

it should be possible to remove the artifact length from the primary key the loader internally uses (i mean possible without touching code, it will then only use the couple [version, artifact url] as internal key for each artifact (internal detail for download optimization purpose).

the loader needs to be adapted to accept that the timestamp of the revision could be null (again because it's currently ok from the revision model). [2] The webapp already allows the timestamp to be null if it's not found.

And then that should unblock the following use case.

What do you think?

[1] It can provide the key pattern to use for the ingestion.

[2] If that's an acceptable trade-off.

@ardumont, maybe we could use the timestamps of the files extracted from an archive to compute the author/committer date when not available from the HEAD response ?

For instance, with the tarball downladed from https://web.archive.org/web/20190318010151/https://www.cs.unm.edu/~mccune/old-ftp/eqp-09d.tar.gz
the creation date of folders and files are available, see screenshot below:

In the build_revision method of the ArchiveLoader, we have access to the uncompressed path of the archive, in case of missing date we could walk on the archive content and use either:

date of creation for root repository if the archive content has such a layout
use the most recent date of the files otherwise

What do you think ?

Yes, that sounds like something achievable from afar.

The upper root directory sounds like a minimal fuss but there might be no upper root
directory (we have no control on how people are currently creating artifacts). Then, if
not present, we could fallback as walking the arborescence tree to determine the most
recent date (but i have no idea how long that could take for big archives). Might be
that's marginal enough in regards to the remaining time it takes to load the actual
tarball, no idea...

So one implem could be, for the loader when the timestamp provided is null:

try to determine it from the root directory if present
if not, try to determine the most recent date from the arborescence tree
if not, then fallback to null for the revision date

thoughs?

Then, if not present, we could fallback as walking the arborescence tree to determine the most
recent date (but i have no idea how long that could take for big archives)

I guess we could avoid walking on subtrees in that case and just take the most recent date on the first level of an archive content.

I guess we could avoid walking on subtrees in that case and just take the most recent
date on the first level of an archive content.

Since we are talking of heuristic here, i guess so as well ;)

@ardumont, @rdicosmo, I just figured out that data we are missing (Content-Length, Last-Modified) from tarballs archived by the Internet archive are in fact available in x-archive-orig-* HTTP response headers, see below:

10:57 $ curl -I https://web.archive.org/web/20161203222044/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09d.tar.gz
HTTP/2 200 
server: nginx/1.15.8
date: Fri, 11 Jun 2021 09:14:20 GMT
content-type: application/x-gzip
x-archive-orig-date: Sat, 03 Dec 2016 22:17:06 GMT
x-archive-orig-server: Apache/2.2.22 (Debian)
x-archive-orig-last-modified: Fri, 02 Apr 1999 20:54:16 GMT
x-archive-orig-etag: "140c53f-1c585-3478ad04e8e00"
x-archive-orig-accept-ranges: bytes
x-archive-orig-content-length: 116101
x-archive-orig-connection: close
cache-control: max-age=1800
x-archive-guessed-content-type: application/x-gzip
memento-datetime: Sat, 03 Dec 2016 22:20:44 GMT
link: <http://www.cs.unm.edu/~mccune/old-ftp/eqp-09d.tar.gz>; rel="original", <https://web.archive.org/web/timemap/link/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09d.tar.gz>; rel="timemap"; type="application/link-format", <https://web.archive.org/web/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09d.tar.gz>; rel="timegate", <https://web.archive.org/web/20061219023759/http://www.cs.unm.edu:80/~mccune/old-ftp/eqp-09d.tar.gz>; rel="first memento"; datetime="Tue, 19 Dec 2006 02:37:59 GMT", <https://web.archive.org/web/20160315104605/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09d.tar.gz>; rel="prev memento"; datetime="Tue, 15 Mar 2016 10:46:05 GMT", <https://web.archive.org/web/20161203222044/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09d.tar.gz>; rel="memento"; datetime="Sat, 03 Dec 2016 22:20:44 GMT", <https://web.archive.org/web/20161203222044/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09d.tar.gz>; rel="last memento"; datetime="Sat, 03 Dec 2016 22:20:44 GMT"
content-security-policy: default-src 'self' 'unsafe-eval' 'unsafe-inline' data: blob: archive.org web.archive.org analytics.archive.org pragma.archivelab.org
x-archive-src: HNEWS-20161203220339-crawl890/HNEWS-20161203221959-08720.warc.gz
server-timing: captures_list;dur=103.118880, exclusion.robots;dur=0.085143, exclusion.robots.policy;dur=0.078349, RedisCDXSource;dur=1.690296, esindex;dur=0.008498, LoadShardBlock;dur=81.465509, PetaboxLoader3.datanode;dur=65.012442, CDXLines.iter;dur=17.253994, load_resource;dur=76.338845, PetaboxLoader3.resolve;dur=29.485151
x-app-server: wwwb-app202
x-ts: 200
x-tr: 194
x-location: All
x-cache-key: httpsweb.archive.org/web/20161203222044/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09d.tar.gzFR
x-rl: 0
x-na: 0
x-page-cache: MISS
x-nid: -
referrer-policy: no-referrer-when-downgrade
permissions-policy: interest-cohort=()

So we should also process those in the swh-web code creating save code now requests with archives visit type.

That's great news!

I think it should be enough to at first do the extra parsing in the archive.

Then try and do the extra adaptations we talked earlier only if we come across other servers not responding with the expected headers.

In T3365#66091, @anlambert wrote:

@ardumont, @rdicosmo, I just figured out that data we are missing (Content-Length, Last-Modified) from tarballs archived by the Internet archive are in fact available in x-archive-orig-* HTTP response headers,

Oh, it makes sense: the IA is explaining that this is information it got from the archived source :-)
Good catch!

I just figured out that data we are missing (Content-Length, Last-Modified) from tarballs archived by the Internet archive are in fact available in x-archive-orig-* HTTP response headers

Just for the record, we miss those headers because the original tarball URL from the Internet Archive is redirected to another Internet Archive one, see below:

anlambert@carnavalet:/tmp$ curl -I https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09c.tar.gz
HTTP/2 302 
server: nginx/1.15.8
date: Fri, 11 Jun 2021 12:57:24 GMT
content-type: text/plain; charset=utf-8
content-length: 0
x-archive-redirect-reason: found capture at 20100729045924
location: https://web.archive.org/web/20100729045924/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09c.tar.gz
server-timing: captures_list;dur=120.218105, CDXLines.iter;dur=26.994936, PetaboxLoader3.datanode;dur=62.329525, LoadShardBlock;dur=85.709518, esindex;dur=0.060370, exclusion.robots;dur=0.467327, exclusion.robots.policy;dur=0.450315, RedisCDXSource;dur=1.384665
x-app-server: wwwb-app44
x-ts: 302
x-tr: 130
x-location: All
x-cache-key: httpsweb.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09c.tar.gzFR
x-rl: 0
x-na: 0
x-page-cache: MISS
x-nid: -
referrer-policy: no-referrer-when-downgrade
permissions-policy: interest-cohort=()

anlambert@carnavalet:/tmp$ curl -LI https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09c.tar.gz
HTTP/2 302 
server: nginx/1.15.8
date: Fri, 11 Jun 2021 12:57:31 GMT
content-type: text/plain; charset=utf-8
content-length: 0
x-archive-redirect-reason: found capture at 20100729045924
location: https://web.archive.org/web/20100729045924/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09c.tar.gz
server-timing: captures_list;dur=120.218105, CDXLines.iter;dur=26.994936, PetaboxLoader3.datanode;dur=62.329525, LoadShardBlock;dur=85.709518, esindex;dur=0.060370, exclusion.robots;dur=0.467327, exclusion.robots.policy;dur=0.450315, RedisCDXSource;dur=1.384665
x-app-server: wwwb-app44
x-ts: 302
x-tr: 130
x-location: All
x-cache-key: httpsweb.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09c.tar.gzFR
x-rl: 0
x-na: 0
x-page-cache: HIT
x-nid: -
referrer-policy: no-referrer-when-downgrade
permissions-policy: interest-cohort=()

HTTP/2 200 
server: nginx/1.15.8
date: Fri, 11 Jun 2021 12:57:32 GMT
content-type: application/x-gzip
x-archive-orig-date: Thu, 29 Jul 2010 04:56:17 GMT
x-archive-orig-server: Apache/2.2.3 (Debian) DAV/2 SVN/1.4.2 PHP/4.4.4-8+etch6 mod_ssl/2.2.3 OpenSSL/0.9.8c
x-archive-orig-last-modified: Wed, 29 Oct 1997 22:38:01 GMT
x-archive-orig-etag: "211710-1c275-31eafa5dd8440"
x-archive-orig-accept-ranges: bytes
x-archive-orig-content-length: 115317
x-archive-orig-connection: close
cache-control: max-age=1800
x-archive-guessed-content-type: application/x-gzip
memento-datetime: Thu, 29 Jul 2010 04:59:24 GMT
link: <http://www.cs.unm.edu/~mccune/old-ftp/eqp-09c.tar.gz>; rel="original", <https://web.archive.org/web/timemap/link/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09c.tar.gz>; rel="timemap"; type="application/link-format", <https://web.archive.org/web/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09c.tar.gz>; rel="timegate", <https://web.archive.org/web/20100729045924/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09c.tar.gz>; rel="first memento"; datetime="Thu, 29 Jul 2010 04:59:24 GMT", <https://web.archive.org/web/20100729045924/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09c.tar.gz>; rel="memento"; datetime="Thu, 29 Jul 2010 04:59:24 GMT", <https://web.archive.org/web/20100729045924/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09c.tar.gz>; rel="last memento"; datetime="Thu, 29 Jul 2010 04:59:24 GMT"
content-security-policy: default-src 'self' 'unsafe-eval' 'unsafe-inline' data: blob: archive.org web.archive.org analytics.archive.org pragma.archivelab.org
x-archive-src: EDG-20100729011318-03540-03548-ia360936-20100729063552-00000/EDG-20100729034221-03547-ia360936.us.archive.org.warc.gz
server-timing: captures_list;dur=220.324379, exclusion.robots;dur=0.146704, exclusion.robots.policy;dur=0.136692, RedisCDXSource;dur=3.613202, esindex;dur=0.009090, LoadShardBlock;dur=194.690976, PetaboxLoader3.datanode;dur=165.769899, CDXLines.iter;dur=19.168372, load_resource;dur=543.301115, PetaboxLoader3.resolve;dur=124.741953
x-app-server: wwwb-app28
x-ts: 200
x-tr: 965
x-location: All
x-cache-key: httpsweb.archive.org/web/20100729045924/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09c.tar.gzFR
x-rl: 0
x-na: 0
x-page-cache: MISS
x-nid: -
referrer-policy: no-referrer-when-downgrade
permissions-policy: interest-cohort=()

So we must ensure to follow redirections in swh-web code sending HEAD request.

anlambert added a revision: D5859: common/origin_save/origin_exists: Handle Internet Archive artifact URLs.Jun 11 2021, 4:43 PM

Fix implemented in D5859 works \o/

This is what I obtained when testing in the docker environment, using the script below to create the save code now request:

save code now script

import os

from bs4 import BeautifulSoup
import requests

url = (
    "https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/"
)
origin_url = "http://www.cs.unm.edu/~mccune/old-ftp/"

response = requests.get(url)

page = BeautifulSoup(response.content, features="html.parser")

supported_extensions = {".tar", ".tar.gz", ".zip", ".tar.Z"}

archive_links = [
    p["href"]
    for p in page.find_all("a")
    if any([p["href"].endswith(ext) for ext in supported_extensions])
]

artifacts = {}

archives_data = []

for archive_link in archive_links:
    if not archive_link.startswith("http"):
        archive_link = url + archive_link
    if not requests.head(archive_link).ok:
        continue
    artifact_version = archive_link.split("/")[-1].split(".")[0]
    if artifact_version not in artifacts:
        archives_data.append(
            {"artifact_url": archive_link, "artifact_version": artifact_version}
        )
        artifacts[artifact_version] = archive_link

save_code_now_url = (
    f"http://localhost:5004/api/1/origin/save/archives/url/{origin_url}/"
)

headers = {"Authorization": f"Bearer {os.environ['SWH_TOKEN']}"}

print(
    requests.post(
        save_code_now_url, json={"archives_data": archives_data}, headers=headers
    ).json()
)

swh-loader service output

swh-loader_1                    | [2021-06-11 15:02:03,561: INFO/MainProcess] Task swh.loader.package.archive.tasks.LoadArchive[dee3a234-61a7-452b-8a73-94054ae3b7a1] received
swh-loader_1                    | [2021-06-11 15:02:03,563: DEBUG/ForkPoolWorker-1] Loading config file /loader.yml
swh-loader_1                    | [2021-06-11 15:02:06,438: DEBUG/ForkPoolWorker-1] last snapshot: None
swh-loader_1                    | [2021-06-11 15:02:06,439: DEBUG/ForkPoolWorker-1] known artifacts: {}
swh-loader_1                    | [2021-06-11 15:02:06,455: DEBUG/ForkPoolWorker-1] package_info: ArchivePackageInfo(url='https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09c.tar.gz', filename='eqp-09c.tar.gz', directory_extrinsic_metadata=[], raw_info={'url': 'https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09c.tar.gz', 'time': '1997-10-29T22:38:01', 'length': 115317, 'version': 'eqp-09c'}, length=115317, time='1997-10-29T22:38:01', version='eqp-09c')
swh-loader_1                    | [2021-06-11 15:02:07,850: DEBUG/ForkPoolWorker-1] filename: eqp-09c.tar.gz
swh-loader_1                    | [2021-06-11 15:02:07,850: DEBUG/ForkPoolWorker-1] filepath: /tmp/tmpovo0r8yj/eqp-09c.tar.gz
swh-loader_1                    | [2021-06-11 15:02:08,711: DEBUG/ForkPoolWorker-1] extrinsic_metadata
swh-loader_1                    | [2021-06-11 15:02:08,753: DEBUG/ForkPoolWorker-1] uncompressed_path: /tmp/tmpovo0r8yj/src
swh-loader_1                    | [2021-06-11 15:02:08,774: DEBUG/ForkPoolWorker-1] Number of skipped contents: 0
swh-loader_1                    | [2021-06-11 15:02:08,774: DEBUG/ForkPoolWorker-1] Number of contents: 76
swh-loader_1                    | [2021-06-11 15:02:08,774: DEBUG/ForkPoolWorker-1] Number of directories: 7
swh-loader_1                    | [2021-06-11 15:02:14,426: DEBUG/ForkPoolWorker-1] Revision: Revision(message=b'swh-loader-package: synthetic revision message', author=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), committer=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), date=TimestampWithTimezone(timestamp=Timestamp(seconds=878164681, microseconds=0), offset=0, negative_utc=False), committer_date=TimestampWithTimezone(timestamp=Timestamp(seconds=878164681, microseconds=0), offset=0, negative_utc=False), type=<RevisionType.TAR: 'tar'>, directory=b'\x16\x1a\xb7\xdf\xb7\xe0&\xe1(v_\xa2\x18`Be4\xf5\xf7\x9b', synthetic=True, metadata=None, parents=(), id=b'\x90\xa0\xe2\xca\x0bVf\x88\xa2\xfe\xd9\xeaK\xde\x1a*\xcb\xc6\xa0`', extra_headers=())
swh-loader_1                    | [2021-06-11 15:02:15,612: DEBUG/MainProcess] heartbeat_tick : for connection 03f3a98f209b464191df421514c25b76
swh-loader_1                    | [2021-06-11 15:02:15,612: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 109/1000, now - 110/1153, monotonic - 21891.167998208, last_heartbeat_sent - 21891.167996711, heartbeat int. - 60 for connection 03f3a98f209b464191df421514c25b76
swh-loader_1                    | [2021-06-11 15:02:18,457: DEBUG/ForkPoolWorker-1] package_info: ArchivePackageInfo(url='https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09d.tar.gz', filename='eqp-09d.tar.gz', directory_extrinsic_metadata=[], raw_info={'url': 'https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09d.tar.gz', 'time': '1999-04-02T20:54:16', 'length': 116101, 'version': 'eqp-09d'}, length=116101, time='1999-04-02T20:54:16', version='eqp-09d')
swh-loader_1                    | [2021-06-11 15:02:19,894: DEBUG/ForkPoolWorker-1] filename: eqp-09d.tar.gz
swh-loader_1                    | [2021-06-11 15:02:19,894: DEBUG/ForkPoolWorker-1] filepath: /tmp/tmpcj1ui00_/eqp-09d.tar.gz
swh-loader_1                    | [2021-06-11 15:02:21,122: DEBUG/ForkPoolWorker-1] extrinsic_metadata
swh-loader_1                    | [2021-06-11 15:02:21,160: DEBUG/ForkPoolWorker-1] uncompressed_path: /tmp/tmpcj1ui00_/src
swh-loader_1                    | [2021-06-11 15:02:21,181: DEBUG/ForkPoolWorker-1] Number of skipped contents: 0
swh-loader_1                    | [2021-06-11 15:02:21,181: DEBUG/ForkPoolWorker-1] Number of contents: 76
swh-loader_1                    | [2021-06-11 15:02:21,181: DEBUG/ForkPoolWorker-1] Number of directories: 7
swh-loader_1                    | [2021-06-11 15:02:24,088: DEBUG/ForkPoolWorker-1] Revision: Revision(message=b'swh-loader-package: synthetic revision message', author=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), committer=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), date=TimestampWithTimezone(timestamp=Timestamp(seconds=923086456, microseconds=0), offset=0, negative_utc=False), committer_date=TimestampWithTimezone(timestamp=Timestamp(seconds=923086456, microseconds=0), offset=0, negative_utc=False), type=<RevisionType.TAR: 'tar'>, directory=b'=l\xbd\x15\x1d\xc8"\xc5$\xaa\x9c+,3\xd0\xb5+\x05_\x94', synthetic=True, metadata=None, parents=(), id=b'\x0b\xcbJ7\x941\xad\xca\xb9\xbcE7\x03\xbeFn\xb8\xca\x81W', extra_headers=())
swh-loader_1                    | [2021-06-11 15:02:27,093: DEBUG/ForkPoolWorker-1] package_info: ArchivePackageInfo(url='https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09e.tar.gz', filename='eqp-09e.tar.gz', directory_extrinsic_metadata=[], raw_info={'url': 'https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/eqp-09e.tar.gz', 'time': '2009-05-12T22:09:43', 'length': 121421, 'version': 'eqp-09e'}, length=121421, time='2009-05-12T22:09:43', version='eqp-09e')
swh-loader_1                    | [2021-06-11 15:02:28,494: DEBUG/ForkPoolWorker-1] filename: eqp-09e.tar.gz
swh-loader_1                    | [2021-06-11 15:02:28,494: DEBUG/ForkPoolWorker-1] filepath: /tmp/tmpht527dbd/eqp-09e.tar.gz
swh-loader_1                    | [2021-06-11 15:02:29,969: DEBUG/ForkPoolWorker-1] extrinsic_metadata
swh-loader_1                    | [2021-06-11 15:02:30,013: DEBUG/ForkPoolWorker-1] uncompressed_path: /tmp/tmpht527dbd/src
swh-loader_1                    | [2021-06-11 15:02:30,032: DEBUG/ForkPoolWorker-1] Number of skipped contents: 0
swh-loader_1                    | [2021-06-11 15:02:30,032: DEBUG/ForkPoolWorker-1] Number of contents: 77
swh-loader_1                    | [2021-06-11 15:02:30,033: DEBUG/ForkPoolWorker-1] Number of directories: 7
swh-loader_1                    | [2021-06-11 15:02:30,073: DEBUG/ForkPoolWorker-1] Revision: Revision(message=b'swh-loader-package: synthetic revision message', author=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), committer=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), date=TimestampWithTimezone(timestamp=Timestamp(seconds=1242166183, microseconds=0), offset=0, negative_utc=False), committer_date=TimestampWithTimezone(timestamp=Timestamp(seconds=1242166183, microseconds=0), offset=0, negative_utc=False), type=<RevisionType.TAR: 'tar'>, directory=b'\x13\x80\xb6\x1b0\x0e\x00&3\xc5\xb7\xce\xf6$\xb2\xb8\x8b\r\t\xb7', synthetic=True, metadata=None, parents=(), id=b'\xc7\x8a\n\xee\x93\xd1o*\xaf \xa4/\xbd\x17\xae\x8e\xady0\x9e', extra_headers=())
swh-loader_1                    | [2021-06-11 15:02:30,176: DEBUG/ForkPoolWorker-1] package_info: ArchivePackageInfo(url='https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/index.tar.Z', filename='index.tar.Z', directory_extrinsic_metadata=[], raw_info={'url': 'https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/index.tar.Z', 'time': '1992-09-15T18:39:06', 'length': 367905, 'version': 'index'}, length=367905, time='1992-09-15T18:39:06', version='index')
swh-loader_1                    | [2021-06-11 15:02:31,564: DEBUG/ForkPoolWorker-1] filename: index.tar.Z
swh-loader_1                    | [2021-06-11 15:02:31,564: DEBUG/ForkPoolWorker-1] filepath: /tmp/tmp1w0rg2wd/index.tar.Z
swh-loader_1                    | [2021-06-11 15:02:35,613: DEBUG/MainProcess] heartbeat_tick : for connection 03f3a98f209b464191df421514c25b76
swh-loader_1                    | [2021-06-11 15:02:35,613: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 110/1153, now - 110/1303, monotonic - 21911.169329286, last_heartbeat_sent - 21891.167996711, heartbeat int. - 60 for connection 03f3a98f209b464191df421514c25b76
swh-loader_1                    | [2021-06-11 15:02:36,114: DEBUG/ForkPoolWorker-1] extrinsic_metadata
swh-loader_1                    | [2021-06-11 15:02:36,167: DEBUG/ForkPoolWorker-1] uncompressed_path: /tmp/tmp1w0rg2wd/src
swh-loader_1                    | [2021-06-11 15:02:36,201: DEBUG/ForkPoolWorker-1] Number of skipped contents: 0
swh-loader_1                    | [2021-06-11 15:02:36,201: DEBUG/ForkPoolWorker-1] Number of contents: 11
swh-loader_1                    | [2021-06-11 15:02:36,202: DEBUG/ForkPoolWorker-1] Number of directories: 1
swh-loader_1                    | [2021-06-11 15:02:36,240: DEBUG/ForkPoolWorker-1] Revision: Revision(message=b'swh-loader-package: synthetic revision message', author=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), committer=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), date=TimestampWithTimezone(timestamp=Timestamp(seconds=716582346, microseconds=0), offset=0, negative_utc=False), committer_date=TimestampWithTimezone(timestamp=Timestamp(seconds=716582346, microseconds=0), offset=0, negative_utc=False), type=<RevisionType.TAR: 'tar'>, directory=b'\x08\xbcr\xba\x07\x9c\xf6\xf9\xeeR\xf1\xbd`\x04\xeb\x9d\xc22\xdd\x82', synthetic=True, metadata=None, parents=(), id=b'v\x03\x86\xf1T7\xe0?\x9c\xd4\xc5\x90\xf8\xaf\xb4\xb1\x8d"\xd2\xd5', extra_headers=())
swh-loader_1                    | [2021-06-11 15:02:36,522: DEBUG/ForkPoolWorker-1] package_info: ArchivePackageInfo(url='https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/mace-1.2.0.tar.Z', filename='mace-1.2.0.tar.Z', directory_extrinsic_metadata=[], raw_info={'url': 'https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/mace-1.2.0.tar.Z', 'time': '1995-08-17T01:54:02', 'length': 132177, 'version': 'mace-1'}, length=132177, time='1995-08-17T01:54:02', version='mace-1')
swh-loader_1                    | [2021-06-11 15:02:37,955: DEBUG/ForkPoolWorker-1] filename: mace-1.2.0.tar.Z
swh-loader_1                    | [2021-06-11 15:02:37,955: DEBUG/ForkPoolWorker-1] filepath: /tmp/tmpke9g_gya/mace-1.2.0.tar.Z
swh-loader_1                    | [2021-06-11 15:02:39,308: DEBUG/ForkPoolWorker-1] extrinsic_metadata
swh-loader_1                    | [2021-06-11 15:02:39,341: DEBUG/ForkPoolWorker-1] uncompressed_path: /tmp/tmpke9g_gya/src
swh-loader_1                    | [2021-06-11 15:02:39,380: DEBUG/ForkPoolWorker-1] Number of skipped contents: 0
swh-loader_1                    | [2021-06-11 15:02:39,381: DEBUG/ForkPoolWorker-1] Number of contents: 70
swh-loader_1                    | [2021-06-11 15:02:39,381: DEBUG/ForkPoolWorker-1] Number of directories: 8
swh-loader_1                    | [2021-06-11 15:02:39,422: DEBUG/ForkPoolWorker-1] Revision: Revision(message=b'swh-loader-package: synthetic revision message', author=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), committer=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), date=TimestampWithTimezone(timestamp=Timestamp(seconds=808624442, microseconds=0), offset=0, negative_utc=False), committer_date=TimestampWithTimezone(timestamp=Timestamp(seconds=808624442, microseconds=0), offset=0, negative_utc=False), type=<RevisionType.TAR: 'tar'>, directory=b'*\x11\x82Lc\xe7,\x0f\x1d\xdb$\xda\x80\x9eC\xd5K!\xd7X', synthetic=True, metadata=None, parents=(), id=b'\xc9-f\xa64Z\xecv\xcb\x08r\xd8\xd4F\xccBV\xe3\xef\x8c', extra_headers=())
swh-loader_1                    | [2021-06-11 15:02:39,752: DEBUG/ForkPoolWorker-1] package_info: ArchivePackageInfo(url='https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/otter-3.0.4.tar.Z', filename='otter-3.0.4.tar.Z', directory_extrinsic_metadata=[], raw_info={'url': 'https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/otter-3.0.4.tar.Z', 'time': '1995-09-25T02:41:13', 'length': 896999, 'version': 'otter-3'}, length=896999, time='1995-09-25T02:41:13', version='otter-3')
swh-loader_1                    | [2021-06-11 15:02:41,151: DEBUG/ForkPoolWorker-1] filename: otter-3.0.4.tar.Z
swh-loader_1                    | [2021-06-11 15:02:41,152: DEBUG/ForkPoolWorker-1] filepath: /tmp/tmptnw0jatf/otter-3.0.4.tar.Z
swh-loader_1                    | [2021-06-11 15:02:47,641: DEBUG/ForkPoolWorker-1] extrinsic_metadata
swh-loader_1                    | [2021-06-11 15:02:47,663: DEBUG/ForkPoolWorker-1] uncompressed_path: /tmp/tmptnw0jatf/src
swh-loader_1                    | [2021-06-11 15:02:47,717: DEBUG/ForkPoolWorker-1] Number of skipped contents: 0
swh-loader_1                    | [2021-06-11 15:02:47,717: DEBUG/ForkPoolWorker-1] Number of contents: 193
swh-loader_1                    | [2021-06-11 15:02:47,717: DEBUG/ForkPoolWorker-1] Number of directories: 12
swh-loader_1                    | [2021-06-11 15:02:47,757: DEBUG/ForkPoolWorker-1] Revision: Revision(message=b'swh-loader-package: synthetic revision message', author=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), committer=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), date=TimestampWithTimezone(timestamp=Timestamp(seconds=811996873, microseconds=0), offset=0, negative_utc=False), committer_date=TimestampWithTimezone(timestamp=Timestamp(seconds=811996873, microseconds=0), offset=0, negative_utc=False), type=<RevisionType.TAR: 'tar'>, directory=b'|\xb4uy\x1f\xf8\xf3p\x0b\xf3\xa8I\x07{]PK*4\xb8', synthetic=True, metadata=None, parents=(), id=b'\x03\xda4\xff4/\xbb]\xd9+\xe4\x0b;O\xe4\xcd\xd5\x8d=\xb2', extra_headers=())
swh-loader_1                    | [2021-06-11 15:02:48,379: DEBUG/ForkPoolWorker-1] package_info: ArchivePackageInfo(url='https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/otter-emacs.tar.gz', filename='otter-emacs.tar.gz', directory_extrinsic_metadata=[], raw_info={'url': 'https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/otter-emacs.tar.gz', 'time': '1996-04-29T19:08:36', 'length': 9185, 'version': 'otter-emacs'}, length=9185, time='1996-04-29T19:08:36', version='otter-emacs')
swh-loader_1                    | [2021-06-11 15:02:49,628: DEBUG/ForkPoolWorker-1] filename: otter-emacs.tar.gz
swh-loader_1                    | [2021-06-11 15:02:49,628: DEBUG/ForkPoolWorker-1] filepath: /tmp/tmpknx1s_9x/otter-emacs.tar.gz
swh-loader_1                    | [2021-06-11 15:02:49,629: DEBUG/ForkPoolWorker-1] extrinsic_metadata
swh-loader_1                    | [2021-06-11 15:02:49,632: DEBUG/ForkPoolWorker-1] uncompressed_path: /tmp/tmpknx1s_9x/src
swh-loader_1                    | [2021-06-11 15:02:49,633: DEBUG/ForkPoolWorker-1] Number of skipped contents: 0
swh-loader_1                    | [2021-06-11 15:02:49,633: DEBUG/ForkPoolWorker-1] Number of contents: 5
swh-loader_1                    | [2021-06-11 15:02:49,633: DEBUG/ForkPoolWorker-1] Number of directories: 1
swh-loader_1                    | [2021-06-11 15:02:49,668: DEBUG/ForkPoolWorker-1] Revision: Revision(message=b'swh-loader-package: synthetic revision message', author=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), committer=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), date=TimestampWithTimezone(timestamp=Timestamp(seconds=830804916, microseconds=0), offset=0, negative_utc=False), committer_date=TimestampWithTimezone(timestamp=Timestamp(seconds=830804916, microseconds=0), offset=0, negative_utc=False), type=<RevisionType.TAR: 'tar'>, directory=b'\x81\x12\xeb\xb4c\xb7\xb9?)\x9fP\xdd%\xaf\xb6\x11\x16nq\x8c', synthetic=True, metadata=None, parents=(), id=b'\xa9M\xa5\x14W\x84\xc4:d\xfd\x06\n8\xfa1a*\xeas\xc0', extra_headers=())
swh-loader_1                    | [2021-06-11 15:02:49,753: DEBUG/ForkPoolWorker-1] package_info: ArchivePackageInfo(url='https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/otter-mode11.tar.gz', filename='otter-mode11.tar.gz', directory_extrinsic_metadata=[], raw_info={'url': 'https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/otter-mode11.tar.gz', 'time': '1996-07-12T18:17:17', 'length': 14926, 'version': 'otter-mode11'}, length=14926, time='1996-07-12T18:17:17', version='otter-mode11')
swh-loader_1                    | [2021-06-11 15:02:52,088: DEBUG/ForkPoolWorker-1] filename: otter-mode11.tar.gz
swh-loader_1                    | [2021-06-11 15:02:52,089: DEBUG/ForkPoolWorker-1] filepath: /tmp/tmp_8pw1awk/otter-mode11.tar.gz
swh-loader_1                    | [2021-06-11 15:02:52,093: DEBUG/ForkPoolWorker-1] extrinsic_metadata
swh-loader_1                    | [2021-06-11 15:02:52,106: DEBUG/ForkPoolWorker-1] uncompressed_path: /tmp/tmp_8pw1awk/src
swh-loader_1                    | [2021-06-11 15:02:52,112: DEBUG/ForkPoolWorker-1] Number of skipped contents: 0
swh-loader_1                    | [2021-06-11 15:02:52,112: DEBUG/ForkPoolWorker-1] Number of contents: 10
swh-loader_1                    | [2021-06-11 15:02:52,112: DEBUG/ForkPoolWorker-1] Number of directories: 1
swh-loader_1                    | [2021-06-11 15:02:52,152: DEBUG/ForkPoolWorker-1] Revision: Revision(message=b'swh-loader-package: synthetic revision message', author=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), committer=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), date=TimestampWithTimezone(timestamp=Timestamp(seconds=837195437, microseconds=0), offset=0, negative_utc=False), committer_date=TimestampWithTimezone(timestamp=Timestamp(seconds=837195437, microseconds=0), offset=0, negative_utc=False), type=<RevisionType.TAR: 'tar'>, directory=b'\xae\xe2\xf0\xd1\xa3\x90\x03x\xf0_\xa4\x94\x99NX\x8e\xc9O\xb3\xbf', synthetic=True, metadata=None, parents=(), id=b'\xf7_\xef\xa0\xea\x1c\x97y4"wYC\xec\x81\x96\xa2\xda\x07\xe3', extra_headers=())
swh-loader_1                    | [2021-06-11 15:02:52,267: DEBUG/ForkPoolWorker-1] package_info: ArchivePackageInfo(url='https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/otter.tar.Z', filename='otter.tar.Z', directory_extrinsic_metadata=[], raw_info={'url': 'https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/otter.tar.Z', 'time': '1998-02-19T22:16:13', 'length': 1035257, 'version': 'otter'}, length=1035257, time='1998-02-19T22:16:13', version='otter')
swh-loader_1                    | [2021-06-11 15:02:53,439: DEBUG/ForkPoolWorker-1] filename: otter.tar.Z
swh-loader_1                    | [2021-06-11 15:02:53,439: DEBUG/ForkPoolWorker-1] filepath: /tmp/tmp0ybl4a6d/otter.tar.Z
swh-loader_1                    | [2021-06-11 15:02:55,615: DEBUG/MainProcess] heartbeat_tick : for connection 03f3a98f209b464191df421514c25b76
swh-loader_1                    | [2021-06-11 15:02:55,615: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 110/1303, now - 110/1453, monotonic - 21931.171280514, last_heartbeat_sent - 21891.167996711, heartbeat int. - 60 for connection 03f3a98f209b464191df421514c25b76
swh-loader_1                    | [2021-06-11 15:03:04,176: DEBUG/ForkPoolWorker-1] extrinsic_metadata
swh-loader_1                    | [2021-06-11 15:03:04,242: DEBUG/ForkPoolWorker-1] uncompressed_path: /tmp/tmp0ybl4a6d/src
swh-loader_1                    | [2021-06-11 15:03:04,306: DEBUG/ForkPoolWorker-1] Number of skipped contents: 0
swh-loader_1                    | [2021-06-11 15:03:04,307: DEBUG/ForkPoolWorker-1] Number of contents: 216
swh-loader_1                    | [2021-06-11 15:03:04,307: DEBUG/ForkPoolWorker-1] Number of directories: 14
swh-loader_1                    | [2021-06-11 15:03:04,352: DEBUG/ForkPoolWorker-1] Revision: Revision(message=b'swh-loader-package: synthetic revision message', author=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), committer=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), date=TimestampWithTimezone(timestamp=Timestamp(seconds=887926573, microseconds=0), offset=0, negative_utc=False), committer_date=TimestampWithTimezone(timestamp=Timestamp(seconds=887926573, microseconds=0), offset=0, negative_utc=False), type=<RevisionType.TAR: 'tar'>, directory=b'\xc04\xc3h\xac[\x8b\xb0\xc5v\xea\xbd\xe4\xce\x07\xfbD\xab\x92P', synthetic=True, metadata=None, parents=(), id=b'\xf3Mv\xc7v6\xf1yJ\x08\xf9\xa4\xef0\xf97\xc5\x07\x7fD', extra_headers=())
swh-loader_1                    | [2021-06-11 15:03:04,784: DEBUG/ForkPoolWorker-1] package_info: ArchivePackageInfo(url='https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/otter304-win95.zip', filename='otter304-win95.zip', directory_extrinsic_metadata=[], raw_info={'url': 'https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/otter304-win95.zip', 'time': '1996-12-12T21:22:11', 'length': 119590, 'version': 'otter304-win95'}, length=119590, time='1996-12-12T21:22:11', version='otter304-win95')
swh-loader_1                    | [2021-06-11 15:03:06,219: DEBUG/ForkPoolWorker-1] filename: otter304-win95.zip
swh-loader_1                    | [2021-06-11 15:03:06,219: DEBUG/ForkPoolWorker-1] filepath: /tmp/tmpgv81mslb/otter304-win95.zip
swh-loader_1                    | [2021-06-11 15:03:07,447: DEBUG/ForkPoolWorker-1] extrinsic_metadata
swh-loader_1                    | [2021-06-11 15:03:07,464: DEBUG/ForkPoolWorker-1] uncompressed_path: /tmp/tmpgv81mslb/src
swh-loader_1                    | [2021-06-11 15:03:07,473: DEBUG/ForkPoolWorker-1] Number of skipped contents: 0
swh-loader_1                    | [2021-06-11 15:03:07,473: DEBUG/ForkPoolWorker-1] Number of contents: 2
swh-loader_1                    | [2021-06-11 15:03:07,473: DEBUG/ForkPoolWorker-1] Number of directories: 1
swh-loader_1                    | [2021-06-11 15:03:07,515: DEBUG/ForkPoolWorker-1] Revision: Revision(message=b'swh-loader-package: synthetic revision message', author=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), committer=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), date=TimestampWithTimezone(timestamp=Timestamp(seconds=850425731, microseconds=0), offset=0, negative_utc=False), committer_date=TimestampWithTimezone(timestamp=Timestamp(seconds=850425731, microseconds=0), offset=0, negative_utc=False), type=<RevisionType.TAR: 'tar'>, directory=b']\xe3\xa0Z0\xb1L\xbf{\xbf\x07z\xfb\x9d\\\x19\xbbh\xca\x01', synthetic=True, metadata=None, parents=(), id=b'\x81\x05\x80t7\x8e\xac\xa6\xeb\x01G\xc0\xf4\xeb\xaf\xde\x97\x9c<\xce', extra_headers=())
swh-loader_1                    | [2021-06-11 15:03:07,639: DEBUG/ForkPoolWorker-1] package_info: ArchivePackageInfo(url='https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/otter305-win98.zip', filename='otter305-win98.zip', directory_extrinsic_metadata=[], raw_info={'url': 'https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/otter305-win98.zip', 'time': '2000-04-12T22:09:55', 'length': 128319, 'version': 'otter305-win98'}, length=128319, time='2000-04-12T22:09:55', version='otter305-win98')
swh-loader_1                    | [2021-06-11 15:03:09,018: DEBUG/ForkPoolWorker-1] filename: otter305-win98.zip
swh-loader_1                    | [2021-06-11 15:03:09,018: DEBUG/ForkPoolWorker-1] filepath: /tmp/tmpkggwq5pl/otter305-win98.zip
swh-loader_1                    | [2021-06-11 15:03:10,151: DEBUG/ForkPoolWorker-1] extrinsic_metadata
swh-loader_1                    | [2021-06-11 15:03:10,163: DEBUG/ForkPoolWorker-1] uncompressed_path: /tmp/tmpkggwq5pl/src
swh-loader_1                    | [2021-06-11 15:03:10,177: DEBUG/ForkPoolWorker-1] Number of skipped contents: 0
swh-loader_1                    | [2021-06-11 15:03:10,178: DEBUG/ForkPoolWorker-1] Number of contents: 1
swh-loader_1                    | [2021-06-11 15:03:10,178: DEBUG/ForkPoolWorker-1] Number of directories: 1
swh-loader_1                    | [2021-06-11 15:03:10,224: DEBUG/ForkPoolWorker-1] Revision: Revision(message=b'swh-loader-package: synthetic revision message', author=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), committer=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), date=TimestampWithTimezone(timestamp=Timestamp(seconds=955577395, microseconds=0), offset=0, negative_utc=False), committer_date=TimestampWithTimezone(timestamp=Timestamp(seconds=955577395, microseconds=0), offset=0, negative_utc=False), type=<RevisionType.TAR: 'tar'>, directory=b'k+#60\x97\xce\xfd\xb3\xe5\x167\xc9\x13e%\xfa\x8d\tZ', synthetic=True, metadata=None, parents=(), id=b'xo\xe4I\xf9&\xf2\xfb\x13,0\x19;W"%\xb70\xa1\x1b', extra_headers=())
swh-loader_1                    | [2021-06-11 15:03:10,350: DEBUG/ForkPoolWorker-1] package_info: ArchivePackageInfo(url='https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/otter_jobs.tar.Z', filename='otter_jobs.tar.Z', directory_extrinsic_metadata=[], raw_info={'url': 'https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/otter_jobs.tar.Z', 'time': '1992-10-15T14:27:39', 'length': 102311, 'version': 'otter_jobs'}, length=102311, time='1992-10-15T14:27:39', version='otter_jobs')
swh-loader_1                    | [2021-06-11 15:03:11,748: DEBUG/ForkPoolWorker-1] filename: otter_jobs.tar.Z
swh-loader_1                    | [2021-06-11 15:03:11,749: DEBUG/ForkPoolWorker-1] filepath: /tmp/tmpsx7tssal/otter_jobs.tar.Z
swh-loader_1                    | [2021-06-11 15:03:12,687: DEBUG/ForkPoolWorker-1] extrinsic_metadata
swh-loader_1                    | [2021-06-11 15:03:12,694: DEBUG/ForkPoolWorker-1] uncompressed_path: /tmp/tmpsx7tssal/src
swh-loader_1                    | [2021-06-11 15:03:12,705: DEBUG/ForkPoolWorker-1] Number of skipped contents: 0
swh-loader_1                    | [2021-06-11 15:03:12,705: DEBUG/ForkPoolWorker-1] Number of contents: 26
swh-loader_1                    | [2021-06-11 15:03:12,705: DEBUG/ForkPoolWorker-1] Number of directories: 2
swh-loader_1                    | [2021-06-11 15:03:12,743: DEBUG/ForkPoolWorker-1] Revision: Revision(message=b'swh-loader-package: synthetic revision message', author=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), committer=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), date=TimestampWithTimezone(timestamp=Timestamp(seconds=719159259, microseconds=0), offset=0, negative_utc=False), committer_date=TimestampWithTimezone(timestamp=Timestamp(seconds=719159259, microseconds=0), offset=0, negative_utc=False), type=<RevisionType.TAR: 'tar'>, directory=b'2X\xfc\xdcP\x9a\x8b\xaa\xb9\x14r`\xe3{L\x90\xd6\xc5L6', synthetic=True, metadata=None, parents=(), id=b' \x8c\x0f\xd2\x02_V\xa7\x08\xb2!\xca2h\r\xdfq\t\r\xd7', extra_headers=())
swh-loader_1                    | [2021-06-11 15:03:12,874: DEBUG/ForkPoolWorker-1] package_info: ArchivePackageInfo(url='https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/otterface-0.1.tar.Z', filename='otterface-0.1.tar.Z', directory_extrinsic_metadata=[], raw_info={'url': 'https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/otterface-0.1.tar.Z', 'time': '1994-08-26T21:27:46', 'length': 189905, 'version': 'otterface-0'}, length=189905, time='1994-08-26T21:27:46', version='otterface-0')
swh-loader_1                    | [2021-06-11 15:03:14,329: DEBUG/ForkPoolWorker-1] filename: otterface-0.1.tar.Z
swh-loader_1                    | [2021-06-11 15:03:14,329: DEBUG/ForkPoolWorker-1] filepath: /tmp/tmpg1c8r_tg/otterface-0.1.tar.Z
swh-loader_1                    | [2021-06-11 15:03:15,617: DEBUG/MainProcess] heartbeat_tick : for connection 03f3a98f209b464191df421514c25b76
swh-loader_1                    | [2021-06-11 15:03:15,617: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 110/1453, now - 110/1603, monotonic - 21951.17286104, last_heartbeat_sent - 21891.167996711, heartbeat int. - 60 for connection 03f3a98f209b464191df421514c25b76
swh-loader_1                    | [2021-06-11 15:03:15,617: DEBUG/MainProcess] heartbeat_tick: sending heartbeat for connection 03f3a98f209b464191df421514c25b76
swh-loader_1                    | [2021-06-11 15:03:16,418: DEBUG/ForkPoolWorker-1] extrinsic_metadata
swh-loader_1                    | [2021-06-11 15:03:16,451: DEBUG/ForkPoolWorker-1] uncompressed_path: /tmp/tmpg1c8r_tg/src
swh-loader_1                    | [2021-06-11 15:03:16,472: DEBUG/ForkPoolWorker-1] Number of skipped contents: 0
swh-loader_1                    | [2021-06-11 15:03:16,473: DEBUG/ForkPoolWorker-1] Number of contents: 11
swh-loader_1                    | [2021-06-11 15:03:16,473: DEBUG/ForkPoolWorker-1] Number of directories: 2
swh-loader_1                    | [2021-06-11 15:03:16,515: DEBUG/ForkPoolWorker-1] Revision: Revision(message=b'swh-loader-package: synthetic revision message', author=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), committer=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), date=TimestampWithTimezone(timestamp=Timestamp(seconds=777936466, microseconds=0), offset=0, negative_utc=False), committer_date=TimestampWithTimezone(timestamp=Timestamp(seconds=777936466, microseconds=0), offset=0, negative_utc=False), type=<RevisionType.TAR: 'tar'>, directory=b'\xde\xbaxb\xa8\x8c\xf7\xac<\xc3\xa3(\n9\xce\xcc\xa4\x97\x19<', synthetic=True, metadata=None, parents=(), id=b'5\x95\xbd\xe9\x8c\xce\xd5\x83\xcf\xb2\x9c\xf7h\xb1%\xa2t\x05\xdeg', extra_headers=())
swh-loader_1                    | [2021-06-11 15:03:16,705: DEBUG/ForkPoolWorker-1] package_info: ArchivePackageInfo(url='https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/pc-1.0.tar.gz', filename='pc-1.0.tar.gz', directory_extrinsic_metadata=[], raw_info={'url': 'https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/pc-1.0.tar.gz', 'time': '1997-04-23T21:16:18', 'length': 66362, 'version': 'pc-1'}, length=66362, time='1997-04-23T21:16:18', version='pc-1')
swh-loader_1                    | [2021-06-11 15:03:18,138: DEBUG/ForkPoolWorker-1] filename: pc-1.0.tar.gz
swh-loader_1                    | [2021-06-11 15:03:18,138: DEBUG/ForkPoolWorker-1] filepath: /tmp/tmprq43559c/pc-1.0.tar.gz
swh-loader_1                    | [2021-06-11 15:03:18,877: DEBUG/ForkPoolWorker-1] extrinsic_metadata
swh-loader_1                    | [2021-06-11 15:03:18,915: DEBUG/ForkPoolWorker-1] uncompressed_path: /tmp/tmprq43559c/src
swh-loader_1                    | [2021-06-11 15:03:18,931: DEBUG/ForkPoolWorker-1] Number of skipped contents: 0
swh-loader_1                    | [2021-06-11 15:03:18,931: DEBUG/ForkPoolWorker-1] Number of contents: 54
swh-loader_1                    | [2021-06-11 15:03:18,931: DEBUG/ForkPoolWorker-1] Number of directories: 7
swh-loader_1                    | [2021-06-11 15:03:18,969: DEBUG/ForkPoolWorker-1] Revision: Revision(message=b'swh-loader-package: synthetic revision message', author=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), committer=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), date=TimestampWithTimezone(timestamp=Timestamp(seconds=861830178, microseconds=0), offset=0, negative_utc=False), committer_date=TimestampWithTimezone(timestamp=Timestamp(seconds=861830178, microseconds=0), offset=0, negative_utc=False), type=<RevisionType.TAR: 'tar'>, directory=b'\x87\x16ciO\x13\xbe\xa1\xfc\xc8m\x05\xd1\x88:%\xd8\x7f\x99\xc1', synthetic=True, metadata=None, parents=(), id=b'\x1e\xe33\r\xdci\x1c\xc9|>\x0e\x06\x97\xb4\x82\xedJs*\xe5', extra_headers=())
swh-loader_1                    | [2021-06-11 15:03:19,245: DEBUG/ForkPoolWorker-1] package_info: ArchivePackageInfo(url='https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/prf_disp.tar.Z', filename='prf_disp.tar.Z', directory_extrinsic_metadata=[], raw_info={'url': 'https://web.archive.org/web/20100705043309/http://www.cs.unm.edu/~mccune/old-ftp/prf_disp.tar.Z', 'time': '1994-06-17T21:01:36', 'length': 34802, 'version': 'prf_disp'}, length=34802, time='1994-06-17T21:01:36', version='prf_disp')
swh-loader_1                    | [2021-06-11 15:03:20,719: DEBUG/ForkPoolWorker-1] filename: prf_disp.tar.Z
swh-loader_1                    | [2021-06-11 15:03:20,719: DEBUG/ForkPoolWorker-1] filepath: /tmp/tmp6aj60eqt/prf_disp.tar.Z
swh-loader_1                    | [2021-06-11 15:03:20,965: DEBUG/ForkPoolWorker-1] extrinsic_metadata
swh-loader_1                    | [2021-06-11 15:03:20,990: DEBUG/ForkPoolWorker-1] uncompressed_path: /tmp/tmp6aj60eqt/src
swh-loader_1                    | [2021-06-11 15:03:21,010: DEBUG/ForkPoolWorker-1] Number of skipped contents: 0
swh-loader_1                    | [2021-06-11 15:03:21,011: DEBUG/ForkPoolWorker-1] Number of contents: 19
swh-loader_1                    | [2021-06-11 15:03:21,011: DEBUG/ForkPoolWorker-1] Number of directories: 1
swh-loader_1                    | [2021-06-11 15:03:21,055: DEBUG/ForkPoolWorker-1] Revision: Revision(message=b'swh-loader-package: synthetic revision message', author=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), committer=Person(fullname=b'Software Heritage', name=b'Software Heritage', email=b'robot@softwareheritage.org'), date=TimestampWithTimezone(timestamp=Timestamp(seconds=771886896, microseconds=0), offset=0, negative_utc=False), committer_date=TimestampWithTimezone(timestamp=Timestamp(seconds=771886896, microseconds=0), offset=0, negative_utc=False), type=<RevisionType.TAR: 'tar'>, directory=b'\x9b\x8e\x120\xcc\x0c\xd84\x98H\xb4\x15U\xe1F\xe9\x19\xed\x91\x95', synthetic=True, metadata=None, parents=(), id=b'V:\xa9\xf9\xf9\x97\xe4\xa2\xd4\x1d?\x13\x19Xf\xc9p\xf4\xec\x88', extra_headers=())
swh-loader_1                    | [2021-06-11 15:03:21,173: DEBUG/ForkPoolWorker-1] default version: prf_disp
swh-loader_1                    | [2021-06-11 15:03:21,177: DEBUG/ForkPoolWorker-1] extra branches: {}
swh-loader_1                    | [2021-06-11 15:03:21,177: DEBUG/ForkPoolWorker-1] revisions: {'eqp-09c': [('releases/eqp-09c', b'\x90\xa0\xe2\xca\x0bVf\x88\xa2\xfe\xd9\xeaK\xde\x1a*\xcb\xc6\xa0`')], 'eqp-09d': [('releases/eqp-09d', b'\x0b\xcbJ7\x941\xad\xca\xb9\xbcE7\x03\xbeFn\xb8\xca\x81W')], 'eqp-09e': [('releases/eqp-09e', b'\xc7\x8a\n\xee\x93\xd1o*\xaf \xa4/\xbd\x17\xae\x8e\xady0\x9e')], 'index': [('releases/index', b'v\x03\x86\xf1T7\xe0?\x9c\xd4\xc5\x90\xf8\xaf\xb4\xb1\x8d"\xd2\xd5')], 'mace-1': [('releases/mace-1', b'\xc9-f\xa64Z\xecv\xcb\x08r\xd8\xd4F\xccBV\xe3\xef\x8c')], 'otter-3': [('releases/otter-3', b'\x03\xda4\xff4/\xbb]\xd9+\xe4\x0b;O\xe4\xcd\xd5\x8d=\xb2')], 'otter-emacs': [('releases/otter-emacs', b'\xa9M\xa5\x14W\x84\xc4:d\xfd\x06\n8\xfa1a*\xeas\xc0')], 'otter-mode11': [('releases/otter-mode11', b'\xf7_\xef\xa0\xea\x1c\x97y4"wYC\xec\x81\x96\xa2\xda\x07\xe3')], 'otter': [('releases/otter', b'\xf3Mv\xc7v6\xf1yJ\x08\xf9\xa4\xef0\xf97\xc5\x07\x7fD')], 'otter304-win95': [('releases/otter304-win95', b'\x81\x05\x80t7\x8e\xac\xa6\xeb\x01G\xc0\xf4\xeb\xaf\xde\x97\x9c<\xce')], 'otter305-win98': [('releases/otter305-win98', b'xo\xe4I\xf9&\xf2\xfb\x13,0\x19;W"%\xb70\xa1\x1b')], 'otter_jobs': [('releases/otter_jobs', b' \x8c\x0f\xd2\x02_V\xa7\x08\xb2!\xca2h\r\xdfq\t\r\xd7')], 'otterface-0': [('releases/otterface-0', b'5\x95\xbd\xe9\x8c\xce\xd5\x83\xcf\xb2\x9c\xf7h\xb1%\xa2t\x05\xdeg')], 'pc-1': [('releases/pc-1', b'\x1e\xe33\r\xdci\x1c\xc9|>\x0e\x06\x97\xb4\x82\xedJs*\xe5')], 'prf_disp': [('releases/prf_disp', b'V:\xa9\xf9\xf9\x97\xe4\xa2\xd4\x1d?\x13\x19Xf\xc9p\xf4\xec\x88')]}
swh-loader_1                    | [2021-06-11 15:03:21,177: DEBUG/ForkPoolWorker-1] snapshot: {'branches': {b'releases/eqp-09c': {'target_type': 'revision', 'target': b'\x90\xa0\xe2\xca\x0bVf\x88\xa2\xfe\xd9\xeaK\xde\x1a*\xcb\xc6\xa0`'}, b'releases/eqp-09d': {'target_type': 'revision', 'target': b'\x0b\xcbJ7\x941\xad\xca\xb9\xbcE7\x03\xbeFn\xb8\xca\x81W'}, b'releases/eqp-09e': {'target_type': 'revision', 'target': b'\xc7\x8a\n\xee\x93\xd1o*\xaf \xa4/\xbd\x17\xae\x8e\xady0\x9e'}, b'releases/index': {'target_type': 'revision', 'target': b'v\x03\x86\xf1T7\xe0?\x9c\xd4\xc5\x90\xf8\xaf\xb4\xb1\x8d"\xd2\xd5'}, b'releases/mace-1': {'target_type': 'revision', 'target': b'\xc9-f\xa64Z\xecv\xcb\x08r\xd8\xd4F\xccBV\xe3\xef\x8c'}, b'releases/otter-3': {'target_type': 'revision', 'target': b'\x03\xda4\xff4/\xbb]\xd9+\xe4\x0b;O\xe4\xcd\xd5\x8d=\xb2'}, b'releases/otter-emacs': {'target_type': 'revision', 'target': b'\xa9M\xa5\x14W\x84\xc4:d\xfd\x06\n8\xfa1a*\xeas\xc0'}, b'releases/otter-mode11': {'target_type': 'revision', 'target': b'\xf7_\xef\xa0\xea\x1c\x97y4"wYC\xec\x81\x96\xa2\xda\x07\xe3'}, b'releases/otter': {'target_type': 'revision', 'target': b'\xf3Mv\xc7v6\xf1yJ\x08\xf9\xa4\xef0\xf97\xc5\x07\x7fD'}, b'releases/otter304-win95': {'target_type': 'revision', 'target': b'\x81\x05\x80t7\x8e\xac\xa6\xeb\x01G\xc0\xf4\xeb\xaf\xde\x97\x9c<\xce'}, b'releases/otter305-win98': {'target_type': 'revision', 'target': b'xo\xe4I\xf9&\xf2\xfb\x13,0\x19;W"%\xb70\xa1\x1b'}, b'releases/otter_jobs': {'target_type': 'revision', 'target': b' \x8c\x0f\xd2\x02_V\xa7\x08\xb2!\xca2h\r\xdfq\t\r\xd7'}, b'releases/otterface-0': {'target_type': 'revision', 'target': b'5\x95\xbd\xe9\x8c\xce\xd5\x83\xcf\xb2\x9c\xf7h\xb1%\xa2t\x05\xdeg'}, b'releases/pc-1': {'target_type': 'revision', 'target': b'\x1e\xe33\r\xdci\x1c\xc9|>\x0e\x06\x97\xb4\x82\xedJs*\xe5'}, b'HEAD': {'target_type': 'alias', 'target': b'releases/prf_disp'}, b'releases/prf_disp': {'target_type': 'revision', 'target': b'V:\xa9\xf9\xf9\x97\xe4\xa2\xd4\x1d?\x13\x19Xf\xc9p\xf4\xec\x88'}}}
swh-loader_1                    | [2021-06-11 15:03:21,178: DEBUG/ForkPoolWorker-1] snapshot: Snapshot(branches=ImmutableDict({b'releases/eqp-09c': SnapshotBranch(target=b'\x90\xa0\xe2\xca\x0bVf\x88\xa2\xfe\xd9\xeaK\xde\x1a*\xcb\xc6\xa0`', target_type=<TargetType.REVISION: 'revision'>), b'releases/eqp-09d': SnapshotBranch(target=b'\x0b\xcbJ7\x941\xad\xca\xb9\xbcE7\x03\xbeFn\xb8\xca\x81W', target_type=<TargetType.REVISION: 'revision'>), b'releases/eqp-09e': SnapshotBranch(target=b'\xc7\x8a\n\xee\x93\xd1o*\xaf \xa4/\xbd\x17\xae\x8e\xady0\x9e', target_type=<TargetType.REVISION: 'revision'>), b'releases/index': SnapshotBranch(target=b'v\x03\x86\xf1T7\xe0?\x9c\xd4\xc5\x90\xf8\xaf\xb4\xb1\x8d"\xd2\xd5', target_type=<TargetType.REVISION: 'revision'>), b'releases/mace-1': SnapshotBranch(target=b'\xc9-f\xa64Z\xecv\xcb\x08r\xd8\xd4F\xccBV\xe3\xef\x8c', target_type=<TargetType.REVISION: 'revision'>), b'releases/otter-3': SnapshotBranch(target=b'\x03\xda4\xff4/\xbb]\xd9+\xe4\x0b;O\xe4\xcd\xd5\x8d=\xb2', target_type=<TargetType.REVISION: 'revision'>), b'releases/otter-emacs': SnapshotBranch(target=b'\xa9M\xa5\x14W\x84\xc4:d\xfd\x06\n8\xfa1a*\xeas\xc0', target_type=<TargetType.REVISION: 'revision'>), b'releases/otter-mode11': SnapshotBranch(target=b'\xf7_\xef\xa0\xea\x1c\x97y4"wYC\xec\x81\x96\xa2\xda\x07\xe3', target_type=<TargetType.REVISION: 'revision'>), b'releases/otter': SnapshotBranch(target=b'\xf3Mv\xc7v6\xf1yJ\x08\xf9\xa4\xef0\xf97\xc5\x07\x7fD', target_type=<TargetType.REVISION: 'revision'>), b'releases/otter304-win95': SnapshotBranch(target=b'\x81\x05\x80t7\x8e\xac\xa6\xeb\x01G\xc0\xf4\xeb\xaf\xde\x97\x9c<\xce', target_type=<TargetType.REVISION: 'revision'>), b'releases/otter305-win98': SnapshotBranch(target=b'xo\xe4I\xf9&\xf2\xfb\x13,0\x19;W"%\xb70\xa1\x1b', target_type=<TargetType.REVISION: 'revision'>), b'releases/otter_jobs': SnapshotBranch(target=b' \x8c\x0f\xd2\x02_V\xa7\x08\xb2!\xca2h\r\xdfq\t\r\xd7', target_type=<TargetType.REVISION: 'revision'>), b'releases/otterface-0': SnapshotBranch(target=b'5\x95\xbd\xe9\x8c\xce\xd5\x83\xcf\xb2\x9c\xf7h\xb1%\xa2t\x05\xdeg', target_type=<TargetType.REVISION: 'revision'>), b'releases/pc-1': SnapshotBranch(target=b'\x1e\xe33\r\xdci\x1c\xc9|>\x0e\x06\x97\xb4\x82\xedJs*\xe5', target_type=<TargetType.REVISION: 'revision'>), b'HEAD': SnapshotBranch(target=b'releases/prf_disp', target_type=<TargetType.ALIAS: 'alias'>), b'releases/prf_disp': SnapshotBranch(target=b'V:\xa9\xf9\xf9\x97\xe4\xa2\xd4\x1d?\x13\x19Xf\xc9p\xf4\xec\x88', target_type=<TargetType.REVISION: 'revision'>)}), id=b'\xb3\xa0\xc0\x11\x99\x83\xa5p\xb7\xb4}\x02H\x9e\x97UX\x8e\x83-')
swh-loader_1                    | [2021-06-11 15:03:26,114: DEBUG/ForkPoolWorker-1] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@6c2485e6a684', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 19.2.1', 'product': 'RabbitMQ', 'version': '3.6.16'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
swh-loader_1                    | [2021-06-11 15:03:26,117: DEBUG/ForkPoolWorker-1] using channel_id: 1
swh-loader_1                    | [2021-06-11 15:03:26,119: DEBUG/ForkPoolWorker-1] Channel open
swh-loader_1                    | [2021-06-11 15:03:26,121: INFO/ForkPoolWorker-1] Task swh.loader.package.archive.tasks.LoadArchive[dee3a234-61a7-452b-8a73-94054ae3b7a1] succeeded in 82.5276818070015s: {'status': 'eventful', 'snapshot_id': 'b3a0c0119983a570b7b47d02489e9755588e832d'}

anlambert added a commit: rDWAPPSa0db251b3280: common/origin_save/origin_exists: Handle Internet Archive artifact URLs.Jun 11 2021, 5:23 PM

Great, it seems we are getting there :-)

Issue has been resolved and tarballs hosted on the Internet Archive can now be properly loaded in production (see example).

This task has been migrated to GitLab.

save code now: Failure to ingest new 'archives' type when head response is incompleteClosed, MigratedEdits LockedActions

Description

Revisions and Commits

Related Objects

Event Timeline

save code now: Failure to ingest new 'archives' type when head response is incomplete
Closed, MigratedEdits Locked
Actions