Change Details

Another issue exists, sometimes the worker just hangs forever... For example, right now a nixguix process (worker0.internal.staging.swh.network) is currently hanging on a download connection [1] The only solution i see is to kill the process. Which will result in an unfinish visit state. Which gives credits to the reaper proposition defined in T2310. Adding some timeout to the download connection sounds sensible to avoid this kind of caveat [2] Note: It's probably shared to other package loaders. Right now, it's more obvious with this one as it treats a lot of artifacts in round. [1] Last log entry as of now: ``` Apr 09 17:37:09 worker0 python3[1914]: [2020-04-09 17:37:09,838: DEBUG/ForkPoolWorker-1] package_info: {'url': 'http://ftp.ebi.ac.uk/pub/software/vertebrategenomics/exonerate/exonerate-2.4.0.tar.gz', 'raw': {'url': 'http://ftp.ebi.ac.uk/pub/software/vertebrategenomics/exonerate/exonerate-2.4.0.tar.gz', 'integrity': 'sha256-+EkmHcfJfvHxXyIulVsNPa+ZTsE8nbd2bxrH53uqQEI='}} ``` Stracing the issue, it's currently waiting on file descriptor 95: ``` # strace -p 1914 strace: Process 1914 attached recvfrom(95, ``` Which leads to socket: ``` # file /proc/1914/fd/95 /proc/1914/fd/95: symbolic link to socket:[74794390] ``` Indeed, it's stuck at the http connection. ``` root@worker0:~# lsof -p 1914 | grep 74794390 python3 1914 swhworker 95u IPv4 74794390 0t0 TCP worker0.internal.staging.swh.network:58952->hx-xfer-prod.ebi.ac.uk:http (ESTABLISHED) ``` [2] Also, relatedly to download, we discussed with @lewo a possibility to improve the download process to be done in parallel.

Another issue exists, sometimes the worker just hangs forever... For example, right now a nixguix process (worker0.internal.staging.swh.network) is currently hanging on a download connection [1] The only solution i see is to kill the process. Which will result in an unfinish visit state (stuck in ongoing state). Which gives credits to the origin visit reaper proposition btw T2310#43199. Adding some timeout to the download connection sounds sensible to avoid this kind of caveat [2] Note: It's probably shared to other package loaders. Right now, it's more obvious with this one as it treats a lot of artifacts in round. [1] Last log entry as of now: ``` Apr 09 17:37:09 worker0 python3[1914]: [2020-04-09 17:37:09,838: DEBUG/ForkPoolWorker-1] package_info: {'url': 'http://ftp.ebi.ac.uk/pub/software/vertebrategenomics/exonerate/exonerate-2.4.0.tar.gz', 'raw': {'url': 'http://ftp.ebi.ac.uk/pub/software/vertebrategenomics/exonerate/exonerate-2.4.0.tar.gz', 'integrity': 'sha256-+EkmHcfJfvHxXyIulVsNPa+ZTsE8nbd2bxrH53uqQEI='}} ``` Stracing the issue, it's currently waiting on file descriptor 95: ``` # strace -p 1914 strace: Process 1914 attached recvfrom(95, ``` Which leads to socket: ``` # file /proc/1914/fd/95 /proc/1914/fd/95: symbolic link to socket:[74794390] ``` Indeed, it's stuck at the http connection. ``` root@worker0:~# lsof -p 1914 | grep 74794390 python3 1914 swhworker 95u IPv4 74794390 0t0 TCP worker0.internal.staging.swh.network:58952->hx-xfer-prod.ebi.ac.uk:http (ESTABLISHED) ``` [2] Also, relatedly to download, we discussed with @lewo a possibility to improve the download process to be done in parallel.

Another issue exists, sometimes the worker just hangs forever... For example, right now a nixguix process (worker0.internal.staging.swh.network) is currently hanging on a download connection [1] The only solution i see is to kill the process. Which will result in an unfinish visit state (stuck in ongoing state). Which gives credits to the origin visit reaper proposition defined in T2310btw T2310#43199. Adding some timeout to the download connection sounds sensible to avoid this kind of caveat [2] Note: It's probably shared to other package loaders. Right now, it's more obvious with this one as it treats a lot of artifacts in round. [1] Last log entry as of now: ``` Apr 09 17:37:09 worker0 python3[1914]: [2020-04-09 17:37:09,838: DEBUG/ForkPoolWorker-1] package_info: {'url': 'http://ftp.ebi.ac.uk/pub/software/vertebrategenomics/exonerate/exonerate-2.4.0.tar.gz', 'raw': {'url': 'http://ftp.ebi.ac.uk/pub/software/vertebrategenomics/exonerate/exonerate-2.4.0.tar.gz', 'integrity': 'sha256-+EkmHcfJfvHxXyIulVsNPa+ZTsE8nbd2bxrH53uqQEI='}} ``` Stracing the issue, it's currently waiting on file descriptor 95: ``` # strace -p 1914 strace: Process 1914 attached recvfrom(95, ``` Which leads to socket: ``` # file /proc/1914/fd/95 /proc/1914/fd/95: symbolic link to socket:[74794390] ``` Indeed, it's stuck at the http connection. ``` root@worker0:~# lsof -p 1914 | grep 74794390 python3 1914 swhworker 95u IPv4 74794390 0t0 TCP worker0.internal.staging.swh.network:58952->hx-xfer-prod.ebi.ac.uk:http (ESTABLISHED) ``` [2] Also, relatedly to download, we discussed with @lewo a possibility to improve the download process to be done in parallel.