Page MenuHomeSoftware Heritage
Paste P1114

regular update without a cron (execution does not exceed one minute)
ActivePublic

Authored by ardumont on Aug 3 2021, 5:52 PM.
swhwebapp@moma:~$ export DJANGO_SETTINGS_MODULE=swh.web.settings.production
swhwebapp@moma:~$ time /usr/bin/django-admin refresh_savecodenow_statuses
Successfully updated 4 save request(s).
real 0m4.973s
user 0m3.102s
sys 0m0.602s
swhwebapp@moma:~$ time /usr/bin/django-admin refresh_savecodenow_statuses
Successfully updated 4 save request(s).
real 0m4.494s
user 0m2.886s
sys 0m0.380s
swhwebapp@moma:~$ while true; do time /usr/bin/django-admin refresh_savecodenow_statuses; sleep 60; done
Successfully updated 4 save request(s).
real 0m4.689s
user 0m2.943s
sys 0m0.494s
Successfully updated 4 save request(s).
real 0m4.979s
user 0m3.255s
sys 0m0.461s
Successfully updated 4 save request(s).
...
real 0m4.811s
user 0m3.006s
sys 0m0.436s
Successfully updated 4 save request(s).
real 0m5.070s
user 0m3.180s
sys 0m0.446s
Successfully updated 4 save request(s).
real 0m5.144s
user 0m3.264s
sys 0m0.434s
Successfully updated 4 save request(s).
real 0m5.095s
user 0m3.191s
sys 0m0.506s
Successfully updated 4 save request(s).
real 0m4.596s
user 0m2.948s
sys 0m0.381s

Event Timeline

ardumont edited the content of this paste. (Show Details)
ardumont changed the title of this paste from regular update without a cron to regular update without a cron (this does not exceed one minute).

4 visit status always popping up over and over again for some reason:

18:05:11 swh-web@belvedere:5432=> select * from save_origin_request where status='accepted' and (visit_date is null or visit_status is null or visit_status in ('ongoing', 'created')) and request_date >= now() - interval '30 days';
+-------+-------------------------------+------------+-------------------------------------------------------+----------+-----------------+-------------------------------+---------------------+--------------+----------+
|  id   |         request_date          | visit_type |                      origin_url                       |  status  | loading_task_id |          visit_date           | loading_task_status | visit_status | user_ids |
+-------+-------------------------------+------------+-------------------------------------------------------+----------+-----------------+-------------------------------+---------------------+--------------+----------+
| 86401 | 2021-07-17 22:43:29.870783+00 | git        | https://github.com/keybase/client                     | accepted |       396435047 | 2021-07-17 22:43:35.725966+00 | running             | created      | (null)   |
| 87538 | 2021-07-26 12:59:50.038714+00 | svn        | https://svn.r-project.org/R-dev-web/trunk/            | accepted |       396985198 | 2021-07-26 12:59:52.299569+00 | running             | created      | (null)   |
| 87536 | 2021-07-26 12:54:25.784339+00 | svn        | svn://svn.code.sf.net/p/codeblocks/code/trunk         | accepted |       396984987 | 2021-07-26 12:54:31.857131+00 | running             | created      | (null)   |
| 87544 | 2021-07-26 13:34:16.987444+00 | git        | https://android.googlesource.com/platform/system/core | accepted |       396985908 | 2021-07-26 13:34:25.182417+00 | running             | created      | (null)   |
+-------+-------------------------------+------------+-------------------------------------------------------+----------+-----------------+-------------------------------+---------------------+--------------+----------+
(4 rows)

Time: 48.099 ms

4 visit status always popping up over and over again for some reason:

"A quick look", those are big repositories that may have been oom killed.

(i've rescheduled those)

root@pergamon:~# clush -b -w @swh-workers systemctl status swh-worker@loader_high_priority | grep keybase
Aug 03 16:15:46 worker11 python3[3836]: [2021-08-03 16:15:46,315: INFO/ForkPoolWorker-1] Load origin 'https://github.com/keybase/client' with type 'git'
Aug 03 16:17:12 worker11 python3[3836]: [2021-08-03 16:17:12,887: INFO/ForkPoolWorker-1] Listed 19843 refs for repo https://github.com/keybase/client
root@pergamon:~# clush -b -w @swh-workers systemctl status swh-worker@loader_high_priority | grep android
Aug 03 16:15:46 worker12 python3[3872]: [2021-08-03 16:15:46,341: INFO/ForkPoolWorker-1] Load origin 'https://android.googlesource.com/platform/system/core' with type 'git'
Aug 03 16:17:00 worker12 python3[3872]: [2021-08-03 16:17:00,671: INFO/ForkPoolWorker-1] Listed 48703 refs for repo https://android.googlesource.com/platform/system/core
root@pergamon:~# clush -b -w @swh-workers systemctl status swh-worker@loader_high_priority | grep trunk
             └─340690 svnrdump dump svn://svn.code.sf.net/p/codeblocks/code/trunk
Aug 03 16:15:46 worker06 python3[3724]: [2021-08-03 16:15:46,888: INFO/ForkPoolWorker-1] Load origin 'svn://svn.code.sf.net/p/codeblocks/code/trunk' with type 'svn'
             ├─334757 cat /tmp/tmp5je6yu23/tmpmi2i2ya_/httpssvnrprojectorgRdevwebtrunk.svndump
Aug 03 16:15:46 worker14 python3[3759]: [2021-08-03 16:15:46,955: INFO/ForkPoolWorker-1] Load origin 'https://svn.r-project.org/R-dev-web/trunk/' with type 'svn'

somehow for keybase, that ends up in:

root@pergamon:~# clush -b -w @swh-workers systemctl status swh-worker@loader_high_priority | grep -A3 keybase
Aug 03 16:15:46 worker11 python3[3836]: [2021-08-03 16:15:46,315: INFO/ForkPoolWorker-1] Load origin 'https://github.com/keybase/client' with type 'git'
Aug 03 16:15:52 worker11 python3[3836]: [62B blob data]
Aug 03 16:15:52 worker11 python3[3836]: [3.4K blob data]
Aug 03 16:15:53 worker11 python3[3836]: [3.7K blob data]
--
Aug 03 16:17:12 worker11 python3[3836]: [2021-08-03 16:17:12,887: INFO/ForkPoolWorker-1] Listed 19843 refs for repo https://github.com/keybase/client
Aug 03 16:22:13 worker11 python3[3775]: Process 'ForkPoolWorker-1' pid:3836 exited with 'signal 9 (SIGKILL)'
Aug 03 16:22:14 worker11 python3[3775]: [2021-08-03 16:22:13,883: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 9 (SIGKILL).')
                                        Traceback (most recent call last):

somehow ^

android one went well though:

Aug 03 16:15:46 worker12 python3[3872]: [2021-08-03 16:15:46,341: INFO/ForkPoolWorker-1] Load origin 'https://android.googlesource.com/platform/system/core' with type 'git'
Aug 03 16:17:00 worker12 python3[3872]: Sending approximately 188.57 MiB ...
Aug 03 16:17:00 worker12 python3[3872]: [51B blob data]
Aug 03 16:17:00 worker12 python3[3872]: [4.0K blob data]
--
Aug 03 16:17:00 worker12 python3[3872]: [2021-08-03 16:17:00,671: INFO/ForkPoolWorker-1] Listed 48703 refs for repo https://android.googlesource.com/platform/system/core
Aug 03 16:24:09 worker12 python3[3872]: [2021-08-03 16:24:09,256: INFO/ForkPoolWorker-1] Task swh.loader.git.tasks.UpdateGitRepository[3e1cdc0c-8d55-4bed-b455-188811852105] succeeded in 503.007593125978s: {'status': 'eventful'}

on worker17 which is a bit more involved as our current worker, this gets killed with oom:

swhworker@worker17:~$ swh loader -C /etc/softwareheritage/loader_oneshot.yml run git https://github.com/keybase/client
INFO:swh.loader.git.loader.GitLoader:Load origin 'https://github.com/keybase/client' with type 'git'
Enumerating objects: 556997, done.
Counting objects: 100% (2700/2700), done.
Compressing objects: 100% (2219/2219), done.
Total 556997 (delta 589), reused 2436 (delta 457), pack-reused 554297
INFO:swh.loader.git.loader.GitLoader:Listed 19843 refs for repo https://github.com/keybase/client
ERROR:swh.loader.git.loader.GitLoader:Loading failure, updating to `failed` status
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 339, in load
    self.store_data()
  File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 463, in store_data
    for release in self.get_releases():
  File "/usr/lib/python3/dist-packages/swh/loader/git/loader.py", line 349, in get_releases
    for raw_obj in self.iter_objects(b"tag"):
  File "/usr/lib/python3/dist-packages/swh/loader/git/loader.py", line 315, in iter_objects
    PackData.from_file(self.pack_buffer, self.pack_size)
  File "/usr/lib/python3/dist-packages/dulwich/pack.py", line 1337, in _walk_all_chains
    for result in self._follow_chain(offset, type_num, None):
  File "/usr/lib/python3/dist-packages/dulwich/pack.py", line 1393, in _follow_chain
    unpacked = self._resolve_object(offset, obj_type_num, base_chunks)
  File "/usr/lib/python3/dist-packages/dulwich/pack.py", line 1385, in _resolve_object
    unpacked.decomp_chunks)
MemoryError
{'status': 'failed'}
ardumont changed the title of this paste from regular update without a cron (this does not exceed one minute) to regular update without a cron (execution does not exceed one minute).Aug 4 2021, 10:16 AM