Page MenuHomeSoftware Heritage
Create Task
Maniphest T1514

MemoryError in loader-git
Closed, MigratedEdits Locked
Actions

Description

Got this (in a docker env):

for the task

swh-loader_1                  | [2019-02-04 22:01:11,762: INFO/ForkPoolWorker-78] Listed 246 refs for repo https://0xacab.org/taggart/check_mk.git
swh-loader_1                  | [2019-02-04 22:06:10,247: DEBUG/ForkPoolWorker-78] Sending 10000 contents
swh-loader_1                  | [2019-02-04 22:06:16,323: DEBUG/ForkPoolWorker-78] Done sending 10000 contents
swh-loader_1                  | [2019-02-04 22:06:18,814: DEBUG/ForkPoolWorker-78] Sending 9266 contents
swh-loader_1                  | [2019-02-04 22:06:45,060: DEBUG/ForkPoolWorker-78] Done sending 9266 contents
swh-loader_1                  | [2019-02-04 22:06:46,581: DEBUG/ForkPoolWorker-78] Sending 7 contents
swh-loader_1                  | [2019-02-04 22:07:01,304: DEBUG/ForkPoolWorker-78] Done sending 7 contents
swh-loader_1                  | [2019-02-04 22:07:02,597: DEBUG/ForkPoolWorker-78] Sending 461 contents
swh-loader_1                  | [2019-02-04 22:07:17,958: DEBUG/ForkPoolWorker-78] Done sending 461 contents
swh-loader_1                  | [2019-02-04 22:07:19,119: DEBUG/ForkPoolWorker-78] Sending 404 contents
swh-loader_1                  | [2019-02-04 22:07:29,656: DEBUG/ForkPoolWorker-78] Done sending 404 contents
swh-loader_1                  | [2019-02-04 22:07:30,867: DEBUG/ForkPoolWorker-78] Sending 26 contents
swh-loader_1                  | [2019-02-04 22:07:43,644: DEBUG/ForkPoolWorker-78] Done sending 26 contents
swh-loader_1                  | [2019-02-04 22:07:44,695: DEBUG/ForkPoolWorker-78] Sending 40 contents
swh-loader_1                  | [2019-02-04 22:07:59,788: DEBUG/ForkPoolWorker-78] Done sending 40 contents
swh-loader_1                  | [2019-02-04 22:08:00,847: DEBUG/ForkPoolWorker-78] Sending 90 contents
swh-loader_1                  | [2019-02-04 22:08:13,685: DEBUG/ForkPoolWorker-78] Done sending 90 contents
swh-loader_1                  | [2019-02-04 22:08:14,720: DEBUG/ForkPoolWorker-78] Sending 356 contents
swh-loader_1                  | [2019-02-04 22:08:24,389: DEBUG/ForkPoolWorker-78] Done sending 356 contents
swh-loader_1                  | [2019-02-04 22:08:25,431: DEBUG/ForkPoolWorker-78] Sending 1010 contents
swh-loader_1                  | [2019-02-04 22:08:31,103: DEBUG/ForkPoolWorker-78] Done sending 1010 contents
swh-loader_1                  | [2019-02-04 22:08:33,886: DEBUG/ForkPoolWorker-78] Sending 10000 contents
swh-loader_1                  | [2019-02-04 22:08:51,682: DEBUG/ForkPoolWorker-78] Done sending 10000 contents
swh-loader_1                  | [2019-02-04 22:08:53,840: DEBUG/ForkPoolWorker-78] Sending 10000 contents
swh-loader_1                  | [2019-02-04 22:09:15,183: DEBUG/ForkPoolWorker-78] Done sending 10000 contents
swh-loader_1                  | [2019-02-04 22:09:16,717: DEBUG/ForkPoolWorker-78] Sending 3163 contents
swh-loader_1                  | [2019-02-04 22:09:28,613: DEBUG/ForkPoolWorker-78] Done sending 3163 contents
swh-loader_1                  | [2019-02-04 22:09:30,575: DEBUG/ForkPoolWorker-78] Sending 4800 contents
swh-loader_1                  | [2019-02-04 22:09:48,590: DEBUG/ForkPoolWorker-78] Done sending 4800 contents
swh-loader_1                  | [2019-02-04 22:09:49,987: DEBUG/ForkPoolWorker-78] Sending 267 contents
swh-loader_1                  | [2019-02-04 22:10:00,342: DEBUG/ForkPoolWorker-78] Done sending 267 contents
swh-loader_1                  | [2019-02-04 22:10:01,339: DEBUG/ForkPoolWorker-78] Sending 38 contents
swh-loader_1                  | [2019-02-04 22:10:14,504: DEBUG/ForkPoolWorker-78] Done sending 38 contents
swh-loader_1                  | [2019-02-04 22:10:15,929: DEBUG/ForkPoolWorker-78] Sending 480 contents
swh-loader_1                  | [2019-02-04 22:10:27,080: DEBUG/ForkPoolWorker-78] Done sending 480 contents
swh-loader_1                  | [2019-02-04 22:10:28,496: DEBUG/ForkPoolWorker-78] Sending 2015 contents
swh-loader_1                  | [2019-02-04 22:10:35,806: DEBUG/ForkPoolWorker-78] Done sending 2015 contents
swh-loader_1                  | [2019-02-04 22:10:37,206: DEBUG/ForkPoolWorker-78] Sending 3443 contents
swh-loader_1                  | [2019-02-04 22:10:50,143: DEBUG/ForkPoolWorker-78] Done sending 3443 contents
swh-loader_1                  | [2019-02-04 22:10:51,400: DEBUG/ForkPoolWorker-78] Sending 957 contents
swh-loader_1                  | [2019-02-04 22:11:08,386: DEBUG/ForkPoolWorker-78] Done sending 957 contents
swh-loader_1                  | [2019-02-04 22:11:09,384: DEBUG/ForkPoolWorker-78] Sending 141 contents
swh-loader_1                  | [2019-02-04 22:11:24,293: DEBUG/ForkPoolWorker-78] Done sending 141 contents
swh-loader_1                  | [2019-02-04 22:11:25,289: DEBUG/ForkPoolWorker-78] Sending 140 contents
swh-loader_1                  | [2019-02-04 22:11:40,312: DEBUG/ForkPoolWorker-78] Done sending 140 contents
swh-loader_1                  | [2019-02-04 22:11:41,288: DEBUG/ForkPoolWorker-78] Sending 140 contents
swh-loader_1                  | [2019-02-04 22:11:56,447: DEBUG/ForkPoolWorker-78] Done sending 140 contents
swh-loader_1                  | [2019-02-04 22:11:57,433: DEBUG/ForkPoolWorker-78] Sending 139 contents
swh-loader_1                  | [2019-02-04 22:12:12,435: DEBUG/ForkPoolWorker-78] Done sending 139 contents
swh-loader_1                  | [2019-02-04 22:12:13,445: DEBUG/ForkPoolWorker-78] Sending 144 contents
swh-loader_1                  | [2019-02-04 22:12:28,304: DEBUG/ForkPoolWorker-78] Done sending 144 contents
swh-loader_1                  | [2019-02-04 22:12:29,259: DEBUG/ForkPoolWorker-78] Sending 143 contents
swh-loader_1                  | [2019-02-04 22:12:44,295: DEBUG/ForkPoolWorker-78] Done sending 143 contents
swh-loader_1                  | [2019-02-04 22:12:45,269: DEBUG/ForkPoolWorker-78] Sending 144 contents
swh-loader_1                  | [2019-02-04 22:13:00,267: DEBUG/ForkPoolWorker-78] Done sending 144 contents
swh-loader_1                  | [2019-02-04 22:13:01,227: DEBUG/ForkPoolWorker-78] Sending 148 contents
swh-loader_1                  | [2019-02-04 22:13:16,331: DEBUG/ForkPoolWorker-78] Done sending 148 contents
swh-loader_1                  | [2019-02-04 22:13:17,336: DEBUG/ForkPoolWorker-78] Sending 591 contents
swh-loader_1                  | [2019-02-04 22:13:32,604: DEBUG/ForkPoolWorker-78] Done sending 591 contents
swh-loader_1                  | [2019-02-04 22:13:33,827: DEBUG/ForkPoolWorker-78] Sending 1824 contents
swh-loader_1                  | [2019-02-04 22:13:43,649: DEBUG/ForkPoolWorker-78] Done sending 1824 contents
swh-loader_1                  | [2019-02-04 22:13:44,853: DEBUG/ForkPoolWorker-78] Sending 1792 contents
swh-loader_1                  | [2019-02-04 22:13:53,726: DEBUG/ForkPoolWorker-78] Done sending 1792 contents
swh-loader_1                  | [2019-02-04 22:13:55,018: DEBUG/ForkPoolWorker-78] Sending 770 contents
swh-loader_1                  | [2019-02-04 22:14:04,946: DEBUG/ForkPoolWorker-78] Done sending 770 contents
swh-loader_1                  | [2019-02-04 22:14:06,155: DEBUG/ForkPoolWorker-78] Sending 114 contents
swh-loader_1                  | [2019-02-04 22:14:17,741: DEBUG/ForkPoolWorker-78] Done sending 114 contents
swh-loader_1                  | [2019-02-04 22:14:19,044: DEBUG/ForkPoolWorker-78] Sending 35 contents
swh-loader_1                  | [2019-02-04 22:14:33,063: DEBUG/ForkPoolWorker-78] Done sending 35 contents
swh-loader_1                  | [2019-02-04 22:14:34,157: DEBUG/ForkPoolWorker-78] Sending 8 contents
swh-loader_1                  | [2019-02-04 22:14:46,962: DEBUG/ForkPoolWorker-78] Done sending 8 contents
swh-loader_1                  | [2019-02-04 22:14:48,005: DEBUG/ForkPoolWorker-78] Sending 8 contents
swh-loader_1                  | [2019-02-04 22:15:00,937: DEBUG/ForkPoolWorker-78] Done sending 8 contents
swh-loader_1                  | [2019-02-04 22:15:01,969: DEBUG/ForkPoolWorker-78] Sending 25 contents
swh-loader_1                  | [2019-02-04 22:15:14,619: DEBUG/ForkPoolWorker-78] Done sending 25 contents
swh-loader_1                  | [2019-02-04 22:15:15,652: DEBUG/ForkPoolWorker-78] Sending 84 contents
swh-loader_1                  | [2019-02-04 22:15:29,427: DEBUG/ForkPoolWorker-78] Done sending 84 contents
swh-loader_1                  | [2019-02-04 22:15:30,392: DEBUG/ForkPoolWorker-78] Sending 86 contents
swh-loader_1                  | [2019-02-04 22:15:42,263: DEBUG/ForkPoolWorker-78] Done sending 86 contents
swh-loader_1                  | [2019-02-04 22:15:43,460: DEBUG/ForkPoolWorker-78] Sending 14 contents
swh-loader_1                  | [2019-02-04 22:15:53,996: DEBUG/ForkPoolWorker-78] Done sending 14 contents
swh-loader_1                  | [2019-02-04 22:15:55,043: DEBUG/ForkPoolWorker-78] Sending 9 contents
swh-loader_1                  | [2019-02-04 22:16:04,859: DEBUG/ForkPoolWorker-78] Done sending 9 contents
swh-loader_1                  | [2019-02-04 22:16:05,900: DEBUG/ForkPoolWorker-78] Sending 10 contents
swh-loader_1                  | [2019-02-04 22:16:15,853: DEBUG/ForkPoolWorker-78] Done sending 10 contents
swh-loader_1                  | [2019-02-04 22:16:17,063: DEBUG/ForkPoolWorker-78] Sending 780 contents
swh-loader_1                  | [2019-02-04 22:16:31,538: DEBUG/ForkPoolWorker-78] Done sending 780 contents
swh-loader_1                  | [2019-02-04 22:16:32,589: DEBUG/ForkPoolWorker-78] Sending 566 contents
swh-loader_1                  | [2019-02-04 22:16:48,293: DEBUG/ForkPoolWorker-78] Done sending 566 contents
swh-loader_1                  | [2019-02-04 22:16:49,360: DEBUG/ForkPoolWorker-78] Sending 605 contents
swh-loader_1                  | [2019-02-04 22:17:11,644: DEBUG/ForkPoolWorker-78] Done sending 605 contents
swh-loader_1                  | [2019-02-04 22:17:12,705: DEBUG/ForkPoolWorker-78] Sending 474 contents
swh-loader_1                  | [2019-02-04 22:17:22,657: DEBUG/ForkPoolWorker-78] Done sending 474 contents
swh-loader_1                  | [2019-02-04 22:17:23,674: DEBUG/ForkPoolWorker-78] Sending 186 contents
swh-loader_1                  | [2019-02-04 22:17:34,255: DEBUG/ForkPoolWorker-78] Done sending 186 contents
swh-loader_1                  | [2019-02-04 22:17:35,392: DEBUG/ForkPoolWorker-78] Sending 645 contents
swh-loader_1                  | [2019-02-04 22:17:45,710: DEBUG/ForkPoolWorker-78] Done sending 645 contents
swh-loader_1                  | [2019-02-04 22:17:46,685: DEBUG/ForkPoolWorker-78] Sending 192 contents
swh-loader_1                  | [2019-02-04 22:17:57,403: DEBUG/ForkPoolWorker-78] Done sending 192 contents
swh-loader_1                  | [2019-02-04 22:17:58,377: DEBUG/ForkPoolWorker-78] Sending 196 contents
swh-loader_1                  | [2019-02-04 22:18:09,049: DEBUG/ForkPoolWorker-78] Done sending 196 contents
swh-loader_1                  | [2019-02-04 22:18:09,998: DEBUG/ForkPoolWorker-78] Sending 200 contents
swh-loader_1                  | [2019-02-04 22:18:21,186: DEBUG/ForkPoolWorker-78] Done sending 200 contents
swh-loader_1                  | [2019-02-04 22:18:22,128: DEBUG/ForkPoolWorker-78] Sending 199 contents
swh-loader_1                  | [2019-02-04 22:18:33,000: DEBUG/ForkPoolWorker-78] Done sending 199 contents
swh-loader_1                  | [2019-02-04 22:18:34,137: DEBUG/ForkPoolWorker-78] Sending 51 contents
swh-loader_1                  | [2019-02-04 22:18:47,720: DEBUG/ForkPoolWorker-78] Done sending 51 contents
swh-loader_1                  | [2019-02-04 22:18:48,803: DEBUG/ForkPoolWorker-78] Sending 8 contents
swh-loader_1                  | [2019-02-04 22:19:02,640: DEBUG/ForkPoolWorker-78] Done sending 8 contents
swh-loader_1                  | [2019-02-04 22:19:03,807: DEBUG/ForkPoolWorker-78] Sending 31 contents
swh-loader_1                  | [2019-02-04 22:19:15,095: DEBUG/ForkPoolWorker-78] Done sending 31 contents
swh-loader_1                  | [2019-02-04 22:19:16,102: DEBUG/ForkPoolWorker-78] Sending 17 contents
swh-loader_1                  | [2019-02-04 22:19:25,826: DEBUG/ForkPoolWorker-78] Done sending 17 contents
swh-loader_1                  | [2019-02-04 22:19:26,920: DEBUG/ForkPoolWorker-78] Sending 1415 contents
swh-loader_1                  | [2019-02-04 22:19:37,548: DEBUG/ForkPoolWorker-78] Done sending 1415 contents
swh-loader_1                  | [2019-02-04 22:19:38,623: DEBUG/ForkPoolWorker-78] Sending 456 contents
swh-loader_1                  | [2019-02-04 22:19:47,249: DEBUG/ForkPoolWorker-78] Done sending 456 contents
swh-loader_1                  | [2019-02-04 22:19:48,316: DEBUG/ForkPoolWorker-78] Sending 339 contents
swh-loader_1                  | [2019-02-04 22:19:58,026: DEBUG/ForkPoolWorker-78] Done sending 339 contents
swh-loader_1                  | [2019-02-04 22:19:59,202: DEBUG/ForkPoolWorker-78] Sending 62 contents
swh-loader_1                  | [2019-02-04 22:20:10,176: DEBUG/ForkPoolWorker-78] Done sending 62 contents
swh-loader_1                  | [2019-02-04 22:20:11,456: DEBUG/ForkPoolWorker-78] Sending 48 contents
swh-loader_1                  | [2019-02-04 22:20:23,843: DEBUG/ForkPoolWorker-78] Done sending 48 contents
swh-loader_1                  | [2019-02-04 22:20:24,960: DEBUG/ForkPoolWorker-78] Sending 65 contents
swh-loader_1                  | [2019-02-04 22:20:36,725: DEBUG/ForkPoolWorker-78] Done sending 65 contents
swh-loader_1                  | [2019-02-04 22:20:37,830: DEBUG/ForkPoolWorker-78] Sending 145 contents
swh-loader_1                  | [2019-02-04 22:20:49,793: DEBUG/ForkPoolWorker-78] Done sending 145 contents
swh-loader_1                  | [2019-02-04 22:20:50,875: DEBUG/ForkPoolWorker-78] Sending 558 contents
swh-loader_1                  | [2019-02-04 22:21:01,788: DEBUG/ForkPoolWorker-78] Done sending 558 contents
swh-loader_1                  | [2019-02-04 22:21:02,829: DEBUG/ForkPoolWorker-78] Sending 242 contents
swh-loader_1                  | [2019-02-04 22:21:16,667: DEBUG/ForkPoolWorker-78] Done sending 242 contents
swh-loader_1                  | [2019-02-04 22:21:17,621: DEBUG/ForkPoolWorker-78] Sending 233 contents
swh-loader_1                  | [2019-02-04 22:21:28,596: DEBUG/ForkPoolWorker-78] Done sending 233 contents
swh-loader_1                  | [2019-02-04 22:21:29,592: DEBUG/ForkPoolWorker-78] Sending 243 contents
swh-loader_1                  | [2019-02-04 22:21:40,682: DEBUG/ForkPoolWorker-78] Done sending 243 contents
swh-loader_1                  | [2019-02-04 22:21:41,640: DEBUG/ForkPoolWorker-78] Sending 235 contents
swh-loader_1                  | [2019-02-04 22:21:52,469: DEBUG/ForkPoolWorker-78] Done sending 235 contents
swh-loader_1                  | [2019-02-04 22:21:53,441: DEBUG/ForkPoolWorker-78] Sending 249 contents
swh-loader_1                  | [2019-02-04 22:22:04,226: DEBUG/ForkPoolWorker-78] Done sending 249 contents
swh-loader_1                  | [2019-02-04 22:22:05,255: DEBUG/ForkPoolWorker-78] Sending 239 contents
swh-loader_1                  | [2019-02-04 22:22:16,029: DEBUG/ForkPoolWorker-78] Done sending 239 contents
swh-loader_1                  | [2019-02-04 22:22:17,128: DEBUG/ForkPoolWorker-78] Sending 1336 contents
swh-loader_1                  | [2019-02-04 22:22:35,879: DEBUG/ForkPoolWorker-78] Done sending 1336 contents
swh-loader_1                  | [2019-02-04 22:22:37,187: DEBUG/ForkPoolWorker-78] Sending 1489 contents
swh-loader_1                  | [2019-02-04 22:22:48,185: DEBUG/ForkPoolWorker-78] Done sending 1489 contents
swh-loader_1                  | [2019-02-04 22:22:49,282: DEBUG/ForkPoolWorker-78] Sending 49 contents
swh-loader_1                  | [2019-02-04 22:23:02,595: DEBUG/ForkPoolWorker-78] Done sending 49 contents
swh-loader_1                  | [2019-02-04 22:23:03,846: DEBUG/ForkPoolWorker-78] Sending 82 contents
swh-loader_1                  | [2019-02-04 22:23:17,393: DEBUG/ForkPoolWorker-78] Done sending 82 contents
swh-loader_1                  | [2019-02-04 22:23:18,684: DEBUG/ForkPoolWorker-78] Sending 90 contents
swh-loader_1                  | [2019-02-04 22:23:32,770: DEBUG/ForkPoolWorker-78] Done sending 90 contents
swh-loader_1                  | [2019-02-04 22:23:33,901: DEBUG/ForkPoolWorker-78] Sending 263 contents
swh-loader_1                  | [2019-02-04 22:23:44,144: DEBUG/ForkPoolWorker-78] Done sending 263 contents
swh-loader_1                  | [2019-02-04 22:23:45,437: DEBUG/ForkPoolWorker-78] Sending 358 contents
swh-loader_1                  | [2019-02-04 22:23:54,634: DEBUG/ForkPoolWorker-78] Done sending 358 contents
swh-loader_1                  | [2019-02-04 22:23:55,688: DEBUG/ForkPoolWorker-78] Sending 338 contents
swh-loader_1                  | [2019-02-04 22:24:10,234: DEBUG/ForkPoolWorker-78] Done sending 338 contents
swh-loader_1                  | [2019-02-04 22:24:11,246: DEBUG/ForkPoolWorker-78] Sending 339 contents
swh-loader_1                  | [2019-02-04 22:24:22,995: DEBUG/ForkPoolWorker-78] Done sending 339 contents
swh-loader_1                  | [2019-02-04 22:24:24,022: DEBUG/ForkPoolWorker-78] Sending 341 contents
swh-loader_1                  | [2019-02-04 22:24:35,874: DEBUG/ForkPoolWorker-78] Done sending 341 contents
swh-loader_1                  | [2019-02-04 22:24:36,872: DEBUG/ForkPoolWorker-78] Sending 323 contents
swh-loader_1                  | [2019-02-04 22:24:48,593: DEBUG/ForkPoolWorker-78] Done sending 323 contents
swh-loader_1                  | [2019-02-04 22:24:49,650: DEBUG/ForkPoolWorker-78] Sending 317 contents
swh-loader_1                  | [2019-02-04 22:25:01,264: DEBUG/ForkPoolWorker-78] Done sending 317 contents
swh-loader_1                  | [2019-02-04 22:25:02,322: DEBUG/ForkPoolWorker-78] Sending 331 contents
swh-loader_1                  | [2019-02-04 22:25:14,013: DEBUG/ForkPoolWorker-78] Done sending 331 contents
swh-loader_1                  | [2019-02-04 22:25:15,078: DEBUG/ForkPoolWorker-78] Sending 333 contents
swh-loader_1                  | [2019-02-04 22:25:39,098: DEBUG/ForkPoolWorker-78] Done sending 333 contents
swh-loader_1                  | [2019-02-04 22:25:40,139: DEBUG/ForkPoolWorker-78] Sending 341 contents
swh-loader_1                  | [2019-02-04 22:25:51,663: DEBUG/ForkPoolWorker-78] Done sending 341 contents
swh-loader_1                  | [2019-02-04 22:25:52,661: DEBUG/ForkPoolWorker-78] Sending 343 contents
swh-loader_1                  | [2019-02-04 22:26:04,208: DEBUG/ForkPoolWorker-78] Done sending 343 contents
swh-loader_1                  | [2019-02-04 22:26:05,242: DEBUG/ForkPoolWorker-78] Sending 317 contents
swh-loader_1                  | [2019-02-04 22:26:16,853: DEBUG/ForkPoolWorker-78] Done sending 317 contents
swh-loader_1                  | [2019-02-04 22:26:18,052: DEBUG/ForkPoolWorker-78] Sending 633 contents
swh-loader_1                  | [2019-02-04 22:26:28,362: DEBUG/ForkPoolWorker-78] Done sending 633 contents
swh-loader_1                  | [2019-02-04 22:26:29,473: DEBUG/ForkPoolWorker-78] Sending 334 contents
swh-loader_1                  | [2019-02-04 22:26:40,568: DEBUG/ForkPoolWorker-78] Done sending 334 contents
swh-loader_1                  | [2019-02-04 22:26:41,723: DEBUG/ForkPoolWorker-78] Sending 1280 contents
swh-loader_1                  | [2019-02-04 22:27:03,858: DEBUG/ForkPoolWorker-78] Done sending 1280 contents
swh-loader_1                  | [2019-02-04 22:27:04,945: DEBUG/ForkPoolWorker-78] Sending 375 contents
swh-loader_1                  | [2019-02-04 22:27:17,186: DEBUG/ForkPoolWorker-78] Done sending 375 contents
swh-loader_1                  | [2019-02-04 22:27:18,209: DEBUG/ForkPoolWorker-78] Sending 381 contents
swh-loader_1                  | [2019-02-04 22:27:30,194: DEBUG/ForkPoolWorker-78] Done sending 381 contents
swh-loader_1                  | [2019-02-04 22:27:31,232: DEBUG/ForkPoolWorker-78] Sending 379 contents
swh-loader_1                  | [2019-02-04 22:27:43,223: DEBUG/ForkPoolWorker-78] Done sending 379 contents
swh-loader_1                  | [2019-02-04 22:27:44,241: DEBUG/ForkPoolWorker-78] Sending 378 contents
swh-loader_1                  | [2019-02-04 22:27:56,367: DEBUG/ForkPoolWorker-78] Done sending 378 contents
swh-loader_1                  | [2019-02-04 22:27:57,444: DEBUG/ForkPoolWorker-78] Sending 701 contents
swh-loader_1                  | [2019-02-04 22:28:19,520: DEBUG/ForkPoolWorker-78] Done sending 701 contents
swh-loader_1                  | [2019-02-04 22:28:20,526: DEBUG/ForkPoolWorker-78] Sending 179 contents
swh-loader_1                  | [2019-02-04 22:28:35,859: DEBUG/ForkPoolWorker-78] Done sending 179 contents
swh-loader_1                  | [2019-02-04 22:28:36,804: DEBUG/ForkPoolWorker-78] Sending 199 contents
swh-loader_1                  | [2019-02-04 22:28:52,484: DEBUG/ForkPoolWorker-78] Done sending 199 contents
swh-loader_1                  | [2019-02-04 22:28:53,499: DEBUG/ForkPoolWorker-78] Sending 211 contents
swh-loader_1                  | [2019-02-04 22:29:09,308: DEBUG/ForkPoolWorker-78] Done sending 211 contents
swh-loader_1                  | [2019-02-04 22:29:10,400: DEBUG/ForkPoolWorker-78] Sending 801 contents
swh-loader_1                  | [2019-02-04 22:29:24,838: DEBUG/ForkPoolWorker-78] Done sending 801 contents
swh-loader_1                  | [2019-02-04 22:29:26,109: DEBUG/ForkPoolWorker-78] Sending 1887 contents
swh-loader_1                  | [2019-02-04 22:29:37,143: DEBUG/ForkPoolWorker-78] Done sending 1887 contents
swh-loader_1                  | [2019-02-04 22:29:38,338: DEBUG/ForkPoolWorker-78] Sending 980 contents
swh-loader_1                  | [2019-02-04 22:29:50,818: DEBUG/ForkPoolWorker-78] Done sending 980 contents
swh-loader_1                  | [2019-02-04 22:29:51,921: DEBUG/ForkPoolWorker-78] Sending 812 contents
swh-loader_1                  | [2019-02-04 22:30:18,332: DEBUG/ForkPoolWorker-78] Done sending 812 contents
swh-loader_1                  | [2019-02-04 22:30:18,700: DEBUG/ForkPoolWorker-78] Sending 767 contents
swh-loader_1                  | [2019-02-04 22:30:22,050: DEBUG/ForkPoolWorker-78] Done sending 767 contents
swh-loader_1                  | [2019-02-04 22:32:15,691: DEBUG/ForkPoolWorker-78] Sending 25000 directories
swh-loader_1                  | [2019-02-04 22:32:20,984: ERROR/ForkPoolWorker-78] Loading failure, updating to `partial` status
swh-loader_1                  | Traceback (most recent call last):
swh-loader_1                  |   File "/usr/local/lib/python3.6/site-packages/swh/loader/core/loader.py", line 894, in load
swh-loader_1                  |     self.store_data()
swh-loader_1                  |   File "/usr/local/lib/python3.6/site-packages/swh/loader/core/loader.py", line 1003, in store_data
swh-loader_1                  |     self.send_batch_directories(self.get_directories())
swh-loader_1                  |   File "/usr/local/lib/python3.6/site-packages/swh/loader/core/loader.py", line 652, in send_batch_directories
swh-loader_1                  |     send_in_packets(directories, self.send_directories, packet_size)
swh-loader_1                  |   File "/usr/local/lib/python3.6/site-packages/swh/loader/core/loader.py", line 41, in send_in_packets
swh-loader_1                  |     sender(formatted_objects)
swh-loader_1                  |   File "/usr/local/lib/python3.6/site-packages/retrying.py", line 49, in wrapped_f
swh-loader_1                  |     return Retrying(*dargs, **dkw).call(f, *args, **kw)
swh-loader_1                  |   File "/usr/local/lib/python3.6/site-packages/retrying.py", line 206, in call
swh-loader_1                  |     return attempt.get(self._wrap_exception)
swh-loader_1                  |   File "/usr/local/lib/python3.6/site-packages/retrying.py", line 247, in get
swh-loader_1                  |     six.reraise(self.value[0], self.value[1], self.value[2])
swh-loader_1                  |   File "/usr/local/lib/python3.6/site-packages/six.py", line 693, in reraise
swh-loader_1                  |     raise value
swh-loader_1                  |   File "/usr/local/lib/python3.6/site-packages/retrying.py", line 200, in call
swh-loader_1                  |     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
swh-loader_1                  |   File "/usr/local/lib/python3.6/site-packages/swh/loader/core/loader.py", line 425, in send_directories
swh-loader_1                  |     self.storage.directory_add(directory_list)
swh-loader_1                  |   File "/usr/local/lib/python3.6/site-packages/swh/storage/api/client.py", line 51, in directory_add
swh-loader_1                  |     return self.post('directory/add', {'directories': directories})
swh-loader_1                  |   File "/home/swh/.local/lib/python3.6/site-packages/swh/core/api/__init__.py", line 180, in post
swh-loader_1                  |     data = encode_data(data)
swh-loader_1                  |   File "/home/swh/.local/lib/python3.6/site-packages/swh/core/api/serializers.py", line 19, in encode_data_client
swh-loader_1                  |     return msgpack_dumps(data)
swh-loader_1                  |   File "/home/swh/.local/lib/python3.6/site-packages/swh/core/api/serializers.py", line 170, in msgpack_dumps
swh-loader_1                  |     return msgpack.packb(data, use_bin_type=True, default=encode_types)
swh-loader_1                  |   File "/usr/local/lib/python3.6/site-packages/msgpack/__init__.py", line 47, in packb
swh-loader_1                  |     return Packer(**kwargs).pack(o)
swh-loader_1                  |   File "msgpack/_packer.pyx", line 284, in msgpack._packer.Packer.pack
swh-loader_1                  |   File "msgpack/_packer.pyx", line 294, in msgpack._packer.Packer.pack
swh-loader_1                  | MemoryError
swh-loader_1                  | [2019-02-04 22:32:21,042: DEBUG/ForkPoolWorker-78] Updating origin_visit for origin 769 with status partial
swh-loader_1                  | [2019-02-04 22:32:21,052: DEBUG/ForkPoolWorker-78] Done updating origin_visit for origin 769 with status partial

Event Timeline

douardda triaged this task as Normal priority.Feb 5 2019, 9:47 AM
douardda created this task.
olasd added a subscriber: olasd.Feb 5 2019, 3:40 PM

That's a fairly large repo (as seen with how the content bundles get spread out to limit their size). It looks like it has some large directories (e.g. the .bugs directory looks like it has a lot of entries) so I'm not too surprised.

We should either tune the default "directory bundle size" to be smaller, or we should add a size heuristic as we do with contents.

gitlab-migration closed this task as Migrated.Jan 8 2023, 5:00 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal