Page MenuHomeSoftware Heritage
Paste P1189

draft: ryout runs on loader git improvments
ActivePublic
Actions

Authored by ardumont on Sep 30 2021, 6:46 PM.
* cozy/cozy-stack
** docker-dev with override
#+begin_src sql
18:16:09 swh-storage@localhost:5434=# select now(), * from origin o inner join origin_visit_status ovs on o.id=ovs.origin where o.url = 'https://github.com/cozy/cozy-stack' and ovs.type='git' order by date desc limit 4;
+-------------------------------+----+------------------------------------+--------+-------+-------------------------------+------+---------+----------+--------------------------------------------+
| now | id | url | origin | visit | date | type | status | metadata | snapshot |
+-------------------------------+----+------------------------------------+--------+-------+-------------------------------+------+---------+----------+--------------------------------------------+
| 2021-09-30 16:16:24.727779+00 | 1 | https://github.com/cozy/cozy-stack | 1 | 2 | 2021-09-30 16:16:00.604747+00 | git | full | (null) | \x68dba01dbcf0afdb6130d183d2e8d3e420f15b1b |
| 2021-09-30 16:16:24.727779+00 | 1 | https://github.com/cozy/cozy-stack | 1 | 2 | 2021-09-30 16:15:59.32989+00 | git | created | (null) | (null) |
+-------------------------------+----+------------------------------------+--------+-------+-------------------------------+------+---------+----------+--------------------------------------------+
(4 rows)
Time: 0.612 ms
#+end_src
logs:
#+begin_src sh
$ swh-doco exec swh-loader /bin/bash
+ cd /home/tony/work/inria/repo/swh/swh-environment/docker
+ docker-compose -f docker-compose.yml -f docker-compose.override.yml exec swh-loader /bin/bash
swh@1da9a15ff22e:/$ /usr/bin/time -v swh loader run git https://github.com/cozy/cozy-stack
INFO:swh.loader.git.loader.GitLoader:Load origin 'https://github.com/cozy/cozy-stack' with type 'git'
INFO:swh.loader.git.loader:##### transport url: git://github.com/cozy/cozy-stack
INFO:swh.loader.git.loader:##### limit the refs to: 2000
INFO:swh.loader.git.loader:##### wanted_refs: 3117
INFO:swh.loader.git.loader:##### Fetch range [0 : 2000]
INFO:swh.loader.git.loader:##### nb of fetched refs: 2000
INFO:swh.loader.git.loader:##### nb of fetched refs: 2000
Enumerating objects: 66765, done.
Counting objects: 100% (3877/3877), done.
Compressing objects: 100% (1348/1348), done.
Total 66765 (delta 2588), reused 3624 (delta 2473), pack-reused 62888
INFO:swh.loader.git.loader:##### Fetched pack size: 181200078
INFO:swh.loader.git.loader:#### remote_refs: 3197
INFO:swh.loader.git.loader:#### filtered_remote_refs: 3127
INFO:swh.loader.git.loader:#### symbolic_refs: 1
INFO:swh.loader.git.loader:#### filtered_symbolic_refs: 1
INFO:swh.loader.git.loader:Listed 3126 refs for repo https://github.com/cozy/cozy-stack
INFO:swh.loader.git.loader:##### transport url: git://github.com/cozy/cozy-stack
INFO:swh.loader.git.loader:##### limit the refs to: 2000
INFO:swh.loader.git.loader:##### wanted_refs: 3117
INFO:swh.loader.git.loader:##### Fetch range [2000 : 3117]
INFO:swh.loader.git.loader:##### nb of fetched refs: 1117
Enumerating objects: 64307, done.
Counting objects: 100% (3740/3740), done.
Compressing objects: 100% (1268/1268), done.
Total 64307 (delta 2523), reused 3525 (delta 2418), pack-reused 60567
INFO:swh.loader.git.loader:##### Fetched pack size: 149331238
INFO:swh.loader.git.loader:#### remote_refs: 3197
INFO:swh.loader.git.loader:#### filtered_remote_refs: 3127
INFO:swh.loader.git.loader:#### symbolic_refs: 1
INFO:swh.loader.git.loader:#### filtered_symbolic_refs: 1
INFO:swh.loader.git.loader:Listed 3126 refs for repo https://github.com/cozy/cozy-stack
{'status': 'eventful'}
Command being timed: "swh loader run git https://github.com/cozy/cozy-stack"
User time (seconds): 119.58
System time (seconds): 4.29
Percent of CPU this job got: 28%
Elapsed (wall clock) time (h:mm:ss or m:ss): 7:08.98
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2784576
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 625433
Voluntary context switches: 36303
Involuntary context switches: 1842
Swaps: 0
File system inputs: 0
File system outputs: 645584
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
#+end_src
** docker without override
#+begin_src sh
18:24:44 swh-storage@localhost:5434=# select now(), * from origin o inner join origin_visit_status ovs on o.id=ovs.origin where o.url = 'https://github.com/cozy/cozy-stack' and ovs.type='git' order by date desc limit 2;
+-------------------------------+----+------------------------------------+--------+-------+-------------------------------+------+---------+----------+--------------------------------------------+
| now | id | url | origin | visit | date | type | status | metadata | snapshot |
+-------------------------------+----+------------------------------------+--------+-------+-------------------------------+------+---------+----------+--------------------------------------------+
| 2021-09-30 16:24:45.270929+00 | 1 | https://github.com/cozy/cozy-stack | 1 | 1 | 2021-09-30 16:24:35.706488+00 | git | full | (null) | \x68dba01dbcf0afdb6130d183d2e8d3e420f15b1b |
| 2021-09-30 16:24:45.270929+00 | 1 | https://github.com/cozy/cozy-stack | 1 | 1 | 2021-09-30 16:18:07.55629+00 | git | created | (null) | (null) |
+-------------------------------+----+------------------------------------+--------+-------+-------------------------------+------+---------+----------+--------------------------------------------+
(2 rows)
Time: 2.139 ms
#+end_src
logs:
#+begin_src sh
swh-doco exec swh-loader /bin/bash
+ cd /home/tony/work/inria/repo/swh/swh-environment/docker
+ docker-compose -f docker-compose.yml -f docker-compose.override.yml exec swh-loader /bin/bash
swh@d902ac731424:/$ swh^C
swh@d902ac731424:/$ /usr/bin/time -v swh loader run git https://github.com/cozy/cozy-stack
INFO:swh.loader.git.loader.GitLoader:Load origin 'https://github.com/cozy/cozy-stack' with type 'git'
Enumerating objects: 68843, done.
Counting objects: 100% (4028/4028), done.
Compressing objects: 100% (1412/1412), done.
Total 68843 (delta 2677), reused 3773 (delta 2560), pack-reused 64815
INFO:swh.loader.git.loader.GitLoader:Listed 3127 refs for repo https://github.com/cozy/cozy-stack
{'status': 'eventful'}
Command being timed: "swh loader run git https://github.com/cozy/cozy-stack"
User time (seconds): 104.66
System time (seconds): 4.96
Percent of CPU this job got: 28%
Elapsed (wall clock) time (h:mm:ss or m:ss): 6:31.43
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 766780
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 528418
Voluntary context switches: 18964
Involuntary context switches: 1679
Swaps: 0
File system inputs: 0
File system outputs: 389928
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
swh@a6ddb051b815:/$ /usr/bin/time -v swh loader run git https://github.com/cozy/cozy-stack
INFO:swh.loader.git.loader.GitLoader:Load origin 'https://github.com/cozy/cozy-stack' with type 'git'
Enumerating objects: 68841, done.
Counting objects: 100% (4026/4026), done.
Compressing objects: 100% (1412/1412), done.
Total 68841 (delta 2677), reused 3772 (delta 2558), pack-reused 64815
INFO:swh.loader.git.loader.GitLoader:Listed 3128 refs for repo https://github.com/cozy/cozy-stack
{'status': 'eventful'}
Command being timed: "swh loader run git https://github.com/cozy/cozy-stack"
User time (seconds): 104.65
System time (seconds): 5.08
Percent of CPU this job got: 27%
Elapsed (wall clock) time (h:mm:ss or m:ss): 6:32.20
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 775628
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 560836
Voluntary context switches: 20797
Involuntary context switches: 1396
Swaps: 0
File system inputs: 0
File system outputs: 389992
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
#+end_src
* Run simple
** staging
- Run on staging
logs:
#+begin_src sh
swhworker@worker1:~$ time SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_git.yml swh loader run git https://github.coswhworker@worker1:~$ time SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_git.yml swh loader run git https://github.com/cozy/cozy-stack
INFO:swh.loader.git.loader.GitLoader:Load origin 'https://github.com/cozy/cozy-stack' with type 'git'
Enumerating objects: 68841, done.
Counting objects: 100% (4026/4026), done.
Compressing objects: 100% (1411/1411), done.
Total 68841 (delta 2676), reused 3773 (delta 2559), pack-reused 64815
INFO:swh.loader.git.loader.GitLoader:Listed 3128 refs for repo https://github.com/cozy/cozy-stack
{'status': 'eventful'}
real 58m26.944s
user 1m53.073s
sys 0m8.366s
#+end_src
sql:
#+begin_src sql
18:45:04 swh@db1:5432=> select now(), * from origin o inner join origin_visit_status ovs on o.id=ovs.origin where o.url = 'https://github.com/cozy/cozy-stack' and ovs.type='git' order by date desc limit 2;
+-------------------------------+---------+------------------------------------+---------+-------+-------------------------------+---------+----------+--------------------------------------------+------+
| now | id | url | origin | visit | date | status | metadata | snapshot | type |
+-------------------------------+---------+------------------------------------+---------+-------+-------------------------------+---------+----------+--------------------------------------------+------+
| 2021-09-30 16:45:17.564742+00 | 1411008 | https://github.com/cozy/cozy-stack | 1411008 | 1 | 2021-09-30 16:43:52.188045+00 | full | (null) | \x3e22041530f79efd27359cf8c25fc6f5b014abec | git |
| 2021-09-30 16:45:17.564742+00 | 1411008 | https://github.com/cozy/cozy-stack | 1411008 | 1 | 2021-09-30 15:45:32.462213+00 | created | (null) | (null) | git |
+-------------------------------+---------+------------------------------------+---------+-------+-------------------------------+---------+----------+--------------------------------------------+------+
(2 rows)
Time: 19.525 ms
#+end_src
* simple dot-files
** staging
#+begin_src sql
16:06:33 swh@db1:5432=> select now(), * from origin o inner join origin_visit_status ovs on o.id=ovs.origin where o.url = 'https://github.com/ardumont/dot-files' and ovs.type='git' order by date desc limit 2;
+-------------------------------+---------+---------------------------------------+---------+-------+-------------------------------+---------+----------+--------------------------------------------+------+
| now | id | url | origin | visit | date | status | metadata | snapshot | type |
+-------------------------------+---------+---------------------------------------+---------+-------+-------------------------------+---------+----------+--------------------------------------------+------+
| 2021-09-30 14:06:34.712062+00 | 1410211 | https://github.com/ardumont/dot-files | 1410211 | 1 | 2021-09-30 14:05:23.908049+00 | full | (null) | \x8172027a50ac8d92844a7c462c4913cfdce56b92 | git |
| 2021-09-30 14:06:34.712062+00 | 1410211 | https://github.com/ardumont/dot-files | 1410211 | 1 | 2021-09-30 13:59:09.933579+00 | created | (null) | (null) | git |
+-------------------------------+---------+---------------------------------------+---------+-------+-------------------------------+---------+----------+--------------------------------------------+------+
(2 rows)
Time: 5.185 ms
#+end_src
** docker
#+begin_src sql
16:07:34 swh-storage@localhost:5434=# select now(), * from origin o inner join origin_visit_status ovs on o.id=ovs.origin where o.url = 'https://github.com/ardumont/dot-files' and ovs.type='git' order by date desc limit 2;
+-------------------------------+----+---------------------------------------+--------+-------+-------------------------------+------+---------+----------+--------------------------------------------+
| now | id | url | origin | visit | date | type | status | metadata | snapshot |
+-------------------------------+----+---------------------------------------+--------+-------+-------------------------------+------+---------+----------+--------------------------------------------+
| 2021-09-30 14:07:36.528774+00 | 1 | https://github.com/ardumont/dot-files | 1 | 6 | 2021-09-30 13:59:09.784122+00 | git | full | (null) | \x8172027a50ac8d92844a7c462c4913cfdce56b92 |
| 2021-09-30 14:07:36.528774+00 | 1 | https://github.com/ardumont/dot-files | 1 | 6 | 2021-09-30 13:58:34.500716+00 | git | created | (null) | (null) |
+-------------------------------+----+---------------------------------------+--------+-------+-------------------------------+------+---------+----------+--------------------------------------------+
(2 rows)
#+end_src
About · Developer information · Legal