Page MenuHomeSoftware Heritage

Smoke test ingestion of bitbucket repositories with latest loader mercurial
Closed, ResolvedPublic

Description

Smoke test it with remote and local (archived from BitBucket) repositories.
loader mercurial version v2.1

Event Timeline

ardumont renamed this task from Deploy swh.loader.mercurial 1.0 in production to Smoke test ingestion of bitbucket repositories with latest loader mercurial.Jul 29 2021, 6:36 PM
ardumont updated the task description. (Show Details)

Smoke test it with remote repositories.

Well, our check end to end loading the hgview [1] repository is now green again and no longer flip-floping so that takes care of it.

Some other remote repositories were tested as well without any issues [2]

[1] https://foss.heptapod.net/mercurial/hgview

[2] P1107

Smoke test it with local bitbucket repositories

that's next.

ardumont changed the task status from Open to Work in Progress.Jul 30 2021, 12:16 PM
ardumont moved this task from Backlog to in-progress on the System administration board.

Smoke test it with local bitbucket repositories

that's next.

It's working as expected [1].

This is ready to be deployed in production.

Here is their visit status result and analysis next to visit status that matters.

12:46:22 softwareheritage@belvedere:5432=> select o.url, ovs.date, ovs.status, ovs.snapshot from origin o inner join origin_visit_status ovs on o.id=ovs.origin where o.url in ('https://bitbucket.org/sunithacharan/pocketbook1', 'https://bit
bucket.org/alienonland/ownershipandriskweb', 'https://bitbucket.org/alienonland/ownershipandriskweb/wiki', 'https://bitbucket.org/rarchk/jsmn', 'https://bitbucket.org/rarchk/jsmn/wiki', 'https://bitbucket.org/k4nagatsuki/cw_wsn1example-hg'
, 'https://bitbucket.org/csimpson/pythononrails', 'https://bitbucket.org/tindelli/polytope') and status='full' order by origin desc,visit desc;
+------------------------------------------------------------+-------------------------------+--------+--------------------------------------------+
|                            url                             |             date              | status |                  snapshot                  |
+------------------------------------------------------------+-------------------------------+--------+--------------------------------------------+
| https://bitbucket.org/k4nagatsuki/cw_wsn1example-hg        | 2021-07-30 10:30:36.532165+00 | full   | (null)                                     |  2nd run uneventful
| https://bitbucket.org/k4nagatsuki/cw_wsn1example-hg        | 2021-07-30 10:29:41.131538+00 | full   | \x4e85afea09148d757966425d12026f99c40cd181 |  1st run evenfful

| https://bitbucket.org/rarchk/jsmn/wiki                     | 2021-07-30 10:30:30.356632+00 | full   | (null)                                     |  2nd run uneventful
| https://bitbucket.org/rarchk/jsmn/wiki                     | 2021-07-30 10:29:32.817241+00 | full   | \x9731b94489737bc6a459a591dedcbeb4e2adde1d |  1st run evenfful

| https://bitbucket.org/alienonland/ownershipandriskweb/wiki | 2021-07-30 10:30:26.201182+00 | full   | (null)                                     |  2nd run uneventful
| https://bitbucket.org/alienonland/ownershipandriskweb/wiki | 2021-07-30 10:29:27.681315+00 | full   | \x564df4553f129c77c7f683931929907266f9f072 |  2nd run uneventful

| https://bitbucket.org/tindelli/polytope                    | 2021-07-30 10:43:12.058411+00 | full   | (null)                                     |  3rd run uneventful (i had another run independently from the paste, overlapping range)
| https://bitbucket.org/tindelli/polytope                    | 2021-07-30 10:30:41.082185+00 | full   | (null)                                     |  2nd run uneventful
| https://bitbucket.org/tindelli/polytope                    | 2021-07-30 10:30:07.399909+00 | full   | \x4cfe3bcc2a05672715ca91b2634eaf7e923a5e71 |  1st run uneventful
| https://bitbucket.org/tindelli/polytope                    | 2019-09-23 16:33:01.485429+00 | full   | \x5d36d39afc5ff7685c8e3ad69229eb3a4add9e27 |  previous snapshot prior to the new release so snapshot was different because of the "default" extid version 0
| https://bitbucket.org/tindelli/polytope                    | 2019-09-19 13:29:25.313042+00 | full   | \x5d36d39afc5ff7685c8e3ad69229eb3a4add9e27 |  ...
| https://bitbucket.org/tindelli/polytope                    | 2019-09-11 13:28:52.070991+00 | full   | \x5d36d39afc5ff7685c8e3ad69229eb3a4add9e27 |
| https://bitbucket.org/tindelli/polytope                    | 2019-08-25 11:10:14.362887+00 | full   | \x5d36d39afc5ff7685c8e3ad69229eb3a4add9e27 |
| https://bitbucket.org/tindelli/polytope                    | 2019-07-06 03:26:54.404194+00 | full   | \x5d36d39afc5ff7685c8e3ad69229eb3a4add9e27 |
| https://bitbucket.org/rarchk/jsmn                          | 2021-07-30 10:30:28.378383+00 | full   | (null)                                     |
| https://bitbucket.org/rarchk/jsmn                          | 2021-07-30 10:29:30.613391+00 | full   | \xc07b7f6b2c84a59b2ed38f867831cec466db02cb |
| https://bitbucket.org/rarchk/jsmn                          | 2019-12-10 11:18:00.14157+00  | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-12-08 19:09:42.092199+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-12-07 05:46:00.746191+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-12-05 10:56:01.93435+00  | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-12-02 20:46:13.740019+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-12-01 08:25:17.093066+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-11-29 18:12:02.422935+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-11-28 00:27:55.639441+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-11-26 06:37:10.417357+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-11-24 18:23:50.054938+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-11-23 08:01:28.024955+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-11-21 15:57:13.062139+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-11-19 21:53:58.539205+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-11-18 03:55:22.896125+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-11-16 16:12:55.214658+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-11-15 00:06:22.283524+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-11-13 04:10:10.904844+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-11-11 11:37:13.496159+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-11-09 22:42:06.663271+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-11-08 06:31:49.302642+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-11-06 13:37:37.053864+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-11-04 21:04:16.742105+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-11-01 21:49:58.214468+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-10-29 16:40:36.82471+00  | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-10-17 07:24:10.446816+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-10-15 00:00:33.547458+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-10-12 07:08:42.233412+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-10-09 22:56:17.834132+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-10-07 21:31:22.227515+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-10-05 14:08:13.437975+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-10-03 06:17:19.175246+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-09-30 09:57:31.611388+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-09-26 00:01:59.286519+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-09-08 22:29:07.337007+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-08-23 12:34:55.080055+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |
| https://bitbucket.org/rarchk/jsmn                          | 2019-07-01 12:53:13.721234+00 | full   | \x61320da4a3980cca57f93a5106b70126078680e1 |

| https://bitbucket.org/sunithacharan/pocketbook1            | 2021-07-30 10:30:21.770151+00 | full   | (null)                                     | empty repository [2]
| https://bitbucket.org/sunithacharan/pocketbook1            | 2021-07-30 10:29:22.310871+00 | full   | (null)                                     |

| https://bitbucket.org/csimpson/pythononrails               | 2021-07-30 10:30:38.998849+00 | full   | (null)                                     |  same pattern as before...
| https://bitbucket.org/csimpson/pythononrails               | 2021-07-30 10:29:44.559665+00 | full   | \x042f695a1768bff19b8dfdd497ad4f6504e0fc6f |

| https://bitbucket.org/csimpson/pythononrails               | 2019-12-10 17:42:53.270323+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-12-08 23:35:46.74614+00  | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-12-07 10:55:28.856216+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-12-05 17:08:36.001726+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-12-03 02:07:46.069676+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-12-01 12:46:48.708118+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-11-29 23:23:17.158027+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-11-28 05:46:14.40683+00  | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-11-26 12:20:18.439113+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-11-24 22:17:57.685404+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-11-23 12:39:16.899187+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-11-21 21:10:34.829363+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-11-20 02:42:45.542426+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-11-18 08:05:14.350113+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-11-16 20:41:21.411309+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-11-15 04:40:16.795394+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-11-13 10:13:04.192558+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-11-11 16:21:47.134495+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-11-10 02:49:53.025858+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-11-08 11:29:23.697376+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-11-06 18:48:27.53713+00  | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-11-05 01:43:06.324477+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-11-02 06:24:22.09214+00  | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-10-30 01:47:24.573198+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-10-17 17:33:33.105511+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-10-15 06:26:47.392683+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-10-12 13:37:10.587124+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-10-10 04:59:49.034164+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-10-08 03:51:15.326945+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-10-05 22:06:30.09897+00  | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-10-03 23:20:14.156245+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-10-01 16:11:35.54927+00  | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-09-29 01:41:35.957731+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-09-24 23:01:59.883073+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-09-07 09:03:17.9129+00   | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-08-13 02:03:07.679798+00 | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |
| https://bitbucket.org/csimpson/pythononrails               | 2019-06-30 15:38:58.06108+00  | full   | \x34446e03d09755b3e48569a575f0eec7fe149abe |

| https://bitbucket.org/alienonland/ownershipandriskweb      | 2021-07-30 10:30:24.049421+00 | full   | (null)                                     |
| https://bitbucket.org/alienonland/ownershipandriskweb      | 2021-07-30 10:29:25.438277+00 | full   | \x4f429edb601ea9aa1ab1d48aab4e961c3eab159e |
| https://bitbucket.org/alienonland/ownershipandriskweb      | 2019-10-12 20:16:01.14027+00  | full   | \x661d74b97dfd5519d8840e3f1fdd42978db10a98 |
| https://bitbucket.org/alienonland/ownershipandriskweb      | 2019-10-10 10:40:15.296953+00 | full   | \x661d74b97dfd5519d8840e3f1fdd42978db10a98 |
| https://bitbucket.org/alienonland/ownershipandriskweb      | 2019-10-08 08:12:09.836132+00 | full   | \x661d74b97dfd5519d8840e3f1fdd42978db10a98 |
| https://bitbucket.org/alienonland/ownershipandriskweb      | 2019-10-06 03:57:51.121655+00 | full   | \x661d74b97dfd5519d8840e3f1fdd42978db10a98 |
| https://bitbucket.org/alienonland/ownershipandriskweb      | 2019-10-04 04:05:41.913737+00 | full   | \x661d74b97dfd5519d8840e3f1fdd42978db10a98 |
| https://bitbucket.org/alienonland/ownershipandriskweb      | 2019-10-01 19:54:23.079177+00 | full   | \x661d74b97dfd5519d8840e3f1fdd42978db10a98 |
| https://bitbucket.org/alienonland/ownershipandriskweb      | 2019-09-30 04:12:31.91622+00  | full   | \x661d74b97dfd5519d8840e3f1fdd42978db10a98 |
| https://bitbucket.org/alienonland/ownershipandriskweb      | 2019-09-28 17:51:05.256742+00 | full   | \x661d74b97dfd5519d8840e3f1fdd42978db10a98 |
| https://bitbucket.org/alienonland/ownershipandriskweb      | 2019-09-27 15:44:13.862242+00 | full   | \x661d74b97dfd5519d8840e3f1fdd42978db10a98 |
| https://bitbucket.org/alienonland/ownershipandriskweb      | 2019-09-26 03:56:42.959891+00 | full   | \x661d74b97dfd5519d8840e3f1fdd42978db10a98 |
| https://bitbucket.org/alienonland/ownershipandriskweb      | 2019-09-24 00:32:53.056976+00 | full   | \x661d74b97dfd5519d8840e3f1fdd42978db10a98 |
| https://bitbucket.org/alienonland/ownershipandriskweb      | 2019-09-22 20:49:03.454986+00 | full   | \x661d74b97dfd5519d8840e3f1fdd42978db10a98 |
| https://bitbucket.org/alienonland/ownershipandriskweb      | 2019-09-20 19:37:44.966056+00 | full   | \x661d74b97dfd5519d8840e3f1fdd42978db10a98 |
| https://bitbucket.org/alienonland/ownershipandriskweb      | 2019-09-08 15:52:01.564134+00 | full   | \x661d74b97dfd5519d8840e3f1fdd42978db10a98 |
| https://bitbucket.org/alienonland/ownershipandriskweb      | 2019-08-31 01:27:53.877789+00 | full   | \x661d74b97dfd5519d8840e3f1fdd42978db10a98 |
| https://bitbucket.org/alienonland/ownershipandriskweb      | 2019-07-30 01:52:57.936951+00 | full   | \x661d74b97dfd5519d8840e3f1fdd42978db10a98 |
| https://bitbucket.org/alienonland/ownershipandriskweb      | 2019-06-28 01:51:50.188403+00 | full   | \x661d74b97dfd5519d8840e3f1fdd42978db10a98 |
+------------------------------------------------------------+-------------------------------+--------+--------------------------------------------+
(112 rows)

[1] P1108

[2] Emtpy repository

ardumont@worker16:/srv/storage/space/mirrors/boatbucket$ base_dir=/srv/storage/space/mirrors/boatbucket
ardumont@worker16:/srv/storage/space/mirrors/boatbucket$ grep sunithacharan/pocketbook1 mapping-to-repos.txt
refugees/data/f6/f6f2cb2b-7bb5-482a-9974-c86fc28a1ef8/main-repo.mercurial https://bitbucket.org/sunithacharan/pocketbook1
ardumont@worker16:/srv/storage/space/mirrors/boatbucket$ repo=$(grep sunithacharan/pocketbook1 mapping-to-repos.txt | awk '{print $1}')
ardumont@worker16:/tmp% hg clone $base_dir/$repo
not trusting file /srv/storage/space/mirrors/boatbucket/refugees/data/f6/f6f2cb2b-7bb5-482a-9974-c86fc28a1ef8/main-repo.mercurial/.hg/hgrc from untrusted user 1002, group aeviso
destination directory: main-repo.mercurial
no changes found
updating to branch default
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
swhworker@worker0:/tmp$ cd main-repo.mercurial/
swhworker@worker0:/tmp/main-repo.mercurial$ hg log  -> nothing to show here
ardumont moved this task from in-progress to done on the System administration board.