Page MenuHomeSoftware Heritage
Create Task
Maniphest T1875

staging infra: Reproduce existing production setup in a compact way
Closed, MigratedEdits Locked
Actions

Description

Dependency order to follow

  • gateway (routing)
  • swh-storage (db, swh-storage service)
  • swh-objstorage (disks, swh-objstorage service)
  • swh-indexer storage (db, service)
  • swh-web
  • swh-scheduler (db, rabbitmq, swh-scheduler services)
  • swh-deposit (db, service)
  • 1 worker with at least 1 loader (worker0 & worker1 with loader-git)
  • swh-vault
  • update workers with checker/loader deposit
  • update workers with 1 lister (forge=gitlab, instance=inria)
  • update workers with 1 indexer
  • Make icinga checks green on the expected origins (parmap, cpython, etc...)

Note:

  • Should be a matter of creating the right branch ('staging' for example) in swh-site's repository
  • calling the right environment when `puppet agent --test --noop --environment='staging'
  • production code should only be modified if there are issues identified (there was ;)
  • in the end the staging branch should be merged into production (without any impact on production, that's somehow has been my implicit plan)

Plan:
From provisioning node (creation in our hypervisor) to delegating puppet for the configuration/installation services
For some of our services, this is ambitious:

  • postgres: puppetized the db creation and user done
  • postgres: bootstrap db schema (-> this one needs work upfront in our different services to have a common use interface) [1]
  • rabbitmq: server installation with users setup
  • rabbitmq: improve the configuration option to be in sync with our current instance (P493)

Tests:

  • from webapp: save code now -> browsing ok
  • push a deposit via deposit client cli -> browsing in webapp ok
  • from webapp: request vault cooking -> download in webapp ok
  • listing one forge and see new origins from that forge [2]
  • origin-content-metadata running

[1] I consider this out of scope (i need to draw a line somewhere ;)
Also we as a team started this, that needs to be finalized IIRC.

[2] https://webapp.internal.staging.swh.network/browse/search/?q=gitlab.inria.fr&with_visit&with_content

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ardumont mentioned this in D1807: staging: Add db0 node.Aug 2 2019, 7:57 PM
ardumont mentioned this in rSPRE3ff4f0b6fd27: staging: Modularize node creation.Aug 5 2019, 10:41 AM
ardumont mentioned this in rSPRE6bb4ddd72455: staging: Add db0 node.
ardumont mentioned this in rSPSITE74d46561849f: gateway.internal.staging: Add networks info up to the louvre route.Aug 5 2019, 5:20 PM
ardumont mentioned this in rSPSITE91a9647f5405: site.pp: Reference gateway.internal.staging's role.
ardumont mentioned this in rSPSITEe75d23f6534e: gateway.internal.staging: Add networks info up to the louvre route.Aug 5 2019, 6:03 PM
ardumont mentioned this in rSPSITE9d7d5697168d: site.pp: Reference gateway.internal.staging's role.
ardumont mentioned this in rSPSITE4516b6b558a9: gateway.internal.staging: Add networks info up to the louvre route.Aug 6 2019, 11:29 AM
ardumont mentioned this in rSPSITE98bf9590dfc5: site.pp: Reference gateway.internal.staging's role.
ardumont mentioned this in rSPSITEcbbe50ab64bd: gateway.internal.staging: Add networks info up to the louvre route.Aug 6 2019, 2:35 PM
ardumont mentioned this in rSPSITE3a79a4bfec14: site.pp: Reference gateway.internal.staging's role.
ardumont mentioned this in rSPSITE4fa136f34fe5: gateway.internal.staging: Add networks info up to the louvre route.Aug 6 2019, 9:41 PM
ardumont mentioned this in rSPSITEbdc25d96bde1: site.pp: Reference gateway.internal.staging's role.
ardumont mentioned this in rSPSITE2e22e6f3368d: gateway.internal.staging: Add networks info up to the louvre route.Aug 7 2019, 4:37 PM
ardumont mentioned this in rSPSITE1fed76536252: site.pp: Reference gateway.internal.staging's role.
ardumont mentioned this in rSPSITE47780a6e3b44: gateway.internal.staging: Add networks info up to the louvre route.Aug 7 2019, 4:43 PM
ardumont mentioned this in rSPSITE1161dd952677: site.pp: Reference gateway.internal.staging's role.
ardumont updated the task description. (Show Details)Aug 7 2019, 4:59 PM
ardumont mentioned this in rSPSITE4935624bce51: dns: Add last staging dns records.Aug 7 2019, 6:36 PM
ardumont mentioned this in rSPSITE4552bca31ecf: gateway.internal.staging: Add networks info up to the louvre route.
ardumont mentioned this in rSPSITE2fa06675a306: site.pp: Reference gateway.internal.staging's role.
ardumont updated the task description. (Show Details)Aug 7 2019, 6:41 PM
ardumont updated the task description. (Show Details)Aug 8 2019, 4:49 AM
ardumont mentioned this in rSPSITE1d05d781bc1e: staging: Install webapp node's role.Aug 8 2019, 6:45 AM
ardumont mentioned this in rSPSITEb0d9e37db421: staging: Install webapp node's role.
ardumont mentioned this in rSPSITE02dd154c383a: staging: Install webapp node's role.Aug 8 2019, 9:21 AM
ardumont mentioned this in rSPSITE16fa0f055a14: gateway.internal.staging: Add networks info up to the louvre route.Aug 8 2019, 9:29 AM
ardumont mentioned this in rSPSITEafa1b99fede8: site.pp: Reference gateway.internal.staging's role.
ardumont mentioned this in rSPSITEcedbaee520b9: staging: Install webapp node's role.
ardumont updated the task description. (Show Details)Aug 8 2019, 3:15 PM
ardumont mentioned this in rSPSITE32f73b5f7fe0: gateway.internal.staging: Add networks info up to the louvre route.Aug 8 2019, 4:04 PM
ardumont mentioned this in rSPSITE4143abbc68dd: site.pp: Reference gateway.internal.staging's role.
ardumont mentioned this in rSPSITE6919e83046d1: staging: Install webapp node's role.
ardumont updated the task description. (Show Details)Aug 8 2019, 5:10 PM
ardumont mentioned this in rSPSITEf651ca7abcd7: gateway.internal.staging: Add networks info up to the louvre route.Aug 8 2019, 5:44 PM
ardumont mentioned this in rSPSITEcf99f9566cb4: site.pp: Reference gateway.internal.staging's role.
ardumont mentioned this in rSPSITEc7d9ada09990: staging: Install webapp node's role.
ardumont updated the task description. (Show Details)Aug 8 2019, 7:28 PM
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)Aug 8 2019, 7:47 PM
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)Aug 8 2019, 7:50 PM
ardumont updated the task description. (Show Details)
ardumont mentioned this in rSPSITE0796e1b95f9d: gateway.internal.staging: Add networks info up to the louvre route.Aug 8 2019, 9:04 PM
ardumont mentioned this in rSPSITE7b3bc38295d7: site.pp: Reference gateway.internal.staging's role.
ardumont mentioned this in rSPSITEcf342bbbb925: staging: Install webapp node's role.
ardumont mentioned this in rSPSITEd961819d8c36: gateway.internal.staging: Add networks info up to the louvre route.Aug 8 2019, 9:07 PM
ardumont mentioned this in rSPSITEc128e8324518: site.pp: Reference gateway.internal.staging's role.
ardumont mentioned this in rSPSITE24e74d106a64: staging: Install webapp node's role.
ardumont updated the task description. (Show Details)Aug 8 2019, 10:17 PM
ardumont claimed this task.Aug 8 2019, 11:52 PM
ardumont updated the task description. (Show Details)
ardumont mentioned this in rSPSITE3fcaa352548d: deposit: Allow loader/checker deposit configuration reuse.Aug 29 2019, 1:58 PM
ardumont mentioned this in rSPSITE37b7717f6aee: staging: Add checker and deposit workers.
ardumont mentioned this in rSPSITEc7711b75ac98: deposit: Allow loader/checker deposit configuration reuse.Aug 29 2019, 2:21 PM
ardumont mentioned this in rSPSITEa2db672ae9c5: staging: Add checker and deposit workers.
ardumont mentioned this in rSPSITE2b8da6a447a4: worker/*_deposit: Fix missing base_deposit.
ardumont updated the task description. (Show Details)Aug 29 2019, 2:27 PM
ardumont updated the task description. (Show Details)Aug 29 2019, 3:57 PM
ardumont updated the task description. (Show Details)Aug 29 2019, 4:05 PM
ardumont added a comment.Aug 30 2019, 4:20 PM

Progress on the deposit part:

$ swh deposit upload --url http://deposit.internal.staging.swh.network \
    --username hal-preprod \
    --password <pass> \
    --archive jesuisgpl.tgz \
    --name jesuisgpl \
    --author zack
INFO:swh.deposit.cli.client:{'deposit_id': '2', 'deposit_status': 'deposited', 'deposit_status_detail': None, 'deposit_date': 'Aug. 30, 2019, 2:08 p.m.'}

# deposit-id 1 fails because the metadata we incomplete 
$ swh deposit status --url http://deposit.internal.staging.swh.network \
    --username hal-preprod \
    --password <pass> \
    --deposit-id 1
INFO:swh.deposit.cli.client:{'deposit_id': '1', 'deposit_status': 'rejected', 'deposit_status_detail': '- Mandatory fields are missing (author)\n- Mandatory alternate fields are missing (name or title)', 'deposit_swh_id': None, 'deposit_swh_id_context': None, 'deposit_swh_anchor_id': None, 'deposit_swh_anchor_id_context': None, 'deposit_external_id': '8740a2d3-d11c-4daf-8968-a6554dacbbc2'}

# deposit-id 2 is complete (so should work)
$ swh deposit status --url http://deposit.internal.staging.swh.network \
    --username hal-preprod \
    --password <pass> \
    --deposit-id 2
INFO:swh.deposit.cli.client:{'deposit_id': '2', 'deposit_status': 'verified', 'deposit_status_detail': 'Deposit is fully received, checked, and ready for loading', 'deposit_swh_id': None, 'deposit_swh_id_context': None, 'deposit_swh_anchor_id': None, 'deposit_swh_anchor_id_context': None, 'deposit_external_id': '8a60afdb-f424-46ec-abc6-49a03ccefce3'}

# but it fails for storage migration reasons
$ swh deposit status --url http://deposit.internal.staging.swh.network \
    --username hal-preprod \
    --password <pass> \
    --deposit-id 2
INFO:swh.deposit.cli.client:{'deposit_id': '2', 'deposit_status': 'failed', 'deposit_status_detail': 'The deposit loading into the Software Heritage archive failed', 'deposit_swh_id': None, 'deposit_swh_id_context': None, 'deposit_swh_anchor_id': None, 'deposit_swh_anchor_id_context': None, 'deposit_external_id': '8a60afdb-f424-46ec-abc6-49a03ccefce3'}

Now the deposit loading fails because we have some changes in the storage layer (origin-id are no longer ids but url):

Aug 30 14:08:34 worker1 python3[11461]: [2019-08-30 14:08:34,917: ERROR/ForkPoolWorker-1] Loading failure, updating to `partial` status
                                        Traceback (most recent call last):
                                          File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 876, in load
                                            self.store_metadata()
                                          File "/usr/lib/python3/dist-packages/swh/deposit/loader/loader.py", line 93, in store_metadata
                                            tool_id, metadata)
                                          File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f
                                            return Retrying(*dargs, **dkw).call(f, *args, **kw)
                                          File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call
                                            return attempt.get(self._wrap_exception)
                                          File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get
                                            six.reraise(self.value[0], self.value[1], self.value[2])
                                          File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
                                            raise value
                                          File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call
                                            attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
                                          File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 352, in send_origin_metadata
                                            self.origin['url'], visit_date, provider_id, tool_id, metadata)
                                          File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 241, in origin_metadata_add
                                            'metadata': metadata})
                                          File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 205, in post
                                            return self._decode_response(response)
                                          File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 237, in _decode_response
                                            raise pickle.loads(decode_response(response))
                                        psycopg2.DataError: invalid input syntax for integer: "https://inria.halpreprod.archives-ouvertes.fr/8a60afdb-f424-46ec-abc6-49a03ccefce3"
                                        LINE 2: ...          provider_id, tool_id, metadata) values ('https://i...
ardumont added a subscriber: vlorentz.Aug 30 2019, 5:47 PM

@vlorentz (stacktrace) ^

I think we missed the storage origin_metadata_add endpoint during the migration out of the origin id (both in-memory and pg storages).
The loader-core (0.0.44) has been changed to provide the url instead of the id but the storage endpoint still expects the origin-id.
That's my understanding, do you concur?

TIA

vlorentz added a comment.Sep 2 2019, 11:18 AM

Indeed, I did not change it because I expected to do it at the same time as a refactoring of origin_metadata_* (aka implementing D1614).
I'll fix it asap

vlorentz added a comment.Sep 2 2019, 11:28 AM

D1935

ardumont added a comment.Sep 5 2019, 10:34 AM

Finally, after updating plenty of our modules (model, storage, ...), loader-deposit is fixed, so i can finally cross that box! \m/

Sep 05 08:31:03 worker1 python3[11272]: [2019-09-05 08:31:03,260: INFO/ForkPoolWorker-1] Task swh.deposit.loader.tasks.LoadDepositArchiveTsk[da9f066e-4c44-4053-81a1-e38db35d6149] succeeded in 2.102082097902894s: {'status': 'eventful'}

Cheers,

ardumont updated the task description. (Show Details)Sep 5 2019, 10:39 AM
ardumont created subtask T1995: swh-lister: Unstuck debian build failures.Sep 10 2019, 1:33 PM
olasd closed subtask T1995: swh-lister: Unstuck debian build failures as Resolved.Sep 10 2019, 2:51 PM
ardumont updated the task description. (Show Details)Sep 11 2019, 3:18 PM
ardumont mentioned this in rSPSITE430ce3c4a92c: staging: Add journal0 node.Sep 13 2019, 2:17 PM
ardumont mentioned this in rSPRE8760b5d10019: staging: Add journal0 node.Sep 13 2019, 2:34 PM
ardumont mentioned this in rSPSITE8631d5885abb: staging: Make storage writes to journal0's broker.Sep 13 2019, 2:40 PM
ardumont mentioned this in rSPSITE64aa3c348b8c: staging: Set zookeeper instance on journal0.
ardumont mentioned this in rSPSITEcdd8fb222e92: staging: Fix journal broker reference.Sep 13 2019, 2:52 PM
ardumont mentioned this in rSPSITE624b41c72c64: staging: Add missing kafka::cluster setup.
ardumont mentioned this in rSPSITE67237f1efc00: staging: Add missing kafka::cluster setup.Sep 13 2019, 3:02 PM
ardumont mentioned this in rSPSITE31df4b0f4a9d: staging: Deploy indexer origin intrinsic metadata.Sep 13 2019, 3:32 PM
ardumont mentioned this in rSPSITEc25b001b8862: staging: Add journal0 node.Oct 9 2019, 10:54 PM
ardumont mentioned this in rSPSITEe25f76602fbc: staging: Make storage writes to journal0's broker.
ardumont mentioned this in rSPSITEd594c1a1049a: staging: Set zookeeper instance on journal0.
ardumont mentioned this in rSPSITE15ec1eff1887: staging: Fix journal broker reference.
ardumont mentioned this in rSPSITE41214d7f20b2: staging: Add missing kafka::cluster setup.
ardumont mentioned this in rSPSITEf5f76a8ccd0f: staging: Deploy indexer origin intrinsic metadata.
ardumont added a comment.Oct 22 2019, 9:10 AM

Stand-by as package-loader priority T1389 took over.
Staging infra will be used when package-loader land (which will need some work debian packaging + configuration updates).

And then starting back the work to finalize it (one indexer which pulls a kafka).

vlorentz added a project: Staging environment.Jan 22 2020, 4:28 PM
ardumont closed this task as Resolved.Apr 24 2020, 3:38 PM
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)

It's fairly complete already.

For the remaining indexer which pulls a kafka, we can always improve on this later.

gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 5:56 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration changed the status of subtask T1995: swh-lister: Unstuck debian build failures from Resolved to Migrated.Oct 19 2022, 5:56 PM
About · Developer information · Legal