Page MenuHomeSoftware Heritage

Normalize storage and objstorage configuration schema
Closed, ResolvedPublic

Description

For the first time, we need to be able to deploy a mixed swh-storage. So far, we had only on swh-storage (swh's db + local objstorage as pathslicing).

The use case is the loader-svn test working with a dedicated db and our standard objstorage (accessed remotely so not a pathslicing one).
As of now, the objstorage for our storage is hard-coded to a pathslicing.
So, impossible in its current state to setup.

Also, it turns out that our current objstorage and storage configuration are not yet unified.
This, in effect, limits the previous steps (we cannot express a different objstorage with the current storage configuration scheme).
So we need to also overcome this.

Related Objects

Mentioned In
R65:b84a083330ae: Update configuration schema
rSPSITEc542f00299ee: swh::deploy::webapp: Fix removing one variable too much
rSPSITEba36d3b1c92e: swh::deploy::webapp: Fix missing configuration inlining
rSPSITE12371cae6aca: swh::deploy::webapp: Update configuration file from ini to yml
rSPSITE933cbffea99c: swh::deploy::storage: Update conf file from ini to yml
rDLDSVNf524040db91f: d/control: Bump to latest versions
rDLDSVN028952d55b1a: Fix generator issue when using local storage
rDLDBASE8f6d0f7f0d6c: Fix missing default configuration adaptation
rDOBJSf8a15537d2bd: Unify argument 'url' name for objstorage
rSPPROFc542f00299ee: swh::deploy::webapp: Fix removing one variable too much
rSPPROFba36d3b1c92e: swh::deploy::webapp: Fix missing configuration inlining
rSPSITE8b68ab25a6d2: data/defaults.yaml: Update configuration data for webapp
rSPPROF12371cae6aca: swh::deploy::webapp: Update configuration file from ini to yml
rDWAPPSb84a083330ae: Update configuration schema
rSPSITEbbee096ff3f4: data/default.yaml: Update configuration for storage
rSPPROF933cbffea99c: swh::deploy::storage: Update conf file from ini to yml
rSPSITE020b7d511350: worker01*: Deploy loader-svn testing instance
rSPSITE25f212addfca: data/defaults: Update storage configuration in multiple tools
rDLSb217f55cfe00: Update storage configuration reading
rDLDTAR08e7f00c16be: d/control: Bump dependency to latest swh-loader-core
rDLDDIRd45330a5d775: d/control: Bump dependency to latest swh-loader-core
rDLDBASEc211f04479f9: d/control: Bump to latest version on swh-storage
rDLDBASE4ebe84c31d54: Unify storage instantiation configuration
rDLDGa57dc1e999f2: d/control: Bump to latest version on swh-storage
rDLDG82eeb4008e10: Unify storage instantiation configuration
rDSTOd5f4640d38bf: Unify objstorage and storage configuration
rDSTOafbdb14b001f: Adapt storage's objstorage parameter as a setup property

Event Timeline

Deployment finished by olasd.

Well, finished up to the azure vms (i forgot he mentioned on irc that it was up to that part).

The azure vms are not solid enough. We lose contact with them on an unknown frequency basis.
I don't know why that is. My only workaround is to leverage az tool to reboot them.

for w in {1..8}; do echo "restart worker0$w" ; az vm restart --name worker0$w-euwest -g euwest-workers; done

(well only with the one not responding).

They were not supposed to do anything.
So i rebooted the ones lost (4, 6, 7, 8).
And then passed the puppet manifest updates.

So now, the deployment is actually finished.