A lot of service configurations need to integrate credentials, which are usually sensitive. We currently just dump all the sensitive values in the main yaml configuration file, and we manually try to ensure that these files aren't readable for the world (with patchy results).

We should consider having a mechanism to override config file entries through a reference to a "secrets files" which can be generated on the fly by docker secrets or other config management tools.

If we implement such a mechanism, managing the main config file can be much simpler, as the sensitive data will be clearly separated.

TBD:

• review existing implementations for secret management to ensure a file-based mechanism will be useful (e.g. compatibility with docker secrets, which was the trigger for writing this down in the first place, but also review what's possible with kubernetes and other things)
• define syntax for the "external" config variables
• decide in what swh.core.config function the susbstitution of external config variables should happen
• decide whether the substitution should be opt-in for certain keys, or just done on the whole config data structure recursively