Use requests.Response.raise_for_status to handle errors generically

Rebase

Thanks for the remarks, will go through them.

Fix copyrights and rebase

Improved tests

refactor auth and backoff

• Take into account feedback from ardumont and vlorentz
• Take into account feedback from douardda
• still TODO: rename "singletons"

@douardda: taking into account your suggestions.

• use directory_ls rather than dir_iterator to lower overhead and control iteration
• use a set to test early for known objects
• only update once master head with the given revision id
• do not chunk files ids because in the end the storage requests are not chunked
• partition new and old files to adapt processing

Be more consistent in naming and abbreviating

Take into account feedback from ardumont and vlorentz

Used a "note" directive

In D4731#119235, @vlorentz wrote:

But for components, we wanted to instantiate every instance only one time

What is the benefit of doing that (vs. the cost of the extra complexity)?

First, thanks for the feedback, every help is welcome because I am still pretty the newbie in the project as a whole, and this is a more complex topic that initially anticipated.

Until now, configuration handling was implemented independently for each service.

What do you mean by that? It has always been handled by SWHConfig in swh.core.

@vlorentz wrote:

btw, you don't have to expose SWHID_RE_RAW directly, it's available as SWHID_RE.pattern

Little cleanup

Rebase

Finally fixed this PREsistent typo!

Reflow the whole source to 88 chars lines

Added a TODO about a section on guidelines

Cleaned and wrote up several sections

In addition to failed cooking retry, failed RPC calls are now also retried.

Removed usage of external host volumes, as asked by @douardda, for brievety.

Keep postgresql without major version pin, as we follow debian versions.

Usa a docker-compose.override definition.
Rephrased where appropriate.

Another strategy preferred: using retrying proxies for storage (existing and activated) and scheduler (to be done).

Tested in docker env, the config is valid after D4685 applied.

Suggest to adapt according to policy

Precision about ownership

Improved phrasing