Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/780/ for more details.

I like the way that looks.

I guess the lowest friction way of wiring this up to the rest of our infra would be adding a 'pipeline' class to STORAGE_IMPLEMENTATION, which would make the top-level config look like

storage:
  cls: pipeline
  args:
    steps:
    - filter
    - buffer
    - cls: remote
      args:
        url: http://foo/

Which is a bit less terse, but has the advantage of being fully backwards-compatible.

What do you mean by "backwards-compatible"? get_storage_pipeline accepts a superset of get_storage's syntax.

I'm ok with either syntax though.

I'm not convinced this really is a good idea, it makes it difficult to document. Why not just replace get_xxx(cls, args): by get_xxx(cls, **args), then we can write this yaml as (with a little bit of manipulation on the yaml->arguments transformation):

storage:
  - filter
  - storage:
    - buffer
    - storage:
      - remote
      - url: http://swh-storage:5002/

or (with no under-the-carpet transformation):

storage:
  cls:filter
  storage:
    cls: buffer
    storage:
      cls: remote
      url: http://swh-storage:5002/

which looks a reasonable compromise between specific argument manipulation and yaml readability.

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/781/ for more details.

! In D2276#53517, @douardda wrote:
I'm not convinced this really is a good idea, it makes it difficult to document. Why not just replace get_xxx(cls, args): by get_xxx(cls, **args), then we can write this yaml as (with a little bit of manipulation on the yaml->arguments transformation):

storage:
  - filter
  - storage:
    - buffer
    - storage:
      - remote
      - url: http://swh-storage:5002/

I don't like this, notably for what is going to happen for classes with several arguments. Do I need to add a dash or not?

or (with no under-the-carpet transformation):

storage:
  cls:filter
  storage:
    cls: buffer
    storage:
      cls: remote
      url: http://swh-storage:5002/

which looks a reasonable compromise between specific argument manipulation and yaml readability.

I agree that we should make the **args implicit. We can even do it in a backwards compatible way by checking for args['args'] at the beginning of get_storage.

I still think there's *also* value in the pipeline proposal.

@douardda Your first proposition is essentially moving to positional arguments, which is step backward.

The second one looks indeed like a good compromise between code readability and config readability.

In D2276#53524, @olasd wrote:

I still think there's *also* value in the pipeline proposal.

By combining the two proposals, you can remove the special casing for strings in the pipeline steps and get to:

storage:
  cls: pipeline
  steps:
  - cls: filter
  - cls: buffer
  - cls: remote
    url: http://foo/

Which looks pretty neat, and fairly simple to understand/document to me

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/782/ for more details.

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/783/ for more details.

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/784/ for more details.

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/788/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/788/console