Migrate test cli config to a dict instead of raw yaml
In preparation of being able to override parts of the configuration.