Page MenuHomeSoftware Heritage

Add scheduler services
ClosedPublic

Authored by douardda on Dec 14 2018, 2:36 PM.

Details

Summary
  • scheduler-api
  • scheduler-listener
  • scheduler-runner

this comes with a dedicated database and a rabbitmq service.

Related T1443

Diff Detail

Repository
rCDFD Dockerfiles for developers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ah we stepped on each other's toes, i'll close D819.
this one is more complete.

I'll start the review soon after closing the other one.

dockerfiles/swh-scheduler-worker/entrypoint.sh
34–38

Why exec sh -c instead of directly running the loop?

dockerfiles/swh-scheduler-api/entrypoint.sh
30

In D816, i improved the initialization part to be idempotent (not perfect but it works ;)
In D817, i used it so the init takes place prior to the server's startup.

This is usable as long as we use the docker-compose.override.yml for now.
the real fix would be to cascade a tag version from swh.core to swh.storage and now to swh.scheduler (tagging them also).

Should i go do the cascading?

dockerfiles/swh-scheduler-worker/entrypoint.sh
38

lol

isn't there a flag to the runner so that it does it itself?

Checked and no!

docker-compose.yml
17

can't we move that in dockerfiles/swh-scheduler/scheduler.env?

keeping the configuration file in one location.

37

What do you mean?

I thought the runner and listener were unrelated, checking.

docker-compose.yml
37

They are independent, but if we run stuff.
We would also want that the workers consuming stuff have their tasks updated when they are done (that's the listener's job).

Is that why you drew the deps here?

douardda added inline comments.
dockerfiles/swh-scheduler-api/entrypoint.sh
30

Should i go do the cascading?

yes!

dockerfiles/swh-scheduler-worker/entrypoint.sh
34–38

Why exec sh -c instead of directly running the loop?

I just used what's in the systemd service file.

Sounds good, i have some questions and remarks.

dockerfiles/swh-scheduler-worker/entrypoint.sh
34–38

@douardda That's because systemd doesn't run commands in an actual shell. It's not needed here.

dockerfiles/swh-scheduler-api/entrypoint.sh
30

yes!

It's in progress btw (core is deployed and pushed to pypi, storage and scheduler should be soon too).

This revision is now accepted and ready to land.Dec 14 2018, 4:14 PM
dockerfiles/swh-scheduler-api/entrypoint.sh
30

Done, we can now merge the db initialization step and the starting one (cf. D817).

30

And remove the --no-create!

It remains the entrypoint.sh to adapt so that the scheduler's db is initialized prior to starting up the service as in D819.

This revision now requires changes to proceed.Dec 14 2018, 4:29 PM
douardda added inline comments.
docker-compose.yml
17

can't we move that in dockerfiles/swh-scheduler/scheduler.env?

we could be this env file is not part of the docker image building process, it's really part of the compose orchestration one. So it's not about 'put all config files in one location' actually.

At some point we may need/want to split the docker image generation away from the compose (orchestration) part.

37

What do you mean?
I thought the runner and listener were unrelated, checking.

yes they are. The dependency is there just so the swh/scheduler-worker image is built when starting the runner. Which is a reason why I said in my previous comment:

At some point we may need/want to split the docker image generation away from the compose (orchestration) part.

37

Is that why you drew the deps here?

it's really is just a simple technical deps due to the fact both services share the same image and the fact we do use docker-compose to build them.

dockerfiles/swh-scheduler-worker/entrypoint.sh
34–38

It's not needed here.

if we want to keep the exec command, we need it. And I'm inclined to think it's best to keep the exec statement instead of running the for loop from within the case block.

docker-compose.yml
37

What if we just repeat build: ./dockerfiles/swh-scheduler-worker in this service declaration?

merge init command in service starting one in scheduler-api's entrypoint

And kill the --no-create option from the swh-db-init cmd line in entrypoint

This revision is now accepted and ready to land.Dec 17 2018, 10:46 AM
dockerfiles/swh-scheduler-api/entrypoint.sh
30

log says swh-storage, should say swh-scheduler.

Fix scheduler's entrypoint echo line (s/storage/scheduler/)

Add the active wait on postgres in scheduler-api's entrypoint

and ensure pg password is not asked...

This revision was automatically updated to reflect the committed changes.
dockerfiles/swh-scheduler-api/scheduler.yml
4

i missed that, should have been scheduling_db: swh-scheduler, D838 fixes that.

dockerfiles/swh-scheduler-worker/scheduler.yml
2

i missed that, should have been scheduling_db: service=swh-scheduler, D838 fixes that.