Page MenuHomeSoftware Heritage

Add scheduler services
ClosedPublic

Authored by douardda on Dec 14 2018, 2:36 PM.

Details

Summary
  • scheduler-api
  • scheduler-listener
  • scheduler-runner

this comes with a dedicated database and a rabbitmq service.

Related T1443

Diff Detail

Repository
rCDFD Dockerfiles for developers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.Dec 14 2018, 2:36 PM

ah we stepped on each other's toes, i'll close D819.
this one is more complete.

I'll start the review soon after closing the other one.

ardumont edited the summary of this revision. (Show Details)Dec 14 2018, 2:48 PM
vlorentz added inline comments.Dec 14 2018, 2:53 PM
dockerfiles/swh-scheduler-worker/entrypoint.sh
35–39

Why exec sh -c instead of directly running the loop?

ardumont added inline comments.Dec 14 2018, 2:57 PM
dockerfiles/swh-scheduler-api/entrypoint.sh
31

In D816, i improved the initialization part to be idempotent (not perfect but it works ;)
In D817, i used it so the init takes place prior to the server's startup.

This is usable as long as we use the docker-compose.override.yml for now.
the real fix would be to cascade a tag version from swh.core to swh.storage and now to swh.scheduler (tagging them also).

Should i go do the cascading?

dockerfiles/swh-scheduler-worker/entrypoint.sh
39

lol

isn't there a flag to the runner so that it does it itself?

Checked and no!

ardumont added inline comments.Dec 14 2018, 3:00 PM
docker-compose.yml
18

can't we move that in dockerfiles/swh-scheduler/scheduler.env?

keeping the configuration file in one location.

38

What do you mean?

I thought the runner and listener were unrelated, checking.

ardumont added inline comments.Dec 14 2018, 3:10 PM
docker-compose.yml
38

They are independent, but if we run stuff.
We would also want that the workers consuming stuff have their tasks updated when they are done (that's the listener's job).

Is that why you drew the deps here?

douardda marked 2 inline comments as done.Dec 14 2018, 3:37 PM
douardda added inline comments.
dockerfiles/swh-scheduler-api/entrypoint.sh
31

Should i go do the cascading?

yes!

dockerfiles/swh-scheduler-worker/entrypoint.sh
35–39

Why exec sh -c instead of directly running the loop?

I just used what's in the systemd service file.

Sounds good, i have some questions and remarks.

vlorentz added inline comments.Dec 14 2018, 3:51 PM
dockerfiles/swh-scheduler-worker/entrypoint.sh
35–39

@douardda That's because systemd doesn't run commands in an actual shell. It's not needed here.

ardumont added inline comments.Dec 14 2018, 4:06 PM
dockerfiles/swh-scheduler-api/entrypoint.sh
31

yes!

It's in progress btw (core is deployed and pushed to pypi, storage and scheduler should be soon too).

vlorentz accepted this revision.Dec 14 2018, 4:14 PM
This revision is now accepted and ready to land.Dec 14 2018, 4:14 PM
ardumont added inline comments.Dec 14 2018, 4:21 PM
dockerfiles/swh-scheduler-api/entrypoint.sh
31

Done, we can now merge the db initialization step and the starting one (cf. D817).

31

And remove the --no-create!

ardumont requested changes to this revision.Dec 14 2018, 4:29 PM

It remains the entrypoint.sh to adapt so that the scheduler's db is initialized prior to starting up the service as in D819.

This revision now requires changes to proceed.Dec 14 2018, 4:29 PM
douardda marked 4 inline comments as done.Dec 17 2018, 10:20 AM
douardda added inline comments.
docker-compose.yml
18

can't we move that in dockerfiles/swh-scheduler/scheduler.env?

we could be this env file is not part of the docker image building process, it's really part of the compose orchestration one. So it's not about 'put all config files in one location' actually.

At some point we may need/want to split the docker image generation away from the compose (orchestration) part.

38

What do you mean?
I thought the runner and listener were unrelated, checking.

yes they are. The dependency is there just so the swh/scheduler-worker image is built when starting the runner. Which is a reason why I said in my previous comment:

At some point we may need/want to split the docker image generation away from the compose (orchestration) part.
38

Is that why you drew the deps here?

it's really is just a simple technical deps due to the fact both services share the same image and the fact we do use docker-compose to build them.

dockerfiles/swh-scheduler-worker/entrypoint.sh
35–39

It's not needed here.

if we want to keep the exec command, we need it. And I'm inclined to think it's best to keep the exec statement instead of running the for loop from within the case block.

vlorentz added inline comments.Dec 17 2018, 10:24 AM
docker-compose.yml
38

What if we just repeat build: ./dockerfiles/swh-scheduler-worker in this service declaration?

douardda updated this revision to Diff 2632.Dec 17 2018, 10:26 AM

merge init command in service starting one in scheduler-api's entrypoint

--no-create

should go away.

douardda updated this revision to Diff 2635.Dec 17 2018, 10:35 AM

And kill the --no-create option from the swh-db-init cmd line in entrypoint

ardumont accepted this revision.Dec 17 2018, 10:46 AM
This revision is now accepted and ready to land.Dec 17 2018, 10:46 AM
ardumont added inline comments.Dec 17 2018, 10:47 AM
dockerfiles/swh-scheduler-api/entrypoint.sh
31

log says swh-storage, should say swh-scheduler.

douardda updated this revision to Diff 2641.Dec 17 2018, 11:51 AM

Fix scheduler's entrypoint echo line (s/storage/scheduler/)

douardda updated this revision to Diff 2643.Dec 17 2018, 11:56 AM

rebase on master

douardda updated this revision to Diff 2644.Dec 17 2018, 12:08 PM

Add the active wait on postgres in scheduler-api's entrypoint

and ensure pg password is not asked...

This revision was automatically updated to reflect the committed changes.
ardumont added inline comments.Dec 17 2018, 3:45 PM
dockerfiles/swh-scheduler-api/scheduler.yml
4

i missed that, should have been scheduling_db: swh-scheduler, D838 fixes that.

dockerfiles/swh-scheduler-worker/scheduler.yml
2

i missed that, should have been scheduling_db: service=swh-scheduler, D838 fixes that.