Page MenuHomeSoftware Heritage

Add lister workers services
ClosedPublic

Authored by douardda on Mon, Dec 17, 3:18 PM.

Details

Summary

This includes the npm worker which is not yet published.

The celery workers run under a dedicated swh user otherwise celery
complains quite loudly.

This has a pretty annoying drawback: we cannot pip install swh packages
from volume mounted source in 'develop mode' because vcversioner will
then attempt to write 2 files in the source directory (version.txt and
the _version.py module) in which we do not have write access when
the entrypoint is run from the docker as the swh user.

There is no easy solution for this problem, so for now, we install the
package from sources, bit NOT in develop mode (non editable).

Related to T1443.

Diff Detail

Repository
rCDFD Dockerfiles for developers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.Mon, Dec 17, 3:18 PM
vlorentz added a subscriber: vlorentz.EditedMon, Dec 17, 4:22 PM

There is no easy solution for this problem

You can mount an extra layer of overlayfs with a tmpfs

vlorentz added inline comments.Mon, Dec 17, 4:57 PM
dockerfiles/swh-listers-worker/entrypoint.sh
63–64

indent

ardumont requested changes to this revision.Mon, Dec 17, 5:53 PM
ardumont added a subscriber: ardumont.

Sounds good.
There are typos which will prevent this from working though.

dockerfiles/swh-listers-worker/Dockerfile
18

We do not expose anything here, don't we?

dockerfiles/swh-listers-worker/entrypoint.sh
54

already

60

POSTGRES_

listers.env
2

I found it easier to read when the concurrency is 1, ymmv though ;)

This revision now requires changes to proceed.Mon, Dec 17, 5:53 PM
douardda marked an inline comment as done.Tue, Dec 18, 10:30 AM
douardda added inline comments.
dockerfiles/swh-listers-worker/Dockerfile
18

indeed

douardda updated this revision to Diff 2681.Tue, Dec 18, 10:31 AM

Fix typos

ardumont accepted this revision.Tue, Dec 18, 10:34 AM
This revision is now accepted and ready to land.Tue, Dec 18, 10:34 AM
This revision was automatically updated to reflect the committed changes.