Page MenuHomeSoftware Heritage

Add lister workers services
ClosedPublic

Authored by douardda on Dec 17 2018, 3:18 PM.

Details

Summary

This includes the npm worker which is not yet published.

The celery workers run under a dedicated swh user otherwise celery
complains quite loudly.

This has a pretty annoying drawback: we cannot pip install swh packages
from volume mounted source in 'develop mode' because vcversioner will
then attempt to write 2 files in the source directory (version.txt and
the _version.py module) in which we do not have write access when
the entrypoint is run from the docker as the swh user.

There is no easy solution for this problem, so for now, we install the
package from sources, bit NOT in develop mode (non editable).

Related to T1443.

Diff Detail

Repository
rCDFD Dockerfiles for developers
Branch
listers
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 3115
Build 3986: arc lint + arc unit

Event Timeline

There is no easy solution for this problem

You can mount an extra layer of overlayfs with a tmpfs

dockerfiles/swh-listers-worker/entrypoint.sh
62–63

indent

ardumont added a subscriber: ardumont.

Sounds good.
There are typos which will prevent this from working though.

dockerfiles/swh-listers-worker/Dockerfile
17

We do not expose anything here, don't we?

dockerfiles/swh-listers-worker/entrypoint.sh
53

already

59

POSTGRES_

listers.env
1

I found it easier to read when the concurrency is 1, ymmv though ;)

This revision now requires changes to proceed.Dec 17 2018, 5:53 PM
douardda added inline comments.
dockerfiles/swh-listers-worker/Dockerfile
17

indeed

This revision is now accepted and ready to land.Dec 18 2018, 10:34 AM
This revision was automatically updated to reflect the committed changes.