Page MenuHomeSoftware Heritage
Differential D800

New initial structure of Docker images.
ClosedPublic
Actions

Authored by vlorentz on Dec 11 2018, 3:04 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Commits
rDENVfb03c3ee5a2f: New initial structure of Docker images.
rDENVe35ad06dc46f: New initial structure of Docker images.
rCDFDfb03c3ee5a2f: New initial structure of Docker images.
rCDFDe35ad06dc46f: New initial structure of Docker images.
Summary

Creates each Docker image with the 'entrypoint' fetched locally,
and all its dependencies via PyPI.

Diff Detail

Repository
rCDFD Dockerfiles for developers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Dec 11 2018, 3:04 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptDec 11 2018, 3:04 PM
Harbormaster completed remote builds in B2970: Diff 2546.Dec 11 2018, 3:04 PM
ardumont added a subscriber: ardumont.Dec 11 2018, 3:14 PM
ardumont added inline comments.
docker-compose.yml
7

Commenting on why you need the magic word (about applying updates for password, etc...) would be nice.

dockerfiles/Dockerfile-swh-storage-pg
3

Same here, explicitely saying it's running the swh running script would be great.

vlorentz updated this revision to Diff 2549.Dec 11 2018, 3:16 PM
  • Remove useless config.
ardumont added a comment.Dec 11 2018, 3:16 PM

Trying to build it locally

dockerfiles/Dockerfile-swh-storage
6

When dust settled, maybe having this up to this comment centralized as a startup image would be good?

Harbormaster completed remote builds in B2973: Diff 2549.Dec 11 2018, 3:16 PM
vlorentz updated this revision to Diff 2550.Dec 11 2018, 3:17 PM
vlorentz marked 2 inline comments as done.
  • comment.
Harbormaster completed remote builds in B2974: Diff 2550.Dec 11 2018, 3:17 PM
vlorentz marked an inline comment as done.Dec 11 2018, 3:17 PM
vlorentz added inline comments.
docker-compose.yml
7

Actually, I don't need it. Removed.

vlorentz marked an inline comment as done.Dec 11 2018, 3:18 PM
vlorentz added inline comments.
dockerfiles/Dockerfile-swh-storage
6

What do you mean?

ardumont added inline comments.Dec 11 2018, 3:21 PM
docker-compose.yml
7

Ah yes, because docker-compose down --volumes (from make run) does what's expected.
At next build, the newly declared updates (e.g. pg password) will actually be applied.

ardumont added inline comments.Dec 11 2018, 3:24 PM
dockerfiles/Dockerfile-swh-storage
6

first, when i commented, it was from the start of the dockerfile to line 6 (not 7).
second, i meant, this is duplicated between the storage and objstorage dockerfiles.
I guess, it would also be the case for other new swh modules later.
So i thought of having a base image for that part but yagni for now.

ardumont added inline comments.Dec 11 2018, 3:27 PM
dockerfiles/Dockerfile-swh-objstorage
13

Does that folder exist?

I have the following:

Step 8/12 : COPY ./swh-docker-dev/config/ /etc/softwareheritage/
COPY failed: stat /var/lib/docker/tmp/docker-builder755492407/swh-docker-dev/config: no such file or directory
Makefile:13: recipe for target 'build-swh-objstorage' failed
make: *** [build-swh-objstorage] Error 1
vlorentz marked 2 inline comments as done.Dec 11 2018, 3:30 PM
vlorentz added inline comments.
dockerfiles/Dockerfile-swh-objstorage
13

I created in that diff. Make sure you cloned swh-docker-dev (no longer swh-jenkins-dev!) in swh-environment.

dockerfiles/Dockerfile-swh-storage
6

I thought about it, but there are just four trivial lines in common, I don't think it's worse the extra complexity.

ardumont added inline comments.Dec 11 2018, 3:32 PM
dockerfiles/Dockerfile-swh-objstorage
13

yes, i did that but the local repository is named swh-dockerfiles-dev, not swh-docker-dev.

vlorentz marked an inline comment as done.Dec 11 2018, 3:34 PM
vlorentz added inline comments.
dockerfiles/Dockerfile-swh-objstorage
13

My bad. I'll fix the name of the repo, I believe swh-docker-dev is better (dockerfiles are not the only files here). You should rename it on your machine as well

vlorentz updated this revision to Diff 2552.Dec 11 2018, 3:35 PM
  • rebase
Harbormaster completed remote builds in B2976: Diff 2552.Dec 11 2018, 3:35 PM
ardumont added inline comments.Dec 11 2018, 3:37 PM
dockerfiles/Dockerfile-swh-objstorage
13

fwiw, i created a symlink with that name and it works indeed:

Step 8/12 : COPY ./swh-docker-dev/config/ /etc/softwareheritage/
 ---> d735b074f273
Step 9/12 : COPY . .
ardumont added inline comments.Dec 11 2018, 3:39 PM
dockerfiles/Dockerfile-swh-objstorage
13

My bad. I'll fix the name of the repo, I believe swh-docker-dev is better

ack.

You should rename it on your machine as well

done

ardumont added a comment.Dec 11 2018, 3:46 PM

Trying to build it locally

still running

ardumont accepted this revision.Dec 11 2018, 3:59 PM

still running

done!

tl; dr; it works! \m/

Here is how i tested:

$ make run
pgsql-storage_1   | 2018-12-11 14:48:35.187 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
pgsql-storage_1   | 2018-12-11 14:48:35.188 UTC [1] LOG:  listening on IPv6 address "::", port 5432
pgsql-storage_1   | 2018-12-11 14:48:35.214 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
pgsql-storage_1   | 2018-12-11 14:48:35.272 UTC [106] LOG:  database system was shut down at 2018-12-11 14:48:35 UTC
pgsql-storage_1   | 2018-12-11 14:48:35.294 UTC [1] LOG:  database system is ready to accept connections
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:15] "POST /origin/add HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:15] "POST /origin/visit/add HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:15] "POST /fetch_history/start HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:15] "POST /snapshot/latest HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:16] "POST /object/find_by_sha1_git HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:16] "POST /object/find_by_sha1_git HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:21] "POST /object/find_by_sha1_git HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:21] "POST /object/find_by_sha1_git HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:24] "POST /content/missing HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:43] "POST /content/add HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:43] "POST /directory/missing HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:52] "POST /directory/add HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:52] "POST /revision/missing HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:56] "POST /revision/add HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:56] "POST /release/missing HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:57] "POST /release/add HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:57] "POST /snapshot/add HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:57] "POST /fetch_history/end HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:57] "POST /origin/visit/update HTTP/1.1" 200 -

And a running updater git from the host machine on another shell (tmux pane):

$ cat ~/.config/loader/git-updater.yml
storage:
  cls: remote
  args:
    url: http://localhost:5002/
$ source .venv/bin/activate
python3 -m swh.loader.git.updater --origin-url https://github.com/hylang/hy
/home/tony/work/inria/repo/swh/swh-environment/.venv/lib/python3.5/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
2018-12-11 15:54:15,692 7226 Creating git origin for https://github.com/hylang/hy
2018-12-11 15:54:15,697 7226 Starting new HTTP connection (1): localhost:5002
2018-12-11 15:54:15,713 7226 http://localhost:5002 "POST /origin/add HTTP/1.1" 200 1
2018-12-11 15:54:15,714 7226 Done creating git origin for https://github.com/hylang/hy
2018-12-11 15:54:15,715 7226 Creating origin_visit for origin 1 at time 2018-12-11 14:54:15.692572+00:00
2018-12-11 15:54:15,717 7226 Resetting dropped connection: localhost
2018-12-11 15:54:15,731 7226 http://localhost:5002 "POST /origin/visit/add HTTP/1.1" 200 16
2018-12-11 15:54:15,732 7226 Done Creating origin_visit for origin 1 at time 2018-12-11 14:54:15.692572+00:00
2018-12-11 15:54:15,734 7226 Resetting dropped connection: localhost
2018-12-11 15:54:15,746 7226 http://localhost:5002 "POST /fetch_history/start HTTP/1.1" 200 1
2018-12-11 15:54:15,749 7226 Resetting dropped connection: localhost
2018-12-11 15:54:15,754 7226 http://localhost:5002 "POST /snapshot/latest HTTP/1.1" 200 1
2018-12-11 15:54:15,755 7226 Starting new HTTPS connection (1): github.com:443
2018-12-11 15:54:16,336 7226 https://github.com:443 "GET /hylang/hy/info/refs?service=git-upload-pack HTTP/1.1" 200 None
2018-12-11 15:54:16,465 7226 Resetting dropped connection: localhost
2018-12-11 15:54:16,493 7226 http://localhost:5002 "POST /object/find_by_sha1_git HTTP/1.1" 200 23003
2018-12-11 15:54:16,497 7226 Resetting dropped connection: localhost
2018-12-11 15:54:16,506 7226 http://localhost:5002 "POST /object/find_by_sha1_git HTTP/1.1" 200 7685
2018-12-11 15:54:16,923 7226 https://github.com:443 "POST /hylang/hy/git-upload-pack HTTP/1.1" 200 None
Enumerating objects: 175, done.
Counting objects: 100% (175/175), done.
Compressing objects: 100% (99/99), done.
Total 18590 (delta 89), reused 128 (delta 76), pack-reused 18415
2018-12-11 15:54:20,975 7226 Resetting dropped connection: localhost
2018-12-11 15:54:21,001 7226 http://localhost:5002 "POST /object/find_by_sha1_git HTTP/1.1" 200 23003
2018-12-11 15:54:21,007 7226 Resetting dropped connection: localhost
2018-12-11 15:54:21,019 7226 http://localhost:5002 "POST /object/find_by_sha1_git HTTP/1.1" 200 7685
2018-12-11 15:54:21,025 7226 Listed 907 refs for repo https://github.com/hylang/hy
2018-12-11 15:54:24,740 7226 Resetting dropped connection: localhost
2018-12-11 15:54:24,933 7226 http://localhost:5002 "POST /content/missing HTTP/1.1" 200 126745
2018-12-11 15:54:27,264 7226 Sending 5761 contents
2018-12-11 15:54:27,403 7226 Resetting dropped connection: localhost
2018-12-11 15:54:43,171 7226 http://localhost:5002 "POST /content/add HTTP/1.1" 200 1
2018-12-11 15:54:43,181 7226 Done sending 5761 contents
2018-12-11 15:54:43,215 7226 Resetting dropped connection: localhost
2018-12-11 15:54:43,325 7226 http://localhost:5002 "POST /directory/missing HTTP/1.1" 200 209091
2018-12-11 15:54:45,002 7226 Sending 9504 directories
2018-12-11 15:54:45,075 7226 Resetting dropped connection: localhost
2018-12-11 15:54:52,642 7226 http://localhost:5002 "POST /directory/add HTTP/1.1" 200 1
2018-12-11 15:54:52,643 7226 Done sending 9504 directories
2018-12-11 15:54:52,679 7226 Resetting dropped connection: localhost
2018-12-11 15:54:52,715 7226 http://localhost:5002 "POST /revision/missing HTTP/1.1" 200 72625
2018-12-11 15:54:54,088 7226 Sending 3301 revisions
2018-12-11 15:54:54,099 7226 Resetting dropped connection: localhost
2018-12-11 15:54:56,577 7226 http://localhost:5002 "POST /revision/add HTTP/1.1" 200 1
2018-12-11 15:54:56,578 7226 Done sending 3301 revisions
2018-12-11 15:54:56,584 7226 Resetting dropped connection: localhost
2018-12-11 15:54:56,588 7226 http://localhost:5002 "POST /release/missing HTTP/1.1" 200 531
2018-12-11 15:54:57,816 7226 Sending 24 releases
2018-12-11 15:54:57,818 7226 Resetting dropped connection: localhost
2018-12-11 15:54:57,868 7226 http://localhost:5002 "POST /release/add HTTP/1.1" 200 1
2018-12-11 15:54:57,868 7226 Done sending 24 releases
2018-12-11 15:54:57,876 7226 Resetting dropped connection: localhost
2018-12-11 15:54:57,947 7226 http://localhost:5002 "POST /snapshot/add HTTP/1.1" 200 1
2018-12-11 15:54:57,950 7226 Resetting dropped connection: localhost
2018-12-11 15:54:57,962 7226 http://localhost:5002 "POST /fetch_history/end HTTP/1.1" 200 1
2018-12-11 15:54:57,963 7226 Updating origin_visit for origin 1 with status full
2018-12-11 15:54:57,965 7226 Resetting dropped connection: localhost
2018-12-11 15:54:57,975 7226 http://localhost:5002 "POST /origin/visit/update HTTP/1.1" 200 1
2018-12-11 15:54:57,975 7226 Done updating origin_visit for origin 1 with status full
This revision is now accepted and ready to land.Dec 11 2018, 3:59 PM
Closed by commit rCDFDe35ad06dc46f: New initial structure of Docker images. (authored by vlorentz). · Explain WhyDec 11 2018, 5:10 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDENVe35ad06dc46f: New initial structure of Docker images..Nov 26 2019, 1:34 PM
vlorentz added a commit: rDENVfb03c3ee5a2f: New initial structure of Docker images..

Revision Contents
Changeset List

PathSize
18 lines
10 lines
config/
objstorage/
5 lines
storage/
8 lines
18 lines
dockerfiles/
20 lines
21 lines
4 lines

Diff 2553

View Options

Makefile

Loading...
View Options

README.md

Loading...
View Options

config/objstorage/server.yml

Loading...
View Options

config/storage/storage.yml

Loading...
View Options

docker-compose.yml

Loading...
View Options

dockerfiles/Dockerfile-swh-objstorage

Loading...
View Options

dockerfiles/Dockerfile-swh-storage

Loading...
View Options

dockerfiles/Dockerfile-swh-storage-pg

Loading...
About · Developer information · Legal