Page MenuHomeSoftware Heritage

New initial structure of Docker images.
ClosedPublic

Authored by vlorentz on Dec 11 2018, 3:04 PM.

Diff Detail

Repository
rCDFD Dockerfiles for developers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont added inline comments.
docker-compose.yml
7

Commenting on why you need the magic word (about applying updates for password, etc...) would be nice.

dockerfiles/Dockerfile-swh-storage-pg
3

Same here, explicitely saying it's running the swh running script would be great.

Trying to build it locally

dockerfiles/Dockerfile-swh-storage
6

When dust settled, maybe having this up to this comment centralized as a startup image would be good?

vlorentz marked 2 inline comments as done.
  • comment.
vlorentz added inline comments.
docker-compose.yml
7

Actually, I don't need it. Removed.

vlorentz added inline comments.
dockerfiles/Dockerfile-swh-storage
6

What do you mean?

docker-compose.yml
7

Ah yes, because docker-compose down --volumes (from make run) does what's expected.
At next build, the newly declared updates (e.g. pg password) will actually be applied.

dockerfiles/Dockerfile-swh-storage
6

first, when i commented, it was from the start of the dockerfile to line 6 (not 7).
second, i meant, this is duplicated between the storage and objstorage dockerfiles.
I guess, it would also be the case for other new swh modules later.
So i thought of having a base image for that part but yagni for now.

dockerfiles/Dockerfile-swh-objstorage
13

Does that folder exist?

I have the following:

Step 8/12 : COPY ./swh-docker-dev/config/ /etc/softwareheritage/
COPY failed: stat /var/lib/docker/tmp/docker-builder755492407/swh-docker-dev/config: no such file or directory
Makefile:13: recipe for target 'build-swh-objstorage' failed
make: *** [build-swh-objstorage] Error 1
vlorentz added inline comments.
dockerfiles/Dockerfile-swh-objstorage
13

I created in that diff. Make sure you cloned swh-docker-dev (no longer swh-jenkins-dev!) in swh-environment.

dockerfiles/Dockerfile-swh-storage
6

I thought about it, but there are just four trivial lines in common, I don't think it's worse the extra complexity.

dockerfiles/Dockerfile-swh-objstorage
13

yes, i did that but the local repository is named swh-dockerfiles-dev, not swh-docker-dev.

vlorentz added inline comments.
dockerfiles/Dockerfile-swh-objstorage
13

My bad. I'll fix the name of the repo, I believe swh-docker-dev is better (dockerfiles are not the only files here). You should rename it on your machine as well

dockerfiles/Dockerfile-swh-objstorage
13

fwiw, i created a symlink with that name and it works indeed:

Step 8/12 : COPY ./swh-docker-dev/config/ /etc/softwareheritage/
 ---> d735b074f273
Step 9/12 : COPY . .
dockerfiles/Dockerfile-swh-objstorage
13

My bad. I'll fix the name of the repo, I believe swh-docker-dev is better

ack.

You should rename it on your machine as well

done

Trying to build it locally

still running

still running

done!

tl; dr; it works! \m/

Here is how i tested:

$ make run
pgsql-storage_1   | 2018-12-11 14:48:35.187 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
pgsql-storage_1   | 2018-12-11 14:48:35.188 UTC [1] LOG:  listening on IPv6 address "::", port 5432
pgsql-storage_1   | 2018-12-11 14:48:35.214 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
pgsql-storage_1   | 2018-12-11 14:48:35.272 UTC [106] LOG:  database system was shut down at 2018-12-11 14:48:35 UTC
pgsql-storage_1   | 2018-12-11 14:48:35.294 UTC [1] LOG:  database system is ready to accept connections
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:15] "POST /origin/add HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:15] "POST /origin/visit/add HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:15] "POST /fetch_history/start HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:15] "POST /snapshot/latest HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:16] "POST /object/find_by_sha1_git HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:16] "POST /object/find_by_sha1_git HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:21] "POST /object/find_by_sha1_git HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:21] "POST /object/find_by_sha1_git HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:24] "POST /content/missing HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:43] "POST /content/add HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:43] "POST /directory/missing HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:52] "POST /directory/add HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:52] "POST /revision/missing HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:56] "POST /revision/add HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:56] "POST /release/missing HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:57] "POST /release/add HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:57] "POST /snapshot/add HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:57] "POST /fetch_history/end HTTP/1.1" 200 -
swh-storage_1     | 172.18.0.1 - - [11/Dec/2018 14:54:57] "POST /origin/visit/update HTTP/1.1" 200 -

And a running updater git from the host machine on another shell (tmux pane):

$ cat ~/.config/loader/git-updater.yml
storage:
  cls: remote
  args:
    url: http://localhost:5002/
$ source .venv/bin/activate
python3 -m swh.loader.git.updater --origin-url https://github.com/hylang/hy
/home/tony/work/inria/repo/swh/swh-environment/.venv/lib/python3.5/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
2018-12-11 15:54:15,692 7226 Creating git origin for https://github.com/hylang/hy
2018-12-11 15:54:15,697 7226 Starting new HTTP connection (1): localhost:5002
2018-12-11 15:54:15,713 7226 http://localhost:5002 "POST /origin/add HTTP/1.1" 200 1
2018-12-11 15:54:15,714 7226 Done creating git origin for https://github.com/hylang/hy
2018-12-11 15:54:15,715 7226 Creating origin_visit for origin 1 at time 2018-12-11 14:54:15.692572+00:00
2018-12-11 15:54:15,717 7226 Resetting dropped connection: localhost
2018-12-11 15:54:15,731 7226 http://localhost:5002 "POST /origin/visit/add HTTP/1.1" 200 16
2018-12-11 15:54:15,732 7226 Done Creating origin_visit for origin 1 at time 2018-12-11 14:54:15.692572+00:00
2018-12-11 15:54:15,734 7226 Resetting dropped connection: localhost
2018-12-11 15:54:15,746 7226 http://localhost:5002 "POST /fetch_history/start HTTP/1.1" 200 1
2018-12-11 15:54:15,749 7226 Resetting dropped connection: localhost
2018-12-11 15:54:15,754 7226 http://localhost:5002 "POST /snapshot/latest HTTP/1.1" 200 1
2018-12-11 15:54:15,755 7226 Starting new HTTPS connection (1): github.com:443
2018-12-11 15:54:16,336 7226 https://github.com:443 "GET /hylang/hy/info/refs?service=git-upload-pack HTTP/1.1" 200 None
2018-12-11 15:54:16,465 7226 Resetting dropped connection: localhost
2018-12-11 15:54:16,493 7226 http://localhost:5002 "POST /object/find_by_sha1_git HTTP/1.1" 200 23003
2018-12-11 15:54:16,497 7226 Resetting dropped connection: localhost
2018-12-11 15:54:16,506 7226 http://localhost:5002 "POST /object/find_by_sha1_git HTTP/1.1" 200 7685
2018-12-11 15:54:16,923 7226 https://github.com:443 "POST /hylang/hy/git-upload-pack HTTP/1.1" 200 None
Enumerating objects: 175, done.
Counting objects: 100% (175/175), done.
Compressing objects: 100% (99/99), done.
Total 18590 (delta 89), reused 128 (delta 76), pack-reused 18415
2018-12-11 15:54:20,975 7226 Resetting dropped connection: localhost
2018-12-11 15:54:21,001 7226 http://localhost:5002 "POST /object/find_by_sha1_git HTTP/1.1" 200 23003
2018-12-11 15:54:21,007 7226 Resetting dropped connection: localhost
2018-12-11 15:54:21,019 7226 http://localhost:5002 "POST /object/find_by_sha1_git HTTP/1.1" 200 7685
2018-12-11 15:54:21,025 7226 Listed 907 refs for repo https://github.com/hylang/hy
2018-12-11 15:54:24,740 7226 Resetting dropped connection: localhost
2018-12-11 15:54:24,933 7226 http://localhost:5002 "POST /content/missing HTTP/1.1" 200 126745
2018-12-11 15:54:27,264 7226 Sending 5761 contents
2018-12-11 15:54:27,403 7226 Resetting dropped connection: localhost
2018-12-11 15:54:43,171 7226 http://localhost:5002 "POST /content/add HTTP/1.1" 200 1
2018-12-11 15:54:43,181 7226 Done sending 5761 contents
2018-12-11 15:54:43,215 7226 Resetting dropped connection: localhost
2018-12-11 15:54:43,325 7226 http://localhost:5002 "POST /directory/missing HTTP/1.1" 200 209091
2018-12-11 15:54:45,002 7226 Sending 9504 directories
2018-12-11 15:54:45,075 7226 Resetting dropped connection: localhost
2018-12-11 15:54:52,642 7226 http://localhost:5002 "POST /directory/add HTTP/1.1" 200 1
2018-12-11 15:54:52,643 7226 Done sending 9504 directories
2018-12-11 15:54:52,679 7226 Resetting dropped connection: localhost
2018-12-11 15:54:52,715 7226 http://localhost:5002 "POST /revision/missing HTTP/1.1" 200 72625
2018-12-11 15:54:54,088 7226 Sending 3301 revisions
2018-12-11 15:54:54,099 7226 Resetting dropped connection: localhost
2018-12-11 15:54:56,577 7226 http://localhost:5002 "POST /revision/add HTTP/1.1" 200 1
2018-12-11 15:54:56,578 7226 Done sending 3301 revisions
2018-12-11 15:54:56,584 7226 Resetting dropped connection: localhost
2018-12-11 15:54:56,588 7226 http://localhost:5002 "POST /release/missing HTTP/1.1" 200 531
2018-12-11 15:54:57,816 7226 Sending 24 releases
2018-12-11 15:54:57,818 7226 Resetting dropped connection: localhost
2018-12-11 15:54:57,868 7226 http://localhost:5002 "POST /release/add HTTP/1.1" 200 1
2018-12-11 15:54:57,868 7226 Done sending 24 releases
2018-12-11 15:54:57,876 7226 Resetting dropped connection: localhost
2018-12-11 15:54:57,947 7226 http://localhost:5002 "POST /snapshot/add HTTP/1.1" 200 1
2018-12-11 15:54:57,950 7226 Resetting dropped connection: localhost
2018-12-11 15:54:57,962 7226 http://localhost:5002 "POST /fetch_history/end HTTP/1.1" 200 1
2018-12-11 15:54:57,963 7226 Updating origin_visit for origin 1 with status full
2018-12-11 15:54:57,965 7226 Resetting dropped connection: localhost
2018-12-11 15:54:57,975 7226 http://localhost:5002 "POST /origin/visit/update HTTP/1.1" 200 1
2018-12-11 15:54:57,975 7226 Done updating origin_visit for origin 1 with status full
This revision is now accepted and ready to land.Dec 11 2018, 3:59 PM
This revision was automatically updated to reflect the committed changes.