Page MenuHomeSoftware Heritage

Add swh-web docker service (and fix the storage)
ClosedPublic

Authored by ardumont on Dec 13 2018, 6:03 PM.

Details

Summary

This starts a swh-web service ready to speak with a storage.
So now, we could synthesize the getting-started as: docker-compose up

Related T1443

Test Plan
docker-compose up

To actually see some data in the webapp, we need to ingest some repositories.

for this we need to initialize the db first (for now, this is manual but should be improved soon).
In another shell:

docker-compose run swh-storage init

Then, load a repository:

python3 -m swh.loader.git.updater --origin-url https://github.com/SoftwareHeritage/swh-storage.git

Now, browse the archive (http://localhost:5004) and look for your newly ingested data.

Note:
stop your swh-services and other related services first (postgresql).

Diff Detail

Repository
rCDFD Dockerfiles for developers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont added inline comments.
dockerfiles/swh-web/Dockerfile
5

If we don't use gunicorn3 in the end, this probably could go away.

21

This can be changed if i chmod +x entrypoint.sh directly the file and commit it.

Fix rights in entrypoint.sh

Mmm, apparently it works. It seems it took some time to bootstrap?

  • swh-web: Make the webapp actually serve results
ardumont retitled this revision from wip: Add swh-web docker service to Add swh-web docker service.Dec 13 2018, 7:00 PM
  • swh-web: Make the webapp actually serve results
  • swh-web: Make the services run against storage and objstorage
  • swh-storage/entrypoint: Use --no-create as flag
ardumont marked an inline comment as done.
ardumont added inline comments.
dockerfiles/swh-web/web.yml
14

This still needs other services (vault, scheduler, indexer storage, etc...)
but a subset can already work.

Revert the modification on init (that also needs a bump on the storage)

ardumont added inline comments.
dockerfiles/swh-web/entrypoint.sh
17

I initially commented but that broke the command so here it is:

  • --verbosity to have sensible output
  • --insecure to serve the static css/js
  • 0.0.0.0 so that we can actually reach the service.
dockerfiles/swh-web/web.yml
17

To allow any machine to connect (but really your host machine).
If we don't open this, we got a disallowed host error requesting us to add our hostname machine in the django's settings.py's ALLOWED_HOSTS option (this allowed_hosts entry here is just a mapping for that).

In effect, we can now replace the getting started with docker-compose up.

  • swh-storage: Make the storage work again

In effect, we can now replace the getting started with docker-compose up.

  • swh-storage: Make the storage work again

Prior to this, i was no longer able to load a repository, making the webapp pretty much useless ;)

ardumont retitled this revision from Add swh-web docker service to Add swh-web docker service (and fix the storage).Dec 13 2018, 7:48 PM

Nice work!

All lights are green from my side.

✔ ~/swh/swh-environment/swh-docker-dev [arcpatch-D815 L|✔] 
22:49 $ docker-compose up
Starting swhdockerdev_swh-objstorage_1
Starting swhdockerdev_swh-storage-db_1
Starting swhdockerdev_swh-storage_1
Starting swhdockerdev_swh-web_1
Attaching to swhdockerdev_swh-objstorage_1, swhdockerdev_swh-storage-db_1, swhdockerdev_swh-storage_1, swhdockerdev_swh-web_1
swh-storage-db_1  | 2018-12-13 21:49:06.135 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
swh-storage-db_1  | 2018-12-13 21:49:06.135 UTC [1] LOG:  listening on IPv6 address "::", port 5432
swh-storage_1     | Starting the swh-storage API server
swh-objstorage_1  | Step 1
swh-storage-db_1  | 2018-12-13 21:49:06.141 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
swh-storage_1     |  * Serving Flask app "server" (lazy loading)
swh-storage_1     |  * Environment: production
swh-storage-db_1  | 2018-12-13 21:49:06.157 UTC [25] LOG:  database system was shut down at 2018-12-13 21:49:00 UTC
swh-objstorage_1  | Starting the swh-objstorage API server
swh-storage_1     |    WARNING: Do not use the development server in a production environment.
swh-web_1         | starting the swh-web server
swh-storage_1     |    Use a production WSGI server instead.
swh-storage_1     |  * Debug mode: on
swh-storage_1     | /usr/local/lib/python3.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
swh-storage_1     |   """)
swh-storage_1     |  * Running on http://0.0.0.0:5002/ (Press CTRL+C to quit)
swh-storage_1     |  * Restarting with stat
swh-storage-db_1  | 2018-12-13 21:49:06.162 UTC [1] LOG:  database system is ready to accept connections
swh-storage_1     | /usr/local/lib/python3.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
swh-storage_1     |   """)
swh-storage_1     |  * Debugger is active!
swh-storage_1     |  * Debugger PIN: 282-054-931
swh-web_1         | "GET /browse/search/?q=.*&with_visit HTTP/1.1" 200 11313
swh-web_1         | "GET /static/css/vendors.1bb53db16d3be4ade415.css HTTP/1.1" 304 0
swh-web_1         | "GET /static/js/vendors.1bb53db16d3be4ade415.js HTTP/1.1" 304 0
swh-web_1         | "GET /static/js/browse.0914bcf041c43ac18d99.js HTTP/1.1" 304 0
swh-web_1         | "GET /static/css/webapp.255faff414598b3635fe.css HTTP/1.1" 304 0
swh-web_1         | "GET /static/css/browse.0914bcf041c43ac18d99.css HTTP/1.1" 304 0
swh-web_1         | "GET /static/css/vault.5e75f0f4a5fcbfd2fe60.css HTTP/1.1" 304 0
swh-web_1         | "GET /static/js/webapp.255faff414598b3635fe.js HTTP/1.1" 304 0
swh-web_1         | "GET /static/js/vault.5e75f0f4a5fcbfd2fe60.js HTTP/1.1" 304 0
swh-web_1         | "GET /jsreverse/ HTTP/1.1" 200 14225

Nice work!
All lights are green from my side.

\m/

  • swh-web: Remove unneeded dev instruction
  • README: Mention the new webapp service
  • swh-web/Dockerfile: Fix formatting
  • swh-web/entrypoint: Add back comments
This revision is now accepted and ready to land.Dec 14 2018, 10:35 AM
This revision was automatically updated to reflect the committed changes.
dockerfiles/swh-web/entrypoint.sh
8

python3 -m swh.web.manage migrate should run here too.

ardumont added inline comments.
dockerfiles/swh-web/entrypoint.sh
8

Ah yes, it should be tested in the loop, something like:
[ $srcrepo = 'swh-web' ] && ...