Page MenuHomeSoftware Heritage

Add swh-web docker service (and fix the storage)
ClosedPublic

Authored by ardumont on Dec 13 2018, 6:03 PM.

Details

Summary

This starts a swh-web service ready to speak with a storage.
So now, we could synthesize the getting-started as: docker-compose up

Related T1443

Test Plan
docker-compose up

To actually see some data in the webapp, we need to ingest some repositories.

for this we need to initialize the db first (for now, this is manual but should be improved soon).
In another shell:

docker-compose run swh-storage init

Then, load a repository:

python3 -m swh.loader.git.updater --origin-url https://github.com/SoftwareHeritage/swh-storage.git

Now, browse the archive (http://localhost:5004) and look for your newly ingested data.

Note:
stop your swh-services and other related services first (postgresql).

Diff Detail

Repository
rCDFD Dockerfiles for developers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Dec 13 2018, 6:03 PM
ardumont planned changes to this revision.Dec 13 2018, 6:04 PM
ardumont marked 2 inline comments as done.Dec 13 2018, 6:09 PM
ardumont added inline comments.
dockerfiles/swh-web/Dockerfile
6

If we don't use gunicorn3 in the end, this probably could go away.

22

This can be changed if i chmod +x entrypoint.sh directly the file and commit it.

ardumont updated this revision to Diff 2590.Dec 13 2018, 6:10 PM

Fix rights in entrypoint.sh

ardumont updated this revision to Diff 2591.Dec 13 2018, 6:11 PM

Fix format issue

Mmm, apparently it works. It seems it took some time to bootstrap?

ardumont updated this revision to Diff 2592.Dec 13 2018, 6:55 PM
  • swh-web: Make the webapp actually serve results
ardumont retitled this revision from wip: Add swh-web docker service to Add swh-web docker service.Dec 13 2018, 7:00 PM
ardumont updated this revision to Diff 2593.Dec 13 2018, 7:17 PM
  • swh-web: Make the webapp actually serve results
  • swh-web: Make the services run against storage and objstorage
ardumont updated this revision to Diff 2595.Dec 13 2018, 7:23 PM
  • swh-storage/entrypoint: Use --no-create as flag
ardumont edited the summary of this revision. (Show Details)Dec 13 2018, 7:24 PM
ardumont marked an inline comment as done.
ardumont added inline comments.
dockerfiles/swh-web/web.yml
15

This still needs other services (vault, scheduler, indexer storage, etc...)
but a subset can already work.

ardumont updated this revision to Diff 2596.Dec 13 2018, 7:29 PM

Revert the modification on init (that also needs a bump on the storage)

ardumont marked 2 inline comments as done.Dec 13 2018, 7:33 PM
ardumont added inline comments.
dockerfiles/swh-web/entrypoint.sh
18

I initially commented but that broke the command so here it is:

  • --verbosity to have sensible output
  • --insecure to serve the static css/js
  • 0.0.0.0 so that we can actually reach the service.
dockerfiles/swh-web/web.yml
18

To allow any machine to connect (but really your host machine).
If we don't open this, we got a disallowed host error requesting us to add our hostname machine in the django's settings.py's ALLOWED_HOSTS option (this allowed_hosts entry here is just a mapping for that).

ardumont updated this revision to Diff 2597.Dec 13 2018, 7:46 PM

In effect, we can now replace the getting started with docker-compose up.

  • swh-storage: Make the storage work again
ardumont edited the summary of this revision. (Show Details)Dec 13 2018, 7:47 PM

In effect, we can now replace the getting started with docker-compose up.

  • swh-storage: Make the storage work again

Prior to this, i was no longer able to load a repository, making the webapp pretty much useless ;)

ardumont retitled this revision from Add swh-web docker service to Add swh-web docker service (and fix the storage).Dec 13 2018, 7:48 PM

Nice work!

All lights are green from my side.

✔ ~/swh/swh-environment/swh-docker-dev [arcpatch-D815 L|✔] 
22:49 $ docker-compose up
Starting swhdockerdev_swh-objstorage_1
Starting swhdockerdev_swh-storage-db_1
Starting swhdockerdev_swh-storage_1
Starting swhdockerdev_swh-web_1
Attaching to swhdockerdev_swh-objstorage_1, swhdockerdev_swh-storage-db_1, swhdockerdev_swh-storage_1, swhdockerdev_swh-web_1
swh-storage-db_1  | 2018-12-13 21:49:06.135 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
swh-storage-db_1  | 2018-12-13 21:49:06.135 UTC [1] LOG:  listening on IPv6 address "::", port 5432
swh-storage_1     | Starting the swh-storage API server
swh-objstorage_1  | Step 1
swh-storage-db_1  | 2018-12-13 21:49:06.141 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
swh-storage_1     |  * Serving Flask app "server" (lazy loading)
swh-storage_1     |  * Environment: production
swh-storage-db_1  | 2018-12-13 21:49:06.157 UTC [25] LOG:  database system was shut down at 2018-12-13 21:49:00 UTC
swh-objstorage_1  | Starting the swh-objstorage API server
swh-storage_1     |    WARNING: Do not use the development server in a production environment.
swh-web_1         | starting the swh-web server
swh-storage_1     |    Use a production WSGI server instead.
swh-storage_1     |  * Debug mode: on
swh-storage_1     | /usr/local/lib/python3.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
swh-storage_1     |   """)
swh-storage_1     |  * Running on http://0.0.0.0:5002/ (Press CTRL+C to quit)
swh-storage_1     |  * Restarting with stat
swh-storage-db_1  | 2018-12-13 21:49:06.162 UTC [1] LOG:  database system is ready to accept connections
swh-storage_1     | /usr/local/lib/python3.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
swh-storage_1     |   """)
swh-storage_1     |  * Debugger is active!
swh-storage_1     |  * Debugger PIN: 282-054-931
swh-web_1         | "GET /browse/search/?q=.*&with_visit HTTP/1.1" 200 11313
swh-web_1         | "GET /static/css/vendors.1bb53db16d3be4ade415.css HTTP/1.1" 304 0
swh-web_1         | "GET /static/js/vendors.1bb53db16d3be4ade415.js HTTP/1.1" 304 0
swh-web_1         | "GET /static/js/browse.0914bcf041c43ac18d99.js HTTP/1.1" 304 0
swh-web_1         | "GET /static/css/webapp.255faff414598b3635fe.css HTTP/1.1" 304 0
swh-web_1         | "GET /static/css/browse.0914bcf041c43ac18d99.css HTTP/1.1" 304 0
swh-web_1         | "GET /static/css/vault.5e75f0f4a5fcbfd2fe60.css HTTP/1.1" 304 0
swh-web_1         | "GET /static/js/webapp.255faff414598b3635fe.js HTTP/1.1" 304 0
swh-web_1         | "GET /static/js/vault.5e75f0f4a5fcbfd2fe60.js HTTP/1.1" 304 0
swh-web_1         | "GET /jsreverse/ HTTP/1.1" 200 14225

Nice work!
All lights are green from my side.

\m/

ardumont edited the summary of this revision. (Show Details)Dec 13 2018, 11:04 PM
ardumont edited the test plan for this revision. (Show Details)Dec 13 2018, 11:22 PM
ardumont updated this revision to Diff 2598.Dec 13 2018, 11:22 PM
  • swh-web: Remove unneeded dev instruction
  • README: Mention the new webapp service
  • swh-web/Dockerfile: Fix formatting
  • swh-web/entrypoint: Add back comments
ardumont edited the test plan for this revision. (Show Details)Dec 14 2018, 9:20 AM
douardda accepted this revision.Dec 14 2018, 10:35 AM
This revision is now accepted and ready to land.Dec 14 2018, 10:35 AM
This revision was automatically updated to reflect the committed changes.
vlorentz added inline comments.Dec 14 2018, 12:28 PM
dockerfiles/swh-web/entrypoint.sh
8

python3 -m swh.web.manage migrate should run here too.

ardumont marked an inline comment as done.Dec 14 2018, 2:09 PM
ardumont added inline comments.
dockerfiles/swh-web/entrypoint.sh
8

Ah yes, it should be tested in the loop, something like:
[ $srcrepo = 'swh-web' ] && ...