Page MenuHomeSoftware Heritage
Differential D815

Add swh-web docker service (and fix the storage)
ClosedPublic
Actions

Authored by ardumont on Dec 13 2018, 6:03 PM.

Details

Reviewers
vlorentz
douardda
Group Reviewers
Reviewers
Commits
rDENVb247c11ee016: swh-web/Dockerfile: Fix formatting
rDENV4ab1c7da2e8e: swh-web/entrypoint: Add back comments
rDENV2d6c5c3d9f3d: README: Mention the new webapp service
rDENV97557cb11569: swh-web: Remove unneeded dev instruction
rDENVbf6ed65cdde6: swh-storage/entrypoint.sh: Remove tabs and use spaces
rDENV639d8647acff: swh-web: Make the webapp actually serve results
rDENVd7ace96a3f1f: Remove unused dependency
rDENV56d6f44e2a6d: swh-web: Make the services run against storage and objstorage
rDENVcb44100fa872: swh-storage: Make the storage work again
rDENVb7673d0a4f43: swh-web: Bootstrap swh-web service
rCDFDb247c11ee016: swh-web/Dockerfile: Fix formatting
rCDFD4ab1c7da2e8e: swh-web/entrypoint: Add back comments
rCDFD97557cb11569: swh-web: Remove unneeded dev instruction
rCDFD2d6c5c3d9f3d: README: Mention the new webapp service
rCDFDbf6ed65cdde6: swh-storage/entrypoint.sh: Remove tabs and use spaces
rCDFDcb44100fa872: swh-storage: Make the storage work again
rCDFD639d8647acff: swh-web: Make the webapp actually serve results
rCDFD56d6f44e2a6d: swh-web: Make the services run against storage and objstorage
rCDFDd7ace96a3f1f: Remove unused dependency
rCDFDb7673d0a4f43: swh-web: Bootstrap swh-web service
Summary

This starts a swh-web service ready to speak with a storage.
So now, we could synthesize the getting-started as: docker-compose up

Related T1443

Test Plan
docker-compose up

To actually see some data in the webapp, we need to ingest some repositories.

for this we need to initialize the db first (for now, this is manual but should be improved soon).
In another shell:

docker-compose run swh-storage init

Then, load a repository:

python3 -m swh.loader.git.updater --origin-url https://github.com/SoftwareHeritage/swh-storage.git

Now, browse the archive (http://localhost:5004) and look for your newly ingested data.

Note:
stop your swh-services and other related services first (postgresql).

Diff Detail

Repository
rCDFD Dockerfiles for developers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Dec 13 2018, 6:03 PM
Harbormaster completed remote builds in B3015: Diff 2589.Dec 13 2018, 6:03 PM
ardumont planned changes to this revision.Dec 13 2018, 6:04 PM
ardumont marked 2 inline comments as done.Dec 13 2018, 6:09 PM
ardumont added inline comments.
dockerfiles/swh-web/Dockerfile
5

If we don't use gunicorn3 in the end, this probably could go away.

21

This can be changed if i chmod +x entrypoint.sh directly the file and commit it.

ardumont updated this revision to Diff 2590.Dec 13 2018, 6:10 PM

Fix rights in entrypoint.sh

Harbormaster completed remote builds in B3016: Diff 2590.Dec 13 2018, 6:10 PM
ardumont updated this revision to Diff 2591.Dec 13 2018, 6:11 PM

Fix format issue

Harbormaster completed remote builds in B3017: Diff 2591.Dec 13 2018, 6:11 PM
ardumont added a comment.Dec 13 2018, 6:30 PM

Mmm, apparently it works. It seems it took some time to bootstrap?

ardumont updated this revision to Diff 2592.Dec 13 2018, 6:55 PM
  • swh-web: Make the webapp actually serve results
Harbormaster completed remote builds in B3018: Diff 2592.Dec 13 2018, 6:55 PM
ardumont retitled this revision from wip: Add swh-web docker service to Add swh-web docker service.Dec 13 2018, 7:00 PM
ardumont updated this revision to Diff 2593.Dec 13 2018, 7:17 PM
  • swh-web: Make the webapp actually serve results
  • swh-web: Make the services run against storage and objstorage
Harbormaster completed remote builds in B3019: Diff 2593.Dec 13 2018, 7:17 PM
ardumont updated this revision to Diff 2595.Dec 13 2018, 7:23 PM
  • swh-storage/entrypoint: Use --no-create as flag
Harbormaster completed remote builds in B3020: Diff 2595.Dec 13 2018, 7:23 PM
ardumont edited the summary of this revision. (Show Details)Dec 13 2018, 7:24 PM
ardumont marked an inline comment as done.
ardumont added inline comments.
dockerfiles/swh-web/web.yml
14

This still needs other services (vault, scheduler, indexer storage, etc...)
but a subset can already work.

ardumont updated this revision to Diff 2596.Dec 13 2018, 7:29 PM

Revert the modification on init (that also needs a bump on the storage)

Harbormaster completed remote builds in B3021: Diff 2596.Dec 13 2018, 7:29 PM
ardumont marked 2 inline comments as done.Dec 13 2018, 7:33 PM
ardumont added inline comments.
dockerfiles/swh-web/entrypoint.sh
17

I initially commented but that broke the command so here it is:

  • --verbosity to have sensible output
  • --insecure to serve the static css/js
  • 0.0.0.0 so that we can actually reach the service.
dockerfiles/swh-web/web.yml
17

To allow any machine to connect (but really your host machine).
If we don't open this, we got a disallowed host error requesting us to add our hostname machine in the django's settings.py's ALLOWED_HOSTS option (this allowed_hosts entry here is just a mapping for that).

ardumont updated this revision to Diff 2597.Dec 13 2018, 7:46 PM

In effect, we can now replace the getting started with docker-compose up.

  • swh-storage: Make the storage work again
Harbormaster completed remote builds in B3022: Diff 2597.Dec 13 2018, 7:46 PM
ardumont edited the summary of this revision. (Show Details)Dec 13 2018, 7:47 PM
In D815#17308, @ardumont wrote:

In effect, we can now replace the getting started with docker-compose up.

  • swh-storage: Make the storage work again

Prior to this, i was no longer able to load a repository, making the webapp pretty much useless ;)

ardumont retitled this revision from Add swh-web docker service to Add swh-web docker service (and fix the storage).Dec 13 2018, 7:48 PM
anlambert added a subscriber: anlambert.Dec 13 2018, 10:51 PM

Nice work!

All lights are green from my side.

✔ ~/swh/swh-environment/swh-docker-dev [arcpatch-D815 L|✔] 
22:49 $ docker-compose up
Starting swhdockerdev_swh-objstorage_1
Starting swhdockerdev_swh-storage-db_1
Starting swhdockerdev_swh-storage_1
Starting swhdockerdev_swh-web_1
Attaching to swhdockerdev_swh-objstorage_1, swhdockerdev_swh-storage-db_1, swhdockerdev_swh-storage_1, swhdockerdev_swh-web_1
swh-storage-db_1  | 2018-12-13 21:49:06.135 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
swh-storage-db_1  | 2018-12-13 21:49:06.135 UTC [1] LOG:  listening on IPv6 address "::", port 5432
swh-storage_1     | Starting the swh-storage API server
swh-objstorage_1  | Step 1
swh-storage-db_1  | 2018-12-13 21:49:06.141 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
swh-storage_1     |  * Serving Flask app "server" (lazy loading)
swh-storage_1     |  * Environment: production
swh-storage-db_1  | 2018-12-13 21:49:06.157 UTC [25] LOG:  database system was shut down at 2018-12-13 21:49:00 UTC
swh-objstorage_1  | Starting the swh-objstorage API server
swh-storage_1     |    WARNING: Do not use the development server in a production environment.
swh-web_1         | starting the swh-web server
swh-storage_1     |    Use a production WSGI server instead.
swh-storage_1     |  * Debug mode: on
swh-storage_1     | /usr/local/lib/python3.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
swh-storage_1     |   """)
swh-storage_1     |  * Running on http://0.0.0.0:5002/ (Press CTRL+C to quit)
swh-storage_1     |  * Restarting with stat
swh-storage-db_1  | 2018-12-13 21:49:06.162 UTC [1] LOG:  database system is ready to accept connections
swh-storage_1     | /usr/local/lib/python3.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
swh-storage_1     |   """)
swh-storage_1     |  * Debugger is active!
swh-storage_1     |  * Debugger PIN: 282-054-931
swh-web_1         | "GET /browse/search/?q=.*&with_visit HTTP/1.1" 200 11313
swh-web_1         | "GET /static/css/vendors.1bb53db16d3be4ade415.css HTTP/1.1" 304 0
swh-web_1         | "GET /static/js/vendors.1bb53db16d3be4ade415.js HTTP/1.1" 304 0
swh-web_1         | "GET /static/js/browse.0914bcf041c43ac18d99.js HTTP/1.1" 304 0
swh-web_1         | "GET /static/css/webapp.255faff414598b3635fe.css HTTP/1.1" 304 0
swh-web_1         | "GET /static/css/browse.0914bcf041c43ac18d99.css HTTP/1.1" 304 0
swh-web_1         | "GET /static/css/vault.5e75f0f4a5fcbfd2fe60.css HTTP/1.1" 304 0
swh-web_1         | "GET /static/js/webapp.255faff414598b3635fe.js HTTP/1.1" 304 0
swh-web_1         | "GET /static/js/vault.5e75f0f4a5fcbfd2fe60.js HTTP/1.1" 304 0
swh-web_1         | "GET /jsreverse/ HTTP/1.1" 200 14225
ardumont added a comment.Dec 13 2018, 10:57 PM

Nice work!
All lights are green from my side.

\m/

ardumont edited the summary of this revision. (Show Details)Dec 13 2018, 11:04 PM
ardumont edited the test plan for this revision. (Show Details)Dec 13 2018, 11:22 PM
ardumont updated this revision to Diff 2598.Dec 13 2018, 11:22 PM
  • swh-web: Remove unneeded dev instruction
  • README: Mention the new webapp service
  • swh-web/Dockerfile: Fix formatting
  • swh-web/entrypoint: Add back comments
Harbormaster completed remote builds in B3023: Diff 2598.Dec 13 2018, 11:23 PM
ardumont mentioned this in T1443: Make swh services run within docker and docker-compose.Dec 14 2018, 8:41 AM
ardumont edited the test plan for this revision. (Show Details)Dec 14 2018, 9:20 AM
douardda accepted this revision.Dec 14 2018, 10:35 AM
This revision is now accepted and ready to land.Dec 14 2018, 10:35 AM
Closed by commit rCDFDb7673d0a4f43: swh-web: Bootstrap swh-web service (authored by ardumont). · Explain WhyDec 14 2018, 10:41 AM
This revision was automatically updated to reflect the committed changes.
ardumont mentioned this in D816: swh-db-init: Make the db initialization idempotent.Dec 14 2018, 12:03 PM
vlorentz added inline comments.Dec 14 2018, 12:28 PM
dockerfiles/swh-web/entrypoint.sh
8

python3 -m swh.web.manage migrate should run here too.

ardumont marked an inline comment as done.Dec 14 2018, 2:09 PM
ardumont added inline comments.
dockerfiles/swh-web/entrypoint.sh
8

Ah yes, it should be tested in the loop, something like:
[ $srcrepo = 'swh-web' ] && ...

ardumont mentioned this in D822: Align swh.indexer.storage.api.server.launch with swh.storage.api.server.launch.Dec 14 2018, 4:03 PM
ardumont added a commit: rDENVb7673d0a4f43: swh-web: Bootstrap swh-web service.Nov 26 2019, 1:34 PM
ardumont added a commit: rDENVd7ace96a3f1f: Remove unused dependency.
ardumont added a commit: rDENV639d8647acff: swh-web: Make the webapp actually serve results.
ardumont added a commit: rDENV56d6f44e2a6d: swh-web: Make the services run against storage and objstorage.
ardumont added a commit: rDENVcb44100fa872: swh-storage: Make the storage work again.
ardumont added a commit: rDENVbf6ed65cdde6: swh-storage/entrypoint.sh: Remove tabs and use spaces.
ardumont added a commit: rDENV97557cb11569: swh-web: Remove unneeded dev instruction.
ardumont added a commit: rDENV2d6c5c3d9f3d: README: Mention the new webapp service.
ardumont added a commit: rDENVb247c11ee016: swh-web/Dockerfile: Fix formatting.
ardumont added a commit: rDENV4ab1c7da2e8e: swh-web/entrypoint: Add back comments.

Revision Contents
Changeset List

PathSize
9 lines
dockerfiles/
swh-web/
23 lines
19 lines
14 lines

Diff 2599

View Options

docker-compose.yml

Loading...
View Options

dockerfiles/swh-web/Dockerfile

Loading...
View Options

dockerfiles/swh-web/entrypoint.sh

Loading...
View Options

dockerfiles/swh-web/web.yml

Loading...
About · Developer information · Legal