Page MenuHomeSoftware Heritage

docker: Add keycloak service and configure its use in swh-web
ClosedPublic

Authored by anlambert on Mar 5 2020, 9:15 PM.

Details

Summary

Add keycloak service in order to manage users authentication and authorization in swh services.

A sample SoftwareHeritage realm exported to JSON format is loaded when the service starts.

Keycloak admin console is available to host through this URL: http://localhost:5080/keycloak/
(user: admin, password: admin).

Some test users are also created in the SoftwareHeritage realm when the swh-web service starts.
Their username/password are listed below:

  • admin/admin (also member of the staff group, which makes swh-web admin features available once logged in)
  • johndoe/johndoe-swh
  • janedoe/janedoe-swh

Login process in swh-web can be tested by applying D2775 locally and using the docker-compose.override.yml file.

Diff Detail

Repository
rDENV Development environment
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Mar 5 2020, 9:15 PM
anlambert retitled this revision from docker: Add keycloak service and configure its use in swh-web Add keycloak service in order to manage users authentication and authorization in swh services. A sample SoftwareHeritage realm exported to JSON format is loaded when the service... to docker: Add keycloak service and configure its use in swh-web.Mar 5 2020, 9:16 PM
anlambert edited the summary of this revision. (Show Details)
anlambert edited the summary of this revision. (Show Details)Mar 5 2020, 9:29 PM

Could we make this optional, by using an extra docker-compose file as override (like we currently do for ES or Cassandra)?

It will prevent some bloat, as most users shouldn't need Keycloak.

Could we make this optional, by using an extra docker-compose file as override (like we currently do for ES or Cassandra)?

It will prevent some bloat, as most users shouldn't need Keycloak.

Sure, that diff is far away to be in a landable state. The idea is to make it evolved as I progress on integrating OpenID Connect in swh-web
as it is helpful to simulate a production environment.

anlambert updated this revision to Diff 10075.Mar 16 2020, 3:05 PM

Update:

  • make keycloak service optional, can be launched using $ docker-compose -f docker-compose.yml -f docker-compose.keycloak.yml up -d
  • add new config file for swh-web setting up keycloak and removing local exempted ips from throttling (in order to test permissions)
vlorentz requested changes to this revision.Mar 18 2020, 10:49 AM

I wonder if we should remove all default values from swh-realm.json to make it shorter, so it's more obvious what the changes are compared to the default.

.gitignore
2 ↗(On Diff #10075)

Why is this needed?

docker/services/keycloak/Dockerfile
4–8

Could you add a comment explaining this is needed to work with the non-root HTTP path?

docker/services/swh-web/keycloak_create_test_users.py
78 ↗(On Diff #10075)

please don't use a domain we don't own. You can use @example.org instead

This revision now requires changes to proceed.Mar 18 2020, 10:49 AM

I wonder if we should remove all default values from swh-realm.json to make it shorter, so it's more obvious what the changes are compared to the default.

This sounds complicated as the file is generated from the keycloak admin console and the default values will appear again each time a new realm export is performed.
Another solution would be to write a Python script creating and configuring the Software realm from scratch using Keycloak admin REST API (wrapped by python-keycloak)
but this will slow down the startup of the keycloak service.

.gitignore
2 ↗(On Diff #10075)

The file docker/services/swh-web/keycloak_create_test_users.py can not be added to the repository without that line

docker/services/keycloak/Dockerfile
4–8

Sure, I will add reference to why that hack is needed when using a reverse proxy

docker/services/swh-web/keycloak_create_test_users.py
78 ↗(On Diff #10075)

ack

vlorentz added inline comments.Mar 18 2020, 12:02 PM
.gitignore
2 ↗(On Diff #10075)

hmm. What about replacing the first line (swh-*/) with ^swh-*/?

anlambert added inline comments.Mar 18 2020, 12:10 PM
.gitignore
2 ↗(On Diff #10075)

I already tried that but then all swh modules fetched with mr became trackable by git and we do not want that, see below:

12:08 $ git status
Sur la branche docker-add-keycloak-service
Modifications qui ne seront pas validées :
  (utilisez "git add <fichier>..." pour mettre à jour ce qui sera validé)
  (utilisez "git checkout -- <fichier>..." pour annuler les modifications dans la copie de travail)

        modifié :         .gitignore
        modifié :         docker/conf/nginx.conf
        modifié :         docker/services/swh-web/entrypoint.sh

Fichiers non suivis:
  (utilisez "git add <fichier>..." pour inclure dans ce qui sera validé)

        docker/conf/keycloak/
        docker/conf/web-keycloak.yml
        docker/docker-compose.keycloak.yml
        docker/env/keycloak-db.env
        docker/env/keycloak.env
        docker/services/keycloak/
        docker/services/swh-web/keycloak_create_test_users.py
        swh-core/
        swh-dataset/
        swh-deposit/
        swh-docs/
        swh-graph/
        swh-icinga-plugins/
        swh-indexer/
        swh-journal/
        swh-lister/
        swh-loader-core/
        swh-loader-git/
        swh-loader-mercurial/
        swh-loader-svn/
        swh-model/
        swh-objstorage/
        swh-py-template/
        swh-scanner/
        swh-scheduler/
        swh-search/
        swh-storage/
        swh-vault/
        swh-web/
anlambert updated this revision to Diff 10130.Mar 18 2020, 3:23 PM

Update:

  • move long JAVA_TOOL_OPTIONS definition from keycloak.env file to docker-compose.keycloak.yml file
  • move custom Keycloak configuration script execution from swh-web service to the keycloak one and rename the file
vlorentz added inline comments.Mar 18 2020, 3:27 PM
.gitignore
2 ↗(On Diff #10075)

Oh, right. It should be /swh-*/

anlambert added inline comments.Mar 18 2020, 4:06 PM
.gitignore
2 ↗(On Diff #10075)

Indeed it works. I moved the file I could not track before so the .gitignore modification is no more needed.

Nevertheless, currently no new files can be added and tracked in docker/services/swh-*/ folders. I will create a separate diff to fix the issue.

anlambert updated this revision to Diff 10133.Mar 18 2020, 4:08 PM

Update: remove .gitignore modification

anlambert added inline comments.Mar 18 2020, 4:15 PM
.gitignore
2 ↗(On Diff #10075)
vlorentz accepted this revision.Mar 18 2020, 4:51 PM
This revision is now accepted and ready to land.Mar 18 2020, 4:51 PM
anlambert updated this revision to Diff 10306.Mar 26 2020, 11:22 PM

Update: Remove SoftwareHeritage realm JSON export file and prefer to use
Keycloak admin API through python-keycloak to define realm and swh-web
client when keycloak service starts.

anlambert updated this revision to Diff 10334.Mar 27 2020, 12:19 PM

Update commit message