Details

Reviewers

vlorentz

Group Reviewers

Reviewers

Commits

rDENV2979481c2b28: docker: Add optional keycloak service and configure its use in swh-web

Summary

Add keycloak service in order to manage users authentication and authorization in swh services.

A sample SoftwareHeritage realm exported to JSON format is loaded when the service starts.

Keycloak admin console is available to host through this URL: http://localhost:5080/keycloak/
(user: admin, password: admin).

Some test users are also created in the SoftwareHeritage realm when the swh-web service starts.
Their username/password are listed below:

admin/admin (also member of the staff group, which makes swh-web admin features available once logged in)
johndoe/johndoe-swh
janedoe/janedoe-swh

Diff Detail

Repository

rDENV Development environment

Branch

docker-add-keycloak-service

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 11206
Build 16921: arc lint + arc unit

Event Timeline

anlambert created this revision.Mar 5 2020, 9:15 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptMar 5 2020, 9:15 PM

Harbormaster completed remote builds in B10938: Diff 9871.Mar 5 2020, 9:15 PM

anlambert retitled this revision from docker: Add keycloak service and configure its use in swh-web Add keycloak service in order to manage users authentication and authorization in swh services. A sample SoftwareHeritage realm exported to JSON format is loaded when the service... to docker: Add keycloak service and configure its use in swh-web.Mar 5 2020, 9:16 PM

anlambert edited the summary of this revision. (Show Details)

anlambert mentioned this in D2131: [POC] Add keycloak service and sample SoftwareHeritage realm.

anlambert mentioned this in D2746: Add OpenID Connect autentication backend and login/logout views.Mar 5 2020, 9:23 PM

anlambert edited the summary of this revision. (Show Details)Mar 5 2020, 9:29 PM

Could we make this optional, by using an extra docker-compose file as override (like we currently do for ES or Cassandra)?

It will prevent some bloat, as most users shouldn't need Keycloak.

In D2774#66324, @vlorentz wrote:

Could we make this optional, by using an extra docker-compose file as override (like we currently do for ES or Cassandra)?

It will prevent some bloat, as most users shouldn't need Keycloak.

Sure, that diff is far away to be in a landable state. The idea is to make it evolved as I progress on integrating OpenID Connect in swh-web
as it is helpful to simulate a production environment.

Update:

make keycloak service optional, can be launched using $ docker-compose -f docker-compose.yml -f docker-compose.keycloak.yml up -d
add new config file for swh-web setting up keycloak and removing local exempted ips from throttling (in order to test permissions)

Harbormaster completed remote builds in B11144: Diff 10075.Mar 16 2020, 3:05 PM

I wonder if we should remove all default values from swh-realm.json to make it shorter, so it's more obvious what the changes are compared to the default.

.gitignore
2 ↗	(On Diff #10075)	Why is this needed?
docker/services/keycloak/Dockerfile
3–7	Could you add a comment explaining this is needed to work with the non-root HTTP path?
docker/services/swh-web/keycloak_create_test_users.py
78 ↗	(On Diff #10075)	please don't use a domain we don't own. You can use @example.org instead

This revision now requires changes to proceed.Mar 18 2020, 10:49 AM

I wonder if we should remove all default values from swh-realm.json to make it shorter, so it's more obvious what the changes are compared to the default.

This sounds complicated as the file is generated from the keycloak admin console and the default values will appear again each time a new realm export is performed.
Another solution would be to write a Python script creating and configuring the Software realm from scratch using Keycloak admin REST API (wrapped by python-keycloak)
but this will slow down the startup of the keycloak service.

.gitignore
2 ↗	(On Diff #10075)	The file `docker/services/swh-web/keycloak_create_test_users.py` can not be added to the repository without that line
docker/services/keycloak/Dockerfile
3–7	Sure, I will add reference to why that hack is needed when using a reverse proxy
docker/services/swh-web/keycloak_create_test_users.py
78 ↗	(On Diff #10075)	ack

vlorentz added inline comments.Mar 18 2020, 12:02 PM

.gitignore
2 ↗	(On Diff #10075)	hmm. What about replacing the first line (`swh-/`) with `^swh-/`?

anlambert added inline comments.Mar 18 2020, 12:10 PM

.gitignore

2 ↗

(On Diff #10075)

I already tried that but then all swh modules fetched with mr became trackable by git and we do not want that, see below:

12:08 $ git status
Sur la branche docker-add-keycloak-service
Modifications qui ne seront pas validées :
  (utilisez "git add <fichier>..." pour mettre à jour ce qui sera validé)
  (utilisez "git checkout -- <fichier>..." pour annuler les modifications dans la copie de travail)

        modifié :         .gitignore
        modifié :         docker/conf/nginx.conf
        modifié :         docker/services/swh-web/entrypoint.sh

Fichiers non suivis:
  (utilisez "git add <fichier>..." pour inclure dans ce qui sera validé)

        docker/conf/keycloak/
        docker/conf/web-keycloak.yml
        docker/docker-compose.keycloak.yml
        docker/env/keycloak-db.env
        docker/env/keycloak.env
        docker/services/keycloak/
        docker/services/swh-web/keycloak_create_test_users.py
        swh-core/
        swh-dataset/
        swh-deposit/
        swh-docs/
        swh-graph/
        swh-icinga-plugins/
        swh-indexer/
        swh-journal/
        swh-lister/
        swh-loader-core/
        swh-loader-git/
        swh-loader-mercurial/
        swh-loader-svn/
        swh-model/
        swh-objstorage/
        swh-py-template/
        swh-scanner/
        swh-scheduler/
        swh-search/
        swh-storage/
        swh-vault/
        swh-web/

Update:

address @vlorentz comments

move long JAVA_TOOL_OPTIONS definition from keycloak.env file to docker-compose.keycloak.yml file

move custom Keycloak configuration script execution from swh-web service to the keycloak one and rename the file

Harbormaster completed remote builds in B11197: Diff 10130.Mar 18 2020, 3:23 PM

vlorentz added inline comments.Mar 18 2020, 3:27 PM

.gitignore
2 ↗	(On Diff #10075)	Oh, right. It should be `/swh-*/`

anlambert added inline comments.Mar 18 2020, 4:06 PM

.gitignore
2 ↗	(On Diff #10075)	Indeed it works. I moved the file I could not track before so the `.gitignore` modification is no more needed. Nevertheless, currently no new files can be added and tracked in `docker/services/swh-*/` folders. I will create a separate diff to fix the issue.

Update: remove .gitignore modification

Harbormaster completed remote builds in B11201: Diff 10133.Mar 18 2020, 4:08 PM

anlambert added inline comments.Mar 18 2020, 4:15 PM

.gitignore
2 ↗	(On Diff #10075)	D2849

vlorentz accepted this revision.Mar 18 2020, 4:51 PM

This revision is now accepted and ready to land.Mar 18 2020, 4:51 PM

Rebase

Harbormaster completed remote builds in B11206: Diff 10138.Mar 18 2020, 4:53 PM

Update: Remove SoftwareHeritage realm JSON export file and prefer to use
Keycloak admin API through python-keycloak to define realm and swh-web
client when keycloak service starts.

Harbormaster completed remote builds in B11387: Diff 10306.Mar 26 2020, 11:22 PM

Update commit message

Harbormaster completed remote builds in B11414: Diff 10334.Mar 27 2020, 12:19 PM

Closed by commit rDENV2979481c2b28: docker: Add optional keycloak service and configure its use in swh-web (authored by anlambert). · Explain WhyMar 27 2020, 12:25 PM

This revision was automatically updated to reflect the committed changes.

anlambert added a commit: rDENV2979481c2b28: docker: Add optional keycloak service and configure its use in swh-web.

docker: Add keycloak service and configure its use in swh-web
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 10138

docker/conf/keycloak/swh-realm.json

docker/conf/nginx.conf

docker/conf/web-keycloak.yml

docker/docker-compose.keycloak.yml

docker/env/keycloak-db.env

docker/env/keycloak.env

docker/services/keycloak/Dockerfile

docker/services/keycloak/entrypoint.sh

docker/services/keycloak/keycloak_swh_setup.py

docker/services/swh-web/entrypoint.sh

docker: Add keycloak service and configure its use in swh-webClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 10138

docker/conf/keycloak/swh-realm.json

docker/conf/nginx.conf

docker/conf/web-keycloak.yml

docker/docker-compose.keycloak.yml

docker/env/keycloak-db.env

docker/env/keycloak.env

docker/services/keycloak/Dockerfile

docker/services/keycloak/entrypoint.sh

docker/services/keycloak/keycloak_swh_setup.py

docker/services/swh-web/entrypoint.sh

docker: Add keycloak service and configure its use in swh-web
ClosedPublic
Actions

Revision Contents
Changeset List