Page MenuHomeSoftware Heritage
Differential D2774

docker: Add keycloak service and configure its use in swh-web
ClosedPublic
Actions

Authored by anlambert on Mar 5 2020, 9:15 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDENV2979481c2b28: docker: Add optional keycloak service and configure its use in swh-web
Summary

Add keycloak service in order to manage users authentication and authorization in swh services.

A sample SoftwareHeritage realm exported to JSON format is loaded when the service starts.

Keycloak admin console is available to host through this URL: http://localhost:5080/keycloak/
(user: admin, password: admin).

Some test users are also created in the SoftwareHeritage realm when the swh-web service starts.
Their username/password are listed below:

  • admin/admin (also member of the staff group, which makes swh-web admin features available once logged in)
  • johndoe/johndoe-swh
  • janedoe/janedoe-swh

Login process in swh-web can be tested by applying D2775 locally and using the docker-compose.override.yml file.

Diff Detail

Repository
rDENV Development environment
Branch
docker-add-keycloak-service
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 10938
Build 16456: arc lint + arc unit

Event Timeline

anlambert created this revision.Mar 5 2020, 9:15 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptMar 5 2020, 9:15 PM
Harbormaster completed remote builds in B10938: Diff 9871.Mar 5 2020, 9:15 PM
anlambert retitled this revision from docker: Add keycloak service and configure its use in swh-web Add keycloak service in order to manage users authentication and authorization in swh services. A sample SoftwareHeritage realm exported to JSON format is loaded when the service... to docker: Add keycloak service and configure its use in swh-web.Mar 5 2020, 9:16 PM
anlambert edited the summary of this revision. (Show Details)
anlambert mentioned this in D2131: [POC] Add keycloak service and sample SoftwareHeritage realm.
anlambert mentioned this in D2746: Add OpenID Connect autentication backend and login/logout views.Mar 5 2020, 9:23 PM
anlambert edited the summary of this revision. (Show Details)Mar 5 2020, 9:29 PM
vlorentz added a subscriber: vlorentz.Mar 6 2020, 1:08 AM

Could we make this optional, by using an extra docker-compose file as override (like we currently do for ES or Cassandra)?

It will prevent some bloat, as most users shouldn't need Keycloak.

anlambert added a comment.Mar 6 2020, 2:07 AM
In D2774#66324, @vlorentz wrote:

Could we make this optional, by using an extra docker-compose file as override (like we currently do for ES or Cassandra)?

It will prevent some bloat, as most users shouldn't need Keycloak.

Sure, that diff is far away to be in a landable state. The idea is to make it evolved as I progress on integrating OpenID Connect in swh-web
as it is helpful to simulate a production environment.

anlambert updated this revision to Diff 10075.Mar 16 2020, 3:05 PM

Update:

  • make keycloak service optional, can be launched using $ docker-compose -f docker-compose.yml -f docker-compose.keycloak.yml up -d
  • add new config file for swh-web setting up keycloak and removing local exempted ips from throttling (in order to test permissions)
Harbormaster completed remote builds in B11144: Diff 10075.Mar 16 2020, 3:05 PM
vlorentz requested changes to this revision.Mar 18 2020, 10:49 AM

I wonder if we should remove all default values from swh-realm.json to make it shorter, so it's more obvious what the changes are compared to the default.

.gitignore
1

Why is this needed?

docker/services/keycloak/Dockerfile
3–7

Could you add a comment explaining this is needed to work with the non-root HTTP path?

docker/services/swh-web/keycloak_create_test_users.py
78

please don't use a domain we don't own. You can use @example.org instead

This revision now requires changes to proceed.Mar 18 2020, 10:49 AM
anlambert added a comment.Mar 18 2020, 11:45 AM

I wonder if we should remove all default values from swh-realm.json to make it shorter, so it's more obvious what the changes are compared to the default.

This sounds complicated as the file is generated from the keycloak admin console and the default values will appear again each time a new realm export is performed.
Another solution would be to write a Python script creating and configuring the Software realm from scratch using Keycloak admin REST API (wrapped by python-keycloak)
but this will slow down the startup of the keycloak service.

.gitignore
1

The file docker/services/swh-web/keycloak_create_test_users.py can not be added to the repository without that line

docker/services/keycloak/Dockerfile
3–7

Sure, I will add reference to why that hack is needed when using a reverse proxy

docker/services/swh-web/keycloak_create_test_users.py
78

ack

vlorentz added inline comments.Mar 18 2020, 12:02 PM
.gitignore
1

hmm. What about replacing the first line (swh-*/) with ^swh-*/?

anlambert added inline comments.Mar 18 2020, 12:10 PM
.gitignore
1

I already tried that but then all swh modules fetched with mr became trackable by git and we do not want that, see below:

12:08 $ git status
Sur la branche docker-add-keycloak-service
Modifications qui ne seront pas validées :
  (utilisez "git add <fichier>..." pour mettre à jour ce qui sera validé)
  (utilisez "git checkout -- <fichier>..." pour annuler les modifications dans la copie de travail)

        modifié :         .gitignore
        modifié :         docker/conf/nginx.conf
        modifié :         docker/services/swh-web/entrypoint.sh

Fichiers non suivis:
  (utilisez "git add <fichier>..." pour inclure dans ce qui sera validé)

        docker/conf/keycloak/
        docker/conf/web-keycloak.yml
        docker/docker-compose.keycloak.yml
        docker/env/keycloak-db.env
        docker/env/keycloak.env
        docker/services/keycloak/
        docker/services/swh-web/keycloak_create_test_users.py
        swh-core/
        swh-dataset/
        swh-deposit/
        swh-docs/
        swh-graph/
        swh-icinga-plugins/
        swh-indexer/
        swh-journal/
        swh-lister/
        swh-loader-core/
        swh-loader-git/
        swh-loader-mercurial/
        swh-loader-svn/
        swh-model/
        swh-objstorage/
        swh-py-template/
        swh-scanner/
        swh-scheduler/
        swh-search/
        swh-storage/
        swh-vault/
        swh-web/
anlambert updated this revision to Diff 10130.Mar 18 2020, 3:23 PM

Update:

  • move long JAVA_TOOL_OPTIONS definition from keycloak.env file to docker-compose.keycloak.yml file
  • move custom Keycloak configuration script execution from swh-web service to the keycloak one and rename the file
Harbormaster completed remote builds in B11197: Diff 10130.Mar 18 2020, 3:23 PM
vlorentz added inline comments.Mar 18 2020, 3:27 PM
.gitignore
1

Oh, right. It should be /swh-*/

anlambert added inline comments.Mar 18 2020, 4:06 PM
.gitignore
1

Indeed it works. I moved the file I could not track before so the .gitignore modification is no more needed.

Nevertheless, currently no new files can be added and tracked in docker/services/swh-*/ folders. I will create a separate diff to fix the issue.

anlambert updated this revision to Diff 10133.Mar 18 2020, 4:08 PM

Update: remove .gitignore modification

Harbormaster completed remote builds in B11201: Diff 10133.Mar 18 2020, 4:08 PM
anlambert added inline comments.Mar 18 2020, 4:15 PM
.gitignore
1

D2849

vlorentz accepted this revision.Mar 18 2020, 4:51 PM
This revision is now accepted and ready to land.Mar 18 2020, 4:51 PM
anlambert updated this revision to Diff 10138.Mar 18 2020, 4:53 PM

Rebase

Harbormaster completed remote builds in B11206: Diff 10138.Mar 18 2020, 4:53 PM
anlambert updated this revision to Diff 10306.Mar 26 2020, 11:22 PM

Update: Remove SoftwareHeritage realm JSON export file and prefer to use
Keycloak admin API through python-keycloak to define realm and swh-web
client when keycloak service starts.

Harbormaster completed remote builds in B11387: Diff 10306.Mar 26 2020, 11:22 PM
anlambert updated this revision to Diff 10334.Mar 27 2020, 12:19 PM

Update commit message

Harbormaster completed remote builds in B11414: Diff 10334.Mar 27 2020, 12:19 PM
Closed by commit rDENV2979481c2b28: docker: Add optional keycloak service and configure its use in swh-web (authored by anlambert). · Explain WhyMar 27 2020, 12:25 PM
This revision was automatically updated to reflect the committed changes.
anlambert added a commit: rDENV2979481c2b28: docker: Add optional keycloak service and configure its use in swh-web.

Revision Contents
Changeset List

PathSize
1 line
docker/
conf/
keycloak/
1982 lines
11 lines
4 lines
21 lines
env/
4 lines
10 lines
services/
keycloak/
7 lines
swh-web/
5 lines
161 lines

Diff 9871

View Options

.gitignore

Loading...
View Options

docker/conf/keycloak/swh-realm.json

Loading...
View Options

docker/conf/nginx.conf

Loading...
View Options

docker/conf/web.yml

Loading...
View Options

docker/docker-compose.yml

Loading...
View Options

docker/env/keycloak-db.env

Loading...
View Options

docker/env/keycloak.env

Loading...
View Options

docker/services/keycloak/Dockerfile

Loading...
View Options

docker/services/swh-web/entrypoint.sh

Loading...
View Options

docker/services/swh-web/keycloak_create_test_users.py

Loading...
About · Developer information · Legal