Page MenuHomeSoftware Heritage

keycloak: Update deployment with puppet
ClosedPublic

Authored by anlambert on Oct 8 2020, 9:08 PM.

Details

Summary

That diff contains four commits related to Keycloak deployment and configuration
with puppet:

  • Set some login related realm options to true
  • Add roles to swh-web client (Roles define permissions for a given client).
  • Override direct grant flow for swh-web client by removing the Conditional OTP flow execution. This will prevent users that have configured OTP in their account to get invalid credentials error when trying to generate a bearer token for Web API authentication (experienced by @haltode that morning).
  • Upgrade Keycloak from 8.0.1 to 10.0.2

I tested those changes locally with pupperware and everything worked fine.

Below is the octocatalog diff:

20:14 $ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details -t staging kelvingrove.internal.softwareheritage.org
Found host kelvingrove.internal.softwareheritage.org
Cloning into '/tmp/swh-ocd.7PWUzRsO/environments/production/data/private'...
done.
Cloning into '/tmp/swh-ocd.7PWUzRsO/environments/staging/data/private'...
done.
*** Running octocatalog-diff on host kelvingrove.internal.softwareheritage.org
I, [2020-10-08T20:14:42.141264 #1878250]  INFO -- : Catalogs compiled for kelvingrove.internal.softwareheritage.org
I, [2020-10-08T20:14:42.349763 #1878250]  INFO -- : Diffs computed for kelvingrove.internal.softwareheritage.org
diff origin/production/kelvingrove.internal.softwareheritage.org current/kelvingrove.internal.softwareheritage.org
*******************************************
+ Archive[keycloak-10.0.2.tar.gz] =>
   parameters =>
      "cleanup": true
      "creates": "/opt/keycloak-10.0.2/bin"
      "ensure": "present"
      "extract": true
      "extract_command": "tar xfz %s --strip-components=1"
      "extract_path": "/opt/keycloak-10.0.2"
      "group": "keycloak"
      "path": "/tmp/keycloak-10.0.2.tar.gz"
      "source": "https://downloads.jboss.org/keycloak/10.0.2/keycloak-10.0.2.tar.gz"
      "user": "keycloak"
*******************************************
- Archive[keycloak-8.0.1.tar.gz]
*******************************************
  Concat::Fragment[config.cli-keycloak] =>
   parameters =>
     target =>
      - /opt/keycloak-8.0.1/config.cli
      + /opt/keycloak-10.0.2/config.cli
*******************************************
+ Concat[/opt/keycloak-10.0.2/config.cli] =>
   parameters =>
      "backup": "puppet"
      "ensure": "present"
      "ensure_newline": false
      "force": false
      "format": "plain"
      "group": "keycloak"
      "mode": "0600"
      "notify": "Exec[jboss-cli.sh --file=config.cli]"
      "order": "alpha"
      "owner": "keycloak"
      "path": "/opt/keycloak-10.0.2/config.cli"
      "replace": true
      "show_diff": false
      "warn": false
*******************************************
- Concat[/opt/keycloak-8.0.1/config.cli]
*******************************************
+ Concat_file[/opt/keycloak-10.0.2/config.cli] =>
   parameters =>
      "backup": "puppet"
      "ensure_newline": false
      "force": false
      "format": "plain"
      "group": "keycloak"
      "mode": "0600"
      "order": "alpha"
      "owner": "keycloak"
      "replace": true
      "show_diff": false
      "tag": "_opt_keycloak-10.0.2_config.cli"
*******************************************
- Concat_file[/opt/keycloak-8.0.1/config.cli]
*******************************************
  Concat_fragment[config.cli-keycloak] =>
   parameters =>
     tag =>
      - _opt_keycloak-8.0.1_config.cli
      + _opt_keycloak-10.0.2_config.cli
     target =>
      - /opt/keycloak-8.0.1/config.cli
      + /opt/keycloak-10.0.2/config.cli
*******************************************
  Exec[create-keycloak-admin] =>
   parameters =>
     command =>
      - /opt/keycloak-8.0.1/bin/add-user-keycloak.sh --user keycloak-admin --password keycloak::admin::password --realm master && touch /opt/keycloak-8.0.1/.create-keycloak-admin-postgresql
      + /opt/keycloak-10.0.2/bin/add-user-keycloak.sh --user keycloak-admin --password keycloak::admin::password --realm master && touch /opt/keycloak-10.0.2/.create-keycloak-admin-postgresql
     creates =>
      - /opt/keycloak-8.0.1/.create-keycloak-admin-postgresql
      + /opt/keycloak-10.0.2/.create-keycloak-admin-postgresql
*******************************************
  Exec[jboss-cli.sh --file=config.cli] =>
   parameters =>
     command =>
      - /opt/keycloak-8.0.1/bin/jboss-cli.sh --file=config.cli
      + /opt/keycloak-10.0.2/bin/jboss-cli.sh --file=config.cli
     cwd =>
      - /opt/keycloak-8.0.1
      + /opt/keycloak-10.0.2
*******************************************
+ Exec[mkdir -p /opt/keycloak-10.0.2/modules/system/layers/keycloak/org/postgresql/main] =>
   parameters =>
      "creates": "/opt/keycloak-10.0.2/modules/system/layers/keycloak/org/postgresql/main"
      "group": "keycloak"
      "path": "/usr/bin:/bin"
      "user": "keycloak"
*******************************************
- Exec[mkdir -p /opt/keycloak-8.0.1/modules/system/layers/keycloak/org/postgresql/main]
*******************************************
  File[/etc/systemd/system/keycloak.service] =>
   parameters =>
     content =>
      @@ -7,5 +7,5 @@
       User=keycloak
       Group=keycloak
      -ExecStart=/opt/keycloak-8.0.1/bin/standalone.sh -b 0.0.0.0 -Djboss.http.port=8080
      +ExecStart=/opt/keycloak-10.0.2/bin/standalone.sh -b 0.0.0.0 -Djboss.http.port=8080
       TimeoutStartSec=600
       TimeoutStopSec=600
*******************************************
+ File[/opt/keycloak-10.0.2/bin/kcadm-wrapper.sh] =>
   parameters =>
      "ensure": "file"
      "group": "keycloak"
      "mode": "0750"
      "owner": "keycloak"
      "show_diff": false
      "content": >>>
#!/bin/bash

KCADM="/opt/keycloak-10.0.2/bin/kcadm.sh"

${KCADM} "$@" --no-config --server http://localhost:8080/auth --realm master --user keycloak-admin --password keycloak::admin::password
<<<
*******************************************
+ File[/opt/keycloak-10.0.2/modules/system/layers/keycloak/org/postgresql/main/module.xml] =>
   parameters =>
      "ensure": "file"
      "group": "keycloak"
      "mode": "0644"
      "owner": "keycloak"
      "content": >>>
<?xml version="1.0" ?>
<module xmlns="urn:jboss:module:1.3" name="org.postgresql">

    <resources>
        <resource-root path="postgresql-jdbc.jar"/>
    </resources>

    <dependencies>
        <module name="javax.api"/>
        <module name="javax.transaction.api"/>
    </dependencies>
</module>
<<<
*******************************************
+ File[/opt/keycloak-10.0.2/modules/system/layers/keycloak/org/postgresql/main/postgresql-jdbc.jar] =>
   parameters =>
      "ensure": "file"
      "group": "keycloak"
      "mode": "0644"
      "owner": "keycloak"
      "source": "/usr/share/java/postgresql.jar"
*******************************************
+ File[/opt/keycloak-10.0.2/modules/system/layers/keycloak/org/postgresql/main] =>
   parameters =>
      "ensure": "directory"
      "group": "keycloak"
      "mode": "0755"
      "owner": "keycloak"
*******************************************
+ File[/opt/keycloak-10.0.2/standalone/configuration/profile.properties] =>
   parameters =>
      "ensure": "file"
      "group": "keycloak"
      "mode": "0644"
      "notify": "Class[Keycloak::Service]"
      "owner": "keycloak"
      "content": >>>
# File managed by Puppet - DO NOT EDIT
<<<
*******************************************
+ File[/opt/keycloak-10.0.2/standalone/configuration] =>
   parameters =>
      "ensure": "directory"
      "group": "keycloak"
      "mode": "0750"
      "owner": "keycloak"
*******************************************
+ File[/opt/keycloak-10.0.2/themes/swh] =>
   parameters =>
      "ensure": "link"
      "target": "/opt/swh-keycloak-theme/swh"
*******************************************
+ File[/opt/keycloak-10.0.2/tmp] =>
   parameters =>
      "ensure": "directory"
      "group": "keycloak"
      "mode": "0755"
      "owner": "keycloak"
*******************************************
+ File[/opt/keycloak-10.0.2] =>
   parameters =>
      "ensure": "directory"
      "group": "keycloak"
      "mode": "0755"
      "owner": "keycloak"
*******************************************
- File[/opt/keycloak-8.0.1/bin/kcadm-wrapper.sh]
*******************************************
- File[/opt/keycloak-8.0.1/modules/system/layers/keycloak/org/postgresql/main/module.xml]
*******************************************
- File[/opt/keycloak-8.0.1/modules/system/layers/keycloak/org/postgresql/main/postgresql-jdbc.jar]
*******************************************
- File[/opt/keycloak-8.0.1/modules/system/layers/keycloak/org/postgresql/main]
*******************************************
- File[/opt/keycloak-8.0.1/standalone/configuration/profile.properties]
*******************************************
- File[/opt/keycloak-8.0.1/standalone/configuration]
*******************************************
- File[/opt/keycloak-8.0.1/themes/swh]
*******************************************
- File[/opt/keycloak-8.0.1/tmp]
*******************************************
- File[/opt/keycloak-8.0.1]
*******************************************
  File[/opt/keycloak] =>
   parameters =>
     target =>
      - /opt/keycloak-8.0.1
      + /opt/keycloak-10.0.2
*******************************************
  File_line[standalone.conf-JAVA_OPTS] =>
   parameters =>
     path =>
      - /opt/keycloak-8.0.1/bin/standalone.conf
      + /opt/keycloak-10.0.2/bin/standalone.conf
*******************************************
  Keycloak_client[swh-web on SoftwareHeritageStaging] =>
   parameters =>
     direct_grant_flow =>
      + direct_grant_no_otp-SoftwareHeritageStaging
     roles =>
      + ["swh.web.api.throtlling_exempted", "swh.web.api.graph"]
*******************************************
  Keycloak_client[swh-web on SoftwareHeritage] =>
   parameters =>
     direct_grant_flow =>
      + direct_grant_no_otp-SoftwareHeritage
     roles =>
      + ["swh.web.api.throtlling_exempted", "swh.web.api.graph"]
*******************************************
+ Keycloak_flow[direct_grant_no_otp on SoftwareHeritageStaging] =>
   parameters =>
      "alias": "direct_grant_no_otp-SoftwareHeritageStaging"
      "description": "Direct grant flow without conditional OTP"
      "ensure": "present"
      "id": "d6a91808-4cad-5e18-a48e-7e48e4281edb"
      "realm": "SoftwareHeritageStaging"
*******************************************
+ Keycloak_flow[direct_grant_no_otp on SoftwareHeritage] =>
   parameters =>
      "alias": "direct_grant_no_otp-SoftwareHeritage"
      "description": "Direct grant flow without conditional OTP"
      "ensure": "present"
      "id": "cff702ba-f497-5298-b244-4b1519bb8799"
      "realm": "SoftwareHeritage"
*******************************************
+ Keycloak_flow_execution[direct-grant-validate-password under direct_grant_no_otp-SoftwareHeritage on SoftwareHeritage] =>
   parameters =>
      "alias": "direct-grant-validate-password-SoftwareHeritage"
      "ensure": "present"
      "flow_alias": "direct_grant_no_otp-SoftwareHeritage"
      "id": "a288cfe6-fa66-585e-8e50-2babc5f764b8"
      "index": 0
      "provider_id": "direct-grant-validate-password"
      "realm": "SoftwareHeritage"
      "requirement": "REQUIRED"
*******************************************
+ Keycloak_flow_execution[direct-grant-validate-password under direct_grant_no_otp-SoftwareHeritageStaging on SoftwareHeritageStaging] =>
   parameters =>
      "alias": "direct-grant-validate-password-SoftwareHeritageStaging"
      "ensure": "present"
      "flow_alias": "direct_grant_no_otp-SoftwareHeritageStaging"
      "id": "39e8b167-5328-5c95-9cd3-d15b6ae85c83"
      "index": 0
      "provider_id": "direct-grant-validate-password"
      "realm": "SoftwareHeritageStaging"
      "requirement": "REQUIRED"
*******************************************
+ Keycloak_flow_execution[direct-grant-validate-username under direct_grant_no_otp-SoftwareHeritage on SoftwareHeritage] =>
   parameters =>
      "alias": "direct-grant-validate-username-SoftwareHeritage"
      "ensure": "present"
      "flow_alias": "direct_grant_no_otp-SoftwareHeritage"
      "id": "d6b5deea-503c-5041-ad59-c9a4ecb21344"
      "index": 0
      "provider_id": "direct-grant-validate-username"
      "realm": "SoftwareHeritage"
      "requirement": "REQUIRED"
*******************************************
+ Keycloak_flow_execution[direct-grant-validate-username under direct_grant_no_otp-SoftwareHeritageStaging on SoftwareHeritageStaging] =>
   parameters =>
      "alias": "direct-grant-validate-username-SoftwareHeritageStaging"
      "ensure": "present"
      "flow_alias": "direct_grant_no_otp-SoftwareHeritageStaging"
      "id": "f77d5d0b-71fc-5c28-aff5-72f32eac709a"
      "index": 0
      "provider_id": "direct-grant-validate-username"
      "realm": "SoftwareHeritageStaging"
      "requirement": "REQUIRED"
*******************************************
  Keycloak_realm[SoftwareHeritageStaging] =>
   parameters =>
     registration_allowed =>
      + true
     reset_password_allowed =>
      + true
     verify_email =>
      + true
*******************************************
  Keycloak_realm[SoftwareHeritage] =>
   parameters =>
     registration_allowed =>
      + true
     reset_password_allowed =>
      + true
     verify_email =>
      + true
*******************************************
  Systemd::Unit_file[keycloak.service] =>
   parameters =>
     content =>
      @@ -7,5 +7,5 @@
       User=keycloak
       Group=keycloak
      -ExecStart=/opt/keycloak-8.0.1/bin/standalone.sh -b 0.0.0.0 -Djboss.http.port=8080
      +ExecStart=/opt/keycloak-10.0.2/bin/standalone.sh -b 0.0.0.0 -Djboss.http.port=8080
       TimeoutStartSec=600
       TimeoutStopSec=600
*******************************************
*** End octocatalog-diff on kelvingrove.internal.softwareheritage.org

Diff Detail

Repository
rSPSITE puppet-swh-site
Branch
staging
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 16098
Build 24762: arc lint + arc unit

Event Timeline

Thanks!

I'm a bit concerned about the keycloak upgrade process:

Have you tested the actual upgrade process in pupperware?

In D4211#104343, @olasd wrote:

Have you tested the actual upgrade process in pupperware?

Btw, it's okay if the answer is "no"; we'll just make sure to do the deployment in prod under supervision, and manually run a migration script if needed (and then we'll *know* :))

The latest upstream version is 11.0.2, but I guess this is blocked on https://github.com/treydock/puppet-module-keycloak/pull/154; Could you add a comment where the keycloak version is defined?

Yes, there is an issue with flows update when using Keycloak 11.x so I preferred to upgrade to 10.x. What do you mean by adding a comment on the version definition ?

I was looking at upstream docs which say a migration script needs to be executed: https://www.keycloak.org/docs/latest/upgrading/, and I got worried. But the puppet module claims it can handle upgrades, although I don't see it setup the postgres connector to do the database upgrades, so we'll need to double check.

The postgres connector seems to be correctly setup when upgrading, see below:

+ File[/opt/keycloak-10.0.2/modules/system/layers/keycloak/org/postgresql/main/module.xml] =>
   parameters =>
      "ensure": "file"
      "group": "keycloak"
      "mode": "0644"
      "owner": "keycloak"
      "content": >>>
<?xml version="1.0" ?>
<module xmlns="urn:jboss:module:1.3" name="org.postgresql">

    <resources>
        <resource-root path="postgresql-jdbc.jar"/>
    </resources>

    <dependencies>
        <module name="javax.api"/>
        <module name="javax.transaction.api"/>
    </dependencies>
</module>
<<<
*******************************************
+ File[/opt/keycloak-10.0.2/modules/system/layers/keycloak/org/postgresql/main/postgresql-jdbc.jar] =>
   parameters =>
      "ensure": "file"
      "group": "keycloak"
      "mode": "0644"
      "owner": "keycloak"
      "source": "/usr/share/java/postgresql.jar"
*******************************************
+ File[/opt/keycloak-10.0.2/modules/system/layers/keycloak/org/postgresql/main] =>
   parameters =>
      "ensure": "directory"
      "group": "keycloak"
      "mode": "0755"
      "owner": "keycloak"

Have you tested the actual upgrade process in pupperware?

Yes of course. I first ran puppet a first time to get to the same state as we currently are in production and created a couple of users. Then I applied the changes in that diff and ran puppet again. The upgrade was successful and created users still available in the database afterwards.

The latest upstream version is 11.0.2, but I guess this is blocked on https://github.com/treydock/puppet-module-keycloak/pull/154; Could you add a comment where the keycloak version is defined?

Yes, there is an issue with flows update when using Keycloak 11.x so I preferred to upgrade to 10.x. What do you mean by adding a comment on the version definition ?

I've added an inline comment to be clearer.

I was looking at upstream docs which say a migration script needs to be executed: https://www.keycloak.org/docs/latest/upgrading/, and I got worried. But the puppet module claims it can handle upgrades, although I don't see it setup the postgres connector to do the database upgrades, so we'll need to double check.

The postgres connector seems to be correctly setup when upgrading, see below:

+ File[/opt/keycloak-10.0.2/modules/system/layers/keycloak/org/postgresql/main/module.xml] =>
   parameters =>
      "ensure": "file"
      "group": "keycloak"
      "mode": "0644"
      "owner": "keycloak"
      "content": >>>
<?xml version="1.0" ?>
<module xmlns="urn:jboss:module:1.3" name="org.postgresql">

    <resources>
        <resource-root path="postgresql-jdbc.jar"/>
    </resources>

    <dependencies>
        <module name="javax.api"/>
        <module name="javax.transaction.api"/>
    </dependencies>
</module>
<<<
*******************************************
+ File[/opt/keycloak-10.0.2/modules/system/layers/keycloak/org/postgresql/main/postgresql-jdbc.jar] =>
   parameters =>
      "ensure": "file"
      "group": "keycloak"
      "mode": "0644"
      "owner": "keycloak"
      "source": "/usr/share/java/postgresql.jar"
*******************************************
+ File[/opt/keycloak-10.0.2/modules/system/layers/keycloak/org/postgresql/main] =>
   parameters =>
      "ensure": "directory"
      "group": "keycloak"
      "mode": "0755"
      "owner": "keycloak"

I meant that I haven't found anything that looks like https://www.keycloak.org/docs/latest/upgrading/#automatic-relational-database-migration (either the spi setting or the CLI command).

Have you tested the actual upgrade process in pupperware?

Yes of course. I first ran puppet a first time to get to the same state as we currently are in production and created a couple of users. Then I applied the changes in that diff and ran puppet again. The upgrade was successful and created users still available in the database afterwards.

Awesome! Then I think we're good to go.

data/common/common.yaml
2902–2904

Add a comment to this line explaining why we stick to 10.0.2 instead of 11.0.2

This revision is now accepted and ready to land.Oct 9 2020, 11:38 AM

Uodate: Rebase and add comment about Keycloak version

Uodate: Rebase and add comment about Keycloak version

Awesome!