Page MenuHomeSoftware Heritage

Allow to configure authentication mechanism per configuration file
ClosedPublic

Authored by ardumont on Mar 19 2021, 4:08 PM.

Details

Summary

This will allow to deploy on staging with keycloak while continue using the deposit with
http basic authentication in production.

If some urgency fix is needed, we won't be forced into migrating keycloak immediately
to fix that potential other issue.

Note that the following transitional code will probably go away when we are done
migrating to keycloak.

Related to T2858
Related to D5284

Test Plan

tox

in docker

Diff Detail

Repository
rDDEP Push deposit
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D5291 (id=18940)

Could not rebase; Attempt merge onto 3a9b2fc4ba...

Updating 3a9b2fc4..db75a381
Fast-forward
 requirements-swh-server.txt                        |   1 +
 swh/deposit/api/common.py                          |  32 ++--
 swh/deposit/api/private/__init__.py                |  12 +-
 swh/deposit/api/service_document.py                |  27 ++-
 swh/deposit/auth.py                                | 112 ++++++++++++-
 swh/deposit/cli/admin.py                           |   8 +-
 swh/deposit/config.py                              |   1 -
 swh/deposit/exception.py                           |   5 +-
 swh/deposit/models.py                              |   8 +-
 swh/deposit/settings/common.py                     |   4 +-
 swh/deposit/settings/production.py                 |  27 ++-
 swh/deposit/settings/testing.py                    |   7 +-
 swh/deposit/tests/api/test_collection.py           |  12 +-
 .../tests/api/test_collection_add_to_origin.py     |  10 +-
 .../tests/api/test_collection_reuse_slug.py        |  22 +--
 swh/deposit/tests/api/test_exception.py            |   4 +-
 swh/deposit/tests/api/test_get_file.py             |  10 +-
 swh/deposit/tests/api/test_service_document.py     |   8 +-
 swh/deposit/tests/cli/test_admin.py                |   2 +-
 swh/deposit/tests/conftest.py                      | 186 ++++++++++++++++-----
 swh/deposit/tests/test_backend.py                  |  71 ++++++++
 21 files changed, 453 insertions(+), 116 deletions(-)
 create mode 100644 swh/deposit/tests/test_backend.py
Changes applied before test
commit db75a381c60687f3dfb848dc60df45ba963071e2
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Mar 19 16:05:48 2021 +0100

    Allow to configure authentication mechanism per configuration file
    
    This will allow to deploy on staging with keycloak and continue using the deposit with
    http basic authentication.
    
    Related to T2858

commit 27efcccbfaea7188b05e6ce18e201ec9a2ddfb79
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Feb 18 16:57:55 2021 +0100

    Delegate authentication to keycloak
    
    This reworks the internal authentication mechanism to delegate that part to keycloak.
    Through a Direct Grant access flow.
    
    It's transparent for the deposit clients. They will continue their authentication
    against the deposit server. Internally, the deposit server will require authentication
    on their behalf.
    
    Related to T2858

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/578/ for more details.

vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/deposit/api/common.py
176–180

missing tests

189

else: error

swh/deposit/api/service_document.py
20–25

why is this not in a middleware?

This revision now requires changes to proceed.Mar 19 2021, 4:15 PM
anlambert added a subscriber: anlambert.
anlambert added inline comments.
swh/deposit/api/service_document.py
20–25

Not a big fan of that either.

I think you can set the AUTH_USER_MODEL django setting and the rest_framework.authentication.BasicAuthentication backend
should return the user properly typed.

swh/deposit/api/common.py
176–180

i know.

But i'm a bit worried:

  1. making the existing tests using both keycloak and basic authentication scheme will take forever.
  2. i'll *never* deploy what i'm aiming at...
189

I Iike your confidence. But I'm not so sure about that.

We still have the private api which wants none of those (thus the override later).
I'm not sure we want the fallback to empty anyway either...

swh/deposit/api/service_document.py
20–25

That's a "fast" workaround.

The new keycloak thing does the right thing.
But the older one (which i'm reintroducing) did not ¯\_(ツ)_/¯.
I'll have a look at what antoine suggested.

swh/deposit/api/service_document.py
20–25

well, i fear that's another rabbit hole.

Adding:

AUTH_USER_MODEL = "swh.deposit.models.DepositClient"

Then, the container won't restart, it's not too happy about the new oidc_user attribute.

TypeError: OIDCUser cannot proxy the swapped model 'swh.deposit.models.DepositClient'.
swh/deposit/settings/common.py
105–106

I am not sure this setting is needed as you explicitly set the authentication backend in the APIView class.

swh/deposit/settings/production.py
99–101

same here, I think you can remove that setting.

swh/deposit/settings/testing.py
44

same here

swh/deposit/settings/common.py
105–106

oh that'd be neat indeed, less code in those files.

swh/deposit/api/common.py
176–180

I'll keep it simple, at least 1 test which loads a config for basic and checks that the classes are the expected ones.
(and maybe another for the keycloak thingy).

swh/deposit/api/service_document.py
20–25

Maybe you could use that workaround ?

  • Drop some redundant settings
  • Activate back the password as explain in comments

Build is green

Patch application report for D5291 (id=18943)

Could not rebase; Attempt merge onto 3a9b2fc4ba...

Updating 3a9b2fc4..2ced88bb
Fast-forward
 requirements-swh-server.txt                        |   1 +
 swh/deposit/api/common.py                          |  32 ++--
 swh/deposit/api/private/__init__.py                |  12 +-
 swh/deposit/api/service_document.py                |  27 ++-
 swh/deposit/auth.py                                | 112 ++++++++++++-
 swh/deposit/cli/admin.py                           |  10 +-
 swh/deposit/config.py                              |   1 -
 swh/deposit/exception.py                           |   5 +-
 swh/deposit/models.py                              |   8 +-
 swh/deposit/settings/common.py                     |   7 +-
 swh/deposit/settings/production.py                 |  23 ++-
 swh/deposit/settings/testing.py                    |   6 +-
 swh/deposit/tests/api/test_collection.py           |  12 +-
 .../tests/api/test_collection_add_to_origin.py     |  10 +-
 .../tests/api/test_collection_reuse_slug.py        |  22 +--
 swh/deposit/tests/api/test_exception.py            |   4 +-
 swh/deposit/tests/api/test_get_file.py             |  10 +-
 swh/deposit/tests/api/test_service_document.py     |   8 +-
 swh/deposit/tests/cli/test_admin.py                |   4 +-
 swh/deposit/tests/conftest.py                      | 186 ++++++++++++++++-----
 swh/deposit/tests/test_backend.py                  |  71 ++++++++
 21 files changed, 452 insertions(+), 119 deletions(-)
 create mode 100644 swh/deposit/tests/test_backend.py
Changes applied before test
commit 2ced88bb12bb38b72726492e3a3702208b0ed272
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Mar 19 16:05:48 2021 +0100

    Allow to configure authentication mechanism per config file
    
    This will allow to deploy on staging with keycloak and continue using the deposit with
    http basic authentication.
    
    Related to T2858

commit 27efcccbfaea7188b05e6ce18e201ec9a2ddfb79
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Feb 18 16:57:55 2021 +0100

    Delegate authentication to keycloak
    
    This reworks the internal authentication mechanism to delegate that part to keycloak.
    Through a Direct Grant access flow.
    
    It's transparent for the deposit clients. They will continue their authentication
    against the deposit server. Internally, the deposit server will require authentication
    on their behalf.
    
    Related to T2858

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/579/ for more details.

ardumont marked 3 inline comments as done.

Add one test to demonstrate http basic auth still works

Build is green

Patch application report for D5291 (id=18949)

Could not rebase; Attempt merge onto 3a9b2fc4ba...

Updating 3a9b2fc4..8753653e
Fast-forward
 requirements-swh-server.txt                        |   1 +
 swh/deposit/api/common.py                          |  32 ++--
 swh/deposit/api/private/__init__.py                |  12 +-
 swh/deposit/api/service_document.py                |  27 ++-
 swh/deposit/auth.py                                | 112 ++++++++++-
 swh/deposit/cli/admin.py                           |  10 +-
 swh/deposit/config.py                              |   1 -
 swh/deposit/exception.py                           |   5 +-
 swh/deposit/models.py                              |   8 +-
 swh/deposit/settings/common.py                     |   7 +-
 swh/deposit/settings/production.py                 |  23 ++-
 swh/deposit/settings/testing.py                    |   6 +-
 swh/deposit/tests/api/test_basic_auth.py           |  32 ++++
 swh/deposit/tests/api/test_collection.py           |  12 +-
 .../tests/api/test_collection_add_to_origin.py     |  10 +-
 .../tests/api/test_collection_reuse_slug.py        |  22 +--
 swh/deposit/tests/api/test_exception.py            |   4 +-
 swh/deposit/tests/api/test_get_file.py             |  10 +-
 swh/deposit/tests/api/test_service_document.py     |  10 +-
 swh/deposit/tests/cli/test_admin.py                |   4 +-
 swh/deposit/tests/conftest.py                      | 212 ++++++++++++++++-----
 swh/deposit/tests/test_backend.py                  |  71 +++++++
 22 files changed, 510 insertions(+), 121 deletions(-)
 create mode 100644 swh/deposit/tests/api/test_basic_auth.py
 create mode 100644 swh/deposit/tests/test_backend.py
Changes applied before test
commit 8753653e2875ca31706e0902f921df1f5d47f912
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Mar 19 16:05:48 2021 +0100

    Allow to configure authentication mechanism per config file
    
    This will allow to deploy on staging with keycloak while continue using the deposit with
    http basic authentication in production.
    
    If some urgency fix is needed, we won't be forced into migrating keycloak immediately to
    fix that potential other issue.
    
    Note that the following transitional code will probably go away when we are done
    migrating to keycloak.
    
    Related to T2858

commit 27efcccbfaea7188b05e6ce18e201ec9a2ddfb79
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Feb 18 16:57:55 2021 +0100

    Delegate authentication to keycloak
    
    This reworks the internal authentication mechanism to delegate that part to keycloak.
    Through a Direct Grant access flow.
    
    It's transparent for the deposit clients. They will continue their authentication
    against the deposit server. Internally, the deposit server will require authentication
    on their behalf.
    
    Related to T2858

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/580/ for more details.

ardumont added inline comments.
swh/deposit/api/common.py
176–180

I added a simple test checking the service document works with http basic authentication as before.
So i'm considering this done.

swh/deposit/api/service_document.py
20–25

i'm in a mood to let this as is for now.
I'll clean up when i'm done with the migration towards keycloak
Which means that this conditional will go away altogether.

Isn't it enough?

anlambert added inline comments.
swh/deposit/api/service_document.py
20–25

fair enough

swh/deposit/api/common.py
189

and same remark as below, this is a transitional step anyway.
This will go away when we will be using keycloak for authentication mechanism.

vlorentz added inline comments.
swh/deposit/api/common.py
189

Then:

elif auth_provider is None:
    pass
else:
    error

I just don't want a typo in the auth_provider to silently disable authentication

This revision now requires changes to proceed.Mar 20 2021, 11:45 AM

Adapt according to last remark (make the initialization fail if the configuration is incomplete)

Build is green

Patch application report for D5291 (id=18962)

Could not rebase; Attempt merge onto 3a9b2fc4ba...

Updating 3a9b2fc4..9e6bb9ac
Fast-forward
 requirements-swh-server.txt                        |   1 +
 swh/deposit/api/common.py                          |  36 ++--
 swh/deposit/api/private/__init__.py                |  12 +-
 swh/deposit/api/service_document.py                |  27 ++-
 swh/deposit/auth.py                                | 112 ++++++++++-
 swh/deposit/cli/admin.py                           |  10 +-
 swh/deposit/config.py                              |   1 -
 swh/deposit/exception.py                           |   5 +-
 swh/deposit/models.py                              |   8 +-
 swh/deposit/settings/common.py                     |   7 +-
 swh/deposit/settings/production.py                 |  23 ++-
 swh/deposit/settings/testing.py                    |   6 +-
 swh/deposit/tests/api/test_basic_auth.py           |  32 ++++
 swh/deposit/tests/api/test_collection.py           |  12 +-
 .../tests/api/test_collection_add_to_origin.py     |  10 +-
 .../tests/api/test_collection_reuse_slug.py        |  22 +--
 swh/deposit/tests/api/test_exception.py            |   4 +-
 swh/deposit/tests/api/test_get_file.py             |  10 +-
 swh/deposit/tests/api/test_service_document.py     |  10 +-
 swh/deposit/tests/cli/test_admin.py                |   4 +-
 swh/deposit/tests/conftest.py                      | 212 ++++++++++++++++-----
 swh/deposit/tests/test_backend.py                  |  71 +++++++
 swh/deposit/tests/test_init.py                     |  18 +-
 23 files changed, 531 insertions(+), 122 deletions(-)
 create mode 100644 swh/deposit/tests/api/test_basic_auth.py
 create mode 100644 swh/deposit/tests/test_backend.py
Changes applied before test
commit 9e6bb9acd8fb46b8545cf68bd39100f56d1c5327
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Mar 19 16:05:48 2021 +0100

    Allow to configure authentication mechanism per config file
    
    This will allow to deploy on staging with keycloak while continue using the deposit with
    http basic authentication in production.
    
    If some urgency fix is needed, we won't be forced into migrating keycloak immediately to
    fix that potential other issue.
    
    Note that the following transitional code will probably go away when we are done
    migrating to keycloak.
    
    Related to T2858

commit 27efcccbfaea7188b05e6ce18e201ec9a2ddfb79
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Feb 18 16:57:55 2021 +0100

    Delegate authentication to keycloak
    
    This reworks the internal authentication mechanism to delegate that part to keycloak.
    Through a Direct Grant access flow.
    
    It's transparent for the deposit clients. They will continue their authentication
    against the deposit server. Internally, the deposit server will require authentication
    on their behalf.
    
    Related to T2858

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/581/ for more details.

swh/deposit/api/common.py
189

Then:

Well, I will check first if the bare else is enough...
In the end, I think it should.

I just don't want a typo in the auth_provider to silently disable authentication

right.

Build is green

Patch application report for D5291 (id=18964)

Could not rebase; Attempt merge onto 3a9b2fc4ba...

Updating 3a9b2fc4..1176de11
Fast-forward
 requirements-swh-server.txt                        |   1 +
 swh/deposit/api/common.py                          |  37 ++--
 swh/deposit/api/private/__init__.py                |  12 +-
 swh/deposit/api/service_document.py                |  27 ++-
 swh/deposit/auth.py                                | 112 ++++++++++-
 swh/deposit/cli/admin.py                           |  10 +-
 swh/deposit/config.py                              |   1 -
 swh/deposit/exception.py                           |   5 +-
 swh/deposit/models.py                              |   8 +-
 swh/deposit/settings/common.py                     |   7 +-
 swh/deposit/settings/production.py                 |  23 ++-
 swh/deposit/settings/testing.py                    |   6 +-
 swh/deposit/tests/api/test_basic_auth.py           |  32 ++++
 swh/deposit/tests/api/test_collection.py           |  12 +-
 .../tests/api/test_collection_add_to_origin.py     |  10 +-
 .../tests/api/test_collection_reuse_slug.py        |  22 +--
 swh/deposit/tests/api/test_exception.py            |   4 +-
 swh/deposit/tests/api/test_get_file.py             |  10 +-
 swh/deposit/tests/api/test_service_document.py     |  10 +-
 swh/deposit/tests/cli/test_admin.py                |   4 +-
 swh/deposit/tests/conftest.py                      | 212 ++++++++++++++++-----
 swh/deposit/tests/test_backend.py                  |  71 +++++++
 swh/deposit/tests/test_init.py                     |  18 +-
 23 files changed, 532 insertions(+), 122 deletions(-)
 create mode 100644 swh/deposit/tests/api/test_basic_auth.py
 create mode 100644 swh/deposit/tests/test_backend.py
Changes applied before test
commit 1176de11ff613fc0a3b4f54c6c5cea86014df0d6
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Mar 19 16:05:48 2021 +0100

    Allow to configure authentication mechanism per config file
    
    This will allow to deploy on staging with keycloak while continue using the deposit with
    http basic authentication in production.
    
    If some urgency fix is needed, we won't be forced into migrating keycloak immediately to
    fix that potential other issue.
    
    Note that the following transitional code will probably go away when we are done
    migrating to keycloak.
    
    Related to T2858

commit 27efcccbfaea7188b05e6ce18e201ec9a2ddfb79
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Feb 18 16:57:55 2021 +0100

    Delegate authentication to keycloak
    
    This reworks the internal authentication mechanism to delegate that part to keycloak.
    Through a Direct Grant access flow.
    
    It's transparent for the deposit clients. They will continue their authentication
    against the deposit server. Internally, the deposit server will require authentication
    on their behalf.
    
    Related to T2858

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/582/ for more details.

vlorentz added inline comments.
swh/deposit/api/common.py
190–193
This revision is now accepted and ready to land.Mar 22 2021, 11:23 AM

Adapt according to last suggestion

Build is green

Patch application report for D5291 (id=18973)

Could not rebase; Attempt merge onto 3a9b2fc4ba...

Updating 3a9b2fc4..6773775a
Fast-forward
 requirements-swh-server.txt                        |   1 +
 swh/deposit/api/common.py                          |  37 ++--
 swh/deposit/api/private/__init__.py                |  12 +-
 swh/deposit/api/service_document.py                |  27 ++-
 swh/deposit/auth.py                                | 112 ++++++++++-
 swh/deposit/cli/admin.py                           |  10 +-
 swh/deposit/config.py                              |   1 -
 swh/deposit/exception.py                           |   5 +-
 swh/deposit/models.py                              |   8 +-
 swh/deposit/settings/common.py                     |   7 +-
 swh/deposit/settings/production.py                 |  23 ++-
 swh/deposit/settings/testing.py                    |   6 +-
 swh/deposit/tests/api/test_basic_auth.py           |  32 ++++
 swh/deposit/tests/api/test_collection.py           |  12 +-
 .../tests/api/test_collection_add_to_origin.py     |  10 +-
 .../tests/api/test_collection_reuse_slug.py        |  22 +--
 swh/deposit/tests/api/test_exception.py            |   4 +-
 swh/deposit/tests/api/test_get_file.py             |  10 +-
 swh/deposit/tests/api/test_service_document.py     |  10 +-
 swh/deposit/tests/cli/test_admin.py                |   4 +-
 swh/deposit/tests/conftest.py                      | 212 ++++++++++++++++-----
 swh/deposit/tests/test_backend.py                  |  71 +++++++
 swh/deposit/tests/test_init.py                     |  18 +-
 23 files changed, 532 insertions(+), 122 deletions(-)
 create mode 100644 swh/deposit/tests/api/test_basic_auth.py
 create mode 100644 swh/deposit/tests/test_backend.py
Changes applied before test
commit 6773775acfcb55b6b3b8e7472eeeecc19853788d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Mar 19 16:05:48 2021 +0100

    Allow to configure authentication mechanism per config file
    
    This will allow to deploy on staging with keycloak while continue using the deposit with
    http basic authentication in production.
    
    If some urgency fix is needed, we won't be forced into migrating keycloak immediately to
    fix that potential other issue.
    
    Note that the following transitional code will probably go away when we are done
    migrating to keycloak.
    
    Related to T2858

commit 27efcccbfaea7188b05e6ce18e201ec9a2ddfb79
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Feb 18 16:57:55 2021 +0100

    Delegate authentication to keycloak
    
    This reworks the internal authentication mechanism to delegate that part to keycloak.
    Through a Direct Grant access flow.
    
    It's transparent for the deposit clients. They will continue their authentication
    against the deposit server. Internally, the deposit server will require authentication
    on their behalf.
    
    Related to T2858

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/583/ for more details.