Page MenuHomeSoftware Heritage
Differential D7551

Prepare the rancher manager bootstrap
ClosedPublic
Actions

Authored by vsellier on Apr 12 2022, 9:57 AM.

Details

Reviewers
ardumont
Group Reviewers
System administrators
Maniphest Tasks
T4046: Bootstrap a gitlab instance hosted on our infrastructure
Commits
rSPRE5292d8c407a5: rancher: Create an aks cluster dedicated to rancher
rSPRE0241d0e4514c: aks: Allow to disable the public ip provisioning
Summary

2 commits:

  • Allow to disable the default public ip assignement
  • Create an aks cluster dedicated to rancher

It will allow to bootstrap the rancher manager without
manually installing a kubernetes on premise.

Related to T4046

Diff Detail

Repository
rSPRE sysadm-provisioning
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 28390
Build 44393: arc lint + arc unit

Event Timeline

vsellier created this revision.Apr 12 2022, 9:57 AM
Herald added a reviewer: System administrators. · View Herald TranscriptApr 12 2022, 9:57 AM
Harbormaster completed remote builds in B28389: Diff 27378.Apr 12 2022, 9:57 AM
vsellier requested review of this revision.Apr 12 2022, 9:57 AM
vsellier updated this revision to Diff 27379.

rebase

Harbormaster completed remote builds in B28390: Diff 27379.Apr 12 2022, 9:57 AM
vsellier updated this revision to Diff 27383.Apr 12 2022, 12:29 PM

don't provision a public ip for rancher

Harbormaster completed remote builds in B28395: Diff 27383.Apr 12 2022, 12:29 PM
ardumont accepted this revision.Apr 12 2022, 3:05 PM
ardumont added a subscriber: ardumont.

ok

one docstring suggestion inline.

azure/terraform/modules/kubernetes/variables.tf
41
This revision is now accepted and ready to land.Apr 12 2022, 3:05 PM
vsellier updated this revision to Diff 27471.Apr 14 2022, 6:57 PM

rebase

Harbormaster completed remote builds in B28491: Diff 27471.Apr 14 2022, 6:57 PM
Closed by commit rSPRE0241d0e4514c: aks: Allow to disable the public ip provisioning (authored by vsellier). · Explain WhyApr 14 2022, 6:57 PM
This revision was automatically updated to reflect the committed changes.
vsellier added a commit: rSPRE0241d0e4514c: aks: Allow to disable the public ip provisioning.
vsellier added a commit: rSPRE5292d8c407a5: rancher: Create an aks cluster dedicated to rancher.

Revision Contents
Changeset List

PathSize
azure/
terraform/
modules/
kubernetes/
1 line
2 lines
6 lines
27 lines

Diff 27379

View Options

azure/terraform/modules/kubernetes/main.tf

Show All 39 Lines) -> Dict[str, str]:
from swh.core.cli.db import populate_database_for_package from swh.core.cli.db import populate_database_for_package
populate_database_for_package( populate_database_for_package(
"swh.provenance", postgresql.dsn, flavor=request.param "swh.provenance", postgresql.dsn, flavor=request.param
) )
return postgresql.get_dsn_parameters() return postgresql.get_dsn_parameters()
@pytest.fixture(params=["mongodb", "postgresql"]) @pytest.fixture(params=["mongodb", "postgresql", "rabbitmq"])
def provenance_storage( def provenance_storage(
request: SubRequest, request: SubRequest,
provenance_postgresqldb: Dict[str, str], provenance_postgresqldb: Dict[str, str],
mongodb: mongomock.database.Database, mongodb: mongomock.database.Database,
) -> Generator[ProvenanceStorageInterface, None, None]: ) -> Generator[ProvenanceStorageInterface, None, None]:
"""Return a working and initialized ProvenanceStorageInterface object""" """Return a working and initialized ProvenanceStorageInterface object"""
if request.param == "mongodb": if request.param == "mongodb":
mongodb_params = { mongodb_params = {
"host": mongodb.client.address[0], "host": mongodb.client.address[0],
"port": mongodb.client.address[1], "port": mongodb.client.address[1],
"dbname": mongodb.name, "dbname": mongodb.name,
} }
with get_provenance_storage( with get_provenance_storage(
cls=request.param, db=mongodb_params, engine="mongomock" cls=request.param, db=mongodb_params, engine="mongomock"
) as storage: ) as storage:
yield storage yield storage
elif request.param == "rabbitmq":
from swh.provenance.api.server import ProvenanceStorageRabbitMQServer
rabbitmq = request.getfixturevalue("rabbitmq")
host = rabbitmq.args["host"]
douarddaUnsubmitted
Done
Inline Actions

what's the %2f doing here?

douardda: what's the `%2f` doing here?
aevisoAuthorUnsubmitted
Done
Inline Actions

is the virtual host to connect to (the URL encoding for /): https://pika.readthedocs.io/en/stable/examples/using_urlparameters.html

aeviso: is the virtual host to connect to (the URL encoding for /): https://pika.readthedocs.
port = rabbitmq.args["port"]
rabbitmq_params: Dict[str, Any] = {
"url": f"amqp://guest:guest@{host}:{port}/%2f",
"storage_config": {
"cls": "postgresql", # TODO: also test with underlying mongodb storage
"db": provenance_postgresqldb,
"raise_on_commit": True,
},
}
server = ProvenanceStorageRabbitMQServer(
url=rabbitmq_params["url"], storage_config=rabbitmq_params["storage_config"]
)
server.start()
with get_provenance_storage(cls=request.param, **rabbitmq_params) as storage:
yield storage
server.stop()
else: else:
# in test sessions, we DO want to raise any exception occurring at commit time # in test sessions, we DO want to raise any exception occurring at commit time
with get_provenance_storage( with get_provenance_storage(
cls=request.param, db=provenance_postgresqldb, raise_on_commit=True cls=request.param, db=provenance_postgresqldb, raise_on_commit=True
) as storage: ) as storage:
yield storage yield storage
▲ Show 20 Lines • Show All 72 Lines • Show Last 20 Lines
View Options

azure/terraform/modules/kubernetes/outputs.tf

Show All 39 Lines) -> Dict[str, str]:
from swh.core.cli.db import populate_database_for_package from swh.core.cli.db import populate_database_for_package
populate_database_for_package( populate_database_for_package(
"swh.provenance", postgresql.dsn, flavor=request.param "swh.provenance", postgresql.dsn, flavor=request.param
) )
return postgresql.get_dsn_parameters() return postgresql.get_dsn_parameters()
@pytest.fixture(params=["mongodb", "postgresql"]) @pytest.fixture(params=["mongodb", "postgresql", "rabbitmq"])
def provenance_storage( def provenance_storage(
request: SubRequest, request: SubRequest,
provenance_postgresqldb: Dict[str, str], provenance_postgresqldb: Dict[str, str],
mongodb: mongomock.database.Database, mongodb: mongomock.database.Database,
) -> Generator[ProvenanceStorageInterface, None, None]: ) -> Generator[ProvenanceStorageInterface, None, None]:
"""Return a working and initialized ProvenanceStorageInterface object""" """Return a working and initialized ProvenanceStorageInterface object"""
if request.param == "mongodb": if request.param == "mongodb":
mongodb_params = { mongodb_params = {
"host": mongodb.client.address[0], "host": mongodb.client.address[0],
"port": mongodb.client.address[1], "port": mongodb.client.address[1],
"dbname": mongodb.name, "dbname": mongodb.name,
} }
with get_provenance_storage( with get_provenance_storage(
cls=request.param, db=mongodb_params, engine="mongomock" cls=request.param, db=mongodb_params, engine="mongomock"
) as storage: ) as storage:
yield storage yield storage
elif request.param == "rabbitmq":
from swh.provenance.api.server import ProvenanceStorageRabbitMQServer
rabbitmq = request.getfixturevalue("rabbitmq")
host = rabbitmq.args["host"]
douarddaUnsubmitted
Done
Inline Actions

what's the %2f doing here?

douardda: what's the `%2f` doing here?
aevisoAuthorUnsubmitted
Done
Inline Actions

is the virtual host to connect to (the URL encoding for /): https://pika.readthedocs.io/en/stable/examples/using_urlparameters.html

aeviso: is the virtual host to connect to (the URL encoding for /): https://pika.readthedocs.
port = rabbitmq.args["port"]
rabbitmq_params: Dict[str, Any] = {
"url": f"amqp://guest:guest@{host}:{port}/%2f",
"storage_config": {
"cls": "postgresql", # TODO: also test with underlying mongodb storage
"db": provenance_postgresqldb,
"raise_on_commit": True,
},
}
server = ProvenanceStorageRabbitMQServer(
url=rabbitmq_params["url"], storage_config=rabbitmq_params["storage_config"]
)
server.start()
with get_provenance_storage(cls=request.param, **rabbitmq_params) as storage:
yield storage
server.stop()
else: else:
# in test sessions, we DO want to raise any exception occurring at commit time # in test sessions, we DO want to raise any exception occurring at commit time
with get_provenance_storage( with get_provenance_storage(
cls=request.param, db=provenance_postgresqldb, raise_on_commit=True cls=request.param, db=provenance_postgresqldb, raise_on_commit=True
) as storage: ) as storage:
yield storage yield storage
▲ Show 20 Lines • Show All 72 Lines • Show Last 20 Lines
View Options

azure/terraform/modules/kubernetes/variables.tf

Show All 39 Lines) -> Dict[str, str]:
from swh.core.cli.db import populate_database_for_package from swh.core.cli.db import populate_database_for_package
populate_database_for_package( populate_database_for_package(
"swh.provenance", postgresql.dsn, flavor=request.param "swh.provenance", postgresql.dsn, flavor=request.param
) )
return postgresql.get_dsn_parameters() return postgresql.get_dsn_parameters()
@pytest.fixture(params=["mongodb", "postgresql"]) @pytest.fixture(params=["mongodb", "postgresql", "rabbitmq"])
def provenance_storage( def provenance_storage(
request: SubRequest, request: SubRequest,
provenance_postgresqldb: Dict[str, str], provenance_postgresqldb: Dict[str, str],
mongodb: mongomock.database.Database, mongodb: mongomock.database.Database,
) -> Generator[ProvenanceStorageInterface, None, None]: ) -> Generator[ProvenanceStorageInterface, None, None]:
"""Return a working and initialized ProvenanceStorageInterface object""" """Return a working and initialized ProvenanceStorageInterface object"""
if request.param == "mongodb": if request.param == "mongodb":
mongodb_params = { mongodb_params = {
"host": mongodb.client.address[0], "host": mongodb.client.address[0],
"port": mongodb.client.address[1], "port": mongodb.client.address[1],
"dbname": mongodb.name, "dbname": mongodb.name,
} }
with get_provenance_storage( with get_provenance_storage(
cls=request.param, db=mongodb_params, engine="mongomock" cls=request.param, db=mongodb_params, engine="mongomock"
) as storage: ) as storage:
yield storage yield storage
elif request.param == "rabbitmq":
from swh.provenance.api.server import ProvenanceStorageRabbitMQServer
rabbitmq = request.getfixturevalue("rabbitmq")
host = rabbitmq.args["host"]
douarddaUnsubmitted
Done
Inline Actions

what's the %2f doing here?

douardda: what's the `%2f` doing here?
aevisoAuthorUnsubmitted
Done
Inline Actions

is the virtual host to connect to (the URL encoding for /): https://pika.readthedocs.io/en/stable/examples/using_urlparameters.html

aeviso: is the virtual host to connect to (the URL encoding for /): https://pika.readthedocs.
port = rabbitmq.args["port"]
rabbitmq_params: Dict[str, Any] = {
"url": f"amqp://guest:guest@{host}:{port}/%2f",
"storage_config": {
"cls": "postgresql", # TODO: also test with underlying mongodb storage
"db": provenance_postgresqldb,
"raise_on_commit": True,
},
}
server = ProvenanceStorageRabbitMQServer(
url=rabbitmq_params["url"], storage_config=rabbitmq_params["storage_config"]
)
server.start()
with get_provenance_storage(cls=request.param, **rabbitmq_params) as storage:
yield storage
server.stop()
else: else:
# in test sessions, we DO want to raise any exception occurring at commit time # in test sessions, we DO want to raise any exception occurring at commit time
with get_provenance_storage( with get_provenance_storage(
cls=request.param, db=provenance_postgresqldb, raise_on_commit=True cls=request.param, db=provenance_postgresqldb, raise_on_commit=True
) as storage: ) as storage:
yield storage yield storage
▲ Show 20 Lines • Show All 72 Lines • Show Last 20 Lines
View Options

azure/terraform/rancher.tf

Show All 39 Lines) -> Dict[str, str]:
from swh.core.cli.db import populate_database_for_package from swh.core.cli.db import populate_database_for_package
populate_database_for_package( populate_database_for_package(
"swh.provenance", postgresql.dsn, flavor=request.param "swh.provenance", postgresql.dsn, flavor=request.param
) )
return postgresql.get_dsn_parameters() return postgresql.get_dsn_parameters()
@pytest.fixture(params=["mongodb", "postgresql"]) @pytest.fixture(params=["mongodb", "postgresql", "rabbitmq"])
def provenance_storage( def provenance_storage(
request: SubRequest, request: SubRequest,
provenance_postgresqldb: Dict[str, str], provenance_postgresqldb: Dict[str, str],
mongodb: mongomock.database.Database, mongodb: mongomock.database.Database,
) -> Generator[ProvenanceStorageInterface, None, None]: ) -> Generator[ProvenanceStorageInterface, None, None]:
"""Return a working and initialized ProvenanceStorageInterface object""" """Return a working and initialized ProvenanceStorageInterface object"""
if request.param == "mongodb": if request.param == "mongodb":
mongodb_params = { mongodb_params = {
"host": mongodb.client.address[0], "host": mongodb.client.address[0],
"port": mongodb.client.address[1], "port": mongodb.client.address[1],
"dbname": mongodb.name, "dbname": mongodb.name,
} }
with get_provenance_storage( with get_provenance_storage(
cls=request.param, db=mongodb_params, engine="mongomock" cls=request.param, db=mongodb_params, engine="mongomock"
) as storage: ) as storage:
yield storage yield storage
elif request.param == "rabbitmq":
from swh.provenance.api.server import ProvenanceStorageRabbitMQServer
rabbitmq = request.getfixturevalue("rabbitmq")
host = rabbitmq.args["host"]
douarddaUnsubmitted
Done
Inline Actions

what's the %2f doing here?

douardda: what's the `%2f` doing here?
aevisoAuthorUnsubmitted
Done
Inline Actions

is the virtual host to connect to (the URL encoding for /): https://pika.readthedocs.io/en/stable/examples/using_urlparameters.html

aeviso: is the virtual host to connect to (the URL encoding for /): https://pika.readthedocs.
port = rabbitmq.args["port"]
rabbitmq_params: Dict[str, Any] = {
"url": f"amqp://guest:guest@{host}:{port}/%2f",
"storage_config": {
"cls": "postgresql", # TODO: also test with underlying mongodb storage
"db": provenance_postgresqldb,
"raise_on_commit": True,
},
}
server = ProvenanceStorageRabbitMQServer(
url=rabbitmq_params["url"], storage_config=rabbitmq_params["storage_config"]
)
server.start()
with get_provenance_storage(cls=request.param, **rabbitmq_params) as storage:
yield storage
server.stop()
else: else:
# in test sessions, we DO want to raise any exception occurring at commit time # in test sessions, we DO want to raise any exception occurring at commit time
with get_provenance_storage( with get_provenance_storage(
cls=request.param, db=provenance_postgresqldb, raise_on_commit=True cls=request.param, db=provenance_postgresqldb, raise_on_commit=True
) as storage: ) as storage:
yield storage yield storage
▲ Show 20 Lines • Show All 72 Lines • Show Last 20 Lines
About · Developer information · Legal