Page MenuHomeSoftware Heritage
Differential D4560

Configure the kafka server for staging
ClosedPublic
Actions

Authored by vsellier on Nov 23 2020, 4:07 PM.

Details

Reviewers
ardumont
Group Reviewers
System administrators
Maniphest Tasks
T2790: [staging] deploy the journal infrastructure
Commits
rSPSITE9074d87bd6dd: Configure the kafka server for staging
rSPSITEbf96179df42d: Ensure java is installed before trying to manipulate the java keystore
Summary

also add a dependency on java before trying to manipulate the java keystore

Related to T2790

Diff Detail

Repository
rSPSITE puppet-swh-site
Branch
staging
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 17424
Build 26922: arc lint + arc unit

Event Timeline

vsellier planned changes to this revision.Nov 23 2020, 4:07 PM
vsellier created this revision.
Harbormaster completed remote builds in B17401: Diff 16201.Nov 23 2020, 4:07 PM
vsellier updated this revision to Diff 16225.Nov 24 2020, 9:21 AM

Fix wrong indentation on the storage service configuration

The journal configuration was not at the right place

Harbormaster completed remote builds in B17424: Diff 16225.Nov 24 2020, 9:21 AM
vsellier added a comment.Nov 24 2020, 9:25 AM

The configuration seems to work pretty well in vagrant :

root@journal0:/opt/kafka/bin# ./kafka-topics.sh --bootstrap-server 10.168.130.70:9092 --list | xargs -n1 -i{} ./kafka-run-class.sh kafka.tools.GetOffsetShell  --broker-list 10.168.130.70:9092 --topic {} | grep -v consumer
swh.journal.objects.content:0:898
swh.journal.objects.directory:0:1172
swh.journal.objects.origin:0:26
swh.journal.objects.origin_visit:0:55
swh.journal.objects.origin_visit_status:0:2
swh.journal.objects.release:0:0
swh.journal.objects.revision:0:209
swh.journal.objects.snapshot:0:1

I let the worker create the topics so there is only one partition. It will need to be adapted for the staging environment

ardumont accepted this revision.Nov 24 2020, 9:56 AM
This revision is now accepted and ready to land.Nov 24 2020, 9:56 AM
vsellier updated this revision to Diff 16231.Nov 24 2020, 11:45 AM

Rename the zookeper cluster to match the kafka cluster name

Harbormaster completed remote builds in B17430: Diff 16231.Nov 24 2020, 11:45 AM
vsellier added a comment.Nov 24 2020, 11:47 AM

precision: the tests were only done on the internal plaintext listener without sasl authentication

ardumont accepted this revision.Nov 24 2020, 11:47 AM
Closed by commit rSPSITEbf96179df42d: Ensure java is installed before trying to manipulate the java keystore (authored by vsellier). · Explain WhyNov 24 2020, 12:07 PM
This revision was automatically updated to reflect the committed changes.
vsellier added a commit: rSPSITEbf96179df42d: Ensure java is installed before trying to manipulate the java keystore.
vsellier added a commit: rSPSITE9074d87bd6dd: Configure the kafka server for staging.

Revision Contents
Changeset List

PathSize
data/
deployments/
staging/
33 lines
2 lines
site-modules/
profile/
manifests/
kafka/
1 line
role/
manifests/
1 line

Diff 16225

View Options

data/deployments/staging/common.yaml

# Copyright (C) 2022 The Software Heritage developers # Copyright (C) 2022 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
import os import os
from pathlib import Path from pathlib import Path
from breezy.builtins import cmd_uncommit from breezy.builtins import cmd_uncommit
import pytest import pytest
import swh.loader.bzr.loader as loader_mod
from swh.loader.bzr.loader import BazaarLoader, BzrDirectory from swh.loader.bzr.loader import BazaarLoader, BzrDirectory
from swh.loader.tests import ( from swh.loader.tests import (
assert_last_visit_matches, assert_last_visit_matches,
get_stats, get_stats,
prepare_repository_from_archive, prepare_repository_from_archive,
) )
from swh.model.from_disk import Content from swh.model.from_disk import Content
from swh.model.hashutil import hash_to_bytes from swh.model.hashutil import hash_to_bytes
▲ Show 20 Lines • Show All 111 Lines • ▼ Show 20 Lines assert revision.to_dict() == {
(b"time_offset_seconds", b"3600"), (b"time_offset_seconds", b"3600"),
(b"bug", b"fixed https://launchpad.net/bugs/1234"), (b"bug", b"fixed https://launchpad.net/bugs/1234"),
(b"bug", b"fixed https://bz.example.com/?show_bug=4321"), (b"bug", b"fixed https://bz.example.com/?show_bug=4321"),
), ),
} }
def test_needs_upgrade(swh_storage, datadir, tmp_path, mocker): def test_needs_upgrade(swh_storage, datadir, tmp_path, mocker):
"""Old bzr repository format should fail the ingestion (upgrade necessary)"""
archive_path = Path(datadir, "needs-upgrade.tgz") archive_path = Path(datadir, "needs-upgrade.tgz")
repo_url = prepare_repository_from_archive(archive_path, "needs-upgrade", tmp_path) repo_url = prepare_repository_from_archive(archive_path, "needs-upgrade", tmp_path)
init_spy = mocker.spy(loader_mod.RepositoryNeedsUpgrade, "__init__")
init_spy.assert_not_called()
res = BazaarLoader(swh_storage, repo_url, directory=repo_url).load() res = BazaarLoader(swh_storage, repo_url, directory=repo_url).load()
init_spy.assert_called()
assert res == {"status": "failed"} assert res == {"status": "failed"}
def test_no_branch(swh_storage, datadir, tmp_path): def test_no_branch(swh_storage, datadir, tmp_path):
"""This should only happen with a broken clone, so the expected result is failure""" """This should only happen with a broken clone, so the expected result is failure"""
archive_path = Path(datadir, "no-branch.tgz") archive_path = Path(datadir, "no-branch.tgz")
repo_url = prepare_repository_from_archive(archive_path, "no-branch", tmp_path) repo_url = prepare_repository_from_archive(archive_path, "no-branch", tmp_path)
▲ Show 20 Lines • Show All 261 Lines • Show Last 20 Lines
View Options

data/deployments/staging/vagrant.yaml

# Copyright (C) 2022 The Software Heritage developers # Copyright (C) 2022 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
import os import os
from pathlib import Path from pathlib import Path
from breezy.builtins import cmd_uncommit from breezy.builtins import cmd_uncommit
import pytest import pytest
import swh.loader.bzr.loader as loader_mod
from swh.loader.bzr.loader import BazaarLoader, BzrDirectory from swh.loader.bzr.loader import BazaarLoader, BzrDirectory
from swh.loader.tests import ( from swh.loader.tests import (
assert_last_visit_matches, assert_last_visit_matches,
get_stats, get_stats,
prepare_repository_from_archive, prepare_repository_from_archive,
) )
from swh.model.from_disk import Content from swh.model.from_disk import Content
from swh.model.hashutil import hash_to_bytes from swh.model.hashutil import hash_to_bytes
▲ Show 20 Lines • Show All 111 Lines • ▼ Show 20 Lines assert revision.to_dict() == {
(b"time_offset_seconds", b"3600"), (b"time_offset_seconds", b"3600"),
(b"bug", b"fixed https://launchpad.net/bugs/1234"), (b"bug", b"fixed https://launchpad.net/bugs/1234"),
(b"bug", b"fixed https://bz.example.com/?show_bug=4321"), (b"bug", b"fixed https://bz.example.com/?show_bug=4321"),
), ),
} }
def test_needs_upgrade(swh_storage, datadir, tmp_path, mocker): def test_needs_upgrade(swh_storage, datadir, tmp_path, mocker):
"""Old bzr repository format should fail the ingestion (upgrade necessary)"""
archive_path = Path(datadir, "needs-upgrade.tgz") archive_path = Path(datadir, "needs-upgrade.tgz")
repo_url = prepare_repository_from_archive(archive_path, "needs-upgrade", tmp_path) repo_url = prepare_repository_from_archive(archive_path, "needs-upgrade", tmp_path)
init_spy = mocker.spy(loader_mod.RepositoryNeedsUpgrade, "__init__")
init_spy.assert_not_called()
res = BazaarLoader(swh_storage, repo_url, directory=repo_url).load() res = BazaarLoader(swh_storage, repo_url, directory=repo_url).load()
init_spy.assert_called()
assert res == {"status": "failed"} assert res == {"status": "failed"}
def test_no_branch(swh_storage, datadir, tmp_path): def test_no_branch(swh_storage, datadir, tmp_path):
"""This should only happen with a broken clone, so the expected result is failure""" """This should only happen with a broken clone, so the expected result is failure"""
archive_path = Path(datadir, "no-branch.tgz") archive_path = Path(datadir, "no-branch.tgz")
repo_url = prepare_repository_from_archive(archive_path, "no-branch", tmp_path) repo_url = prepare_repository_from_archive(archive_path, "no-branch", tmp_path)
▲ Show 20 Lines • Show All 261 Lines • Show Last 20 Lines
View Options

site-modules/profile/manifests/kafka/broker.pp

# Copyright (C) 2022 The Software Heritage developers # Copyright (C) 2022 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
import os import os
from pathlib import Path from pathlib import Path
from breezy.builtins import cmd_uncommit from breezy.builtins import cmd_uncommit
import pytest import pytest
import swh.loader.bzr.loader as loader_mod
from swh.loader.bzr.loader import BazaarLoader, BzrDirectory from swh.loader.bzr.loader import BazaarLoader, BzrDirectory
from swh.loader.tests import ( from swh.loader.tests import (
assert_last_visit_matches, assert_last_visit_matches,
get_stats, get_stats,
prepare_repository_from_archive, prepare_repository_from_archive,
) )
from swh.model.from_disk import Content from swh.model.from_disk import Content
from swh.model.hashutil import hash_to_bytes from swh.model.hashutil import hash_to_bytes
▲ Show 20 Lines • Show All 111 Lines • ▼ Show 20 Lines assert revision.to_dict() == {
(b"time_offset_seconds", b"3600"), (b"time_offset_seconds", b"3600"),
(b"bug", b"fixed https://launchpad.net/bugs/1234"), (b"bug", b"fixed https://launchpad.net/bugs/1234"),
(b"bug", b"fixed https://bz.example.com/?show_bug=4321"), (b"bug", b"fixed https://bz.example.com/?show_bug=4321"),
), ),
} }
def test_needs_upgrade(swh_storage, datadir, tmp_path, mocker): def test_needs_upgrade(swh_storage, datadir, tmp_path, mocker):
"""Old bzr repository format should fail the ingestion (upgrade necessary)"""
archive_path = Path(datadir, "needs-upgrade.tgz") archive_path = Path(datadir, "needs-upgrade.tgz")
repo_url = prepare_repository_from_archive(archive_path, "needs-upgrade", tmp_path) repo_url = prepare_repository_from_archive(archive_path, "needs-upgrade", tmp_path)
init_spy = mocker.spy(loader_mod.RepositoryNeedsUpgrade, "__init__")
init_spy.assert_not_called()
res = BazaarLoader(swh_storage, repo_url, directory=repo_url).load() res = BazaarLoader(swh_storage, repo_url, directory=repo_url).load()
init_spy.assert_called()
assert res == {"status": "failed"} assert res == {"status": "failed"}
def test_no_branch(swh_storage, datadir, tmp_path): def test_no_branch(swh_storage, datadir, tmp_path):
"""This should only happen with a broken clone, so the expected result is failure""" """This should only happen with a broken clone, so the expected result is failure"""
archive_path = Path(datadir, "no-branch.tgz") archive_path = Path(datadir, "no-branch.tgz")
repo_url = prepare_repository_from_archive(archive_path, "no-branch", tmp_path) repo_url = prepare_repository_from_archive(archive_path, "no-branch", tmp_path)
▲ Show 20 Lines • Show All 261 Lines • Show Last 20 Lines
View Options

site-modules/role/manifests/swh_journal_allinone.pp

# Copyright (C) 2022 The Software Heritage developers # Copyright (C) 2022 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
import os import os
from pathlib import Path from pathlib import Path
from breezy.builtins import cmd_uncommit from breezy.builtins import cmd_uncommit
import pytest import pytest
import swh.loader.bzr.loader as loader_mod
from swh.loader.bzr.loader import BazaarLoader, BzrDirectory from swh.loader.bzr.loader import BazaarLoader, BzrDirectory
from swh.loader.tests import ( from swh.loader.tests import (
assert_last_visit_matches, assert_last_visit_matches,
get_stats, get_stats,
prepare_repository_from_archive, prepare_repository_from_archive,
) )
from swh.model.from_disk import Content from swh.model.from_disk import Content
from swh.model.hashutil import hash_to_bytes from swh.model.hashutil import hash_to_bytes
▲ Show 20 Lines • Show All 111 Lines • ▼ Show 20 Lines assert revision.to_dict() == {
(b"time_offset_seconds", b"3600"), (b"time_offset_seconds", b"3600"),
(b"bug", b"fixed https://launchpad.net/bugs/1234"), (b"bug", b"fixed https://launchpad.net/bugs/1234"),
(b"bug", b"fixed https://bz.example.com/?show_bug=4321"), (b"bug", b"fixed https://bz.example.com/?show_bug=4321"),
), ),
} }
def test_needs_upgrade(swh_storage, datadir, tmp_path, mocker): def test_needs_upgrade(swh_storage, datadir, tmp_path, mocker):
"""Old bzr repository format should fail the ingestion (upgrade necessary)"""
archive_path = Path(datadir, "needs-upgrade.tgz") archive_path = Path(datadir, "needs-upgrade.tgz")
repo_url = prepare_repository_from_archive(archive_path, "needs-upgrade", tmp_path) repo_url = prepare_repository_from_archive(archive_path, "needs-upgrade", tmp_path)
init_spy = mocker.spy(loader_mod.RepositoryNeedsUpgrade, "__init__")
init_spy.assert_not_called()
res = BazaarLoader(swh_storage, repo_url, directory=repo_url).load() res = BazaarLoader(swh_storage, repo_url, directory=repo_url).load()
init_spy.assert_called()
assert res == {"status": "failed"} assert res == {"status": "failed"}
def test_no_branch(swh_storage, datadir, tmp_path): def test_no_branch(swh_storage, datadir, tmp_path):
"""This should only happen with a broken clone, so the expected result is failure""" """This should only happen with a broken clone, so the expected result is failure"""
archive_path = Path(datadir, "no-branch.tgz") archive_path = Path(datadir, "no-branch.tgz")
repo_url = prepare_repository_from_archive(archive_path, "no-branch", tmp_path) repo_url = prepare_repository_from_archive(archive_path, "no-branch", tmp_path)
▲ Show 20 Lines • Show All 261 Lines • Show Last 20 Lines
About · Developer information · Legal