diff --git a/.editorconfig b/.editorconfig index d77700e..ecb10a8 100644 --- a/.editorconfig +++ b/.editorconfig @@ -1,14 +1,15 @@ # editorconfig.org -# MANAGED BY MODULESYNC +# Managed by modulesync - DO NOT EDIT +# https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ root = true [*] charset = utf-8 end_of_line = lf indent_size = 2 tab_width = 2 indent_style = space insert_final_newline = true trim_trailing_whitespace = true diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index f1f88cc..8011013 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -1,282 +1,252 @@ # Contribution guidelines ## Table of contents * [Contributing](#contributing) * [Writing proper commits - short version](#writing-proper-commits-short-version) * [Writing proper commits - long version](#writing-proper-commits-long-version) * [Dependencies](#dependencies) * [Note for OS X users](#note-for-os-x-users) * [The test matrix](#the-test-matrix) * [Syntax and style](#syntax-and-style) * [Running the unit tests](#running-the-unit-tests) * [Unit tests in docker](#unit-tests-in-docker) * [Integration tests](#integration-tests) This module has grown over time based on a range of contributions from people using it. If you follow these contributing guidelines your patch will likely make it into a release a little more quickly. ## Contributing Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms. [Contributor Code of Conduct](https://voxpupuli.org/coc/). * Fork the repo. * Create a separate branch for your change. -* We only take pull requests with passing tests, and documentation. [travis-ci](http://travis-ci.org) runs the tests for us. You can also execute them locally. This is explained [in a later section](#the-test-matrix). +* We only take pull requests with passing tests, and documentation. GitHub Actions run the tests for us (check the .github/workflows/ directory). You can also execute them locally. This is explained [in a later section](#the-test-matrix). * Checkout [our docs](https://voxpupuli.org/docs/reviewing_pr/) we use to review a module and the [official styleguide](https://puppet.com/docs/puppet/6.0/style_guide.html). They provide some guidance for new code that might help you before you submit a pull request. * Add a test for your change. Only refactoring and documentation changes require no new tests. If you are adding functionality or fixing a bug, please add a test. * Squash your commits down into logical components. Make sure to rebase against our current master. * Push the branch to your fork and submit a pull request. Please be prepared to repeat some of these steps as our contributors review your code. Also consider sending in your profile code that calls this component module as an acceptance test or provide it via an issue. This helps reviewers a lot to test your use case and prevents future regressions! ## Writing proper commits - short version * Make commits of logical units. * Check for unnecessary whitespace with "git diff --check" before committing. * Commit using Unix line endings (check the settings around "crlf" in git-config(1)). * Do not check in commented out code or unneeded files. * The first line of the commit message should be a short description (50 characters is the soft limit, excluding ticket number(s)), and should skip the full stop. * Associate the issue in the message. The first line should include the issue number in the form "(#XXXX) Rest of message". * The body should provide a meaningful commit message, which: *uses the imperative, present tense: `change`, not `changed` or `changes`. * includes motivation for the change, and contrasts its implementation with the previous behavior. * Make sure that you have tests for the bug you are fixing, or feature you are adding. * Make sure the test suites passes after your commit: * When introducing a new feature, make sure it is properly documented in the README.md ## Writing proper commits - long version 1. Make separate commits for logically separate changes. Please break your commits down into logically consistent units which include new or changed tests relevant to the rest of the change. The goal of doing this is to make the diff easier to read for whoever is reviewing your code. In general, the easier your diff is to read, the more likely someone will be happy to review it and get it into the code base. If you are going to refactor a piece of code, please do so as a separate commit from your feature or bug fix changes. We also really appreciate changes that include tests to make sure the bug is not re-introduced, and that the feature is not accidentally broken. Describe the technical detail of the change(s). If your description starts to get too long, that is a good sign that you probably need to split up your commit into more finely grained pieces. Commits which plainly describe the things which help reviewers check the patch and future developers understand the code are much more likely to be merged in with a minimum of bike-shedding or requested changes. Ideally, the commit message would include information, and be in a form suitable for inclusion in the release notes for the version of Puppet that includes them. Please also check that you are not introducing any trailing whitespace or other "whitespace errors". You can do this by running "git diff --check" on your changes before you commit. 2. Sending your patches To submit your changes via a GitHub pull request, we _highly_ recommend that you have them on a topic branch, instead of directly on `master`. It makes things much easier to keep track of, especially if you decide to work on another thing before your first change is merged in. GitHub has some pretty good [general documentation](http://help.github.com/) on using their site. They also have documentation on [creating pull requests](http://help.github.com/send-pull-requests/). In general, after pushing your topic branch up to your repository on GitHub, you can switch to the branch in the GitHub UI and click "Pull Request" towards the top of the page in order to open a pull request. 3. Update the related GitHub issue. If there is a GitHub issue associated with the change you submitted, then you should update the ticket to include the location of your branch, along with any other commentary you may wish to make. ## Dependencies The testing and development tools have a bunch of dependencies, all managed by [bundler](http://bundler.io/) according to the [Puppet support matrix](http://docs.puppetlabs.com/guides/platforms.html#ruby-versions). By default the tests use a baseline version of Puppet. If you have Ruby 2.x or want a specific version of Puppet, you must set an environment variable such as: ```sh export PUPPET_VERSION="~> 5.5.6" ``` You can install all needed gems for spec tests into the modules directory by running: ```sh bundle install --path .vendor/ --without development system_tests release --jobs "$(nproc)" ``` If you also want to run acceptance tests: ```sh bundle install --path .vendor/ --with system_tests --without development release --jobs "$(nproc)" ``` Our all in one solution if you don't know if you need to install or update gems: ```sh bundle install --path .vendor/ --with system_tests --without development release --jobs "$(nproc)"; bundle update; bundle clean ``` As an alternative to the `--jobs "$(nproc)` parameter, you can set an environment variable: ```sh BUNDLE_JOBS="$(nproc)" ``` ### Note for OS X users `nproc` isn't a valid command under OS x. As an alternative, you can do: ```sh --jobs "$(sysctl -n hw.ncpu)" ``` ## The test matrix ### Syntax and style The test suite will run [Puppet Lint](http://puppet-lint.com/) and [Puppet Syntax](https://github.com/gds-operations/puppet-syntax) to check various syntax and style things. You can run these locally with: ```sh bundle exec rake lint bundle exec rake validate ``` It will also run some [Rubocop](http://batsov.com/rubocop/) tests against it. You can run those locally ahead of time with: ```sh bundle exec rake rubocop ``` ### Running the unit tests The unit test suite covers most of the code, as mentioned above please add tests if you're adding new functionality. If you've not used [rspec-puppet](http://rspec-puppet.com/) before then feel free to ask about how best to test your new feature. To run the linter, the syntax checker and the unit tests: ```sh bundle exec rake test ``` To run your all the unit tests ```sh bundle exec rake spec ``` To run a specific spec test set the `SPEC` variable: ```sh bundle exec rake spec SPEC=spec/foo_spec.rb ``` #### Unit tests in docker Some people don't want to run the dependencies locally or don't want to install ruby. We ship a Dockerfile that enables you to run all unit tests and linting. You only need to run: ```sh docker build . ``` Please ensure that a docker daemon is running and that your user has the permission to talk to it. You can specify a remote docker host by setting the `DOCKER_HOST` environment variable. it will copy the content of the module into the docker image. So it will not work if a Gemfile.lock exists. ### Integration tests The unit tests just check the code runs, not that it does exactly what we want on a real machine. For that we're using [beaker](https://github.com/puppetlabs/beaker). This fires up a new virtual machine (using vagrant) and runs a series of simple tests against it after applying the module. You can run this with: ```sh -bundle exec rake beaker -``` - -This will run the tests on the module's default nodeset. You can override the -nodeset used, e.g., - -```sh -BEAKER_set=centos-7-x64 bundle exec rake beaker -``` - -There are default rake tasks for the various acceptance test modules, e.g., - -```sh -bundle exec rake beaker:centos-7-x64 -bundle exec rake beaker:ssh:centos-7-x64 -``` - -If you don't want to have to recreate the virtual machine every time you can -use `BEAKER_destroy=no` and `BEAKER_provision=no`. On the first run you will at -least need `BEAKER_provision` set to yes (the default). The Vagrantfile for the -created virtual machines will be in `.vagrant/beaker_vagrant_files`. - -Beaker also supports docker containers. We also use that in our automated CI -pipeline at [travis-ci](http://travis-ci.org). To use that instead of Vagrant: - -```sh -PUPPET_INSTALL_TYPE=agent BEAKER_IS_PE=no BEAKER_PUPPET_COLLECTION=puppet6 BEAKER_debug=true BEAKER_setfile=debian10-64{hypervisor=docker} BEAKER_destroy=yes bundle exec rake beaker +BEAKER_setfile=debian10-x64 bundle exec rake beaker ``` You can replace the string `debian10` with any common operating system. The following strings are known to work: * ubuntu1604 * ubuntu1804 * ubuntu2004 * debian9 * debian10 * centos7 * centos8 -The easiest way to debug in a docker container is to open a shell: - -```sh -docker exec -it -u root ${container_id_or_name} bash -``` +For more information and tips & tricks, see [voxpupuli-acceptance's documentation](https://github.com/voxpupuli/voxpupuli-acceptance#running-tests). The source of this file is in our [modulesync_config](https://github.com/voxpupuli/modulesync_config/blob/master/moduleroot/.github/CONTRIBUTING.md.erb) repository. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b4f47e8..150e6ac 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,75 +1,79 @@ +--- +# Managed by modulesync - DO NOT EDIT +# https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ + name: CI on: pull_request jobs: setup_matrix: name: 'Setup Test Matrix' runs-on: ubuntu-latest + timeout-minutes: 40 outputs: beaker_setfiles: ${{ steps.get-outputs.outputs.beaker_setfiles }} puppet_major_versions: ${{ steps.get-outputs.outputs.puppet_major_versions }} puppet_unit_test_matrix: ${{ steps.get-outputs.outputs.puppet_unit_test_matrix }} env: - BUNDLE_WITHOUT: development:test:release + BUNDLE_WITHOUT: development:system_tests:release steps: - uses: actions/checkout@v2 - name: Setup ruby uses: ruby/setup-ruby@v1 with: - ruby-version: '2.7' + ruby-version: '3.0' bundler-cache: true - - name: Run rake validate - run: bundle exec rake validate + - name: Run static validations + run: bundle exec rake validate lint check + - name: Run rake rubocop + run: bundle exec rake rubocop - name: Setup Test Matrix id: get-outputs run: bundle exec metadata2gha --use-fqdn --pidfile-workaround false unit: needs: setup_matrix runs-on: ubuntu-latest + timeout-minutes: 40 strategy: fail-fast: false matrix: include: ${{fromJson(needs.setup_matrix.outputs.puppet_unit_test_matrix)}} env: BUNDLE_WITHOUT: development:system_tests:release PUPPET_VERSION: "~> ${{ matrix.puppet }}.0" name: Puppet ${{ matrix.puppet }} (Ruby ${{ matrix.ruby }}) steps: - uses: actions/checkout@v2 - name: Setup ruby uses: ruby/setup-ruby@v1 with: ruby-version: ${{ matrix.ruby }} bundler-cache: true - name: Run tests - run: bundle exec rake + run: bundle exec rake parallel_spec acceptance: needs: setup_matrix runs-on: ubuntu-latest env: BUNDLE_WITHOUT: development:test:release strategy: fail-fast: false matrix: setfile: ${{fromJson(needs.setup_matrix.outputs.beaker_setfiles)}} puppet: ${{fromJson(needs.setup_matrix.outputs.puppet_major_versions)}} name: ${{ matrix.puppet.name }} - ${{ matrix.setfile.name }} steps: - - name: Enable IPv6 on docker - run: | - echo '{"ipv6":true,"fixed-cidr-v6":"2001:db8:1::/64"}' | sudo tee /etc/docker/daemon.json - sudo service docker restart - uses: actions/checkout@v2 - name: Setup ruby uses: ruby/setup-ruby@v1 with: - ruby-version: '2.7' + ruby-version: '3.0' bundler-cache: true - name: Run tests run: bundle exec rake beaker env: BEAKER_PUPPET_COLLECTION: ${{ matrix.puppet.collection }} BEAKER_setfile: ${{ matrix.setfile.value }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 68b8528..664ba69 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -1,29 +1,34 @@ +--- +# Managed by modulesync - DO NOT EDIT +# https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ + name: Release on: push: tags: - '*' env: BUNDLE_WITHOUT: development:test:system_tests jobs: deploy: name: 'deploy to forge' runs-on: ubuntu-latest + if: github.repository_owner == 'voxpupuli' steps: - name: Checkout repository uses: actions/checkout@v2 - name: Setup Ruby uses: ruby/setup-ruby@v1 with: ruby-version: '2.7' bundler-cache: true - name: Build and Deploy env: # Configure secrets here: # https://docs.github.com/en/free-pro-team@latest/actions/reference/encrypted-secrets BLACKSMITH_FORGE_USERNAME: '${{ secrets.PUPPET_FORGE_USERNAME }}' BLACKSMITH_FORGE_API_KEY: '${{ secrets.PUPPET_FORGE_API_KEY }}' run: bundle exec rake module:push diff --git a/.gitignore b/.gitignore index e9b3cf4..9b95224 100644 --- a/.gitignore +++ b/.gitignore @@ -1,20 +1,23 @@ +# Managed by modulesync - DO NOT EDIT +# https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ + pkg/ Gemfile.lock Gemfile.local vendor/ .vendor/ spec/fixtures/manifests/ spec/fixtures/modules/ .vagrant/ .bundle/ .ruby-version coverage/ log/ .idea/ .dependencies/ .librarian/ Puppetfile.lock *.iml .*.sw? .yardoc/ Guardfile diff --git a/.msync.yml b/.msync.yml index a0770a8..9c9f18f 100644 --- a/.msync.yml +++ b/.msync.yml @@ -1,2 +1,5 @@ --- -modulesync_config_version: '4.0.0' +# Managed by modulesync - DO NOT EDIT +# https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ + +modulesync_config_version: '4.1.0' diff --git a/.overcommit.yml b/.overcommit.yml index 0af0fdc..d367ada 100644 --- a/.overcommit.yml +++ b/.overcommit.yml @@ -1,64 +1,65 @@ -# Managed by https://github.com/voxpupuli/modulesync_configs +# Managed by modulesync - DO NOT EDIT +# https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ # # Hooks are only enabled if you take action. # # To enable the hooks run: # # ``` # bundle exec overcommit --install # # ensure .overcommit.yml does not harm to you and then # bundle exec overcommit --sign # ``` # # (it will manage the .git/hooks directory): # # Examples howto skip a test for a commit or push: # # ``` # SKIP=RuboCop git commit # SKIP=PuppetLint git commit # SKIP=RakeTask git push # ``` # # Don't invoke overcommit at all: # # ``` # OVERCOMMIT_DISABLE=1 git commit # ``` # # Read more about overcommit: https://github.com/brigade/overcommit # # To manage this config yourself in your module add # # ``` # .overcommit.yml: # unmanaged: true # ``` # # to your modules .sync.yml config --- PreCommit: RuboCop: enabled: true description: 'Runs rubocop on modified files only' command: ['bundle', 'exec', 'rubocop'] PuppetLint: enabled: true description: 'Runs puppet-lint on modified files only' command: ['bundle', 'exec', 'puppet-lint'] YamlSyntax: enabled: true JsonSyntax: enabled: true TrailingWhitespace: enabled: true PrePush: RakeTarget: enabled: true description: 'Run rake targets' targets: - 'validate' - 'test' - 'rubocop' command: ['bundle', 'exec', 'rake'] diff --git a/.pmtignore b/.pmtignore index 33a8c65..65f5051 100644 --- a/.pmtignore +++ b/.pmtignore @@ -1,34 +1,37 @@ +# Managed by modulesync - DO NOT EDIT +# https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ + docs/ pkg/ Gemfile Gemfile.lock Gemfile.local vendor/ .vendor/ spec/ Rakefile .vagrant/ .bundle/ .ruby-version coverage/ log/ .idea/ .dependencies/ .github/ .librarian/ Puppetfile.lock *.iml .editorconfig .fixtures.yml .gitignore .msync.yml .overcommit.yml .pmtignore .rspec .rspec_parallel .rubocop.yml .sync.yml .*.sw? .yardoc/ .yardopts Dockerfile diff --git a/.rspec b/.rspec index 8c18f1a..f634583 100644 --- a/.rspec +++ b/.rspec @@ -1,2 +1,5 @@ +# Managed by modulesync - DO NOT EDIT +# https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ + --format documentation --color diff --git a/.rspec_parallel b/.rspec_parallel index e4d136b..a9a84f8 100644 --- a/.rspec_parallel +++ b/.rspec_parallel @@ -1 +1,4 @@ +# Managed by modulesync - DO NOT EDIT +# https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ + --format progress diff --git a/.rubocop.yml b/.rubocop.yml index 198a359..53ac189 100644 --- a/.rubocop.yml +++ b/.rubocop.yml @@ -1,3 +1,6 @@ --- +# Managed by modulesync - DO NOT EDIT +# https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ + inherit_gem: voxpupuli-test: rubocop.yml diff --git a/.yardopts b/.yardopts deleted file mode 100644 index 3687f51..0000000 --- a/.yardopts +++ /dev/null @@ -1,2 +0,0 @@ ---markup markdown ---output-dir docs/ diff --git a/Dockerfile b/Dockerfile index 6fd6342..e3cf307 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,21 +1,24 @@ -FROM ruby:2.5.3 +# MANAGED BY MODULESYNC +# https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ + +FROM ruby:2.7 WORKDIR /opt/puppet # https://github.com/puppetlabs/puppet/blob/06ad255754a38f22fb3a22c7c4f1e2ce453d01cb/lib/puppet/provider/service/runit.rb#L39 RUN mkdir -p /etc/sv ARG PUPPET_VERSION="~> 6.0" ARG PARALLEL_TEST_PROCESSORS=4 # Cache gems COPY Gemfile . RUN bundle install --without system_tests development release --path=${BUNDLE_PATH:-vendor/bundle} COPY . . RUN bundle install RUN bundle exec rake release_checks # Container should not saved RUN exit 1 diff --git a/Gemfile b/Gemfile index 0d0a9fb..1a09527 100644 --- a/Gemfile +++ b/Gemfile @@ -1,33 +1,34 @@ +# Managed by modulesync - DO NOT EDIT +# https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ + source ENV['GEM_SOURCE'] || "https://rubygems.org" group :test do - gem 'voxpupuli-test', '~> 2.1', :require => false - gem 'coveralls', :require => false - gem 'simplecov-console', :require => false + gem 'voxpupuli-test', '~> 2.1', :require => false + gem 'coveralls', :require => false + gem 'simplecov-console', :require => false + gem 'puppet_metadata', '~> 0.3.0', :require => false end group :development do gem 'guard-rake', :require => false gem 'overcommit', '>= 0.39.1', :require => false end group :system_tests do - gem 'puppet_metadata', '~> 0.3.0', :require => false - gem 'voxpupuli-acceptance', :require => false + gem 'voxpupuli-acceptance', '~> 1.0', :require => false end group :release do - gem 'github_changelog_generator', :require => false, :git => 'https://github.com/voxpupuli/github-changelog-generator', :branch => 'voxpupuli_essential_fixes' - gem 'puppet-blacksmith', :require => false - gem 'voxpupuli-release', :require => false - gem 'puppet-strings', '>= 2.2', :require => false + gem 'github_changelog_generator', '>= 1.16.1', :require => false + gem 'voxpupuli-release', '>= 1.0.2', :require => false + gem 'puppet-strings', '>= 2.2', :require => false end -gem 'puppetlabs_spec_helper', '~> 2.0', :require => false gem 'rake', :require => false gem 'facter', ENV['FACTER_GEM_VERSION'], :require => false, :groups => [:test] -puppetversion = ENV['PUPPET_VERSION'] || '~> 6.0' +puppetversion = ENV['PUPPET_VERSION'] || '>= 6.0' gem 'puppet', puppetversion, :require => false, :groups => [:test] # vim: syntax=ruby diff --git a/README.md b/README.md index 853c03a..590c781 100644 --- a/README.md +++ b/README.md @@ -1,132 +1,136 @@ # Kafka module for Puppet -[![Build Status](https://travis-ci.org/voxpupuli/puppet-kafka.png?branch=master)](https://travis-ci.org/voxpupuli/puppet-kafka) +[![Build Status](https://github.com/voxpupuli/puppet-kafka/workflows/CI/badge.svg)](https://github.com/voxpupuli/puppet-kafka/actions?query=workflow%3ACI) +[![Release](https://github.com/voxpupuli/puppet-kafka/actions/workflows/release.yml/badge.svg)](https://github.com/voxpupuli/puppet-kafka/actions/workflows/release.yml) [![Puppet Forge](https://img.shields.io/puppetforge/v/puppet/kafka.svg)](https://forge.puppetlabs.com/puppet/kafka) [![Puppet Forge - downloads](https://img.shields.io/puppetforge/dt/puppet/kafka.svg)](https://forge.puppetlabs.com/puppet/kafka) [![Puppet Forge - endorsement](https://img.shields.io/puppetforge/e/puppet/kafka.svg)](https://forge.puppetlabs.com/puppet/kafka) [![Puppet Forge - scores](https://img.shields.io/puppetforge/f/puppet/kafka.svg)](https://forge.puppetlabs.com/puppet/kafka) +[![puppetmodule.info docs](http://www.puppetmodule.info/images/badge.png)](http://www.puppetmodule.info/m/puppet-kafka) +[![MIT License](https://img.shields.io/github/license/voxpupuli/puppet-kafka.svg)](LICENSE) +[![Donated by Liam Bennett](https://img.shields.io/badge/donated%20by-Liam%20Bennett-fb7047.svg)](#Author) ## Table of Contents 1. [Overview](#overview) 1. [Module Description - What the module does and why it is useful](#module-description) 1. [Setup - The basics of getting started with Kafka](#setup) * [What Kafka affects](#what-kafka-affects) * [Setup requirements](#setup-requirements) * [Beginning with Kafka](#beginning-with-kafka) 1. [Usage - Configuration options and additional functionality](#usage) 1. [Reference - An under-the-hood peek at what the module is doing and how](#reference) 1. [Limitations - OS compatibility, etc.](#limitations) 1. [Development - Guide for contributing to the module](#development) ## Overview The Kafka module for managing the installation and configuration of [Apache Kafka](http://kafka.apache.org). ## Module Description The Kafka module for managing the installation and configuration of Apache Kafka: it's brokers, producers and consumers. ## Setup ### What Kafka affects Installs the Kafka package and creates a new service. ### Setup requirements This module has the following dependencies: * [deric/zookeeper](https://github.com/deric/puppet-zookeeper) * [camptocamp/systemd](https://github.com/camptocamp/puppet-systemd) * [puppet/archive](https://github.com/voxpupuli/puppet-archive) * [puppetlabs/java](https://github.com/puppetlabs/puppetlabs-java) * [puppetlabs/stdlib](https://github.com/puppetlabs/puppetlabs-stdlib) ### Beginning with Kafka To successfully install Kafka using this module you need to have Apache ZooKeeper already running at localhost:2181. You can specify another ZooKeeper host:port configuration using the config hash of the kafka::broker class. The default configuration installs Kafka 0.11.0.3 binaries with Scala 2.11: ```puppet class { 'kafka': } ``` If you want a Kafka broker server that connects to ZooKeeper listening on port 2181: ```puppet class { 'kafka::broker': config => { 'broker.id' => '0', 'zookeeper.connect' => 'localhost:2181' } } ``` ## Usage You can specify different Kafka binaries packages versions to install. Please take a look at the different Scala and Kafka versions combinations at the [Apache Kafka Website](http://kafka.apache.org/downloads.html) ### Installing Kafka version 1.1.0 with scala 2.12 We first install the binary package with: ```puppet class { 'kafka': version => '1.1.0', scala_version => '2.12' } ``` Then we set a minimal Kafka broker configuration with: ```puppet class { 'kafka::broker': config => { 'broker.id' => '0', 'zookeeper.connect' => 'localhost:2181' } } ``` ## Reference The [reference][1] documentation of this module is generated using [puppetlabs/puppetlabs-strings][2]. ## Limitations This module only supports Kafka >= 0.9.0.0. This module is tested on the following platforms: * Debian 8 * Debian 9 * Debian 10 * Ubuntu 16.04 * Ubuntu 18.04 * CentOS 7 It is tested with the OSS version of Puppet (>= 5.5) only. ## Development This module has grown over time based on a range of contributions from people using it. If you follow these [contributing][3] guidelines your patch will likely make it into a release a little more quickly. ## Author This module is maintained by [Vox Pupuli][4]. It was originally written and maintained by [Liam Bennett][5]. [1]: https://github.com/voxpupuli/puppet-kafka/blob/master/REFERENCE.md [2]: https://github.com/puppetlabs/puppetlabs-strings [3]: https://github.com/voxpupuli/puppet-kafka/blob/master/.github/CONTRIBUTING.md [4]: https://voxpupuli.org [5]: https://www.opentable.com diff --git a/Rakefile b/Rakefile index d1bf749..80b799d 100644 --- a/Rakefile +++ b/Rakefile @@ -1,61 +1,72 @@ +# Managed by modulesync - DO NOT EDIT +# https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ + # Attempt to load voxupuli-test (which pulls in puppetlabs_spec_helper), # otherwise attempt to load it directly. begin require 'voxpupuli/test/rake' rescue LoadError - require 'puppetlabs_spec_helper/rake_tasks' + begin + require 'puppetlabs_spec_helper/rake_tasks' + rescue LoadError + end +end + +# load optional tasks for acceptance +# only available if gem group releases is installed +begin + require 'voxpupuli/acceptance/rake' +rescue LoadError end # load optional tasks for releases # only available if gem group releases is installed begin require 'voxpupuli/release/rake_tasks' rescue LoadError end desc "Run main 'test' task and report merged results to coveralls" task test_with_coveralls: [:test] do if Dir.exist?(File.expand_path('../lib', __FILE__)) require 'coveralls/rake/task' Coveralls::RakeTask.new Rake::Task['coveralls:push'].invoke else puts 'Skipping reporting to coveralls. Module has no lib dir' end end desc 'Generate REFERENCE.md' task :reference, [:debug, :backtrace] do |t, args| patterns = '' Rake::Task['strings:generate:reference'].invoke(patterns, args[:debug], args[:backtrace]) end begin require 'github_changelog_generator/task' require 'puppet_blacksmith' GitHubChangelogGenerator::RakeTask.new :changelog do |config| - version = (Blacksmith::Modulefile.new).version - config.future_release = "v#{version}" if version =~ /^\d+\.\d+.\d+$/ + metadata = Blacksmith::Modulefile.new + config.future_release = "v#{metadata.version}" if metadata.version =~ /^\d+\.\d+.\d+$/ config.header = "# Changelog\n\nAll notable changes to this project will be documented in this file.\nEach new release typically also includes the latest modulesync defaults.\nThese should not affect the functionality of the module." config.exclude_labels = %w{duplicate question invalid wontfix wont-fix modulesync skip-changelog} config.user = 'voxpupuli' - metadata_json = File.join(File.dirname(__FILE__), 'metadata.json') - metadata = JSON.load(File.read(metadata_json)) - config.project = metadata['name'] + config.project = metadata.metadata['name'] end # Workaround for https://github.com/github-changelog-generator/github-changelog-generator/issues/715 require 'rbconfig' if RbConfig::CONFIG['host_os'] =~ /linux/ task :changelog do puts 'Fixing line endings...' changelog_file = File.join(__dir__, 'CHANGELOG.md') changelog_txt = File.read(changelog_file) new_contents = changelog_txt.gsub(%r{\r\n}, "\n") File.open(changelog_file, "w") {|file| file.puts new_contents } end end rescue LoadError end # vim: syntax=ruby diff --git a/spec/classes/broker_spec.rb b/spec/classes/broker_spec.rb index 2758b33..0c3d309 100644 --- a/spec/classes/broker_spec.rb +++ b/spec/classes/broker_spec.rb @@ -1,100 +1,102 @@ require 'spec_helper' require 'shared_examples_param_validation' describe 'kafka::broker', type: :class do on_supported_os.each do |os, os_facts| context "on #{os}" do let(:facts) do os_facts end let :params do { config: { 'zookeeper.connect' => 'localhost:2181' } } end it { is_expected.to contain_class('kafka::broker::install').that_comes_before('Class[kafka::broker::config]') } it { is_expected.to contain_class('kafka::broker::config').that_comes_before('Class[kafka::broker::service]') } it { is_expected.to contain_class('kafka::broker::service').that_comes_before('Class[kafka::broker]') } it { is_expected.to contain_class('kafka::broker') } context 'with manage_log4j => true' do - let(:params) { {'manage_log4j' => true} } + let(:params) { { 'manage_log4j' => true } } + it { is_expected.to contain_class('kafka::broker::config').with('log_file_size' => '50MB', 'log_file_count' => 7) } end describe 'kafka::broker::install' do context 'defaults' do it { is_expected.to contain_class('kafka') } end end describe 'kafka::broker::config' do context 'defaults' do it { is_expected.to contain_file('/opt/kafka/config/server.properties') } end context 'with manage_log4j => true' do - let(:params) { {'manage_log4j' => true} } + let(:params) { { 'manage_log4j' => true } } + it { is_expected.to contain_file('/opt/kafka/config/log4j.properties').with_content(%r{^log4j.appender.kafkaAppender.MaxFileSize=50MB$}) } it { is_expected.to contain_file('/opt/kafka/config/log4j.properties').with_content(%r{^log4j.appender.kafkaAppender.MaxBackupIndex=7$}) } end end describe 'kafka::broker::service' do context 'manage_service false' do let(:params) { super().merge(manage_service: false) } it { is_expected.not_to contain_file('/etc/init.d/kafka') } it { is_expected.not_to contain_file('/etc/systemd/system/kafka.service') } it { is_expected.not_to contain_service('kafka') } end context 'defaults' do if os_facts[:service_provider] == 'systemd' it { is_expected.to contain_file('/etc/init.d/kafka').with_ensure('absent') } it { is_expected.to contain_file('/etc/systemd/system/kafka.service').with_content %r{^After=network\.target syslog\.target$} } it { is_expected.to contain_file('/etc/systemd/system/kafka.service').with_content %r{^Wants=network\.target syslog\.target$} } it { is_expected.not_to contain_file('/etc/systemd/system/kafka.service').with_content %r{^LimitNOFILE=} } it { is_expected.not_to contain_file('/etc/systemd/system/kafka.service').with_content %r{^LimitCORE=} } else it { is_expected.to contain_file('/etc/init.d/kafka') } end it { is_expected.to contain_service('kafka') } end context 'limit_nofile set' do let(:params) { super().merge(limit_nofile: '65536') } if os_facts[:service_provider] == 'systemd' it { is_expected.to contain_file('/etc/systemd/system/kafka.service').with_content %r{^LimitNOFILE=65536$} } else it { is_expected.to contain_file('/etc/init.d/kafka').with_content %r{ulimit -n 65536$} } end end context 'limit_core set' do let(:params) { super().merge(limit_core: 'infinity') } if os_facts[:service_provider] == 'systemd' it { is_expected.to contain_file('/etc/systemd/system/kafka.service').with_content %r{^LimitCORE=infinity$} } else it { is_expected.to contain_file('/etc/init.d/kafka').with_content %r{ulimit -c infinity$} } end end context 'service_requires set', if: os_facts[:service_provider] == 'systemd' do let(:params) { super().merge(service_requires: ['dummy.target']) } it { is_expected.to contain_file('/etc/systemd/system/kafka.service').with_content %r{^After=dummy\.target$} } it { is_expected.to contain_file('/etc/systemd/system/kafka.service').with_content %r{^Wants=dummy\.target$} } end end it_validates_parameter 'mirror_url' end end end diff --git a/spec/classes/consumer_spec.rb b/spec/classes/consumer_spec.rb index a023950..137972e 100644 --- a/spec/classes/consumer_spec.rb +++ b/spec/classes/consumer_spec.rb @@ -1,62 +1,64 @@ require 'spec_helper' require 'shared_examples_param_validation' describe 'kafka::consumer', type: :class do on_supported_os.each do |os, os_facts| context "on #{os}" do let(:facts) do os_facts end let :params do { service_config: { 'topic' => 'demo', 'zookeeper' => 'localhost:2181' } } end it { is_expected.to contain_class('kafka::consumer::install').that_comes_before('Class[kafka::consumer::config]') } it { is_expected.to contain_class('kafka::consumer::config').that_comes_before('Class[kafka::consumer::service]') } it { is_expected.to contain_class('kafka::consumer::service').that_comes_before('Class[kafka::consumer]') } it { is_expected.to contain_class('kafka::consumer') } context 'with manage_log4j => true' do - let(:params) { {'manage_log4j' => true} } + let(:params) { { 'manage_log4j' => true } } + it { is_expected.to contain_class('kafka::consumer::config').with('log_file_size' => '50MB', 'log_file_count' => 7) } end describe 'kafka::consumer::install' do context 'defaults' do it { is_expected.to contain_class('kafka') } end end describe 'kafka::consumer::config' do context 'defaults' do it { is_expected.to contain_file('/opt/kafka/config/consumer.properties') } end context 'with manage_log4j => true' do - let(:params) { {'manage_log4j' => true} } + let(:params) { { 'manage_log4j' => true } } + it { is_expected.to contain_file('/opt/kafka/config/log4j.properties').with_content(%r{^log4j.appender.kafkaAppender.MaxFileSize=50MB$}) } it { is_expected.to contain_file('/opt/kafka/config/log4j.properties').with_content(%r{^log4j.appender.kafkaAppender.MaxBackupIndex=7$}) } end end describe 'kafka::consumer::service' do context 'defaults' do if os_facts[:service_provider] == 'systemd' it { is_expected.to contain_file('/etc/init.d/kafka-consumer').with_abent('absent') } else it { is_expected.to contain_file('/etc/init.d/kafka-consumer') } end it { is_expected.to contain_service('kafka-consumer') } end end it_validates_parameter 'mirror_url' end end end diff --git a/spec/classes/mirror_spec.rb b/spec/classes/mirror_spec.rb index b1ce157..e76caf5 100644 --- a/spec/classes/mirror_spec.rb +++ b/spec/classes/mirror_spec.rb @@ -1,63 +1,64 @@ require 'spec_helper' require 'shared_examples_param_validation' describe 'kafka::mirror', type: :class do on_supported_os.each do |os, os_facts| context "on #{os}" do let(:facts) do os_facts end let :params do { consumer_config: { 'group.id' => 'kafka-mirror', 'zookeeper.connect' => 'localhost:2181' }, producer_config: { 'bootstrap.servers' => 'localhost:9092' } } end it { is_expected.to contain_class('kafka::mirror::install').that_comes_before('Class[kafka::mirror::config]') } it { is_expected.to contain_class('kafka::mirror::config').that_comes_before('Class[kafka::mirror::service]') } it { is_expected.to contain_class('kafka::mirror::service').that_comes_before('Class[kafka::mirror]') } it { is_expected.to contain_class('kafka::mirror') } context 'with manage_log4j => true' do - let(:params) { {'manage_log4j' => true} } + let(:params) { { 'manage_log4j' => true } } + it { is_expected.to contain_class('kafka::mirror::config').with('log_file_size' => '50MB', 'log_file_count' => 7) } end describe 'kafka::mirror::install' do context 'defaults' do it { is_expected.to contain_class('kafka') } end end describe 'kafka::mirror::config' do context 'defaults' do it { is_expected.to contain_class('kafka::consumer::config') } it { is_expected.to contain_class('kafka::producer::config') } end end describe 'kafka::mirror::service' do context 'defaults' do if os_facts[:service_provider] == 'systemd' it { is_expected.to contain_file('/etc/init.d/kafka-mirror').with_ensure('absent') } it { is_expected.to contain_file('/etc/systemd/system/kafka-mirror.service').with_content %r{/opt/kafka/config/(?=.*consumer)|(?=.*producer).propertie} } else it { is_expected.to contain_file('/etc/init.d/kafka-mirror') } it { is_expected.to contain_file('/etc/init.d/kafka-mirror').with_content %r{/opt/kafka/config/(?=.*consumer)|(?=.*producer).properties} } end it { is_expected.to contain_service('kafka-mirror') } end end it_validates_parameter 'mirror_url' end end end diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index d266f6b..fb5f0cb 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -1,18 +1,17 @@ -# This file is managed via modulesync -# https://github.com/voxpupuli/modulesync -# https://github.com/voxpupuli/modulesync_config +# Managed by modulesync - DO NOT EDIT +# https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ # puppetlabs_spec_helper will set up coverage if the env variable is set. # We want to do this if lib exists and it hasn't been explicitly set. ENV['COVERAGE'] ||= 'yes' if Dir.exist?(File.expand_path('../../lib', __FILE__)) require 'voxpupuli/test/spec_helper' if File.exist?(File.join(__dir__, 'default_module_facts.yml')) facts = YAML.safe_load(File.read(File.join(__dir__, 'default_module_facts.yml'))) if facts facts.each do |name, value| add_custom_fact name.to_sym, value end end end