diff --git a/README.md b/README.md index ceb2556..afbb9a7 100644 --- a/README.md +++ b/README.md @@ -1,380 +1,381 @@ # puppet-zookeeper [![Puppet Forge](http://img.shields.io/puppetforge/v/deric/zookeeper.svg)](https://forge.puppetlabs.com/deric/zookeeper) [![Build Status](https://travis-ci.org/deric/puppet-zookeeper.png?branch=master)](https://travis-ci.org/deric/puppet-zookeeper) [![Puppet Forge Downloads](http://img.shields.io/puppetforge/dt/deric/zookeeper.svg)](https://forge.puppetlabs.com/deric/zookeeper/scores) A puppet receipt for [Apache Zookeeper](http://zookeeper.apache.org/). ZooKeeper is a high-performance coordination service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. ## Requirements * Puppet * Binary or ZooKeeper source code archive Compatibility matrix: | `puppet-zookeeper`| Puppet 3.x | Puppet 4.x | Puppet 5.x | Puppet 6.x | | ----------------- | ------------- |--------------| -----------|----------------| | `0.7.x` | :heavy_check_mark: | :heavy_check_mark: | :question: | :x: | | `0.8.x` | :x: | :heavy_check_mark: | :heavy_check_mark: | :x: | | `1.0.x` | :x: | :x: | :heavy_check_mark: | :heavy_check_mark: | ## Basic Usage: ```puppet class { 'zookeeper': } ``` ## Cluster setup When running ZooKeeper in the distributed mode each node must have unique ID (`1-255`). The easiest way how to setup multiple ZooKeepers, is by using Hiera. `hiera/host/zk1.example.com.yaml`: ```yaml zookeeper::id: '1' ``` `hiera/host/zk2.example.com.yaml`: ```yaml zookeeper::id: '2' ``` `hiera/host/zk3.example.com.yaml`: ```yaml zookeeper::id: '3' ``` A ZooKeeper quorum should consist of odd number of nodes (usually `3` or `5`). For defining a quorum it is enough to list all IP addresses of all its members. ```puppet class { 'zookeeper': servers => { 1 => '192.168.1.1', 2 => '192.168.1.2', 3 => '192.168.1.3', }, } ``` In case that an array is passed as `servers`, first ZooKeeper will be assigned `ID = 1`. This would produce following configuration: ``` server.1=192.168.1.1:2888:3888 server.2=192.168.1.2:2888:3888 server.3=192.168.1.3:2888:3888 ``` where first port is `election_port` and second one `leader_port`. Both ports could be customized for each ZooKeeper instance. ```puppet class { 'zookeeper': election_port => 2889, leader_port => 3889, servers => { 1 => '192.168.1.1', 2 => '192.168.1.2', 3 => '192.168.1.3', } } ``` ### Observers [Observers](http://zookeeper.apache.org/doc/r3.3.0/zookeeperObservers.html) were introduced in ZooKeeper 3.3.0. To enable this feature simply state which of ZooKeeper servers are observing: ```puppet class { 'zookeeper': servers => ['192.168.1.1', '192.168.1.2', '192.168.1.3', '192.168.1.4', '192.168.1.5'], observers => ['192.168.1.4', '192.168.1.5'] } ``` **Note**: Currently observer server needs to be listed between standard servers (this behavior might change in feature). ### Set binding interface By default ZooKeeper should bind to all interfaces. When you specify `client_ip` only single interface will be used. If `$::ipaddress` is not your public IP (e.g. you are using Docker) make sure to setup correct IP: ```puppet class { 'zookeeper': client_ip => $::ipaddress_eth0 } ``` or in Hiera: ```yaml zookeeper::client_ip: "%{::ipaddress_eth0}" ``` This is a workaround for a a [Facter issue](https://tickets.puppetlabs.com/browse/FACT-380). ### ZooKeeper service Use `service_provider` to override Puppet detection for starting service. ```puppet class { 'zookeeper': service_provider => 'init', manage_service_file => false, } ``` Some reasonable values are: * `init` * `systemd` * `runit` * `exhibitor` - zookeeper process and config will be managed by exhibitor (https://github.com/soabase/exhibitor). Exhibitor is not managed by this module. * `none` - service won't be installed Parameter `manage_service_file` controls whether service definition should be managed by Puppet (default: `false`). Currently supported for `systemd` and `init`. ### Systemd Unit 'After' and 'Want' control By default the module will create the following Unit section in /etc/systemd/system/multi-user.target.wants/zookeeper.service ```` [Unit] Description=Apache ZooKeeper After=network.target ```` Both After and Want (omitted when using the module defaults) can be controled using this module. E.g on CentOS 7 those might have to be configured for 'netwrok-online.target' using the following syntax: ```puppet class { 'zookeeper': systemd_unit_after => 'network-online.target', systemd_unit_want => 'network-online.target', } ``` Which will modify the Unit section to look like: ```` [Unit] Description=Apache ZooKeeper Want=network-online.target After=network-online.target ```` ## Parameters - `id` - cluster-unique zookeeper's instance id (1-255) - `datastore` - `datalogstore` - specifying this configures the `dataLogDir` ZooKeeper config values and allows for transaction logs to be stored in a different location, improving IO performance - `log_dir` - `purge_interval` - automatically will delete ZooKeeper logs (available since ZooKeeper 3.4.0) - `snap_retain_count` - number of snapshots that will be kept after purging (since ZooKeeper 3.4.0) - `min_session_timeout` - the minimum session timeout in milliseconds that the server will allow the client to negotiate. Defaults to 2 times the **tickTime** (since ZooKeeper 3.3.0) - `max_session_timeout` - the maximum session timeout in milliseconds that the server will allow the client to negotiate. Defaults to 20 times the **tickTime** (since ZooKeeper 3.3.0) - `global_outstanding_limit` - ZooKeeper will throttle clients so that there is no more than `global_outstanding_limit` outstanding requests in the system. - `manage_service` (default: `true`) whether Puppet should ensure running service - `manage_service_file` when enabled on RHEL 7.0 a systemd config will be managed - `ensure_account` controls whether `zookeeper` user and group will be ensured (set to `false` to disable this feature) - `install_method` controls whether ZooKeeper is installed from binary (`package`) or source (`archive`) packages - `archive_version` allows to specify an arbitrary version of ZooKeeper when using source packages - `archive_install_dir` controls the installation directory when using source packages (defaults to `/opt`) - `archive_symlink` controls the name of a version-independent symlink when using source packages - `archive_dl_url` allows to change the download URL for source packages (defaults to apache.org) - `systemd_path` where to put `systemd` service files (applies only if `manage_service_file` and `service_provider == 'systemd'`) - `restart_on_change` whether ZooKeeper service should be restarted on configuration files change (default: `true`) - `remove_host_principal` whether to remove host from Kerberos principal (default: `false`) - `remove_realm_principal` whether to remove relam from Kerberos principal (default: `false`) - `whitelist_4lw` Fine grained control over the set of commands ZooKeeper can execute (an array e.g. `whitelist_4lw = ['*']`) and many others, see the `params.pp` file for more details. If your distribution has multiple packages for ZooKeeper, you can provide all package names as an array. ```puppet class { 'zookeeper': packages => ['zookeeper', 'zookeeper-java'] } ``` ## Logging ZooKeeper uses log4j, following variables can be configured: ```puppet class { 'zookeeper': console_threshold => 'INFO', rollingfile_threshold => 'INFO', tracefile_threshold => 'TRACE', maxfilesize => '256MB', maxbackupindex => 20, } ``` Threshold supported values are: `ALL`, `DEBUG`, `ERROR`, `FATAL`, `INFO`, `OFF`, `TRACE` and `WARN`. [Maxfilesize](https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/RollingFileAppender.html#maxFileSize) [MaxBackupIndex](https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/RollingFileAppender.html#maxBackupIndex) By default console, rolling file and trace logging can be configured. Additional log appenders (vulgo log methods) can be configured by adding a hash `extra_appenders`. The following sets up syslog logging and points the root logger towards syslog (note that you must have syslog listening on port 514/udp for this to work): ```puppet class { 'zookeeper': log4j_prop => 'INFO,SYSLOG', extra_appenders => { 'Syslog' => { 'class' => 'org.apache.log4j.net.SyslogAppender', 'layout' => 'org.apache.log4j.PatternLayout', 'layout.conversionPattern' => "${hostname} zookeeper[id:%X{myid}] - %-5p [%t:%C{1}@%L][%x] - %m%n", 'syslogHost' => 'localhost', 'facility' => 'user', }, }, } ``` ## Hiera Support All parameters could be defined in hiera files, e.g. `common.yaml`, `Debian.yaml` or `zookeeper.yaml`: ```yaml zookeeper::id: 1 zookeeper::client_port: 2181 zookeeper::datastore: '/var/lib/zookeeper' zookeeper::datalogstore: '/disk2/zookeeper' ``` ## Cloudera package In Cloudera distribution ZooKeeper package does not provide init scripts (same as in Debian). Package containing init scripts is called `zookeeper-server` and the service as well. Moreover there's initialization script which should be called after installation. So, the configuration might look like this: ```puppet class { 'zookeeper': packages => ['zookeeper', 'zookeeper-server'], service_name => 'zookeeper-server', initialize_datastore => true } ``` ### Managing repository For RedHat family currently we support also managing a `cloudera` yum repo versions 4, and 5. It can be enabled with `repo` parameter: ```puppet class { 'zookeeper': repo => 'cloudera', cdhver => '5', } ``` #### Custom RPM repository Optionally you can specify a custom repository, using a hash configuration. ```puppet class { 'zookeeper': cdhver => '5', repo => { name => 'myrepo', url => 'http://cusom.url', descr => 'description' } } ``` ## Source package Source packages provide the ability to install arbitrary versions of ZooKeeper on any platform. Note that you'll likely have to use the `manage_service_file` in order to be able to control the ZooKeeper service (because source packages do not install service files). ```puppet class { 'zookeeper': install_method => 'archive', archive_version => '3.4.8', } ``` Optionally you can specify a `proxy_server`: ```puppet class { 'zookeeper': install_method => 'archive', archive_version => '3.4.8', proxy_server => 'http://10.0.0.1:8080' } ``` ## Java installation Default: `false` By changing these two parameters you can ensure, that given Java package will be installed before ZooKeeper packages. ```puppet class { 'zookeeper': install_java => true, java_package => 'openjdk-7-jre-headless' } ``` ## Install ### Librarian (recommended) For [puppet-librarian](https://github.com/rodjek/librarian-puppet) just add to `Puppetfile` from Forge: ```ruby mod 'deric-zookeeper' ``` latest (development) version from GitHub ```ruby mod 'deric-zookeeper', git: 'git://github.com/deric/puppet-zookeeper.git' ``` ### submodules If you are versioning your puppet conf with git just add it as submodule, from your repository root: git submodule add git://github.com/deric/puppet-zookeeper.git modules/zookeeper ## Dependencies * stdlib `> 2.3.3` - function `ensure_resources` is required * puppet-archive `> 0.4.4` - provides capabilities to use archives instead of binary packages ## Acceptance testing Fastest way is to run tests on prepared Docker images: ``` BEAKER_set=debian9-6.3 bundle exec rake acceptance ``` For examining system state set Beaker's ENV variable `BEAKER_destroy=no`: ``` BEAKER_destroy=no BEAKER_set=default bundle exec rake acceptance ``` and after finishing tests connect to container: ``` docker exec -it adoring_shirley bash ``` When host machine is NOT provisioned (puppet installed, etc.): ``` PUPPET_install=yes BEAKER_set=debian-8 bundle exec rake acceptance ``` Run on specific OS (see `spec/acceptance/nodesets`), to see available sets: ``` rake beaker:sets ``` ## Supported platforms * Debian/Ubuntu * RedHat/CentOS/Fedora ### Tested on: * Debian (8, 9, 10) * Ubuntu (16.04, 18.04) * RHEL (6, 7) * CentOS (6, 7) + * SLES (12) diff --git a/manifests/install.pp b/manifests/install.pp index 1a4ade1..d83028b 100644 --- a/manifests/install.pp +++ b/manifests/install.pp @@ -1,66 +1,66 @@ # Class: zookeeper::install # # This module manages installation tasks. # # PRIVATE CLASS - do not use directly (use main `zookeeper` class). class zookeeper::install inherits zookeeper { anchor { 'zookeeper::install::begin': } $os_family = $facts['os']['family'] # Repo management case $os_family { - 'RedHat': { + 'RedHat', 'Suse': { include zookeeper::install::repo Anchor['zookeeper::install::begin'] -> Class['zookeeper::install::repo'] } default: {} # nothing to do } # Java installation if ($zookeeper::install_java) { if !$zookeeper::java_package { fail('Java installation is required, but no java package was provided.') } validate_string($zookeeper::java_package) # Make sure the Java package is only installed once. anchor { 'zookeeper::install::java': } ensure_resource('package', $zookeeper::java_package, {'ensure' => $zookeeper::ensure, 'allow_virtual' => true, 'before' => Anchor['zookeeper::install::intermediate'], 'require' => Anchor['zookeeper::install::begin']} ) } anchor { 'zookeeper::install::intermediate': } # Package installation case $zookeeper::install_method { 'archive': { include zookeeper::install::archive Anchor['zookeeper::install::intermediate'] -> Class['zookeeper::install::archive'] -> Anchor['zookeeper::install::end'] } 'package': { include zookeeper::install::package Anchor['zookeeper::install::intermediate'] -> Class['zookeeper::install::package'] -> Anchor['zookeeper::install::end'] } default: { fail('Valid installation methods are `package` or `archive`') } } anchor { 'zookeeper::install::end': } # Post installation tasks class { 'zookeeper::post_install': require => Anchor['zookeeper::install::end'], } } diff --git a/manifests/install/repo.pp b/manifests/install/repo.pp index 01f0288..09f9f48 100644 --- a/manifests/install/repo.pp +++ b/manifests/install/repo.pp @@ -1,99 +1,104 @@ # == Class zookeeper::install::repo # # This class manages yum repository for Zookeeper packages # # PRIVATE CLASS - do not use directly (use main `zookeeper` class). class zookeeper::install::repo inherits zookeeper::install { + if $facts['os']['family'] == 'RedHat' { + $os_name = downcase($facts['os']['family']) + } else { + $os_name = downcase($facts['os']['name']) + } $os_release = $facts['os']['release']['major'] $os_hardware = $facts['os']['hardware'] if $zookeeper::repo_source { case $zookeeper::repo_source { undef: {} # nothing to do 'cloudera': { if $zookeeper::cdhver == undef { fail('Cloudera repo is required, but no CDH version is provided.') } case $zookeeper::cdhver { '4': { case $os_hardware { 'i386', 'x86_64': { case $os_release { '6', '7': { $release = '6' } default: { - fail("Yum repository '${zookeeper::repo_source}' is not supported for redhat version ${os_release}") + fail("Yum repository '${zookeeper::repo_source}' is not supported for ${os_name} version ${os_release}") } } } default: { fail("Yum repository '${zookeeper::repo_source}' is not supported for architecture ${os_hardware}") } } } '5': { case $os_hardware { 'x86_64': { case $os_release { # CentOS uses os_release=2015 - '6', '7', '2015': { + '6', '7', '12', '2015': { $release = $os_release } default: { - fail("Yum repository '${zookeeper::repo_source}' is not supported for redhat version ${os_release}") + fail("Yum repository '${zookeeper::repo_source}' is not supported for ${os_name} version ${os_release}") } } } default: { fail("Yum repository '${zookeeper::repo_source}' is not supported for architecture ${os_hardware}") } } } default: { fail("CDH version'${zookeeper::cdhver}' is not a supported cloudera repo.") } } # Puppet 4 compatibility: force variable to be a String yumrepo { "cloudera-cdh${zookeeper::cdhver}": ensure => $zookeeper::ensure, descr => "Cloudera's Distribution for Hadoop, Version ${zookeeper::cdhver}", - baseurl => "http://archive.cloudera.com/cdh${zookeeper::cdhver}/redhat/${release}/${os_hardware}/cdh/${zookeeper::cdhver}/", - gpgkey => "http://archive.cloudera.com/cdh${zookeeper::cdhver}/redhat/${release}/${os_hardware}/cdh/RPM-GPG-KEY-cloudera", + baseurl => "http://archive.cloudera.com/cdh${zookeeper::cdhver}/${os_name}/${release}/${os_hardware}/cdh/${zookeeper::cdhver}/", + gpgkey => "http://archive.cloudera.com/cdh${zookeeper::cdhver}/${os_name}/${release}/${os_hardware}/cdh/RPM-GPG-KEY-cloudera", gpgcheck => 1, } } 'custom':{ $_config = $zookeeper::repo validate_hash($_config) if $_config['name'] == undef or $_config['url'] == undef or $_config['descr'] == undef { fail('Invalid parameter settings for custom repo') } case $os_release { - '6', '7': { + '6', '7', '12': { # Puppet 4 compatibility: force variable to be a String yumrepo { $_config['name']: ensure => $zookeeper::ensure, descr => $_config['descr'], baseurl => $_config['url'], enabled => 1, sslverify => empty($_config['sslverify']) ? { true => 0, false => $_config['sslverify'] }, gpgcheck => empty($_config['gpgcheck']) ? { true => 0, false => $_config['gpgcheck'] }, } } default: { - fail("Redhat '${os_release}' is not a supported.") + fail("${facts['os']['family']} '${os_release}' is not a supported.") } } } default: { fail("\"${module_name}\" provides no repository information for yum repository \"${zookeeper::repo_source}\"") } } } } diff --git a/manifests/params.pp b/manifests/params.pp index de1845c..e3b140a 100644 --- a/manifests/params.pp +++ b/manifests/params.pp @@ -1,164 +1,180 @@ # OS specific configuration should be defined here # # PRIVATE CLASS - do not use directly (use main `zookeeper` class). class zookeeper::params { $_defaults = { 'packages' => ['zookeeper'], } $os_family = $facts['os']['family'] $os_name = $facts['os']['name'] $os_release = $facts['os']['release']['major'] case $os_family { 'Debian': { case $os_name { 'Debian', 'Ubuntu': { $initstyle = 'systemd' } default: { $initstyle = undef } } $_os_overrides = { 'packages' => ['zookeeper', 'zookeeperd'], 'service_name' => 'zookeeper', 'service_provider' => $initstyle, 'shell' => '/bin/false', } # 'environment' file probably read just by Debian # see #16, #81 $environment_file = 'environment' } 'RedHat': { case $os_name { 'RedHat', 'CentOS': { if versioncmp($os_release, '7') < 0 { $initstyle = 'redhat' } else { $initstyle = 'systemd' } } default: { $initstyle = undef } } $_os_overrides = { 'packages' => ['zookeeper', 'zookeeper-server'], 'service_name' => 'zookeeper-server', 'service_provider' => $initstyle, 'shell' => '/sbin/nologin', } $environment_file = 'java.env' } + 'Suse': { + case $os_name { + 'SLES': { + $initstyle = 'systemd' + } + default: { $initstyle = undef } + } + + $_os_overrides = { + 'packages' => ['zookeeper', 'zookeeper-server'], + 'service_name' => 'zookeeper-server', + 'service_provider' => $initstyle, + 'shell' => '/bin/false', + } + $environment_file = 'java.env' + } default: { fail("Module '${module_name}' is not supported on OS: '${os_name}', family: '${os_family}'") } } $_params = merge($_defaults, $_os_overrides) # meta options $ensure = present $ensure_account = present $ensure_cron = true $group = 'zookeeper' $system_group = false $packages = $_params['packages'] $shell = $_params['shell'] $user = 'zookeeper' $system_user = false # installation options $archive_checksum = {} $archive_dl_site = undef $archive_dl_timeout = 600 $archive_dl_url = undef $archive_install_dir = '/opt' $archive_symlink = true $archive_symlink_name = "${archive_install_dir}/zookeeper" $archive_version = '3.4.8' $cdhver = '5' $install_java = false $install_method = 'package' $java_bin = '/usr/bin/java' $java_opts = '' $java_package = undef $repo = undef $proxy_server = undef $proxy_type = undef # service options $manage_service = true $manage_service_file = false $pid_dir = '/var/run' $pid_file = undef $restart_on_change = true $service_ensure = 'running' $service_name = $_params['service_name'] $service_provider = $_params['service_provider'] # systemd_unit_want and _after can be overridden to # donate the matching directives in the [Unit] section $systemd_unit_want = undef $systemd_unit_after = 'network.target' $systemd_path = '/etc/systemd/system' $zk_dir = '/etc/zookeeper' # zookeeper config $cfg_dir = '/etc/zookeeper/conf' $cleanup_sh = '/usr/share/zookeeper/bin/zkCleanup.sh' $client_ip = undef # use e.g. $::ipaddress if you want to bind to single interface $client_port = 2181 $datastore = '/var/lib/zookeeper' # datalogstore used to put transaction logs in separate location than snapshots $datalogstore = undef $election_port = 2888 $export_tag = 'zookeeper' $id = '1' $init_limit = 10 $initialize_datastore = false $leader = true $leader_port = 3888 $log_dir = '/var/log/zookeeper' $manual_clean = false $max_allowed_connections = undef $max_session_timeout = undef $min_session_timeout = undef $observers = [] # interval in hours, purging enabled when >= 1 $purge_interval = 0 $servers = [] $snap_count = 10000 # since zookeeper 3.4, for earlier version cron task might be used $snap_retain_count = 3 $sync_limit = 5 $tick_time = 2000 $global_outstanding_limit = 1000 $use_sasl_auth = false $zoo_dir = '/usr/lib/zookeeper' $zoo_main = 'org.apache.zookeeper.server.quorum.QuorumPeerMain' # log4j properties $log4j_prop = 'INFO,ROLLINGFILE' $peer_type = 'UNSET' $rollingfile_threshold = 'INFO' $console_threshold = 'INFO' $tracefile_threshold = 'TRACE' $maxfilesize = '256MB' $maxbackupindex = 20 $extra_appenders = {} # sasl options $sasl_krb5 = true $sasl_users = {} $keytab_path = '/etc/zookeeper/conf/zookeeper.keytab' $principal = "zookeeper/${facts['networking']['fqdn']}" $realm = pick($trusted['domain'], $trusted['certname']) $store_key = true $use_keytab = true $use_ticket_cache = false $remove_host_principal = false $remove_realm_principal = false # whitelist of Four Letter Words commands, see https://zookeeper.apache.org/doc/r3.4.12/zookeeperAdmin.html#sc_zkCommands $whitelist_4lw = [] } diff --git a/manifests/post_install.pp b/manifests/post_install.pp index 13d938b..452f77e 100644 --- a/manifests/post_install.pp +++ b/manifests/post_install.pp @@ -1,74 +1,84 @@ # Class: zookeeper::post_install # # In order to maintain compatibility with older releases, there are # some post-install task to ensure same behaviour on all platforms. # # PRIVATE CLASS - do not use directly (use main `zookeeper` class). class zookeeper::post_install inherits zookeeper { $os_family = $facts['os']['family'] $os_name = $facts['os']['name'] $os_release = $facts['os']['release']['major'] if ($zookeeper::manual_clean) { # User defined value $_clean = $zookeeper::manual_clean } else { # Autodetect: # Since ZooKeeper 3.4 there's no need for purging snapshots with cron case $os_family { 'Debian': { case $os_name { 'Debian', 'Ubuntu': { $_clean = false } default: { fail ("Family: '${os_family}' OS: '${os_name}' is not supported yet") } } } 'Redhat': { $_clean = false } default: { fail ("Family: '${os_family}' OS: '${os_name}' is not supported yet") } + 'Suse': { + case $os_name { + 'SLES': { + $_clean = false + } + default: { + fail ("Family: '${os_family}' OS: '${os_name}' is not supported yet") + } + } + } } } # If !$cleanup_count, then ensure this cron is absent. if ($_clean and $zookeeper::snap_retain_count > 0 and $zookeeper::ensure != 'absent') { if ($zookeeper::ensure_cron){ include cron cron::job { 'zookeeper-cleanup': ensure => present, command => "${zookeeper::cleanup_sh} ${zookeeper::datastore} ${zookeeper::snap_retain_count}", hour => 2, minute => 42, user => $zookeeper::user, } }else { file { '/etc/cron.daily/zkcleanup': ensure => present, content => "${zookeeper::cleanup_sh} ${zookeeper::datastore} ${zookeeper::snap_retain_count}", } } } # Package removal if($_clean and $zookeeper::ensure == 'absent'){ if ($zookeeper::ensure_cron){ class { 'cron': manage_package => false, } cron::job { 'zookeeper-cleanup': ensure => $zookeeper::ensure, } }else{ file { '/etc/cron.daily/zkcleanup': ensure => $zookeeper::ensure, } } } } diff --git a/metadata.json b/metadata.json index 6563da7..ba0edc3 100644 --- a/metadata.json +++ b/metadata.json @@ -1,66 +1,72 @@ { "name": "deric-zookeeper", "version": "1.0.0", "author": "deric", "summary": "Module for managing Apache Zookeeper", "license": "Apache-2.0", "source": "https://github.com/deric/puppet-zookeeper", "project_page": "https://github.com/deric/puppet-zookeeper", "issues_url": "https://github.com/deric/puppet-zookeeper/issues", "description": "ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.", "dependencies": [ { "name": "puppet/archive", "version_requirement": ">= 0.4.4 < 5.0.0" }, { "name": "puppet/cron", "version_requirement": ">= 1.3.1" }, { "name": "puppetlabs/stdlib", "version_requirement": ">= 2.3.3 < 7.0.0" }, { "name": "puppetlabs/yumrepo_core", "version_requirement": ">= 1.0.3" } ], "operatingsystem_support": [ { "operatingsystem": "RedHat", "operatingsystemrelease": [ "6", "7" ] }, { "operatingsystem": "CentOS", "operatingsystemrelease": [ "6", "7" ] }, { "operatingsystem": "Debian", "operatingsystemrelease": [ "8", "9", "10" ] }, { "operatingsystem": "Ubuntu", "operatingsystemrelease": [ "16.04", "18.04" ] + }, + { + "operatingsystem": "SLES", + "operatingsystemrelease": [ + "12" + ] } ], "requirements": [ { "name": "puppet", "version_requirement": ">= 5.5.8 < 7.0.0" } ] } diff --git a/spec/acceptance/zookeeper_spec.rb b/spec/acceptance/zookeeper_spec.rb index ae1dcb5..4d21bbb 100644 --- a/spec/acceptance/zookeeper_spec.rb +++ b/spec/acceptance/zookeeper_spec.rb @@ -1,83 +1,92 @@ require 'spec_helper_acceptance' case fact('osfamily') when 'Debian' service_name = 'zookeeper' -when 'RedHat' +when 'RedHat', 'Suse' service_name = 'zookeeper-server' end describe 'zookeeper defintion' do context 'basic setup' do it 'install zookeeper' do case fact('osfamily') when 'Debian' pp = <<-EOS class { 'zookeeper': } EOS when 'RedHat' pp = <<-EOS class { 'zookeeper': install_java => true, java_package => 'java-1.8.0-openjdk-headless', initialize_datastore => true, service_provider => 'redhat', } EOS + when 'Suse' + pp = <<-EOS + class { 'zookeeper': + install_java => true, + java_package => 'java-1_8_0-openjdk-headless', + initialize_datastore => true, + service_provider => 'systemd', + } + EOS end expect(apply_manifest(pp, catch_failures: false, debug: false).exit_code).to be_zero end describe file('/etc/zookeeper') do it { is_expected.to be_directory } end describe file('/etc/zookeeper/conf') do it { is_expected.to be_directory } end describe user('zookeeper') do it { is_expected.to exist } end describe group('zookeeper') do it { is_expected.to exist } end describe command("/etc/init.d/#{service_name} status") do its(:exit_status) { is_expected.to eq 0 } its(:stdout) { is_expected.to match %r{running} } end # give zookeeper some time to boot describe command('sleep 2 && netstat -tulpn') do its(:exit_status) { is_expected.to eq 0 } its(:stdout) { is_expected.to match %r{2181} } end describe command('cat /etc/zookeeper/conf/myid') do its(:exit_status) { is_expected.to eq 0 } its(:stdout) { is_expected.to match %r{^1$} } end describe file('/etc/zookeeper/conf/zoo.cfg') do it { is_expected.to be_file } it { is_expected.to be_writable.by('owner') } it { is_expected.to be_readable.by('group') } it { is_expected.to be_readable.by('others') } end describe command('cat /etc/zookeeper/conf/zoo.cfg') do its(:exit_status) { is_expected.to eq 0 } its(:stdout) { is_expected.to match %r{^clientPort=2181$} } end describe command('echo stat | nc 0.0.0.0 2181') do its(:exit_status) { is_expected.to eq 0 } its(:stdout) { is_expected.to match %r{^Mode: standalone$} } end end end diff --git a/spec/classes/init_spec.rb b/spec/classes/init_spec.rb index abb4d33..51659b6 100644 --- a/spec/classes/init_spec.rb +++ b/spec/classes/init_spec.rb @@ -1,280 +1,280 @@ require 'spec_helper' shared_examples 'zookeeper' do |os_facts| let(:user) { 'zookeeper' } let(:group) { 'zookeeper' } os_info = get_os_info(os_facts) service_name = os_info[:service_name] environment_file = os_info[:environment_file] init_provider = os_info[:init_provider] should_install_zookeeperd = os_info[:should_install_zookeeperd] it { is_expected.to contain_class('zookeeper::config') } it { is_expected.to contain_class('zookeeper::install') } it { is_expected.to contain_class('zookeeper::service') } it { is_expected.to compile.with_all_deps } it { is_expected.to contain_service(service_name) } it { is_expected.to contain_service(service_name).that_subscribes_to('File[/etc/zookeeper/conf/myid]') } it { is_expected.to contain_service(service_name).that_subscribes_to('File[/etc/zookeeper/conf/zoo.cfg]') } it { is_expected.to contain_service(service_name).that_subscribes_to("File[#{environment_file}]") } it { is_expected.to contain_service(service_name).that_subscribes_to('File[/etc/zookeeper/conf/log4j.properties]') } context 'skip service restart' do let(:params) do { restart_on_change: false } end it { is_expected.to contain_service(service_name) } it { is_expected.not_to contain_service(service_name).that_subscribes_to('File[/etc/zookeeper/conf/myid]') } it { is_expected.not_to contain_service(service_name).that_subscribes_to('File[/etc/zookeeper/conf/zoo.cfg]') } it { is_expected.not_to contain_service(service_name).that_subscribes_to("File[#{environment_file}]") } it { is_expected.not_to contain_service(service_name).that_subscribes_to('File[/etc/zookeeper/conf/log4j.properties]') } end context 'allow installing multiple packages' do let(:params) do { packages: ['zookeeper', 'zookeeper-bin'] } end it { is_expected.to compile.with_all_deps } it { is_expected.to contain_package('zookeeper').with(ensure: 'present') } it { is_expected.to contain_package('zookeeper-bin').with(ensure: 'present') } it { is_expected.to contain_service(service_name).with(ensure: 'running') } # datastore exec is not included by default it { is_expected.not_to contain_exec('initialize_datastore') } it { is_expected.to contain_user('zookeeper').with(ensure: 'present') } it { is_expected.to contain_group('zookeeper').with(ensure: 'present') } end context 'Cloudera packaging' do let(:params) do { packages: ['zookeeper', 'zookeeper-server'], service_name: 'zookeeper-server', initialize_datastore: true } end it { is_expected.to contain_package('zookeeper').with(ensure: 'present') } it { is_expected.to contain_package('zookeeper-server').with(ensure: 'present') } it { is_expected.to contain_service('zookeeper-server').with(ensure: 'running') } it { is_expected.to contain_exec('initialize_datastore') } end context 'setting minSessionTimeout' do let(:params) do { min_session_timeout: 3000 } end it do is_expected.to contain_file( '/etc/zookeeper/conf/zoo.cfg' ).with_content(%r{minSessionTimeout=3000}) end end context 'setting maxSessionTimeout' do let(:params) do { max_session_timeout: 60_000 } end it do is_expected.to contain_file( '/etc/zookeeper/conf/zoo.cfg' ).with_content(%r{maxSessionTimeout=60000}) end end context 'disable service management' do let(:params) do { manage_service: false } end it { is_expected.to contain_package('zookeeper').with(ensure: 'present') } it { is_expected.not_to contain_service(service_name).with(ensure: 'running') } it { is_expected.not_to contain_class('zookeeper::service') } end - if os_facts[:osfamily] == 'RedHat' + if os_facts[:os]['family'] =~ %r{RedHat|Suse} context 'Do not use cloudera by default' do it { is_expected.to compile.with_all_deps } it { is_expected.to contain_class('zookeeper::install::repo') } it { is_expected.not_to contain_yumrepo('cloudera-cdh5') } end context 'use Cloudera RPM repo' do let(:params) do { repo: 'cloudera', cdhver: '5' } end it { is_expected.to compile.with_all_deps } it { is_expected.to contain_class('zookeeper::install::repo') } it { is_expected.to contain_yumrepo('cloudera-cdh5') } context 'custom RPM repo' do let(:params) do { repo: { 'name' => 'myrepo', 'url' => 'http://repo.url', 'descr' => 'custom repo' }, cdhver: '5' } end it { is_expected.to contain_yumrepo('myrepo').with(baseurl: 'http://repo.url') } end end end context 'service provider' do context 'autodetect provider' do it { is_expected.to contain_package('zookeeper').with(ensure: 'present') } - if os_facts[:osfamily] == 'RedHat' + if os_facts[:os]['family'] =~ %r{RedHat|Suse} it { is_expected.to contain_package('zookeeper-server').with(ensure: 'present') } else it { is_expected.not_to contain_package('zookeeper-server').with(ensure: 'present') } end it do is_expected.to contain_service(service_name).with(ensure: 'running', provider: init_provider) end end it { is_expected.to contain_class('zookeeper::service') } end context 'allow passing specific version' do let(:version) { '3.4.5+dfsg-1' } let(:params) do { ensure: version } end it { is_expected.to contain_package('zookeeper').with(ensure: version) } if should_install_zookeeperd it { is_expected.to contain_package('zookeeperd').with(ensure: version) } else it { is_expected.not_to contain_package('zookeeperd').with(ensure: version) } end it { is_expected.to contain_user('zookeeper').with(ensure: 'present') } end context 'set pid file for init provider' do let(:params) do { zoo_dir: '/usr/lib/zookeeper', log_dir: '/var/log/zookeeper', manage_service: true, manage_service_file: true, service_provider: 'init' } end it do is_expected.to contain_file( '/etc/zookeeper/conf/log4j.properties' ).with_content(%r{zookeeper.log.dir=/var/log/zookeeper}) end context 'set service provider' do it { is_expected.to contain_package('zookeeper').with(ensure: 'present') } it do is_expected.to contain_service(service_name).with(ensure: 'running', provider: 'init') end end - if os_facts[:osfamily] == 'RedHat' + if os_facts[:os]['family'] =~ %r{RedHat|Suse} it do is_expected.to contain_file( "/etc/init.d/#{service_name}" ).with_content(%r{pidfile=/var/run/zookeeper.pid}) end else it do is_expected.to contain_file( environment_file ).with_content(%r{PIDFILE=/var/run/zookeeper.pid}) end end end context 'create env file' do it do is_expected.to contain_file( environment_file ) end end context 'managed by exhibitor' do let(:params) do { service_provider: 'exhibitor', service_name: 'zookeeper', cfg_dir: '/opt/zookeeper/conf' } end it { is_expected.not_to contain_class('zookeeper::service') } it { is_expected.not_to contain_service(service_name) } it { is_expected.not_to contain_file('/opt/zookeeper/conf/zoo.cfg') } it { is_expected.not_to contain_file('/opt/zookeeper/conf/myid') } end context 'install from archive' do let(:params) do { install_method: 'archive', archive_version: '3.4.9' } end it { is_expected.to compile.with_all_deps } it { is_expected.to contain_class('Zookeeper::Install::Archive') } it { is_expected.not_to contain_package('zookeeper').with(ensure: 'present') } it { is_expected.to contain_service(service_name).with(ensure: 'running') } it { is_expected.to contain_user('zookeeper').with(ensure: 'present') } it { is_expected.to contain_group('zookeeper').with(ensure: 'present') } end end describe 'zookeeper', type: :class do on_supported_os.each do |os, os_facts| os_facts[:os]['hardware'] = 'x86_64' context "on #{os}" do let(:facts) do os_facts.merge(ipaddress: '192.168.1.1') end include_examples 'zookeeper', os_facts end end end diff --git a/spec/classes/repo_spec.rb b/spec/classes/repo_spec.rb index 3770c50..5d77012 100644 --- a/spec/classes/repo_spec.rb +++ b/spec/classes/repo_spec.rb @@ -1,140 +1,154 @@ require 'spec_helper' -shared_examples 'zookeeper repo release support' do |_os_facts| +shared_examples 'zookeeper repo release support' do |os_facts| + os_name = if os_facts[:os]['family'] == 'RedHat' + os_facts[:os]['family'].downcase + else + os_facts[:os]['name'].downcase + end + os_release = os_facts[:os]['release']['major'] + context 'fail when release not supported' do let :pre_condition do 'class {"zookeeper": repo => "cloudera", cdhver => "5", }' end it do expect do is_expected.to compile - end.to raise_error(%r{is not supported for redhat version}) end + end.to raise_error(%r{is not supported for #{os_name} version #{os_release}}) end end end -shared_examples 'zookeeper repo arch support' do |_os_facts| +shared_examples 'zookeeper repo arch support' do |os_facts| + os_hardware = os_facts[:os]['hardware'] + context 'fail when architecture not supported' do let :pre_condition do 'class {"zookeeper": repo => "cloudera", cdhver => "5", }' end it do expect do is_expected.to compile - end.to raise_error(%r{is not supported for architecture}) end + end.to raise_error(%r{is not supported for architecture #{os_hardware}}) end end end shared_examples 'zookeeper repo' do |os_facts| let(:user) { 'zookeeper' } let(:group) { 'zookeeper' } + os_name = if os_facts[:os]['family'] == 'RedHat' + os_facts[:os]['family'].downcase + else + os_facts[:os]['name'].downcase + end os_release = os_facts[:os]['release']['major'] os_hardware = os_facts[:os]['hardware'] - if os_facts[:osfamily] == 'RedHat' + if os_facts[:os]['family'] =~ %r{RedHat|Suse} context 'Cloudera repo' do let :pre_condition do 'class {"zookeeper": repo => "cloudera", cdhver => "5", }' end it { - is_expected.to contain_yumrepo('cloudera-cdh5').with(baseurl: "http://archive.cloudera.com/cdh5/redhat/#{os_release}/#{os_hardware}/cdh/5/") + is_expected.to contain_yumrepo('cloudera-cdh5').with(baseurl: "http://archive.cloudera.com/cdh5/#{os_name}/#{os_release}/#{os_hardware}/cdh/5/") } end end context 'fail when CDH version not supported' do let :pre_condition do 'class {"zookeeper": repo => "cloudera", cdhver => "6", }' end it do expect do is_expected.to compile end.to raise_error(%r{is not a supported cloudera repo.}) end end context 'fail when repository source not supported' do let :pre_condition do 'class {"zookeeper": repo => "another-repo", }' end it do expect do is_expected.to compile end.to raise_error(%r{provides no repository information for yum repository}) end end end describe 'zookeeper::install::repo' do on_supported_os.each do |os, os_facts| os_facts[:os]['hardware'] = 'x86_64' context "on #{os}" do let(:facts) do os_facts.merge(ipaddress: '192.168.1.1') end include_examples 'zookeeper repo', os_facts end end context 'test unsupported repo arch' do test_on = { hardwaremodels: ['arc'], supported_os: [ { 'operatingsystem' => 'RedHat', 'operatingsystemrelease' => ['7'] } ] } on_supported_os(test_on).each do |os, os_facts| context "on #{os}" do let(:facts) do os_facts.merge(ipaddress: '192.168.1.1') end include_examples 'zookeeper repo arch support', os_facts end end end context 'test unsupported repo release' do test_on = { supported_os: [ { 'operatingsystem' => 'RedHat', 'operatingsystemrelease' => ['8'] } ] } on_supported_os(test_on).each do |os, os_facts| os_facts[:os]['hardware'] = 'x86_64' context "on #{os}" do let(:facts) do os_facts.merge(ipaddress: '192.168.1.1') end include_examples 'zookeeper repo release support', os_facts end end end end diff --git a/spec/spec_helper_acceptance.rb b/spec/spec_helper_acceptance.rb index 1a81793..434b957 100644 --- a/spec/spec_helper_acceptance.rb +++ b/spec/spec_helper_acceptance.rb @@ -1,32 +1,36 @@ require 'beaker-rspec' require 'beaker-puppet' require 'beaker/puppet_install_helper' require 'beaker/module_install_helper' run_puppet_install_helper unless ENV['BEAKER_provision'] == 'no' install_ca_certs unless ENV['PUPPET_INSTALL_TYPE'] =~ %r{pe}i install_module_on(hosts) install_module_dependencies_on(hosts) RSpec.configure do |c| # Readable test descriptions c.formatter = :documentation hosts.each do |host| if fact_on(host, 'osfamily') == 'Debian' on host, puppet('resource', 'package', 'net-tools', 'ensure=installed') on host, puppet('resource', 'package', 'netcat', 'ensure=installed') end if fact_on(host, 'osfamily') == 'RedHat' case fact('os.release.major') when '6' on host, puppet('resource', 'package', 'nc', 'ensure=installed') when '7' on host, puppet('resource', 'package', 'net-tools', 'ensure=installed') on host, puppet('resource', 'package', 'nmap-ncat', 'ensure=installed') end end + if fact_on(host, 'osfamily') == 'Suse' + on host, puppet('resource', 'package', 'net-tools', 'ensure=installed') + on host, puppet('resource', 'package', 'netcat-openbsd', 'ensure=installed') + end if host[:platform] =~ %r{el-7-x86_64} && host[:hypervisor] =~ %r{docker} on(host, "sed -i '/nodocs/d' /etc/yum.conf") end end end diff --git a/spec/spec_helper_local.rb b/spec/spec_helper_local.rb index 56db83a..564f67b 100644 --- a/spec/spec_helper_local.rb +++ b/spec/spec_helper_local.rb @@ -1,48 +1,54 @@ def get_os_info(facts) info = { service_name: nil, environment_file: nil, should_install_zookeeperd: nil, init_provider: nil, init_dir: nil, service_file: nil, should_install_cron: false, zookeeper_shell: nil } case facts[:osfamily] when 'Debian' info[:service_name] = 'zookeeper' info[:environment_file] = '/etc/zookeeper/conf/environment' info[:should_install_zookeeperd] = true info[:zookeeper_shell] = '/bin/false' info[:init_provider] = 'systemd' when 'RedHat' info[:service_name] = 'zookeeper-server' info[:environment_file] = '/etc/zookeeper/conf/java.env' info[:should_install_zookeeperd] = false info[:zookeeper_shell] = '/sbin/nologin' info[:init_provider] = if Puppet::Util::Package.versioncmp(facts[:os]['release']['major'], '7') < 0 'redhat' else 'systemd' end + when 'Suse' + info[:service_name] = 'zookeeper-server' + info[:environment_file] = '/etc/zookeeper/conf/java.env' + info[:should_install_zookeeperd] = false + info[:zookeeper_shell] = '/bin/false' + info[:init_provider] = 'systemd' end case info[:init_provider] when 'init' info[:init_dir] = '/etc/init.d' when 'systemd' info[:init_dir] = '/etc/systemd/system' when 'redhat' info[:init_dir] = '/etc/init.d' end info[:service_file] = if info[:init_provider] == 'systemd' "#{info[:init_dir]}/#{info[:service_name]}.service" else "#{info[:init_dir]}/#{info[:service_name]}" end info end diff --git a/templates/zookeeper.Suse.init.erb b/templates/zookeeper.Suse.init.erb new file mode 100644 index 0000000..c62f717 --- /dev/null +++ b/templates/zookeeper.Suse.init.erb @@ -0,0 +1,141 @@ +#!/bin/sh +# +# zookeeper ZooKeeper Server +# +# chkconfig: - 80 05 +# description: Enable ZooKeeper Server +# + +### BEGIN INIT INFO +# Provides: zookeeper +# Default-Start: +# Default-Stop: +# Required-Start: $remote_fs $network +# Required-Stop: $remote_fs $network +# Description: zookeeper Server +# Short-Description: Enable zookeeper Server +### END INIT INFO + +# Source function library. +. /etc/rc.d/init.d/functions + +prog=<%= scope.lookupvar("zookeeper::service_name") %> +desc="Zookeeper Service" + +[ -e /etc/sysconfig/$prog ] && . /etc/sysconfig/$prog + +lockfile="/var/lock/subsys/$prog" +pidfile=<%= scope.lookupvar("zookeeper::pid_path") %> + +[ "x$JMXLOCALONLY" = "x" ] && JMXLOCALONLY=false + +if [ "x$JMXDISABLE" = "x" ] +then + # for some reason these two options are necessary on jdk6 on Ubuntu + # accord to the docs they are not necessary, but otw jconsole cannot + # do a local attach + ZOOMAIN="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=$JMXLOCALONLY <%= scope.lookupvar("zookeeper::zoo_main") %>" +else + ZOOMAIN=<%= scope.lookupvar("zookeeper::zoo_main") %> +fi + +ZOOBINDIR="<%= scope.lookupvar("zookeeper::service::_zoo_dir") %>/bin" +ZOOCFGDIR=<%= scope.lookupvar("zookeeper::cfg_dir") %> +ZOOCFG="$ZOOCFGDIR/zoo.cfg" +ZOO_LOG_DIR=<%= scope.lookupvar("zookeeper::log_dir") %> + +[ -e "$ZOOCFGDIR/java.env" ] && . "$ZOOCFGDIR/java.env" +[ -e "$ZOOCFGDIR/<%= scope.lookupvar("zookeeper::environment_file") %>" ] && . "$ZOOCFGDIR/<%= scope.lookupvar("zookeeper::environment_file") %>" + +[ "x$ZOO_LOG4J_PROP" = "x" ] && ZOO_LOG4J_PROP="<%= scope.lookupvar("zookeeper::log4j_prop") %>" + +for f in ${ZOOBINDIR}/../zookeeper-*.jar +do + CLASSPATH="$CLASSPATH:$f" +done + +ZOOLIBDIR=${ZOOLIBDIR:-$ZOOBINDIR/../lib} +for i in "$ZOOLIBDIR"/*.jar +do + CLASSPATH="$CLASSPATH:$i" +done + +#add the zoocfg dir to classpath +CLASSPATH=$ZOOCFGDIR:$CLASSPATH + +cmd="java \"-Dzookeeper.log.dir=${ZOO_LOG_DIR}\" \"-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}\" -cp ${CLASSPATH} ${JAVA_OPTS} ${JVMFLAGS} ${ZOOMAIN} ${ZOOCFG} & echo \$! > ${pidfile}" + + +start() { + echo $pidfile + echo -n "Starting $desc $prog: " + touch $pidfile && chown <%= scope.lookupvar("zookeeper::user") %> $pidfile + daemon --user <%= scope.lookupvar("zookeeper::user") %> --pidfile $pidfile "$cmd" + retval=$? + echo + [ $retval -eq 0 ] && touch $lockfile + return $retval +} + +stop() { + echo -n $"Stopping $prog: " + killproc -p $pidfile $prog + retval=$? + echo + [ $retval -eq 0 ] && rm -f $lockfile + return $retval +} + +restart() { + stop + start +} + +reload() { + restart +} + +get_status() { + if [ -f $pidfile ] + then + PID=`cat $pidfile` + if [ -z "`ps -ef | awk '{print $2}' | grep "^$PID$"`" ] + then + echo "$prog stopped but pid file exists" + exit 1 + else + echo "$prog running with pid $PID" + exit 0 + fi + else + echo "$prog stopped" + exit 1 + fi +} + +case "$1" in + start) + start + ;; + stop) + stop + ;; + restart) + restart + ;; + reload) + reload + ;; + condrestart) + [ -e /var/lock/subsys/$prog ] && restart + RETVAL=$? + ;; + status) + get_status + ;; + *) + echo $"Usage: $0 {start|stop|restart|reload|condrestart|status}" + RETVAL=1 +esac + +exit $RETVAL