diff --git a/README.md b/README.md index daacfd2..e1eb0b0 100644 --- a/README.md +++ b/README.md @@ -1,672 +1,670 @@ # Cassandra [![Puppet Forge](http://img.shields.io/puppetforge/v/locp/cassandra.svg)](https://forge.puppetlabs.com/locp/cassandra) [![Github Tag](https://img.shields.io/github/tag/locp/cassandra.svg)](https://github.com/locp/cassandra) [![Build Status](https://travis-ci.org/locp/cassandra.png?branch=master)](https://travis-ci.org/locp/cassandra) [![Coverage Status](https://coveralls.io/repos/locp/cassandra/badge.svg?branch=master&service=github)](https://coveralls.io/github/locp/cassandra?branch=master) [![Join the chat at https://gitter.im/locp/cassandra](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/locp/cassandra?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) [![CircleCI](https://circleci.com/gh/locp/cassandra.svg?style=svg)](https://circleci.com/gh/locp/cassandra) [![Puppet Forge Downloads](http://img.shields.io/puppetforge/dt/locp/cassandra.svg)](https://forge.puppetlabs.com/locp/cassandra) [![Puppet Forge Endorsement](https://img.shields.io/puppetforge/e/locp/cassandra.svg)](https://forge.puppetlabs.com/locp/cassandra) ## Table of Contents 1. [Overview](#overview) 1. [Setup - The basics of getting started with Cassandra](#setup) * [What Cassandra affects](#what-cassandra-affects) * [Upgrading](#upgrading) * [Beginning with Cassandra](#beginning-with-cassandra) 1. [Usage - Configuration options and additional functionality](#usage) * [Setup a keyspace and users](#setup-a-keyspace-and-users) * [Create a Cluster in a Single Data Center](#create-a-cluster-in-a-single-data-center) * [Create a Cluster in Multiple Data Centers](#create-a-cluster-in-multiple-data-centers) * [DataStax Enterprise](#datastax-enterprise) * [Apache Cassandra](#apache-cassandra) 1. [Reference](#reference) 1. [Limitations - OS compatibility, etc.](#limitations) 1. [Development](#development) ## Overview A Puppet module to install and manage Cassandra, DataStax Agent & OpsCenter ## Setup ### What Cassandra affects #### What the Cassandra class affects * Installs the Cassandra package (default **cassandra22** on Red Hat and **cassandra** on Debian). * Configures settings in `${config_path}/cassandra.yaml`. * On CentOS 7 if the `init` service provider is used, then cassandra is added as a system service. * Optionally ensures that the Cassandra service is enabled and running. * On Debian systems: * Optionally replace ```/etc/init.d/cassandra``` with a workaround for [CASSANDRA-9822](https://issues.apache.org/jira/browse/CASSANDRA-9822). #### What the cassandra::datastax_agent class affects * Optionally installs the DataStax agent. * Optionally sets JAVA_HOME in **/etc/default/datastax-agent**. #### What the cassandra::datastax_repo class affects * Optionally configures a Yum repository to install the Cassandra packages from (on Red Hat). * Optionally configures an Apt repository to install the Cassandra packages from (on Debian). #### What the cassandra::dse class affects * Optionally configures files in the `/etc/dse` directory if one is using DataStax Enterprise. #### What the cassandra::firewall_ports class affects * Optionally configures the firewall for the Cassandra related network ports. #### What the cassandra::java class affects * Optionally installs a JRE/JDK package (e.g. java-1.7.0-openjdk) and the Java Native Access (JNA). #### What the cassandra::optutils class affects * Optionally installs the Cassandra support tools (e.g. cassandra22-tools). ### Upgrading We follow [SemVer Versioning](http://semver.org/) and an update of the major release (i.e. from 1.*Y*.*Z* to 2.*Y*.*Z*) will indicate a significant change to the API which will most probably require a change to your manifest. #### Changes in 2.0.0 This is a major change to the API and you will more than likely need to change your manifest to accomodate these changes. The `service_ensure` attribute of the cassandra class now defaults to *undef*, users who do want to manage service status in Puppet can still set it to true. If leaving the value at the default and setting `service_refresh` and `service_enable` to false will mean that the user and not Puppet running will control the running state of Cassandra. This currently works OK on the Red Hat family, but has issues on Debian due to [CASSANDRA-2356](https://issues.apache.org/jira/browse/CASSANDRA-2356) during an initial install or package upgrade. All the functionality relating to OpsCenter has been divested to the [locp/opscenter](https://forge.puppet.com/locp/opscenter) module on Puppet Forge. It should also be noted that the module no longer creates directories for the `data`, `commitlog`, `saved_caches` and for Cassandra 3 the `hints` directory. These resources will now need to be defined in your manifest/profile. For a list of features that have been deprecated in this release, please see https://github.com/locp/cassandra/wiki/Deprecations For details on migrating from the version 1.X.X attributes to the `settings` hash, see (https://github.com/locp/cassandra/wiki/Suggested-Baseline-Settings) Please also see the notes for 2.0.0 in the [CHANGELOG](https://forge.puppet.com/locp/cassandra/changelog). #### Changes in 1.19.0 The hints_directory documentation will cause a change in the cassandra.yaml file regardless of the value you set it to. If you do not wish this to result in a refesh of the Cassandra service, please set service_refresh to false. #### Changes in 1.9.2 Now that Cassandra 3 is available from the DataStax repositories, there is a problem (especially on Debian) with the operating system package manager attempting to install Cassandra 3. This can be mitigated against using something similar to the code in this modules acceptance test. Please note that the default Cassandra package name has now been changed from 'dsc'. See the documentation for cassandra::package_name below for details. ```puppet if $::osfamily == 'RedHat' { $version = '2.2.4-1' } else { $version = '2.2.4' } class { 'cassandra': package_ensure => $version, } ``` #### Changes in 1.8.0 A somewhat embarrassing correction to the spelling of the cassandra::fail_on_non_suppoted_os to cassandra::fail_on_non_supported_os. #### Issues when Upgrading to 1.4.0 Unfortunately both releases 1.3.7 and 1.4.0 have subsequently been found to call a refresh service even when no changes had been made to the underlying configuration. In release 1.8.0 (somewhat belatedly) the service_refresh flag has been introduced to mitigate against similar problems. #### Issues When Upgrading to 1.3.7 * Please see the notes for 1.4.0. #### Changes in 1.0.0 * cassandra::cassandra_package_ensure has been renamed to cassandra::package_ensure. * cassandra::cassandra_package_name has been renamed to cassandra::package_name. #### Changes in 0.4.0 There is now a cassandra::datastax_agent class, therefore: * cassandra::datastax_agent_package_ensure has now been replaced with cassandra::datastax_agent::package_ensure. * cassandra::datastax_agent_service_enable has now been replaced with cassandra::datastax_agent::service_enable. * cassandra::datastax_agent_service_ensure has now been replaced with cassandra::datastax_agent::service_ensure. * cassandra::datastax_agent_package_name has now been replaced with cassandra::datastax_agent::package_name. * cassandra::datastax_agent_service_name has now been replaced with cassandra::datastax_agent::service_name. Likewise now there is a new class for handling the installation of Java: * cassandra::java_package_ensure has now been replaced with cassandra::java::ensure. * cassandra::java_package_name has now been replaced with cassandra::java::package_name. Also there is now a class for installing the optional utilities: * cassandra::cassandra_opt_package_ensure has now been replaced with cassandra::optutils:ensure. * cassandra::cassandra_opt_package_name has now been replaced with cassandra::optutils:package_name. #### Changes in 0.3.0 * cassandra_opt_package_ensure changed from 'present' to undef. * The manage_service option has been replaced with service_enable and service_ensure. ### Beginning with Cassandra Create a Cassandra 2.X cluster called MyCassandraCluster which uses the GossipingPropertyFileSnitch and password authentication. In this very basic example the node itself becomes a seed for the cluster and the credentials will default to a user called cassandra with a password called of cassandra. ```puppet # Cassandra pre-requisites include cassandra::datastax_repo include cassandra::java class { 'cassandra': settings => { 'authenticator' => 'PasswordAuthenticator', 'cluster_name' => 'MyCassandraCluster', 'commitlog_directory' => '/var/lib/cassandra/commitlog', 'commitlog_sync' => 'periodic', 'commitlog_sync_period_in_ms' => 10000, 'data_file_directories' => ['/var/lib/cassandra/data'], 'endpoint_snitch' => 'GossipingPropertyFileSnitch', 'listen_address' => $::ipaddress, 'partitioner' => 'org.apache.cassandra.dht.Murmur3Partitioner', 'saved_caches_directory' => '/var/lib/cassandra/saved_caches', 'seed_provider' => [ { 'class_name' => 'org.apache.cassandra.locator.SimpleSeedProvider', 'parameters' => [ { 'seeds' => $::ipaddress, }, ], }, ], 'start_native_transport' => true, }, require => Class['cassandra::datastax_repo', 'cassandra::java'], } ``` However, **PLEASE** note that this is the **ABSOLUTE MINIMUM** configuration to get Cassandra up and running but will probably give you a rather badly configured node. Please see [Suggested Baseline Settings](https://github.com/locp/cassandra/wiki/Suggested-Baseline-Settings) for details on making your configuration a lot more robust. For this code to run with version 3.X of Cassandra, the `hints_directory` will also need to be specified: ```puppet ... class { 'cassandra': settings => { ... 'hints_directory' => '/var/lib/cassandra/hints', ... }, require => Class['cassandra::datastax_repo', 'cassandra::java'], } ``` ### Hiera In your top level node classification (usually `common.yaml`), add the settings hash and all the tweaks you want all the clusters to use: ```YAML cassandra::baseline_settings: authenticator: AllowAllAuthenticator authorizer: AllowAllAuthorizer auto_bootstrap: true auto_snapshot: true ... ``` Then, in the individual node classification add the parts which define the cluster: ```YAML cassandra::settings: cluster_name: developer playground cassandra cluster cassandra::dc: Onsite1 cassandra::rack: RAC1 cassandra::package_ensure: 3.0.5-1 cassandra::package_name: cassandra30 ``` ## Usage ### Setup a keyspace and users We assume that authentication has been enabled for the cassandra cluster and we are connecting with the default user name and password ('cassandra/cassandra'). In this example, we create a keyspace (mykeyspace) with a table called 'users' and an index called 'users_lname_idx'. We also add three users (to Cassandra, not the mykeyspace.users table) called spillman, akers and boone while ensuring that a user called lucan is absent. ```puppet class { 'cassandra': ... } class { 'cassandra::schema': cqlsh_password => 'cassandra', cqlsh_user => 'cassandra', cqlsh_host => $::ipaddress, indexes => { 'users_lname_idx' => { table => 'users', keys => 'lname', keyspace => 'mykeyspace', }, }, keyspaces => { 'mykeyspace' => { durable_writes => false, replication_map => { keyspace_class => 'SimpleStrategy', replication_factor => 1, }, } }, permissions => { 'Grant select permissions to spillman to all keyspaces' => { permission_name => 'SELECT', user_name => 'spillman', }, 'Grant modify to to keyspace mykeyspace to akers' => { keyspace_name => 'mykeyspace', permission_name => 'MODIFY', user_name => 'akers', }, 'Grant alter permissions to mykeyspace to boone' => { keyspace_name => 'mykeyspace', permission_name => 'ALTER', user_name => 'boone', }, 'Grant ALL permissions to mykeyspace.users to gbennet' => { keyspace_name => 'mykeyspace', permission_name => 'ALTER', table_name => 'users', user_name => 'gbennet', }, }, tables => { 'users' => { columns => { user_id => 'int', fname => 'text', lname => 'text', 'PRIMARY KEY' => '(user_id)', }, keyspace => 'mykeyspace', }, }, users => { 'spillman' => { password => 'Niner27', }, 'akers' => { password => 'Niner2', superuser => true, }, 'boone' => { password => 'Niner75', }, 'gbennet' => { 'password' => 'foobar', }, 'lucan' => { 'ensure' => absent }, }, } ``` ### Create a Cluster in a Single Data Center In the DataStax documentation _Initializing a multiple node cluster (single data center)_ there is a basic example of a six node cluster with two seeds to be created in a single data center spanning two racks. The nodes in the cluster are: **Node Name** | **IP Address** | ---------------|----------------| node0 (seed 1) | 110.82.155.0 | node1 | 110.82.155.1 | node2 | 110.82.155.2 | node3 (seed 2) | 110.82.156.3 | node4 | 110.82.156.4 | node5 | 110.82.156.5 | Each node is configured to use the GossipingPropertyFileSnitch and 256 virtual nodes (vnodes). The name of the cluster is _MyCassandraCluster_. Also, while building the initial cluster, we are setting the auto_bootstrap to false. In this initial example, we are going to expand the example by: * Ensuring that the software is installed via the DataStax Community repository by including `cassandra::datastax_repo`. This needs to be executed before the Cassandra package is installed. * That a suitable Java Runtime environment (JRE) is installed with Java Native Access (JNA) by including `cassandra::java`. This need to be executed before the Cassandra service is started. ```puppet node /^node\d+$/ { class { 'cassandra::datastax_repo': before => Class['cassandra'] } class { 'cassandra::java': before => Class['cassandra'] } class { 'cassandra': settings => { 'authenticator' => 'AllowAllAuthenticator', 'auto_bootstrap' => false, 'cluster_name' => 'MyCassandraCluster', 'commitlog_directory' => '/var/lib/cassandra/commitlog', 'commitlog_sync' => 'periodic', 'commitlog_sync_period_in_ms' => 10000, 'data_file_directories' => ['/var/lib/cassandra/data'], 'endpoint_snitch' => 'GossipingPropertyFileSnitch', 'hints_directory' => '/var/lib/cassandra/hints', 'listen_interface' => 'eth1', 'num_tokens' => 256, 'partitioner' => 'org.apache.cassandra.dht.Murmur3Partitioner', 'saved_caches_directory' => '/var/lib/cassandra/saved_caches', 'seed_provider' => [ { 'class_name' => 'org.apache.cassandra.locator.SimpleSeedProvider', 'parameters' => [ { 'seeds' => '110.82.155.0,110.82.156.3', }, ], }, ], 'start_native_transport' => true, }, } } ``` The default value for the num_tokens is already 256, but it is included in the example for clarity. Do not forget to either set auto_bootstrap to true or not set the attribute at all after initializing the cluster. ### Create a Cluster in Multiple Data Centers To continue with the examples provided by DataStax, we look at the example for a cluster across multiple data centers . **Node Name** | **IP Address** | **Data Center** | **Rack** | ---------------|----------------|-----------------|----------| node0 (seed 1) | 10.168.66.41 | DC1 | RAC1 | node1 | 10.176.43.66 | DC1 | RAC1 | node2 | 10.168.247.41 | DC1 | RAC1 | node3 (seed 2) | 10.176.170.59 | DC2 | RAC1 | node4 | 10.169.61.170 | DC2 | RAC1 | node5 | 10.169.30.138 | DC2 | RAC1 | For the sake of simplicity, we will confine this example to the nodes: ```puppet node /^node[012]$/ { class { 'cassandra': dc => 'DC1', settings => { 'authenticator' => 'AllowAllAuthenticator', 'auto_bootstrap' => false, 'cluster_name' => 'MyCassandraCluster', 'commitlog_directory' => '/var/lib/cassandra/commitlog', 'commitlog_sync' => 'periodic', 'commitlog_sync_period_in_ms' => 10000, 'data_file_directories' => ['/var/lib/cassandra/data'], 'endpoint_snitch' => 'GossipingPropertyFileSnitch', 'hints_directory' => '/var/lib/cassandra/hints', 'listen_interface' => 'eth1', 'num_tokens' => 256, 'partitioner' => 'org.apache.cassandra.dht.Murmur3Partitioner', 'saved_caches_directory' => '/var/lib/cassandra/saved_caches', 'seed_provider' => [ { 'class_name' => 'org.apache.cassandra.locator.SimpleSeedProvider', 'parameters' => [ { 'seeds' => '110.82.155.0,110.82.156.3', }, ], }, ], 'start_native_transport' => true, }, } } node /^node[345]$/ { class { 'cassandra': dc => 'DC2', settings => { 'authenticator' => 'AllowAllAuthenticator', 'auto_bootstrap' => false, 'cluster_name' => 'MyCassandraCluster', 'commitlog_directory' => '/var/lib/cassandra/commitlog', 'commitlog_sync' => 'periodic', 'commitlog_sync_period_in_ms' => 10000, 'data_file_directories' => ['/var/lib/cassandra/data'], 'endpoint_snitch' => 'GossipingPropertyFileSnitch', 'hints_directory' => '/var/lib/cassandra/hints', 'listen_interface' => 'eth1', 'num_tokens' => 256, 'partitioner' => 'org.apache.cassandra.dht.Murmur3Partitioner', 'saved_caches_directory' => '/var/lib/cassandra/saved_caches', 'seed_provider' => [ { 'class_name' => 'org.apache.cassandra.locator.SimpleSeedProvider', 'parameters' => [ { 'seeds' => '110.82.155.0,110.82.156.3', }, ], }, ], 'start_native_transport' => true, }, } } ``` We don't need to specify the rack name (with the rack attribute) as RAC1 is the default value. Again, do not forget to either set auto_bootstrap to true or not set the attribute at all after initializing the cluster. ### DataStax Enterprise After configuring the relevant repository, the following snippet works on CentOS 7 to install DSE Cassandra 4.7.0, set the HADOOP_LOG_DIR, set the DSE_HOME and configure DataStax Enterprise to use LDAP for authentication: ```puppet class { 'cassandra::datastax_repo': descr => 'DataStax Repo for DataStax Enterprise', pkg_url => 'https://username:password@rpm.datastax.com/enterprise', before => Class['cassandra'], } class { 'cassandra': cluster_name => 'MyCassandraCluster', config_path => '/etc/dse/cassandra', package_ensure => '4.7.0-1', package_name => 'dse-full', service_name => 'dse', ... } class { 'cassandra::dse': file_lines => { 'Set HADOOP_LOG_DIR directory' => { ensure => present, path => '/etc/dse/dse-env.sh', line => 'export HADOOP_LOG_DIR=/var/log/hadoop', match => '^# export HADOOP_LOG_DIR=', }, 'Set DSE_HOME' => { ensure => present, path => '/etc/dse/dse-env.sh', line => 'export DSE_HOME=/usr/share/dse', match => '^#export DSE_HOME', }, }, settings => { ldap_options => { server_host => localhost, server_port => 389, search_dn => 'cn=Admin', search_password => secret, use_ssl => false, use_tls => false, truststore_type => jks, user_search_base => 'ou=users,dc=example,dc=com', user_search_filter => '(uid={0})', credentials_validity_in_ms => 0, connection_pool => { max_active => 8, max_idle => 8, } } } } ``` ### Apache Cassandra DataStax announced in late October 2016 that it was no longer supporting the community edition of Cassandra or DSC as it was known (see *[Take a bow Planet Cassandra]*(http://www.datastax.com/2016/10/take-a-bow-planet-cassandra) for details). However, the following snippet of code running on Ubuntu 14.04 worked fine without having to change any of the `::cassandra` class settings: ```puppet require cassandra::java include cassandra::optutils class { 'cassandra::apache_repo': release => '310x', before => Class['cassandra', 'cassandra::optutils'], } class { 'cassandra': ... } ``` ## Reference The reference documentation is generated using the [puppet-strings](https://github.com/puppetlabs/puppet-strings) tool. To see all of it, please go to [http://locp.github.io/cassandra](http://locp.github.io/cassandra/_index.html). ## Limitations * When using a Ruby version before 1.9.0, the contents of the Cassandra configuration file may change order of elements due to a problem with to_yaml in earlier versions of Ruby. * When creating key spaces, indexes, cql_types and users the settings will only be used to create a new resource if it does not currently exist. If a change is made to the Puppet manifest but the resource already exits, this change will not be reflected. * At the moment the `cassandra::system::transparent_hugepage` does not persist between reboots. -* Currently Apache Cassandra only provides binary packages for Debian so -the `cassandra::apache_repo` is not of much use to Red Hat family users. * Acceptance for Debian 7 are confined to Cassandra 2.1 and 2.2. There is a conflict between the GLIBC on Debian 7 and the newer releases of Cassandra 3.X. ## Development Contributions will be gratefully accepted. Please go to the project page, fork the project, make your changes locally and then raise a pull request. Details on how to do this are available at https://guides.github.com/activities/contributing-to-open-source. Please also see the [CONTRIBUTING.md](https://github.com/locp/cassandra/blob/master/CONTRIBUTING.md) page for project specific requirements. ### Additional Contributers For a list of contributers see [CONTRIBUTING.md](https://github.com/locp/cassandra/blob/master/CONTRIBUTING.md) and https://github.com/locp/cassandra/graphs/contributors diff --git a/manifests/apache_repo.pp b/manifests/apache_repo.pp index 45bf6f1..0f1be42 100644 --- a/manifests/apache_repo.pp +++ b/manifests/apache_repo.pp @@ -1,85 +1,85 @@ # An optional class that will allow a suitable repository to be configured -# from which packages for Apache Cassandra can be downloaded. Changing -# the defaults will allow any Debian Apt repository to be configured. -# Currently Apache only provide a Debian repository -# (http://cassandra.apache.org/download/) mentions of Yum are for wishful -# thinking. +# from which packages for Apache Cassandra can be downloaded. # @param descr [string] On the Red Hat family, this is passed as the `descr` # attribute to a `yumrepo` resource. On the Debian family, it is passed as # the `comment` attribute to an `apt::source` resource. # @param key_id [string] On the Debian family, this is passed as the `id` # attribute to an `apt::key` resource. On the Red Hat family, it is # ignored. # @param key_url [string] On the Debian family, this is passed as the # `source` attribute to an `apt::key` resource. On the Red Hat family, -# it is ignored. -# @param pkg_url [string] If left as the default, this will set the `baseurl` -# to 'http://rpm.datastax.com/community' on a `yumrepo` resource -# on the Red Hat family. On the Debian family, leaving this as the default +# it is set to the `gpgkey` attribute on the `yumrepo` resource. +# @param pkg_url [string] On the Red Hat family, leaving this as default will +# set the `baseurl` on the `yumrepo` resource to +# 'http://www.apache.org/dist/cassandra/redhat' with whatever is set in the +# 'release' attribute appended. +# On the Debian family, leaving this as the default # will set the `location` attribute on an `apt::source` to # 'http://www.apache.org/dist/cassandra/debian'. # @param release [string] On the Debian family, this is passed as the `release` -# attribute to an `apt::source` resource. On the Red Hat family, it is -# ignored. +# attribute to an `apt::source` resource. On the Red Hat family, it is the +# major version number of Cassandra, without dot, and with an appended 'x' +# (e.g. '311x') class cassandra::apache_repo ( $descr = 'Repo for Apache Cassandra', $key_id = 'A26E528B271F19B9E5D8E19EA278B781FE4B2BDA', $key_url = 'https://www.apache.org/dist/cassandra/KEYS', $pkg_url = undef, $release = 'main', ) { case $::osfamily { 'RedHat': { if $pkg_url != undef { $baseurl = $pkg_url } else { - $baseurl = 'http://www.apache.org/dist/cassandra/redhat' + $url = 'http://www.apache.org/dist/cassandra/redhat' + $baseurl = "${url}/${release}" } yumrepo { 'cassandra_apache': ensure => present, descr => $descr, - baseurl => "${baseurl}/${release}", + baseurl => $baseurl, enabled => 1, gpgcheck => 1, gpgkey => $key_url, } } 'Debian': { include apt include apt::update apt::key {'apache.cassandra': id => $key_id, source => $key_url, before => Apt::Source['cassandra.sources'], } if $pkg_url != undef { $location = $pkg_url } else { $location = 'http://www.apache.org/dist/cassandra/debian' } apt::source {'cassandra.sources': location => $location, comment => $descr, release => $release, include => { 'src' => false, }, notify => Exec['update-apache-cassandra-repo'], } # Required to wrap apt_update exec {'update-apache-cassandra-repo': refreshonly => true, command => '/bin/true', require => Exec['apt_update'], } } default: { warning("OS family ${::osfamily} not supported") } } } diff --git a/spec/acceptance/bootstrap_spec.rb b/spec/acceptance/bootstrap_spec.rb index b887d4a..c92fd61 100644 --- a/spec/acceptance/bootstrap_spec.rb +++ b/spec/acceptance/bootstrap_spec.rb @@ -1,13 +1,13 @@ require 'spec_helper_acceptance' osfamily = fact('osfamily') roles = hosts[0]['roles'] -t = TestManifests.new(roles, 0) +t = TestManifests.new(roles, 0, 0) bootstrap_pp = t.bootstrap_pp() describe 'Test Entry Criteria' do it "Should work with no errors (#{osfamily})" do apply_manifest(bootstrap_pp, catch_failures: true) shell('[ -d /opt/rh/ruby200 ] && /usr/bin/gem install puppet -v 3.8.7 --no-rdoc --no-ri; true') end end diff --git a/spec/acceptance/cassandra_spec.rb b/spec/acceptance/cassandra_spec.rb index 7cff55f..14ec230 100644 --- a/spec/acceptance/cassandra_spec.rb +++ b/spec/acceptance/cassandra_spec.rb @@ -1,128 +1,128 @@ require 'spec_helper_acceptance' describe 'cassandra' do roles = hosts[0]['roles'] versions = [] versions.push(2.1) if roles.include? 'cassandra2' versions.push(2.2) if roles.include? 'cassandra2' versions.push(3.0) if roles.include? 'cassandra3' versions.each do |version| - t = TestManifests.new(roles, version) + t = TestManifests.new(roles, version, fact('operatingsystemmajrelease')) describe "Cassandra #{version} installation." do # firewall_pp = t.firewall_pp() cassandra_install_pp = t.cassandra_install_pp it 'should work with no errors' do apply_manifest(cassandra_install_pp, catch_failures: true) end it 'check code is idempotent' do expect(apply_manifest(cassandra_install_pp, catch_failures: true).exit_code).to be_zero end end describe service('cassandra') do it "check Cassandra-#{version} is running and enabled" do is_expected.to be_running is_expected.to be_enabled end end if fact('osfamily') == 'RedHat' describe service('datastax-agent') do it 'check service status' do is_expected.to be_running is_expected.to be_enabled end end end describe "Create schema for #{version}." do it 'should work with no errors' do apply_manifest(t.schema_create_pp, catch_failures: true) end if version != 2.1 it 'check code is idempotent' do expect(apply_manifest(t.schema_create_pp, catch_failures: true).exit_code).to be_zero end end end describe "Schema drop type for #{version}." do it 'should work with no errors' do apply_manifest(t.schema_drop_type_pp, catch_failures: true) end it 'check code is idempotent' do expect(apply_manifest(t.schema_drop_type_pp, catch_failures: true).exit_code).to be_zero end end describe "Revoke permissions for #{version}." do it 'should work with no errors' do apply_manifest(t.permissions_revoke_pp, catch_failures: true) end it 'check code is idempotent' do expect(apply_manifest(t.permissions_revoke_pp, catch_failures: true).exit_code).to be_zero end end describe "Drop user for #{version}" do it 'should work with no errors' do apply_manifest(t.schema_drop_user_pp, catch_failures: true) end it 'check code is idempotent' do expect(apply_manifest(t.schema_drop_user_pp, catch_failures: true).exit_code).to be_zero end end describe "Drop index for #{version}" do it 'should work with no errors' do apply_manifest(t.schema_drop_index_pp, catch_failures: true) end it 'check code is idempotent' do expect(apply_manifest(t.schema_drop_index_pp, catch_failures: true).exit_code).to be_zero end end describe "Drop table for #{version}" do it 'should work with no errors' do apply_manifest(t.schema_drop_table_pp, catch_failures: true) end it 'check code is idempotent' do expect(apply_manifest(t.schema_drop_table_pp, catch_failures: true).exit_code).to be_zero end end describe "Drop keyspace for #{version}" do it 'should work with no errors' do apply_manifest(t.schema_drop_keyspace_pp, catch_failures: true) end it 'check code is idempotent' do expect(apply_manifest(t.schema_drop_keyspace_pp, catch_failures: true).exit_code).to be_zero end end describe "Facts Tests for #{version}" do it 'should work with no errors' do apply_manifest(t.facts_testing_pp, catch_failures: true) end end next unless version != 3.0 describe "Uninstall #{version}." do it 'should work with no errors' do apply_manifest(t.cassandra_uninstall_pp, catch_failures: true) end end end end diff --git a/spec/classes/apache_repo_spec.rb b/spec/classes/apache_repo_spec.rb new file mode 100644 index 0000000..7d46212 --- /dev/null +++ b/spec/classes/apache_repo_spec.rb @@ -0,0 +1,78 @@ +require 'spec_helper' +describe 'cassandra::apache_repo' do + let(:pre_condition) do + [ + 'class apt () {}', + 'class apt::update () {}', + 'define apt::key ($id, $source) {}', + 'define apt::source ($location, $comment, $release, $include) {}' + ] + end + + context 'On a RedHat OS with defaults for all parameters' do + let :facts do + { + osfamily: 'RedHat' + } + end + + let :params do + { + release: '311x' + } + end + + it do + should have_resource_count(1) + + should contain_class('cassandra::apache_repo').only_with( + 'descr' => 'Repo for Apache Cassandra', + 'key_id' => 'A26E528B271F19B9E5D8E19EA278B781FE4B2BDA', + 'key_url' => 'https://www.apache.org/dist/cassandra/KEYS', + 'release' => '311x' + ) + + should contain_yumrepo('cassandra_apache').with( + ensure: 'present', + descr: 'Repo for Apache Cassandra', + baseurl: 'http://www.apache.org/dist/cassandra/redhat/311x', + enabled: 1, + gpgcheck: 1, + gpgkey: 'https://www.apache.org/dist/cassandra/KEYS' + ) + end + end + + context 'On a Debian OS with defaults for all parameters' do + let :facts do + { + osfamily: 'Debian', + lsbdistid: 'Ubuntu', + lsbdistrelease: '14.04' + } + end + + it do + should have_resource_count(3) + should contain_class('apt') + should contain_class('apt::update') + + should contain_apt__key('apache.cassandra').with( + id: 'A26E528B271F19B9E5D8E19EA278B781FE4B2BDA', + source: 'https://www.apache.org/dist/cassandra/KEYS' + ) + + should contain_apt__source('cassandra.sources').with( + location: 'http://www.apache.org/dist/cassandra/debian', + comment: 'Repo for Apache Cassandra', + release: 'main', + include: { 'src' => false } + ).that_notifies('Exec[update-apache-cassandra-repo]') + + should contain_exec('update-apache-cassandra-repo').with( + refreshonly: true, + command: '/bin/true' + ) + end + end +end diff --git a/spec/spec_helper_acceptance.rb b/spec/spec_helper_acceptance.rb index 6ccfb01..2f97515 100644 --- a/spec/spec_helper_acceptance.rb +++ b/spec/spec_helper_acceptance.rb @@ -1,508 +1,528 @@ require 'beaker-rspec' require 'pry' class TestManifests - def initialize(roles, version) + def initialize(roles, version, operatingsystemmajrelease) # Instance variables @roles = roles @version = version if version == 2.1 init21 elsif version == 2.2 init22 elsif version == 3.0 - init30 + init30(operatingsystemmajrelease) end end def init21 @debian_release = '21x' @debian_package_ensure = '2.1.18' @redhat_package_ensure = '2.1.15-1' @cassandra_optutils_package = 'cassandra21-tools' @cassandra_package = 'cassandra21' end def init22 @debian_release = '22x' @debian_package_ensure = '2.2.10' @redhat_package_ensure = '2.2.8-1' @cassandra_optutils_package = 'cassandra22-tools' @cassandra_package = 'cassandra22' end - def init30 + def init30(operatingsystemmajrelease) @debian_release = '30x' @debian_package_ensure = '3.0.14' - @redhat_package_ensure = '3.0.9-1' - @cassandra_optutils_package = 'cassandra30-tools' - @cassandra_package = 'cassandra30' + print "operatingsystemmajrelease := #{operatingsystemmajrelease}" + + if operatingsystemmajrelease == '6' + @redhat_package_ensure = '3.0.9-1' + @cassandra_optutils_package = 'cassandra30-tools' + @cassandra_package = 'cassandra30' + else + @redhat_release = '30x' + @redhat_package_ensure = '3.0.14-1' + @cassandra_optutils_package = 'cassandra-tools' + @cassandra_package = 'cassandra' + end end def bootstrap_pp <<-EOS Exec { path => [ '/usr/local/bin', '/opt/local/bin', '/usr/bin', '/usr/sbin', '/bin', '/sbin'], logoutput => true, } - notify { "${::operatingsystem}-${::operatingsystemmajrelease}": } + notify { "${::hostname}:${::operatingsystem}-${::operatingsystemmajrelease}": } file { '/etc/dse': ensure => directory, } -> file { '/etc/dse/dse-env.sh': ensure => present, content => "#export DSE_HOME\n# export HADOOP_LOG_DIR=", } case downcase("${::operatingsystem}-${::operatingsystemmajrelease}") { 'centos-6': { package { ['gcc', 'tar', 'yum-utils', 'centos-release-scl']: } -> exec { 'yum-config-manager --enable rhel-server-rhscl-7-rpms': } -> package { 'ruby200': } -> package { 'python27-python': ensure => '2.7.8-18.el6', } -> exec { 'cp /opt/rh/python27/enable /etc/profile.d/python.sh': } -> exec { 'echo "\n" >> /etc/profile.d/python.sh': } -> exec { 'echo "export PYTHONPATH=/usr/lib/python2.7/site-packages" >> /etc/profile.d/python.sh': } -> exec { '/bin/cp /opt/rh/ruby200/enable /etc/profile.d/ruby.sh': } -> exec { '/bin/rm /usr/bin/ruby /usr/bin/gem': } -> exec { '/usr/sbin/alternatives --install /usr/bin/ruby ruby /opt/rh/ruby200/root/usr/bin/ruby 1000': } -> exec { '/usr/sbin/alternatives --install /usr/bin/gem gem /opt/rh/ruby200/root/usr/bin/gem 1000': } } 'centos-7': { package { ['gcc', 'tar', 'initscripts']: } } 'debian-7': { package { ['sudo', 'ufw', 'wget']: } } 'debian-8': { package { ['locales-all', 'net-tools', 'sudo', 'ufw']: } -> file { '/usr/sbin/policy-rc.d': ensure => absent, } } 'ubuntu-12.04': { package {['python-software-properties', 'iptables', 'sudo']:} -> exec {'/usr/bin/apt-add-repository ppa:brightbox/ruby-ng':} -> exec {'/usr/bin/apt-get update': } -> package {'ruby2.0': } -> exec { '/bin/rm /usr/bin/ruby': } -> exec { '/usr/sbin/update-alternatives --install /usr/bin/ruby ruby /usr/bin/ruby2.0 1000': } } 'ubuntu-16.04': { package { ['locales-all', 'net-tools', 'sudo', 'ufw']: } -> file { '/usr/sbin/policy-rc.d': ensure => absent, } } } EOS end def cassandra_install_pp <<-EOS if $::osfamily == 'Debian' { class { 'cassandra::apache_repo': release => '#{@debian_release}', before => Class['cassandra', 'cassandra::optutils'], } $package_ensure = '#{@debian_package_ensure}' $cassandra_package = 'cassandra' $cassandra_optutils_package = 'cassandra-tools' } else { - class { 'cassandra::datastax_repo': - before => Class['cassandra', 'cassandra::optutils'], + if #{@version} >= 3.0 and $::operatingsystemmajrelease >= 7 { + yumrepo { 'datastax': + ensure => absent, + } -> + class { 'cassandra::apache_repo': + release => '#{@redhat_release}', + before => Class['cassandra', 'cassandra::optutils'], + } + } else { + class { 'cassandra::datastax_repo': + before => Class['cassandra', 'cassandra::optutils'], + } } + $package_ensure = '#{@redhat_package_ensure}' $cassandra_package = '#{@cassandra_package}' $cassandra_optutils_package = '#{@cassandra_optutils_package}' } require cassandra::system::swapoff require cassandra::system::transparent_hugepage include cassandra::java if versioncmp($::rubyversion, '1.9.0') < 0 { $service_refresh = false } else { $service_refresh = true } if #{@version} >= 3.0 { class { 'cassandra': hints_directory => '/var/lib/cassandra/hints', package_ensure => $package_ensure, package_name => $cassandra_package, service_refresh => $service_refresh, require => Class['cassandra::java'], } } else { class { 'cassandra': package_ensure => $package_ensure, package_name => $cassandra_package, service_refresh => $service_refresh, require => Class['cassandra::java'], } } class { 'cassandra::optutils': package_ensure => $package_ensure, package_name => $cassandra_optutils_package, require => Class['cassandra'] } if $::osfamily == 'RedHat' { class { 'cassandra::datastax_agent': require => Class['cassandra'] } } #{firewall_pp} include cassandra::dse EOS end def cassandra_uninstall_pp <<-EOS Exec { path => [ '/usr/bin', '/bin' ], logoutput => true, } if $::osfamily == 'RedHat' { $cassandra_optutils_package = '#{@cassandra_optutils_package}' $cassandra_package = '#{@cassandra_package}' } else { $cassandra_optutils_package = 'cassandra-tools' $cassandra_package = 'cassandra' } service { 'cassandra': ensure => stopped, } -> package { $cassandra_optutils_package: ensure => purged, } -> package { $cassandra_package: ensure => purged, } -> exec { 'rm -rf /var/lib/cassandra/*/* /var/log/cassandra/*': } EOS end def facts_testing_pp <<-EOS #{cassandra_install_pp} if $::osfamily == 'Debian' { $package_comparison = $cassandrarelease } else { $package_comparison = "${cassandrarelease}-1" } if $package_comparison != $package_ensure { fail("cassandrarelease: ${package_comparison} != ${package_ensure}") } if $::cassandramaxheapsize <= 0 { fail('cassandramaxheapsize is not set.') } if $::cassandracmsmaxheapsize <= 0 { fail('cassandracmsmaxheapsize is not set.') } if $::cassandraheapnewsize <= 0 { fail('cassandraheapnewsize is not set.') } if $::cassandracmsheapnewsize <= 0 { fail('cassandracmsheapnewsize is not set.') } EOS end def firewall_pp pp = if @roles.include? 'firewall' <<-EOS class { '::cassandra::firewall_ports': require => Class['::cassandra'], } EOS else <<-EOS # Firewall test skipped EOS end pp end def permissions_revoke_pp <<-EOS #{cassandra_install_pp} class { 'cassandra::schema': cqlsh_password => 'Niner2', cqlsh_user => 'akers', cqlsh_client_config => '/root/.puppetcqlshrc', permissions => { 'Revoke select permissions to spillman to all keyspaces' => { ensure => absent, permission_name => 'SELECT', user_name => 'spillman', }, 'Revoke modify to to keyspace mykeyspace to akers' => { ensure => absent, keyspace_name => 'mykeyspace', permission_name => 'MODIFY', user_name => 'akers', }, 'Revoke alter permissions to mykeyspace to boone' => { ensure => absent, keyspace_name => 'mykeyspace', permission_name => 'ALTER', user_name => 'boone', }, 'Revoke ALL permissions to mykeyspace.users to gbennet' => { ensure => absent, keyspace_name => 'mykeyspace', permission_name => 'ALTER', table_name => 'users', user_name => 'gbennet', }, }, } EOS end def schema_create_pp <<-EOS #{cassandra_install_pp} $cql_types = { 'fullname' => { 'keyspace' => 'mykeyspace', 'fields' => { 'fname' => 'text', 'lname' => 'text', }, }, } $keyspaces = { 'mykeyspace' => { ensure => present, replication_map => { keyspace_class => 'SimpleStrategy', replication_factor => 1, }, durable_writes => false, }, } class { 'cassandra::schema': cql_types => $cql_types, cqlsh_password => 'cassandra', cqlsh_user => 'cassandra', indexes => { 'users_lname_idx' => { keyspace => 'mykeyspace', table => 'users', keys => 'lname', }, }, keyspaces => $keyspaces, tables => { 'users' => { 'keyspace' => 'mykeyspace', 'columns' => { 'userid' => 'int', 'fname' => 'text', 'lname' => 'text', 'PRIMARY KEY' => '(userid)', }, }, }, permissions => { 'Grant select permissions to spillman to all keyspaces' => { permission_name => 'SELECT', user_name => 'spillman', }, 'Grant modify to to keyspace mykeyspace to akers' => { keyspace_name => 'mykeyspace', permission_name => 'MODIFY', user_name => 'akers', }, 'Grant alter permissions to mykeyspace to boone' => { keyspace_name => 'mykeyspace', permission_name => 'ALTER', user_name => 'boone', }, 'Grant ALL permissions to mykeyspace.users to gbennet' => { keyspace_name => 'mykeyspace', permission_name => 'ALTER', table_name => 'users', user_name => 'gbennet', }, }, users => { 'akers' => { password => 'Niner2', superuser => true, }, 'boone' => { password => 'Niner75', }, 'gbennet' => { password => 'Strewth', }, 'spillman' => { password => 'Niner27', }, 'bob' => { password => 'kaZe89a', login => false, }, 'john' => { superuser => true, password => 'kaZe89a', login => true, }, }, } EOS end def schema_drop_index_pp <<-EOS #{cassandra_install_pp} class { 'cassandra::schema': cqlsh_user => 'akers', cqlsh_password => 'Niner2', indexes => { 'users_lname_idx' => { ensure => absent, keyspace => 'mykeyspace', table => 'users', }, }, } EOS end def schema_drop_table_pp <<-EOS #{cassandra_install_pp} class { 'cassandra::schema': cqlsh_password => 'Niner2', cqlsh_user => 'akers', tables => { 'users' => { ensure => absent, keyspace => 'mykeyspace', }, }, } EOS end def schema_drop_keyspace_pp <<-EOS #{cassandra_install_pp} $keyspaces = { 'mykeyspace' => { ensure => absent, } } class { 'cassandra::schema': cqlsh_password => 'Niner2', cqlsh_user => 'akers', keyspaces => $keyspaces, } EOS end def schema_drop_type_pp pp = <<-EOS #{cassandra_install_pp} $cql_types = { 'fullname' => { 'keyspace' => 'mykeyspace', 'ensure' => 'absent' } } class { 'cassandra::schema': cql_types => $cql_types, cqlsh_user => 'akers', cqlsh_password => 'Niner2', } EOS pp end def schema_drop_user_pp pp = <<-EOS #{cassandra_install_pp} class { 'cassandra::schema': cqlsh_password => 'Niner2', cqlsh_user => 'akers', cqlsh_client_config => '/root/.puppetcqlshrc', users => { 'boone' => { ensure => absent, }, }, } EOS pp end end hosts.each do |host| case host.name when 'ubuntu1604' host.install_package('puppet') else install_puppet_on(host) end end RSpec.configure do |c| module_root = File.expand_path(File.join(File.dirname(__FILE__), '..')) c.formatter = :documentation # Configure all nodes in nodeset c.before :suite do # Install modules puppet_module_install(source: module_root, module_name: 'cassandra') hosts.each do |host| on host, puppet('module', 'install', 'puppetlabs-apt'), acceptable_exit_codes: [0, 1] on host, puppet('module', 'install', 'puppetlabs-firewall'), acceptable_exit_codes: [0, 1] on host, puppet('module', 'install', 'puppetlabs-inifile'), acceptable_exit_codes: [0, 1] on host, puppet('module', 'install', 'puppetlabs-stdlib'), acceptable_exit_codes: [0, 1] # Install hiera write_hiera_config_on(host, [ 'environments/%{environment}/data/fqdn/%{fqdn}', 'environments/%{environment}/data/osfamily/%{osfamily}/%{lsbdistcodename}', 'environments/%{environment}/data/osfamily/%{osfamily}/%{lsbmajdistrelease}', 'environments/%{environment}/data/osfamily/%{osfamily}/%{architecture}', 'environments/%{environment}/data/osfamily/%{osfamily}/common', # 'environments/%{environment}/data/modules/%{cname}', 'environments/%{environment}/data/modules/%{caller_module_name}', 'environments/%{environment}/data/modules/%{module_name}', 'environments/%{environment}/data/common' ]) copy_hiera_data_to(host, './spec/acceptance/hieradata/') end end end