diff --git a/REFERENCE.md b/REFERENCE.md index c02a3e4..f1a46a8 100644 --- a/REFERENCE.md +++ b/REFERENCE.md @@ -1,1612 +1,1676 @@ # Reference ## Table of Contents **Classes** _Public Classes_ * [`kafka`](#kafka): This class handles the Kafka requirements. * [`kafka::broker`](#kafkabroker): This class handles the Kafka (broker). * [`kafka::consumer`](#kafkaconsumer): This class handles the Kafka (consumer). * [`kafka::consumer::config`](#kafkaconsumerconfig): This class handles the Kafka (consumer) config. * [`kafka::mirror`](#kafkamirror): This class handles the Kafka (mirror). * [`kafka::params`](#kafkaparams): This class provides default parameters. * [`kafka::producer`](#kafkaproducer): This class handles the Kafka (producer). * [`kafka::producer::config`](#kafkaproducerconfig): This class handles the Kafka (producer) config. _Private Classes_ * `kafka::broker::config`: This class handles the Kafka (broker) config. * `kafka::broker::install`: This class handles the Kafka (broker) package. * `kafka::broker::service`: This class handles the Kafka (broker) service. * `kafka::consumer::install`: This class handles the Kafka (consumer) package. * `kafka::consumer::service`: This class handles the Kafka (consumer) service. * `kafka::mirror::config`: This class handles the Kafka (mirror) config. * `kafka::mirror::install`: This class handles the Kafka (mirror) package. * `kafka::mirror::service`: This class handles the Kafka (mirror) service. * `kafka::producer::install`: This class handles the Kafka (producer) package. * `kafka::producer::service`: This class handles the Kafka (producer) service. **Defined types** * [`kafka::topic`](#kafkatopic): This defined type handles the creation of Kafka topics. ## Classes ### kafka This class handles the Kafka requirements. #### Examples ##### Basic usage ```puppet class { 'kafka': } ``` #### Parameters The following parameters are available in the `kafka` class. ##### `kafka_version` Data type: `String[1]` The version of Kafka that should be installed. Default value: $kafka::params::kafka_version ##### `scala_version` Data type: `String[1]` The scala version what Kafka was built with. Default value: $kafka::params::scala_version ##### `install_dir` Data type: `Stdlib::Absolutepath` The directory to install Kafka to. Default value: $kafka::params::install_dir ##### `mirror_url` Data type: `Stdlib::HTTPUrl` The url where the Kafka is downloaded from. Default value: $kafka::params::mirror_url ##### `manage_java` Data type: `Boolean` Install java if it's not already installed. Default value: $kafka::params::manage_java ##### `package_dir` Data type: `Stdlib::Absolutepath` The directory to install Kafka. Default value: $kafka::params::package_dir ##### `package_name` Data type: `Optional[String[1]]` Package name, when installing Kafka from a package. Default value: $kafka::params::package_name ##### `mirror_subpath` Data type: `Optional[String[1]]` The sub directory where the source is downloaded from. Default value: $kafka::params::mirror_subpath ##### `proxy_server` Data type: `Optional[String[1]]` Set proxy server, when installing Kafka from source. Default value: $kafka::params::proxy_server ##### `proxy_port` Data type: `Optional[String[1]]` Set proxy port, when installing Kafka from source. Default value: $kafka::params::proxy_port ##### `proxy_host` Data type: `Optional[String[1]]` Set proxy host, when installing Kafka from source. Default value: $kafka::params::proxy_host ##### `proxy_type` Data type: `Optional[String[1]]` Set proxy type, when installing Kafka from source. Default value: $kafka::params::proxy_type ##### `package_ensure` Data type: `String[1]` Package version or ensure state, when installing Kafka from a package. Default value: $kafka::params::package_ensure ##### `user_name` Data type: `String[1]` User to run Kafka as. Default value: $kafka::params::user_name ##### `user_shell` Data type: `Stdlib::Absolutepath` Login shell of the Kafka user. Default value: $kafka::params::user_shell ##### `group_name` Data type: `String[1]` Group to run Kafka as. Default value: $kafka::params::group_name ##### `system_user` Data type: `Boolean` Whether the Kafka user is a system user or not. Default value: $kafka::params::system_user ##### `system_group` Data type: `Boolean` Whether the Kafka group is a system group or not. Default value: $kafka::params::system_group ##### `user_id` Data type: `Optional[Integer]` Create the Kafka user with this ID. Default value: $kafka::params::user_id ##### `group_id` Data type: `Optional[Integer]` Create the Kafka group with this ID. Default value: $kafka::params::group_id ##### `manage_user` Data type: `Boolean` Create the Kafka user if it's not already present. Default value: $kafka::params::manage_user ##### `manage_group` Data type: `Boolean` Create the Kafka group if it's not already present. Default value: $kafka::params::manage_group ##### `config_dir` Data type: `Stdlib::Absolutepath` The directory to create the Kafka config files to. Default value: $kafka::params::config_dir ##### `log_dir` Data type: `Stdlib::Absolutepath` The directory for Kafka log files. Default value: $kafka::params::log_dir ##### `install_mode` Data type: `Stdlib::Filemode` The permissions for the install directory. Default value: $kafka::params::install_mode ### kafka::broker This class handles the Kafka (broker). #### Examples ##### Basic usage ```puppet class { 'kafka::broker': config => { 'broker.id' => '0', 'zookeeper.connect' => 'localhost:2181' } } ``` #### Parameters The following parameters are available in the `kafka::broker` class. ##### `kafka_version` Data type: `String[1]` The version of Kafka that should be installed. Default value: $kafka::params::kafka_version ##### `scala_version` Data type: `String[1]` The scala version what Kafka was built with. Default value: $kafka::params::scala_version ##### `install_dir` Data type: `Stdlib::Absolutepath` The directory to install Kafka to. Default value: $kafka::params::install_dir ##### `mirror_url` Data type: `Stdlib::HTTPUrl` The url where the Kafka is downloaded from. Default value: $kafka::params::mirror_url ##### `manage_java` Data type: `Boolean` Install java if it's not already installed. Default value: $kafka::params::manage_java ##### `package_dir` Data type: `Stdlib::Absolutepath` The directory to install Kafka. Default value: $kafka::params::package_dir ##### `package_name` Data type: `Optional[String[1]]` Package name, when installing Kafka from a package. Default value: $kafka::params::package_name ##### `package_ensure` Data type: `String[1]` Package version or ensure state, when installing Kafka from a package. Default value: $kafka::params::package_ensure ##### `user_name` Data type: `String[1]` User to run Kafka as. Default value: $kafka::params::user_name ##### `user_shell` Data type: `Stdlib::Absolutepath` Login shell of the Kafka user. Default value: $kafka::params::user_shell ##### `group_name` Data type: `String[1]` Group to run Kafka as. Default value: $kafka::params::group_name ##### `user_id` Data type: `Optional[Integer]` Create the Kafka user with this ID. Default value: $kafka::params::user_id ##### `group_id` Data type: `Optional[Integer]` Create the Kafka group with this ID. Default value: $kafka::params::group_id ##### `manage_user` Data type: `Boolean` Create the Kafka user if it's not already present. Default value: $kafka::params::manage_user ##### `manage_group` Data type: `Boolean` Create the Kafka group if it's not already present. Default value: $kafka::params::manage_group ##### `config_mode` Data type: `Stdlib::Filemode` The permissions for the config files. Default value: $kafka::params::config_mode ##### `config_dir` Data type: `Stdlib::Absolutepath` The directory to create the Kafka config files to. Default value: $kafka::params::config_dir ##### `log_dir` Data type: `Stdlib::Absolutepath` The directory for Kafka log files. Default value: $kafka::params::log_dir ##### `bin_dir` Data type: `Stdlib::Absolutepath` The directory where the Kafka scripts are. Default value: $kafka::params::bin_dir ##### `service_name` Data type: `String[1]` Set the name of the service. Default value: 'kafka' ##### `manage_service` Data type: `Boolean` Install the init.d or systemd service. Default value: $kafka::params::manage_service ##### `service_ensure` Data type: `Enum['running', 'stopped']` Set the ensure state of the service. Default value: $kafka::params::service_ensure ##### `service_restart` Data type: `Boolean` Whether the configuration files should trigger a service restart. Default value: $kafka::params::service_restart ##### `service_requires` Data type: `Array[String[1]]` Set the list of services required to be running before Kafka. Default value: $kafka::params::service_requires ##### `limit_nofile` Data type: `Optional[String[1]]` Set the 'LimitNOFILE' option of the systemd service. Default value: $kafka::params::limit_nofile ##### `limit_core` Data type: `Optional[String[1]]` Set the 'LimitCORE' option of the systemd service. Default value: $kafka::params::limit_core ##### `timeout_stop` Data type: `Optional[String[1]]` Set the 'TimeoutStopSec' option of the systemd service. Default value: $kafka::params::timeout_stop ##### `exec_stop` Data type: `Boolean` Set the 'ExecStop' option of the systemd service to 'kafka-server-stop.sh'. Default value: $kafka::params::exec_stop ##### `daemon_start` Data type: `Boolean` Use the '-daemon' option when starting Kafka with 'kafka-server-start.sh'. Default value: $kafka::params::daemon_start ##### `env` Data type: `Hash` A hash of the environment variables to set. Default value: {} ##### `config` Data type: `Hash[String[1], Any]` A hash of the broker configuration options. Default value: {} ##### `heap_opts` Data type: `String[1]` Set the Java heap size. Default value: $kafka::params::broker_heap_opts ##### `jmx_opts` Data type: `String[1]` Set the JMX options. Default value: $kafka::params::broker_jmx_opts ##### `log4j_opts` Data type: `String[1]` Set the Log4j options. Default value: $kafka::params::broker_log4j_opts ##### `opts` Data type: `String[0]` Set the Kafka options. Default value: $kafka::params::broker_opts +##### `manage_log4j` + +Data type: `Boolean` + +Enable management of `log4j.properties` file. + +Default value: $kafka::params::manage_log4j + +##### `log_file_size` + +Data type: `Pattern[/[1-9][0-9]*[KMG]B/]` + +Maximum allowed log file size before rolling over. Suffixes "KB", "MB" and "GB" are allowed + +Default value: $kafka::params::log_file_size + +##### `log_file_count` + +Data type: `Integer[1, 50]` + +Set a count of rotated files which will be stored in `log_dir` + +Default value: $kafka::params::log_file_count + ### kafka::consumer This class handles the Kafka (consumer). #### Examples ##### Basic usage ```puppet class { 'kafka::consumer': config => { 'client.id' => '0', 'zookeeper.connect' => 'localhost:2181' } } ``` #### Parameters The following parameters are available in the `kafka::consumer` class. ##### `kafka_version` Data type: `String[1]` The version of Kafka that should be installed. Default value: $kafka::params::kafka_version ##### `scala_version` Data type: `String[1]` The scala version what Kafka was built with. Default value: $kafka::params::scala_version ##### `install_dir` Data type: `Stdlib::Absolutepath` The directory to install Kafka to. Default value: $kafka::params::install_dir ##### `mirror_url` Data type: `Stdlib::HTTPUrl` The url where the Kafka is downloaded from. Default value: $kafka::params::mirror_url ##### `manage_java` Data type: `Boolean` Install java if it's not already installed. Default value: $kafka::params::manage_java ##### `package_dir` Data type: `Stdlib::Absolutepath` The directory to install Kafka. Default value: $kafka::params::package_dir ##### `package_name` Data type: `Optional[String[1]]` Package name, when installing Kafka from a package. Default value: $kafka::params::package_name ##### `package_ensure` Data type: `String[1]` Package version or ensure state, when installing Kafka from a package. Default value: $kafka::params::package_ensure ##### `user_name` Data type: `String[1]` User to run Kafka as. Default value: $kafka::params::user_name ##### `user_shell` Data type: `Stdlib::Absolutepath` Login shell of the Kafka user. Default value: $kafka::params::user_shell ##### `group_name` Data type: `String[1]` Group to run Kafka as. Default value: $kafka::params::group_name ##### `user_id` Data type: `Optional[Integer]` Create the Kafka user with this ID. Default value: $kafka::params::user_id ##### `group_id` Data type: `Optional[Integer]` Create the Kafka group with this ID. Default value: $kafka::params::group_id ##### `manage_user` Data type: `Boolean` Create the Kafka user if it's not already present. Default value: $kafka::params::manage_user ##### `manage_group` Data type: `Boolean` Create the Kafka group if it's not already present. Default value: $kafka::params::manage_group ##### `config_mode` Data type: `Stdlib::Filemode` The permissions for the config files. Default value: $kafka::params::config_mode ##### `config_dir` Data type: `Stdlib::Absolutepath` The directory to create the Kafka config files to. Default value: $kafka::params::config_dir ##### `log_dir` Data type: `Stdlib::Absolutepath` The directory for Kafka log files. Default value: $kafka::params::log_dir ##### `bin_dir` Data type: `Stdlib::Absolutepath` The directory where the Kafka scripts are. Default value: $kafka::params::bin_dir ##### `service_name` Data type: `String[1]` Set the name of the service. Default value: 'kafka-consumer' ##### `manage_service` Data type: `Boolean` Install the init.d or systemd service. Default value: $kafka::params::manage_service ##### `service_ensure` Data type: `Enum['running', 'stopped']` Set the ensure state of the service. Default value: $kafka::params::service_ensure ##### `service_restart` Data type: `Boolean` Whether the configuration files should trigger a service restart. Default value: $kafka::params::service_restart ##### `service_requires` Data type: `Array[String[1]]` Set the list of services required to be running before Kafka. Default value: $kafka::params::service_requires ##### `limit_nofile` Data type: `Optional[String[1]]` Set the 'LimitNOFILE' option of the systemd service. Default value: $kafka::params::limit_nofile ##### `limit_core` Data type: `Optional[String[1]]` Set the 'LimitCORE' option of the systemd service. Default value: $kafka::params::limit_core ##### `env` Data type: `Hash` A hash of the environment variables to set. Default value: {} ##### `config` Data type: `Hash[String[1], Any]` A hash of the consumer configuration options. Default value: {} ##### `service_config` Data type: `Hash[String[1],String[1]]` A hash of the `kafka-console-consumer.sh` script options. Default value: {} ##### `jmx_opts` Data type: `String[1]` Set the JMX options. Default value: $kafka::params::consumer_jmx_opts ##### `log4j_opts` Data type: `String[1]` Set the Log4j options. Default value: $kafka::params::consumer_log4j_opts +##### `manage_log4j` + +Data type: `Boolean` + +Enable management of `log4j.properties` file. + +Default value: $kafka::params::manage_log4j + +##### `log_file_size` + +Data type: `Pattern[/[1-9][0-9]*[KMG]B/]` + +Maximum allowed log file size before rolling over. Suffixes "KB", "MB" and "GB" are allowed + +Default value: $kafka::params::log_file_size + +##### `log_file_count` + +Data type: `Integer[1, 50]` + +Set a count of rotated files which will be stored in `log_dir` + +Default value: $kafka::params::log_file_count + ### kafka::consumer::config This class handles the Kafka (consumer) config. #### Parameters The following parameters are available in the `kafka::consumer::config` class. ##### `manage_service` Data type: `Boolean` - - Default value: $kafka::consumer::manage_service ##### `service_name` Data type: `String[1]` - - Default value: $kafka::consumer::service_name ##### `service_restart` Data type: `Boolean` - - Default value: $kafka::consumer::service_restart ##### `config` Data type: `Hash[String[1], Any]` - - Default value: $kafka::consumer::config ##### `config_dir` Data type: `Stdlib::Absolutepath` - - Default value: $kafka::consumer::config_dir ##### `user_name` Data type: `String[1]` - - Default value: $kafka::consumer::user_name ##### `group_name` Data type: `String[1]` - - Default value: $kafka::consumer::group_name ##### `config_mode` Data type: `Stdlib::Filemode` +Default value: $kafka::consumer::config_mode + +##### `manage_log4j` +Data type: `Boolean` -Default value: $kafka::consumer::config_mode +Enable management of `log4j.properties` file. + +Default value: $kafka::consumer::manage_log4j + +##### `log_file_size` + +Data type: `Pattern[/[1-9][0-9]*[KMG]B/]` + +Maximum allowed log file size before rolling over. Suffixes "KB", "MB" and "GB" are allowed + +Default value: $kafka::consumer::log_file_size + +##### `log_file_count` + +Data type: `Integer[1, 50]` + +Set a count of rotated files which will be stored in `log_dir` + +Default value: $kafka::consumer::log_file_count ### kafka::mirror This class handles the Kafka (mirror). #### Examples ##### Basic usage ```puppet class { 'kafka::mirror': consumer_config => { 'group.id' => 'kafka-mirror', 'zookeeper.connect' => 'localhost:2181' }, producer_config => { 'zookeeper.connect' => 'localhost:2181', }, service_config => { 'whitelist' => '.*', } } ``` #### Parameters The following parameters are available in the `kafka::mirror` class. ##### `kafka_version` Data type: `String[1]` The version of Kafka that should be installed. Default value: $kafka::params::kafka_version ##### `scala_version` Data type: `String[1]` The scala version what Kafka was built with. Default value: $kafka::params::scala_version ##### `install_dir` Data type: `Stdlib::Absolutepath` The directory to install Kafka to. Default value: $kafka::params::install_dir ##### `mirror_url` Data type: `Stdlib::HTTPUrl` The url where the Kafka is downloaded from. Default value: $kafka::params::mirror_url ##### `manage_java` Data type: `Boolean` Install java if it's not already installed. Default value: $kafka::params::manage_java ##### `package_dir` Data type: `Stdlib::Absolutepath` The directory to install Kafka. Default value: $kafka::params::package_dir ##### `package_name` Data type: `Optional[String[1]]` Package name, when installing Kafka from a package. Default value: $kafka::params::package_name ##### `package_ensure` Data type: `String[1]` Package version or ensure state, when installing Kafka from a package. Default value: $kafka::params::package_ensure ##### `user_name` Data type: `String[1]` User to run Kafka as. Default value: $kafka::params::user_name ##### `user_shell` Data type: `Stdlib::Absolutepath` Login shell of the Kafka user. Default value: $kafka::params::user_shell ##### `group_name` Data type: `String[1]` Group to run Kafka as. Default value: $kafka::params::group_name ##### `user_id` Data type: `Optional[Integer]` Create the Kafka user with this ID. Default value: $kafka::params::user_id ##### `group_id` Data type: `Optional[Integer]` Create the Kafka group with this ID. Default value: $kafka::params::group_id ##### `manage_user` Data type: `Boolean` Create the Kafka user if it's not already present. Default value: $kafka::params::manage_user ##### `manage_group` Data type: `Boolean` Create the Kafka group if it's not already present. Default value: $kafka::params::manage_group ##### `config_mode` Data type: `Stdlib::Filemode` The permissions for the config files. Default value: $kafka::params::config_mode ##### `config_dir` Data type: `Stdlib::Absolutepath` The directory to create the Kafka config files to. Default value: $kafka::params::config_dir ##### `log_dir` Data type: `Stdlib::Absolutepath` The directory for Kafka log files. Default value: $kafka::params::log_dir ##### `bin_dir` Data type: `Stdlib::Absolutepath` The directory where the Kafka scripts are. Default value: $kafka::params::bin_dir ##### `service_name` Data type: `String[1]` Set the name of the service. Default value: 'kafka-mirror' ##### `manage_service` Data type: `Boolean` Install the init.d or systemd service. Default value: $kafka::params::manage_service ##### `service_ensure` Data type: `Enum['running', 'stopped']` Set the ensure state of the service. Default value: $kafka::params::service_ensure ##### `service_restart` Data type: `Boolean` Whether the configuration files should trigger a service restart. Default value: $kafka::params::service_restart ##### `service_requires` Data type: `Array[String[1]]` Set the list of services required to be running before Kafka. Default value: $kafka::params::service_requires ##### `limit_nofile` Data type: `Optional[String[1]]` Set the 'LimitNOFILE' option of the systemd service. Default value: $kafka::params::limit_nofile ##### `limit_core` Data type: `Optional[String[1]]` Set the 'LimitCORE' option of the systemd service. Default value: $kafka::params::limit_core ##### `env` Data type: `Hash` A hash of the environment variables to set. Default value: {} ##### `consumer_config` Data type: `Hash[String[1],String[1]]` A hash of the consumer configuration options. Default value: {} ##### `producer_config` Data type: `Hash[String[1],String[1]]` A hash of the producer configuration options. Default value: {} ##### `service_config` Data type: `Hash[String[1],String[1]]` A hash of the mirror script options. Default value: {} ##### `heap_opts` Data type: `String[1]` Set the Java heap size. Default value: $kafka::params::mirror_heap_opts ##### `jmx_opts` Data type: `String[1]` Set the JMX options. Default value: $kafka::params::mirror_jmx_opts ##### `log4j_opts` Data type: `String[1]` Set the Log4j options. Default value: $kafka::params::mirror_log4j_opts +##### `manage_log4j` + +Data type: `Boolean` + +Enable management of `log4j.properties` file. + +Default value: $kafka::params::manage_log4j + +##### `log_file_size` + +Data type: `Pattern[/[1-9][0-9]*[KMG]B/]` + +Maximum allowed log file size before rolling over. Suffixes "KB", "MB" and "GB" are allowed + +Default value: $kafka::params::log_file_size + +##### `log_file_count` + +Data type: `Integer[1, 50]` + +Set a count of rotated files which will be stored in `log_dir` + +Default value: $kafka::params::log_file_count + ### kafka::params This class provides default parameters. ### kafka::producer This class handles the Kafka (producer). #### Examples ##### Basic usage ```puppet class { 'kafka::producer': config => { 'client.id' => '0', 'zookeeper.connect' => 'localhost:2181' } } ``` #### Parameters The following parameters are available in the `kafka::producer` class. ##### `input` Data type: `Optional[String[1]]` Set named pipe as input. ##### `kafka_version` Data type: `String[1]` The version of Kafka that should be installed. Default value: $kafka::params::kafka_version ##### `scala_version` Data type: `String[1]` The scala version what Kafka was built with. Default value: $kafka::params::scala_version ##### `install_dir` Data type: `Stdlib::Absolutepath` The directory to install Kafka to. Default value: $kafka::params::install_dir ##### `mirror_url` Data type: `Stdlib::HTTPUrl` The url where the Kafka is downloaded from. Default value: $kafka::params::mirror_url ##### `manage_java` Data type: `Boolean` Install java if it's not already installed. Default value: $kafka::params::manage_java ##### `package_dir` Data type: `Stdlib::Absolutepath` The directory to install Kafka. Default value: $kafka::params::package_dir ##### `package_name` Data type: `Optional[String[1]]` Package name, when installing Kafka from a package. Default value: $kafka::params::package_name ##### `package_ensure` Data type: `String[1]` Package version or ensure state, when installing Kafka from a package. Default value: $kafka::params::package_ensure ##### `user_name` Data type: `String[1]` User to run Kafka as. Default value: $kafka::params::user_name ##### `user_shell` Data type: `Stdlib::Absolutepath` Login shell of the Kafka user. Default value: $kafka::params::user_shell ##### `group_name` Data type: `String[1]` Group to run Kafka as. Default value: $kafka::params::group_name ##### `user_id` Data type: `Optional[Integer]` Create the Kafka user with this ID. Default value: $kafka::params::user_id ##### `group_id` Data type: `Optional[Integer]` Create the Kafka group with this ID. Default value: $kafka::params::group_id ##### `manage_user` Data type: `Boolean` Create the Kafka user if it's not already present. Default value: $kafka::params::manage_user ##### `manage_group` Data type: `Boolean` Create the Kafka group if it's not already present. Default value: $kafka::params::manage_group ##### `config_mode` Data type: `Stdlib::Filemode` The permissions for the config files. Default value: $kafka::params::config_mode ##### `config_dir` Data type: `Stdlib::Absolutepath` The directory to create the Kafka config files to. Default value: $kafka::params::config_dir ##### `log_dir` Data type: `Stdlib::Absolutepath` The directory for Kafka log files. Default value: $kafka::params::log_dir ##### `bin_dir` Data type: `Stdlib::Absolutepath` The directory where the Kafka scripts are. Default value: $kafka::params::bin_dir ##### `service_name` Data type: `String[1]` Set the name of the service. Default value: 'kafka-producer' ##### `manage_service` Data type: `Boolean` Install the init.d or systemd service. Default value: $kafka::params::manage_service ##### `service_ensure` Data type: `Enum['running', 'stopped']` Set the ensure state of the service. Default value: $kafka::params::service_ensure ##### `service_restart` Data type: `Boolean` Whether the configuration files should trigger a service restart. Default value: $kafka::params::service_restart ##### `service_requires` Data type: `Array[String[1]]` Set the list of services required to be running before Kafka. Default value: $kafka::params::service_requires ##### `limit_nofile` Data type: `Optional[String[1]]` Set the 'LimitNOFILE' option of the systemd service. Default value: $kafka::params::limit_nofile ##### `limit_core` Data type: `Optional[String[1]]` Set the 'LimitCORE' option of the systemd service. Default value: $kafka::params::limit_core ##### `env` Data type: `Hash` A hash of the environment variables to set. Default value: {} ##### `config` Data type: `Hash[String[1], Any]` A hash of the producer configuration options. Default value: {} ##### `service_config` Data type: `Hash[String[1],String[1]]` A hash of the `kafka-console-producer.sh` script options. Default value: {} ##### `jmx_opts` Data type: `String[1]` Set the JMX options. Default value: $kafka::params::producer_jmx_opts ##### `log4j_opts` Data type: `String[1]` Set the Log4j options. Default value: $kafka::params::producer_log4j_opts ### kafka::producer::config This class handles the Kafka (producer) config. #### Parameters The following parameters are available in the `kafka::producer::config` class. ##### `manage_service` Data type: `Boolean` - - Default value: $kafka::producer::manage_service ##### `service_name` Data type: `String[1]` - - Default value: $kafka::producer::service_name ##### `service_restart` Data type: `Boolean` - - Default value: $kafka::producer::service_restart ##### `config` Data type: `Hash[String[1], Any]` - - Default value: $kafka::producer::config ##### `config_dir` Data type: `Stdlib::Absolutepath` - - Default value: $kafka::producer::config_dir ##### `user_name` Data type: `String[1]` - - Default value: $kafka::producer::user_name ##### `group_name` Data type: `String[1]` - - Default value: $kafka::producer::group_name ##### `config_mode` Data type: `Stdlib::Filemode` - - Default value: $kafka::producer::config_mode ## Defined types ### kafka::topic This defined type handles the creation of Kafka topics. #### Examples ##### Basic usage ```puppet kafka::topic { 'test': ensure => present, zookeeper => 'localhost:2181', replication_factor => 1, partitions => 1, } ``` #### Parameters The following parameters are available in the `kafka::topic` defined type. ##### `ensure` Data type: `String[1]` Should the topic be created. Default value: '' ##### `zookeeper` Data type: `String[1]` The connection string for the ZooKeeper connection in the form host:port. Multiple hosts can be given to allow fail-over. Default value: '' ##### `replication_factor` Data type: `Integer` The replication factor for each partition in the topic being created. If not supplied, defaults to the cluster default. Default value: 1 ##### `partitions` Data type: `Integer` The number of partitions for the topic being created or altered. If not supplied for create, defaults to the cluster default. Default value: 1 ##### `bin_dir` Data type: `String[1]` The directory where the file kafka-topics.sh is located. Default value: '/opt/kafka/bin' ##### `config` Data type: `Optional[Hash[String[1],String[1]]]` A topic configuration override for the topic being created or altered. See the Kafka documentation for full details on the topic configs. Default value: `undef` diff --git a/manifests/broker.pp b/manifests/broker.pp index e36f9a3..fa32158 100644 --- a/manifests/broker.pp +++ b/manifests/broker.pp @@ -1,158 +1,161 @@ # @summary # This class handles the Kafka (broker). # # @example Basic usage # class { 'kafka::broker': # config => { # 'broker.id' => '0', # 'zookeeper.connect' => 'localhost:2181' # } # } # # @param kafka_version # The version of Kafka that should be installed. # # @param scala_version # The scala version what Kafka was built with. # # @param install_dir # The directory to install Kafka to. # # @param mirror_url # The url where the Kafka is downloaded from. # # @param manage_java # Install java if it's not already installed. # # @param package_dir # The directory to install Kafka. # # @param package_name # Package name, when installing Kafka from a package. # # @param package_ensure # Package version or ensure state, when installing Kafka from a package. # # @param user_name # User to run Kafka as. # # @param user_shell # Login shell of the Kafka user. # # @param group_name # Group to run Kafka as. # # @param user_id # Create the Kafka user with this ID. # # @param group_id # Create the Kafka group with this ID. # # @param manage_user # Create the Kafka user if it's not already present. # # @param manage_group # Create the Kafka group if it's not already present. # # @param config_mode # The permissions for the config files. # # @param config_dir # The directory to create the Kafka config files to. # # @param log_dir # The directory for Kafka log files. # # @param bin_dir # The directory where the Kafka scripts are. # # @param service_name # Set the name of the service. # # @param manage_service # Install the init.d or systemd service. # # @param service_ensure # Set the ensure state of the service. # # @param service_restart # Whether the configuration files should trigger a service restart. # # @param service_requires # Set the list of services required to be running before Kafka. # # @param limit_nofile # Set the 'LimitNOFILE' option of the systemd service. # # @param limit_core # Set the 'LimitCORE' option of the systemd service. # # @param timeout_stop # Set the 'TimeoutStopSec' option of the systemd service. # # @param exec_stop # Set the 'ExecStop' option of the systemd service to 'kafka-server-stop.sh'. # # @param daemon_start # Use the '-daemon' option when starting Kafka with 'kafka-server-start.sh'. # # @param env # A hash of the environment variables to set. # # @param config # A hash of the broker configuration options. # # @param heap_opts # Set the Java heap size. # # @param jmx_opts # Set the JMX options. # # @param log4j_opts # Set the Log4j options. # # @param opts # Set the Kafka options. # class kafka::broker ( - String[1] $kafka_version = $kafka::params::kafka_version, - String[1] $scala_version = $kafka::params::scala_version, - Stdlib::Absolutepath $install_dir = $kafka::params::install_dir, - Stdlib::HTTPUrl $mirror_url = $kafka::params::mirror_url, - Boolean $manage_java = $kafka::params::manage_java, - Stdlib::Absolutepath $package_dir = $kafka::params::package_dir, - Optional[String[1]] $package_name = $kafka::params::package_name, - String[1] $package_ensure = $kafka::params::package_ensure, - String[1] $user_name = $kafka::params::user_name, - Stdlib::Absolutepath $user_shell = $kafka::params::user_shell, - String[1] $group_name = $kafka::params::group_name, - Optional[Integer] $user_id = $kafka::params::user_id, - Optional[Integer] $group_id = $kafka::params::group_id, - Boolean $manage_user = $kafka::params::manage_user, - Boolean $manage_group = $kafka::params::manage_group, - Stdlib::Filemode $config_mode = $kafka::params::config_mode, - Stdlib::Absolutepath $config_dir = $kafka::params::config_dir, - Stdlib::Absolutepath $log_dir = $kafka::params::log_dir, - Stdlib::Absolutepath $bin_dir = $kafka::params::bin_dir, - String[1] $service_name = 'kafka', - Boolean $manage_service = $kafka::params::manage_service, - Enum['running', 'stopped'] $service_ensure = $kafka::params::service_ensure, - Boolean $service_restart = $kafka::params::service_restart, - Array[String[1]] $service_requires = $kafka::params::service_requires, - Optional[String[1]] $limit_nofile = $kafka::params::limit_nofile, - Optional[String[1]] $limit_core = $kafka::params::limit_core, - Optional[String[1]] $timeout_stop = $kafka::params::timeout_stop, - Boolean $exec_stop = $kafka::params::exec_stop, - Boolean $daemon_start = $kafka::params::daemon_start, - Hash $env = {}, - Hash[String[1], Any] $config = {}, - String[1] $heap_opts = $kafka::params::broker_heap_opts, - String[1] $jmx_opts = $kafka::params::broker_jmx_opts, - String[1] $log4j_opts = $kafka::params::broker_log4j_opts, - String[0] $opts = $kafka::params::broker_opts, + String[1] $kafka_version = $kafka::params::kafka_version, + String[1] $scala_version = $kafka::params::scala_version, + Stdlib::Absolutepath $install_dir = $kafka::params::install_dir, + Stdlib::HTTPUrl $mirror_url = $kafka::params::mirror_url, + Boolean $manage_java = $kafka::params::manage_java, + Stdlib::Absolutepath $package_dir = $kafka::params::package_dir, + Optional[String[1]] $package_name = $kafka::params::package_name, + String[1] $package_ensure = $kafka::params::package_ensure, + String[1] $user_name = $kafka::params::user_name, + Stdlib::Absolutepath $user_shell = $kafka::params::user_shell, + String[1] $group_name = $kafka::params::group_name, + Optional[Integer] $user_id = $kafka::params::user_id, + Optional[Integer] $group_id = $kafka::params::group_id, + Boolean $manage_user = $kafka::params::manage_user, + Boolean $manage_group = $kafka::params::manage_group, + Stdlib::Filemode $config_mode = $kafka::params::config_mode, + Stdlib::Absolutepath $config_dir = $kafka::params::config_dir, + Stdlib::Absolutepath $log_dir = $kafka::params::log_dir, + Stdlib::Absolutepath $bin_dir = $kafka::params::bin_dir, + String[1] $service_name = 'kafka', + Boolean $manage_service = $kafka::params::manage_service, + Enum['running', 'stopped'] $service_ensure = $kafka::params::service_ensure, + Boolean $service_restart = $kafka::params::service_restart, + Array[String[1]] $service_requires = $kafka::params::service_requires, + Optional[String[1]] $limit_nofile = $kafka::params::limit_nofile, + Optional[String[1]] $limit_core = $kafka::params::limit_core, + Optional[String[1]] $timeout_stop = $kafka::params::timeout_stop, + Boolean $exec_stop = $kafka::params::exec_stop, + Boolean $daemon_start = $kafka::params::daemon_start, + Hash $env = {}, + Hash[String[1], Any] $config = {}, + String[1] $heap_opts = $kafka::params::broker_heap_opts, + String[1] $jmx_opts = $kafka::params::broker_jmx_opts, + String[1] $log4j_opts = $kafka::params::broker_log4j_opts, + String[0] $opts = $kafka::params::broker_opts, + Boolean $manage_log4j = $kafka::params::manage_log4j, + Pattern[/[1-9][0-9]*[KMG]B/] $log_file_size = $kafka::params::log_file_size, + Integer[1, 50] $log_file_count = $kafka::params::log_file_count ) inherits kafka::params { class { 'kafka::broker::install': } -> class { 'kafka::broker::config': } -> class { 'kafka::broker::service': } -> Class['kafka::broker'] } diff --git a/manifests/broker/config.pp b/manifests/broker/config.pp index 0407ecc..1e207aa 100644 --- a/manifests/broker/config.pp +++ b/manifests/broker/config.pp @@ -1,34 +1,49 @@ # @summary # This class handles the Kafka (broker) config. # # @api private # class kafka::broker::config ( - Boolean $manage_service = $kafka::broker::manage_service, - String[1] $service_name = $kafka::broker::service_name, - Boolean $service_restart = $kafka::broker::service_restart, - Hash[String[1], Any] $config = $kafka::broker::config, - Stdlib::Absolutepath $config_dir = $kafka::broker::config_dir, - String[1] $user_name = $kafka::broker::user_name, - String[1] $group_name = $kafka::broker::group_name, - Stdlib::Filemode $config_mode = $kafka::broker::config_mode, + Boolean $manage_service = $kafka::broker::manage_service, + String[1] $service_name = $kafka::broker::service_name, + Boolean $service_restart = $kafka::broker::service_restart, + Hash[String[1], Any] $config = $kafka::broker::config, + Stdlib::Absolutepath $config_dir = $kafka::broker::config_dir, + String[1] $user_name = $kafka::broker::user_name, + String[1] $group_name = $kafka::broker::group_name, + Stdlib::Filemode $config_mode = $kafka::broker::config_mode, + Boolean $manage_log4j = $kafka::broker::manage_log4j, + Pattern[/[1-9][0-9]*[KMG]B/] $log_file_size = $kafka::broker::log_file_size, + Integer[1, 50] $log_file_count = $kafka::broker::log_file_count, ) { assert_private() if ($manage_service and $service_restart) { $config_notify = Service[$service_name] } else { $config_notify = undef } $doctag = 'brokerconfigs' file { "${config_dir}/server.properties": ensure => file, owner => $user_name, group => $group_name, mode => $config_mode, content => template('kafka/properties.erb'), notify => $config_notify, require => File[$config_dir], } + + if $manage_log4j { + file { "${config_dir}/log4j.properties": + ensure => file, + owner => $user_name, + group => $group_name, + mode => $config_mode, + content => epp('kafka/log4j.properties.epp', { 'log_file_size' => $log_file_size, 'log_file_count' => $log_file_count }), + notify => $config_notify, + require => File[$config_dir], + } + } } diff --git a/manifests/consumer.pp b/manifests/consumer.pp index 06439f9..d94bffb 100644 --- a/manifests/consumer.pp +++ b/manifests/consumer.pp @@ -1,142 +1,145 @@ # @summary # This class handles the Kafka (consumer). # # @example Basic usage # class { 'kafka::consumer': # config => { # 'client.id' => '0', # 'zookeeper.connect' => 'localhost:2181' # } # } # # @param kafka_version # The version of Kafka that should be installed. # # @param scala_version # The scala version what Kafka was built with. # # @param install_dir # The directory to install Kafka to. # # @param mirror_url # The url where the Kafka is downloaded from. # # @param manage_java # Install java if it's not already installed. # # @param package_dir # The directory to install Kafka. # # @param package_name # Package name, when installing Kafka from a package. # # @param package_ensure # Package version or ensure state, when installing Kafka from a package. # # @param user_name # User to run Kafka as. # # @param user_shell # Login shell of the Kafka user. # # @param group_name # Group to run Kafka as. # # @param user_id # Create the Kafka user with this ID. # # @param group_id # Create the Kafka group with this ID. # # @param manage_user # Create the Kafka user if it's not already present. # # @param manage_group # Create the Kafka group if it's not already present. # # @param config_mode # The permissions for the config files. # # @param config_dir # The directory to create the Kafka config files to. # # @param log_dir # The directory for Kafka log files. # # @param bin_dir # The directory where the Kafka scripts are. # # @param service_name # Set the name of the service. # # @param manage_service # Install the init.d or systemd service. # # @param service_ensure # Set the ensure state of the service. # # @param service_restart # Whether the configuration files should trigger a service restart. # # @param service_requires # Set the list of services required to be running before Kafka. # # @param limit_nofile # Set the 'LimitNOFILE' option of the systemd service. # # @param limit_core # Set the 'LimitCORE' option of the systemd service. # # @param env # A hash of the environment variables to set. # # @param config # A hash of the consumer configuration options. # # @param service_config # A hash of the `kafka-console-consumer.sh` script options. # # @param jmx_opts # Set the JMX options. # # @param log4j_opts # Set the Log4j options. # class kafka::consumer ( - String[1] $kafka_version = $kafka::params::kafka_version, - String[1] $scala_version = $kafka::params::scala_version, - Stdlib::Absolutepath $install_dir = $kafka::params::install_dir, - Stdlib::HTTPUrl $mirror_url = $kafka::params::mirror_url, - Boolean $manage_java = $kafka::params::manage_java, - Stdlib::Absolutepath $package_dir = $kafka::params::package_dir, - Optional[String[1]] $package_name = $kafka::params::package_name, - String[1] $package_ensure = $kafka::params::package_ensure, - String[1] $user_name = $kafka::params::user_name, - Stdlib::Absolutepath $user_shell = $kafka::params::user_shell, - String[1] $group_name = $kafka::params::group_name, - Optional[Integer] $user_id = $kafka::params::user_id, - Optional[Integer] $group_id = $kafka::params::group_id, - Boolean $manage_user = $kafka::params::manage_user, - Boolean $manage_group = $kafka::params::manage_group, - Stdlib::Filemode $config_mode = $kafka::params::config_mode, - Stdlib::Absolutepath $config_dir = $kafka::params::config_dir, - Stdlib::Absolutepath $log_dir = $kafka::params::log_dir, - Stdlib::Absolutepath $bin_dir = $kafka::params::bin_dir, - String[1] $service_name = 'kafka-consumer', - Boolean $manage_service = $kafka::params::manage_service, - Enum['running', 'stopped'] $service_ensure = $kafka::params::service_ensure, - Boolean $service_restart = $kafka::params::service_restart, - Array[String[1]] $service_requires = $kafka::params::service_requires, - Optional[String[1]] $limit_nofile = $kafka::params::limit_nofile, - Optional[String[1]] $limit_core = $kafka::params::limit_core, - Hash $env = {}, - Hash[String[1], Any] $config = {}, - Hash[String[1],String[1]] $service_config = {}, - String[1] $jmx_opts = $kafka::params::consumer_jmx_opts, - String[1] $log4j_opts = $kafka::params::consumer_log4j_opts, + String[1] $kafka_version = $kafka::params::kafka_version, + String[1] $scala_version = $kafka::params::scala_version, + Stdlib::Absolutepath $install_dir = $kafka::params::install_dir, + Stdlib::HTTPUrl $mirror_url = $kafka::params::mirror_url, + Boolean $manage_java = $kafka::params::manage_java, + Stdlib::Absolutepath $package_dir = $kafka::params::package_dir, + Optional[String[1]] $package_name = $kafka::params::package_name, + String[1] $package_ensure = $kafka::params::package_ensure, + String[1] $user_name = $kafka::params::user_name, + Stdlib::Absolutepath $user_shell = $kafka::params::user_shell, + String[1] $group_name = $kafka::params::group_name, + Optional[Integer] $user_id = $kafka::params::user_id, + Optional[Integer] $group_id = $kafka::params::group_id, + Boolean $manage_user = $kafka::params::manage_user, + Boolean $manage_group = $kafka::params::manage_group, + Stdlib::Filemode $config_mode = $kafka::params::config_mode, + Stdlib::Absolutepath $config_dir = $kafka::params::config_dir, + Stdlib::Absolutepath $log_dir = $kafka::params::log_dir, + Stdlib::Absolutepath $bin_dir = $kafka::params::bin_dir, + String[1] $service_name = 'kafka-consumer', + Boolean $manage_service = $kafka::params::manage_service, + Enum['running', 'stopped'] $service_ensure = $kafka::params::service_ensure, + Boolean $service_restart = $kafka::params::service_restart, + Array[String[1]] $service_requires = $kafka::params::service_requires, + Optional[String[1]] $limit_nofile = $kafka::params::limit_nofile, + Optional[String[1]] $limit_core = $kafka::params::limit_core, + Hash $env = {}, + Hash[String[1], Any] $config = {}, + Hash[String[1],String[1]] $service_config = {}, + String[1] $jmx_opts = $kafka::params::consumer_jmx_opts, + String[1] $log4j_opts = $kafka::params::consumer_log4j_opts, + Boolean $manage_log4j = $kafka::params::manage_log4j, + Pattern[/[1-9][0-9]*[KMG]B/] $log_file_size = $kafka::params::log_file_size, + Integer[1, 50] $log_file_count = $kafka::params::log_file_count ) inherits kafka::params { class { 'kafka::consumer::install': } -> class { 'kafka::consumer::config': } -> class { 'kafka::consumer::service': } -> Class['kafka::consumer'] } diff --git a/manifests/consumer/config.pp b/manifests/consumer/config.pp index b44f096..ec11ac1 100644 --- a/manifests/consumer/config.pp +++ b/manifests/consumer/config.pp @@ -1,30 +1,45 @@ # @summary # This class handles the Kafka (consumer) config. # class kafka::consumer::config ( - Boolean $manage_service = $kafka::consumer::manage_service, - String[1] $service_name = $kafka::consumer::service_name, - Boolean $service_restart = $kafka::consumer::service_restart, - Hash[String[1], Any] $config = $kafka::consumer::config, - Stdlib::Absolutepath $config_dir = $kafka::consumer::config_dir, - String[1] $user_name = $kafka::consumer::user_name, - String[1] $group_name = $kafka::consumer::group_name, - Stdlib::Filemode $config_mode = $kafka::consumer::config_mode, + Boolean $manage_service = $kafka::consumer::manage_service, + String[1] $service_name = $kafka::consumer::service_name, + Boolean $service_restart = $kafka::consumer::service_restart, + Hash[String[1], Any] $config = $kafka::consumer::config, + Stdlib::Absolutepath $config_dir = $kafka::consumer::config_dir, + String[1] $user_name = $kafka::consumer::user_name, + String[1] $group_name = $kafka::consumer::group_name, + Stdlib::Filemode $config_mode = $kafka::consumer::config_mode, + Boolean $manage_log4j = $kafka::consumer::manage_log4j, + Pattern[/[1-9][0-9]*[KMG]B/] $log_file_size = $kafka::consumer::log_file_size, + Integer[1, 50] $log_file_count = $kafka::consumer::log_file_count, ) { if ($manage_service and $service_restart) { $config_notify = Service[$service_name] } else { $config_notify = undef } $doctag = 'consumerconfigs' file { "${config_dir}/consumer.properties": ensure => file, owner => $user_name, group => $group_name, mode => $config_mode, content => template('kafka/properties.erb'), notify => $config_notify, require => File[$config_dir], } + + if $manage_log4j { + file { "${config_dir}/log4j.properties": + ensure => file, + owner => $user_name, + group => $group_name, + mode => $config_mode, + content => epp('kafka/log4j.properties.epp', { 'log_file_size' => $log_file_size, 'log_file_count' => $log_file_count }), + notify => $config_notify, + require => File[$config_dir], + } + } } diff --git a/manifests/mirror.pp b/manifests/mirror.pp index 0ceddd1..f0493b3 100644 --- a/manifests/mirror.pp +++ b/manifests/mirror.pp @@ -1,156 +1,159 @@ # @summary # This class handles the Kafka (mirror). # # @example Basic usage # class { 'kafka::mirror': # consumer_config => { # 'group.id' => 'kafka-mirror', # 'zookeeper.connect' => 'localhost:2181' # }, # producer_config => { # 'zookeeper.connect' => 'localhost:2181', # }, # service_config => { # 'whitelist' => '.*', # } # } # # @param kafka_version # The version of Kafka that should be installed. # # @param scala_version # The scala version what Kafka was built with. # # @param install_dir # The directory to install Kafka to. # # @param mirror_url # The url where the Kafka is downloaded from. # # @param manage_java # Install java if it's not already installed. # # @param package_dir # The directory to install Kafka. # # @param package_name # Package name, when installing Kafka from a package. # # @param package_ensure # Package version or ensure state, when installing Kafka from a package. # # @param user_name # User to run Kafka as. # # @param user_shell # Login shell of the Kafka user. # # @param group_name # Group to run Kafka as. # # @param user_id # Create the Kafka user with this ID. # # @param group_id # Create the Kafka group with this ID. # # @param manage_user # Create the Kafka user if it's not already present. # # @param manage_group # Create the Kafka group if it's not already present. # # @param config_mode # The permissions for the config files. # # @param config_dir # The directory to create the Kafka config files to. # # @param log_dir # The directory for Kafka log files. # # @param bin_dir # The directory where the Kafka scripts are. # # @param service_name # Set the name of the service. # # @param manage_service # Install the init.d or systemd service. # # @param service_ensure # Set the ensure state of the service. # # @param service_restart # Whether the configuration files should trigger a service restart. # # @param service_requires # Set the list of services required to be running before Kafka. # # @param limit_nofile # Set the 'LimitNOFILE' option of the systemd service. # # @param limit_core # Set the 'LimitCORE' option of the systemd service. # # @param env # A hash of the environment variables to set. # # @param consumer_config # A hash of the consumer configuration options. # # @param producer_config # A hash of the producer configuration options. # # @param service_config # A hash of the mirror script options. # # @param heap_opts # Set the Java heap size. # # @param jmx_opts # Set the JMX options. # # @param log4j_opts # Set the Log4j options. # class kafka::mirror ( - String[1] $kafka_version = $kafka::params::kafka_version, - String[1] $scala_version = $kafka::params::scala_version, - Stdlib::Absolutepath $install_dir = $kafka::params::install_dir, - Stdlib::HTTPUrl $mirror_url = $kafka::params::mirror_url, - Boolean $manage_java = $kafka::params::manage_java, - Stdlib::Absolutepath $package_dir = $kafka::params::package_dir, - Optional[String[1]] $package_name = $kafka::params::package_name, - String[1] $package_ensure = $kafka::params::package_ensure, - String[1] $user_name = $kafka::params::user_name, - Stdlib::Absolutepath $user_shell = $kafka::params::user_shell, - String[1] $group_name = $kafka::params::group_name, - Optional[Integer] $user_id = $kafka::params::user_id, - Optional[Integer] $group_id = $kafka::params::group_id, - Boolean $manage_user = $kafka::params::manage_user, - Boolean $manage_group = $kafka::params::manage_group, - Stdlib::Filemode $config_mode = $kafka::params::config_mode, - Stdlib::Absolutepath $config_dir = $kafka::params::config_dir, - Stdlib::Absolutepath $log_dir = $kafka::params::log_dir, - Stdlib::Absolutepath $bin_dir = $kafka::params::bin_dir, - String[1] $service_name = 'kafka-mirror', - Boolean $manage_service = $kafka::params::manage_service, - Enum['running', 'stopped'] $service_ensure = $kafka::params::service_ensure, - Boolean $service_restart = $kafka::params::service_restart, - Array[String[1]] $service_requires = $kafka::params::service_requires, - Optional[String[1]] $limit_nofile = $kafka::params::limit_nofile, - Optional[String[1]] $limit_core = $kafka::params::limit_core, - Hash $env = {}, - Hash[String[1],String[1]] $consumer_config = {}, - Hash[String[1],String[1]] $producer_config = {}, - Hash[String[1],String[1]] $service_config = {}, - String[1] $heap_opts = $kafka::params::mirror_heap_opts, - String[1] $jmx_opts = $kafka::params::mirror_jmx_opts, - String[1] $log4j_opts = $kafka::params::mirror_log4j_opts, + String[1] $kafka_version = $kafka::params::kafka_version, + String[1] $scala_version = $kafka::params::scala_version, + Stdlib::Absolutepath $install_dir = $kafka::params::install_dir, + Stdlib::HTTPUrl $mirror_url = $kafka::params::mirror_url, + Boolean $manage_java = $kafka::params::manage_java, + Stdlib::Absolutepath $package_dir = $kafka::params::package_dir, + Optional[String[1]] $package_name = $kafka::params::package_name, + String[1] $package_ensure = $kafka::params::package_ensure, + String[1] $user_name = $kafka::params::user_name, + Stdlib::Absolutepath $user_shell = $kafka::params::user_shell, + String[1] $group_name = $kafka::params::group_name, + Optional[Integer] $user_id = $kafka::params::user_id, + Optional[Integer] $group_id = $kafka::params::group_id, + Boolean $manage_user = $kafka::params::manage_user, + Boolean $manage_group = $kafka::params::manage_group, + Stdlib::Filemode $config_mode = $kafka::params::config_mode, + Stdlib::Absolutepath $config_dir = $kafka::params::config_dir, + Stdlib::Absolutepath $log_dir = $kafka::params::log_dir, + Stdlib::Absolutepath $bin_dir = $kafka::params::bin_dir, + String[1] $service_name = 'kafka-mirror', + Boolean $manage_service = $kafka::params::manage_service, + Enum['running', 'stopped'] $service_ensure = $kafka::params::service_ensure, + Boolean $service_restart = $kafka::params::service_restart, + Array[String[1]] $service_requires = $kafka::params::service_requires, + Optional[String[1]] $limit_nofile = $kafka::params::limit_nofile, + Optional[String[1]] $limit_core = $kafka::params::limit_core, + Hash $env = {}, + Hash[String[1],String[1]] $consumer_config = {}, + Hash[String[1],String[1]] $producer_config = {}, + Hash[String[1],String[1]] $service_config = {}, + String[1] $heap_opts = $kafka::params::mirror_heap_opts, + String[1] $jmx_opts = $kafka::params::mirror_jmx_opts, + String[1] $log4j_opts = $kafka::params::mirror_log4j_opts, + Boolean $manage_log4j = $kafka::params::manage_log4j, + Pattern[/[1-9][0-9]*[KMG]B/] $log_file_size = $kafka::params::log_file_size, + Integer[1, 50] $log_file_count = $kafka::params::log_file_count ) inherits kafka::params { class { 'kafka::mirror::install': } -> class { 'kafka::mirror::config': } -> class { 'kafka::mirror::service': } -> Class['kafka::mirror'] } diff --git a/manifests/mirror/config.pp b/manifests/mirror/config.pp index 9bd1bd2..339c6d2 100644 --- a/manifests/mirror/config.pp +++ b/manifests/mirror/config.pp @@ -1,50 +1,56 @@ # @summary # This class handles the Kafka (mirror) config. # # @api private # class kafka::mirror::config ( - Boolean $manage_service = $kafka::mirror::manage_service, - String[1] $service_name = $kafka::mirror::service_name, - Boolean $service_restart = $kafka::mirror::service_restart, - Hash[String[1],String[1]] $consumer_config = $kafka::mirror::consumer_config, - Hash[String[1],String[1]] $producer_config = $kafka::mirror::producer_config, - Stdlib::Absolutepath $config_dir = $kafka::mirror::config_dir, - String[1] $user_name = $kafka::mirror::user_name, - String[1] $group_name = $kafka::mirror::group_name, - Stdlib::Filemode $config_mode = $kafka::mirror::config_mode, + Boolean $manage_service = $kafka::mirror::manage_service, + String[1] $service_name = $kafka::mirror::service_name, + Boolean $service_restart = $kafka::mirror::service_restart, + Hash[String[1],String[1]] $consumer_config = $kafka::mirror::consumer_config, + Hash[String[1],String[1]] $producer_config = $kafka::mirror::producer_config, + Stdlib::Absolutepath $config_dir = $kafka::mirror::config_dir, + String[1] $user_name = $kafka::mirror::user_name, + String[1] $group_name = $kafka::mirror::group_name, + Stdlib::Filemode $config_mode = $kafka::mirror::config_mode, + Boolean $manage_log4j = $kafka::mirror::manage_log4j, + Pattern[/[1-9][0-9]*[KMG]B/] $log_file_size = $kafka::mirror::log_file_size, + Integer[1, 50] $log_file_count = $kafka::mirror::log_file_count, ) { assert_private() if $consumer_config['group.id'] == '' { fail('[Consumer] You need to specify a value for group.id') } if $consumer_config['zookeeper.connect'] == '' { fail('[Consumer] You need to specify a value for zookeeper.connect') } if $producer_config['bootstrap.servers'] == '' { fail('[Producer] You need to specify a value for bootstrap.servers') } class { 'kafka::consumer::config': manage_service => $manage_service, service_name => $service_name, service_restart => $service_restart, config => $consumer_config, config_dir => $config_dir, user_name => $user_name, group_name => $group_name, config_mode => $config_mode, + manage_log4j => $manage_log4j, + log_file_size => $log_file_size, + log_file_count => $log_file_count, } class { 'kafka::producer::config': manage_service => $manage_service, service_name => $service_name, service_restart => $service_restart, config => $producer_config, config_dir => $config_dir, user_name => $user_name, group_name => $group_name, config_mode => $config_mode, } } diff --git a/manifests/params.pp b/manifests/params.pp index d322a8c..e4c4746 100644 --- a/manifests/params.pp +++ b/manifests/params.pp @@ -1,61 +1,64 @@ # @summary # This class provides default parameters. # class kafka::params { unless $facts['os']['family'] =~ /(RedHat|Debian|Suse)/ { warning("${facts['os']['family']} is not supported") } $kafka_version = '2.4.1' $scala_version = '2.12' $install_dir = "/opt/kafka-${scala_version}-${kafka_version}" $config_dir = '/opt/kafka/config' $bin_dir = '/opt/kafka/bin' $log_dir = '/var/log/kafka' $mirror_url = 'https://archive.apache.org/dist/' $mirror_subpath = "kafka/${kafka_version}" $manage_java = false $package_dir = '/var/tmp/kafka' $package_name = undef $proxy_server = undef $proxy_host = undef $proxy_port = undef $proxy_type = undef $package_ensure = 'present' $user_name = 'kafka' $user_shell = '/sbin/nologin' $group_name = 'kafka' $user_id = undef $group_id = undef $system_user = false $system_group = false $manage_user = true $manage_group = true $config_mode = '0644' $install_mode = '0755' + $manage_log4j = true + $log_file_size = '50MB' + $log_file_count = 7 $manage_service = true $service_ensure = 'running' $service_restart = true $service_requires = ['network.target', 'syslog.target'] $limit_nofile = undef $limit_core = undef $timeout_stop = undef $exec_stop = false $daemon_start = false $broker_heap_opts = '-Xmx1G -Xms1G' $broker_jmx_opts = '-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9990' $broker_log4j_opts = "-Dlog4j.configuration=file:${config_dir}/log4j.properties" $broker_opts = '' $mirror_heap_opts = '-Xmx256M' $mirror_jmx_opts = '-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9991' $mirror_log4j_opts = $broker_log4j_opts $producer_jmx_opts = '-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9992' $producer_log4j_opts = $broker_log4j_opts $consumer_jmx_opts = '-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9993' $consumer_log4j_opts = $broker_log4j_opts } diff --git a/manifests/producer/config.pp b/manifests/producer/config.pp index 7869830..a4113c8 100644 --- a/manifests/producer/config.pp +++ b/manifests/producer/config.pp @@ -1,30 +1,30 @@ # @summary # This class handles the Kafka (producer) config. # class kafka::producer::config ( - Boolean $manage_service = $kafka::producer::manage_service, - String[1] $service_name = $kafka::producer::service_name, - Boolean $service_restart = $kafka::producer::service_restart, - Hash[String[1], Any] $config = $kafka::producer::config, - Stdlib::Absolutepath $config_dir = $kafka::producer::config_dir, - String[1] $user_name = $kafka::producer::user_name, - String[1] $group_name = $kafka::producer::group_name, - Stdlib::Filemode $config_mode = $kafka::producer::config_mode, + Boolean $manage_service = $kafka::producer::manage_service, + String[1] $service_name = $kafka::producer::service_name, + Boolean $service_restart = $kafka::producer::service_restart, + Hash[String[1], Any] $config = $kafka::producer::config, + Stdlib::Absolutepath $config_dir = $kafka::producer::config_dir, + String[1] $user_name = $kafka::producer::user_name, + String[1] $group_name = $kafka::producer::group_name, + Stdlib::Filemode $config_mode = $kafka::producer::config_mode, ) { if ($manage_service and $service_restart) { $config_notify = Service[$service_name] } else { $config_notify = undef } $doctag = 'producerconfigs' file { "${config_dir}/producer.properties": ensure => file, owner => $user_name, group => $group_name, mode => $config_mode, content => template('kafka/properties.erb'), notify => $config_notify, require => File[$config_dir], } } diff --git a/spec/classes/broker_spec.rb b/spec/classes/broker_spec.rb index a1eed6b..2758b33 100644 --- a/spec/classes/broker_spec.rb +++ b/spec/classes/broker_spec.rb @@ -1,90 +1,100 @@ require 'spec_helper' require 'shared_examples_param_validation' describe 'kafka::broker', type: :class do on_supported_os.each do |os, os_facts| context "on #{os}" do let(:facts) do os_facts end let :params do { config: { 'zookeeper.connect' => 'localhost:2181' } } end it { is_expected.to contain_class('kafka::broker::install').that_comes_before('Class[kafka::broker::config]') } it { is_expected.to contain_class('kafka::broker::config').that_comes_before('Class[kafka::broker::service]') } it { is_expected.to contain_class('kafka::broker::service').that_comes_before('Class[kafka::broker]') } it { is_expected.to contain_class('kafka::broker') } + context 'with manage_log4j => true' do + let(:params) { {'manage_log4j' => true} } + it { is_expected.to contain_class('kafka::broker::config').with('log_file_size' => '50MB', 'log_file_count' => 7) } + end + describe 'kafka::broker::install' do context 'defaults' do it { is_expected.to contain_class('kafka') } end end describe 'kafka::broker::config' do context 'defaults' do it { is_expected.to contain_file('/opt/kafka/config/server.properties') } end + context 'with manage_log4j => true' do + let(:params) { {'manage_log4j' => true} } + it { is_expected.to contain_file('/opt/kafka/config/log4j.properties').with_content(%r{^log4j.appender.kafkaAppender.MaxFileSize=50MB$}) } + it { is_expected.to contain_file('/opt/kafka/config/log4j.properties').with_content(%r{^log4j.appender.kafkaAppender.MaxBackupIndex=7$}) } + end end describe 'kafka::broker::service' do context 'manage_service false' do let(:params) { super().merge(manage_service: false) } it { is_expected.not_to contain_file('/etc/init.d/kafka') } it { is_expected.not_to contain_file('/etc/systemd/system/kafka.service') } it { is_expected.not_to contain_service('kafka') } end context 'defaults' do if os_facts[:service_provider] == 'systemd' it { is_expected.to contain_file('/etc/init.d/kafka').with_ensure('absent') } it { is_expected.to contain_file('/etc/systemd/system/kafka.service').with_content %r{^After=network\.target syslog\.target$} } it { is_expected.to contain_file('/etc/systemd/system/kafka.service').with_content %r{^Wants=network\.target syslog\.target$} } it { is_expected.not_to contain_file('/etc/systemd/system/kafka.service').with_content %r{^LimitNOFILE=} } it { is_expected.not_to contain_file('/etc/systemd/system/kafka.service').with_content %r{^LimitCORE=} } else it { is_expected.to contain_file('/etc/init.d/kafka') } end it { is_expected.to contain_service('kafka') } end context 'limit_nofile set' do let(:params) { super().merge(limit_nofile: '65536') } if os_facts[:service_provider] == 'systemd' it { is_expected.to contain_file('/etc/systemd/system/kafka.service').with_content %r{^LimitNOFILE=65536$} } else it { is_expected.to contain_file('/etc/init.d/kafka').with_content %r{ulimit -n 65536$} } end end context 'limit_core set' do let(:params) { super().merge(limit_core: 'infinity') } if os_facts[:service_provider] == 'systemd' it { is_expected.to contain_file('/etc/systemd/system/kafka.service').with_content %r{^LimitCORE=infinity$} } else it { is_expected.to contain_file('/etc/init.d/kafka').with_content %r{ulimit -c infinity$} } end end context 'service_requires set', if: os_facts[:service_provider] == 'systemd' do let(:params) { super().merge(service_requires: ['dummy.target']) } it { is_expected.to contain_file('/etc/systemd/system/kafka.service').with_content %r{^After=dummy\.target$} } it { is_expected.to contain_file('/etc/systemd/system/kafka.service').with_content %r{^Wants=dummy\.target$} } end end it_validates_parameter 'mirror_url' end end end diff --git a/spec/classes/consumer_spec.rb b/spec/classes/consumer_spec.rb index 81e6882..a023950 100644 --- a/spec/classes/consumer_spec.rb +++ b/spec/classes/consumer_spec.rb @@ -1,52 +1,62 @@ require 'spec_helper' require 'shared_examples_param_validation' describe 'kafka::consumer', type: :class do on_supported_os.each do |os, os_facts| context "on #{os}" do let(:facts) do os_facts end let :params do { service_config: { 'topic' => 'demo', 'zookeeper' => 'localhost:2181' } } end it { is_expected.to contain_class('kafka::consumer::install').that_comes_before('Class[kafka::consumer::config]') } it { is_expected.to contain_class('kafka::consumer::config').that_comes_before('Class[kafka::consumer::service]') } it { is_expected.to contain_class('kafka::consumer::service').that_comes_before('Class[kafka::consumer]') } it { is_expected.to contain_class('kafka::consumer') } + context 'with manage_log4j => true' do + let(:params) { {'manage_log4j' => true} } + it { is_expected.to contain_class('kafka::consumer::config').with('log_file_size' => '50MB', 'log_file_count' => 7) } + end + describe 'kafka::consumer::install' do context 'defaults' do it { is_expected.to contain_class('kafka') } end end describe 'kafka::consumer::config' do context 'defaults' do it { is_expected.to contain_file('/opt/kafka/config/consumer.properties') } end + context 'with manage_log4j => true' do + let(:params) { {'manage_log4j' => true} } + it { is_expected.to contain_file('/opt/kafka/config/log4j.properties').with_content(%r{^log4j.appender.kafkaAppender.MaxFileSize=50MB$}) } + it { is_expected.to contain_file('/opt/kafka/config/log4j.properties').with_content(%r{^log4j.appender.kafkaAppender.MaxBackupIndex=7$}) } + end end describe 'kafka::consumer::service' do context 'defaults' do if os_facts[:service_provider] == 'systemd' it { is_expected.to contain_file('/etc/init.d/kafka-consumer').with_abent('absent') } else it { is_expected.to contain_file('/etc/init.d/kafka-consumer') } end it { is_expected.to contain_service('kafka-consumer') } end end it_validates_parameter 'mirror_url' end end end diff --git a/spec/classes/mirror_spec.rb b/spec/classes/mirror_spec.rb index d27aceb..b1ce157 100644 --- a/spec/classes/mirror_spec.rb +++ b/spec/classes/mirror_spec.rb @@ -1,58 +1,63 @@ require 'spec_helper' require 'shared_examples_param_validation' describe 'kafka::mirror', type: :class do on_supported_os.each do |os, os_facts| context "on #{os}" do let(:facts) do os_facts end let :params do { consumer_config: { 'group.id' => 'kafka-mirror', 'zookeeper.connect' => 'localhost:2181' }, producer_config: { 'bootstrap.servers' => 'localhost:9092' } } end it { is_expected.to contain_class('kafka::mirror::install').that_comes_before('Class[kafka::mirror::config]') } it { is_expected.to contain_class('kafka::mirror::config').that_comes_before('Class[kafka::mirror::service]') } it { is_expected.to contain_class('kafka::mirror::service').that_comes_before('Class[kafka::mirror]') } it { is_expected.to contain_class('kafka::mirror') } + context 'with manage_log4j => true' do + let(:params) { {'manage_log4j' => true} } + it { is_expected.to contain_class('kafka::mirror::config').with('log_file_size' => '50MB', 'log_file_count' => 7) } + end + describe 'kafka::mirror::install' do context 'defaults' do it { is_expected.to contain_class('kafka') } end end describe 'kafka::mirror::config' do context 'defaults' do it { is_expected.to contain_class('kafka::consumer::config') } it { is_expected.to contain_class('kafka::producer::config') } end end describe 'kafka::mirror::service' do context 'defaults' do if os_facts[:service_provider] == 'systemd' it { is_expected.to contain_file('/etc/init.d/kafka-mirror').with_ensure('absent') } it { is_expected.to contain_file('/etc/systemd/system/kafka-mirror.service').with_content %r{/opt/kafka/config/(?=.*consumer)|(?=.*producer).propertie} } else it { is_expected.to contain_file('/etc/init.d/kafka-mirror') } it { is_expected.to contain_file('/etc/init.d/kafka-mirror').with_content %r{/opt/kafka/config/(?=.*consumer)|(?=.*producer).properties} } end it { is_expected.to contain_service('kafka-mirror') } end end it_validates_parameter 'mirror_url' end end end diff --git a/templates/log4j.properties.epp b/templates/log4j.properties.epp new file mode 100644 index 0000000..9e0bc2f --- /dev/null +++ b/templates/log4j.properties.epp @@ -0,0 +1,102 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Unspecified loggers and loggers with additivity=true output to server.log and stdout +# Note that INFO only applies to unspecified loggers, the log level of the child logger is used otherwise + +################################################################################# +################## THIS FILE IS MANAGED BY PUPPET ############################### +################################################################################# + +log4j.rootLogger=INFO, stdout, kafkaAppender + +log4j.appender.stdout=org.apache.log4j.ConsoleAppender +log4j.appender.stdout.layout=org.apache.log4j.PatternLayout +log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n + +log4j.appender.kafkaAppender=org.apache.log4j.RollingFileAppender +log4j.appender.kafkaAppender.File=${kafka.logs.dir}/server.log +log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout +log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n +log4j.appender.kafkaAppender.MaxFileSize=<%= $log_file_size %> +log4j.appender.kafkaAppender.MaxBackupIndex=<%= $log_file_count %> + +log4j.appender.stateChangeAppender=org.apache.log4j.RollingFileAppender +log4j.appender.stateChangeAppender.File=${kafka.logs.dir}/state-change.log +log4j.appender.stateChangeAppender.layout=org.apache.log4j.PatternLayout +log4j.appender.stateChangeAppender.layout.ConversionPattern=[%d] %p %m (%c)%n +log4j.appender.stateChangeAppender.MaxFileSize=<%= $log_file_size %> +log4j.appender.stateChangeAppender.MaxBackupIndex=<%= $log_file_count %> + +log4j.appender.requestAppender=org.apache.log4j.RollingFileAppender +log4j.appender.requestAppender.File=${kafka.logs.dir}/kafka-request.log +log4j.appender.requestAppender.layout=org.apache.log4j.PatternLayout +log4j.appender.requestAppender.layout.ConversionPattern=[%d] %p %m (%c)%n +log4j.appender.requestAppender.MaxFileSize=<%= $log_file_size %> +log4j.appender.requestAppender.MaxBackupIndex=<%= $log_file_count %> + +log4j.appender.cleanerAppender=org.apache.log4j.RollingFileAppender +log4j.appender.cleanerAppender.File=${kafka.logs.dir}/log-cleaner.log +log4j.appender.cleanerAppender.layout=org.apache.log4j.PatternLayout +log4j.appender.cleanerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n +log4j.appender.cleanerAppender.MaxFileSize=<%= $log_file_size %> +log4j.appender.cleanerAppender.MaxBackupIndex=<%= $log_file_count %> + +log4j.appender.controllerAppender=org.apache.log4j.RollingFileAppender +log4j.appender.controllerAppender.File=${kafka.logs.dir}/controller.log +log4j.appender.controllerAppender.layout=org.apache.log4j.PatternLayout +log4j.appender.controllerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n +log4j.appender.controllerAppender.MaxFileSize=<%= $log_file_size %> +log4j.appender.controllerAppender.MaxBackupIndex=<%= $log_file_count %> + +log4j.appender.authorizerAppender=org.apache.log4j.RollingFileAppender +log4j.appender.authorizerAppender.File=${kafka.logs.dir}/kafka-authorizer.log +log4j.appender.authorizerAppender.layout=org.apache.log4j.PatternLayout +log4j.appender.authorizerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n +log4j.appender.authorizerAppender.MaxFileSize=<%= $log_file_size %> +log4j.appender.authorizerAppender.MaxBackupIndex=<%= $log_file_count %> + +# Change the two lines below to adjust ZK client logging +log4j.logger.org.I0Itec.zkclient.ZkClient=INFO +log4j.logger.org.apache.zookeeper=INFO + +# Change the two lines below to adjust the general broker logging level (output to server.log and stdout) +log4j.logger.kafka=INFO +log4j.logger.org.apache.kafka=INFO + +# Change to DEBUG or TRACE to enable request logging +log4j.logger.kafka.request.logger=WARN, requestAppender +log4j.additivity.kafka.request.logger=false + +# Uncomment the lines below and change log4j.logger.kafka.network.RequestChannel$ to TRACE for additional output +# related to the handling of requests +#log4j.logger.kafka.network.Processor=TRACE, requestAppender +#log4j.logger.kafka.server.KafkaApis=TRACE, requestAppender +#log4j.additivity.kafka.server.KafkaApis=false +log4j.logger.kafka.network.RequestChannel$=WARN, requestAppender +log4j.additivity.kafka.network.RequestChannel$=false + +log4j.logger.kafka.controller=TRACE, controllerAppender +log4j.additivity.kafka.controller=false + +log4j.logger.kafka.log.LogCleaner=INFO, cleanerAppender +log4j.additivity.kafka.log.LogCleaner=false + +log4j.logger.state.change.logger=TRACE, stateChangeAppender +log4j.additivity.state.change.logger=false + +# Access denials are logged at INFO level, change to DEBUG to also log allowed accesses +log4j.logger.kafka.authorizer.logger=INFO, authorizerAppender +log4j.additivity.kafka.authorizer.logger=false