4.11.11. HOWTO Set Up Centralized Log Collection with Rsyslog¶
SIMP provides a pre-built set of classes in the simp-simp_rsyslog
and
simp-rsyslog
Puppet modules for enabling centralized logging within the
infrastructure. simp-simp_rsyslog
is a profile module that uses
simp-rsyslog
.
There are no provisions here for setting up shared storage or deduplication. This is inherently not a use case that Rsyslog is well designed for and we suggest that you look at an alternative.
Note
For an overview of how to use Hiera to manage class parameters, please see Classification and Data.
4.11.11.1. Preparation¶
4.11.11.1.1. The simp_rsyslog
Profile Module¶
SIMP’s profile module, simp_rsyslog, helps configure systems for local and remote logging.
The simp_rsyslog
class is automatically included on systems if the simp
or simp_lite
scenarios are used and, by default,
configures local logging.
If scenarios are not being used, include the simp_rsyslog
class on all
systems including the log server. If you are using the default SIMP install,
you can add it to the simp::classes
array. Otherwise, you will need to use
a standard Puppet include
mechanism.
4.11.11.1.2. What is Logged¶
The simp-simp_rsyslog
module uses the following parameters:
---
simp_rsyslog::default_logs # A Hash of the default system logs to be collected
simp_rsyslog::log_collection # Use this Hash to add logs to the default set
There are also Booleans available to enable collection of certain logs, such as
those from OpenLDAP. See the simp-simp_rsyslog
module for more details.
4.11.11.1.2.1. The Log Hash Format¶
The Hashes mentioned above are complex in nature but provide a clean interface to most aspects of log collection targeted to most users.
The Puppet Data Type representation of the Hashes is as follows:
Hash[
Enum[
'programs',
'facilities',
'msg_starts',
'msg_regex'
],
Array[String]
]
This means that you can have a Hash
, with any of the keys programs
,
facilities
, msg_starts
, or msg_regex
followed by an Array
of
Strings
.
Using the following example Hash
:
{
'programs' => [ 'sudo' ],
'facilities' => [ 'cron.*' ],
'msg_starts' => [ 'IMPORTANT:' ],
'msg_regex' => [ '*bad_guys*' ]
}
The programs
line would match the following due to the highlighted section:
2017-03-14T15:26:53.589793+00:00 sample.host.name sudo: test_user : TTY=pts/0 ; PWD=/home/test_user ; USER=root ; COMMAND=/usr/sbin/visudo |
The facilities
line would match the following because the listed facility
is cron
:
2017-03-14T15:26:53.589793+00:00 sample.host.name CROND[31415]: (root) CMD (run-parts /etc/cron.hourly) |
The msg_starts
line would match the following due to the highlighted section:
2017-03-14T15:26:53.589793+00:00 sample.host.name kernel: IMPORTANT: This is an important message |
The msg_regex
line would match the following due to the highlighted section:
2017-03-14T15:26:53.589793+00:00 sample.host.name kernel: This system was prodded by bad_guys and should be watched |
4.11.11.1.3. Set Log Servers¶
The list of log servers are usually set during simp config
, and placed in
the simp_config_settings.yaml
Hiera file.
If this value needs to be changed, either simp config
can be run again or
the values below can be overridden in default.yaml
or similar Hiera
file to reach all nodes:
---
simp_options::syslog::log_servers:
- 'logserver1.fullyqualified.domain'
- 'logserver2.fullyqualified.domain'
simp_options::syslog::failover_log_servers:
- 'failoverserver1.fullyqualified.domain'
- 'failoverserver2.fullyqualified.domain'
If you list more than one primary log server your logs will be forwarded to all of the log servers in the array.
Failover log servers are optional.
Warning
If log forwarding is enabled on your log server, make sure you override the log server settings to NOT include itself. This will cause looping and will fill the disks on the system very quickly with repeated messages.
Note
It is common in big environments to use DNS aliases or to cluster servers so determining the name a server is using for logging is not straightforward. Because of this SIMP cannot reliably determine if a host is forwarding to itself.
4.11.11.1.4. TLS¶
If encryption is going to be used, make sure the certificates are in place. See the Apply Certificates documentation to understand how SIMP modules distribute certificates.
If SIMP is not being used to distribute certificates, see the documentation
for the following PKI-related parameters in the rsyslog
class of the
simp-rsyslog
module:
rsyslog::pki
rsyslog::app_pki_external_source_dir
rsyslog::app_pki_dir
4.11.11.2. Enable Clients¶
To set up the clients enter the following settings in the default.yaml
or
similar Hiera file to reach all clients:
---
#If using TLS
simp_rsyslog::forward_logs: true
rsyslog::enable_tls_logging: true
or
---
#If not using TLS
simp_rsyslog::forward_logs: true
rsyslog::pki: false
rsyslog::enable_tls_logging: false
4.11.11.3. Enable the Server¶
To set up the server enter the following in the server’s Hiera file:
---
# If using TLS
simp_rsyslog::is_server: true
simp_rsyslog::forward_logs: false
rsyslog::tls_tcp_server: true
or
---
# If NOT using TLS
simp_rsyslog::is_server: true
simp_rsyslog::forward_logs: false
rsyslog::tcp_server: true
rsyslog::tls_tcp_server: false
After puppet
has run on all the systems, the logs from the clients will be
stored in /var/log/hosts/<client name>
directory on the log server.
simp-simp_rsyslog
also sets up log rotation for these files by default using
the simp-logrotate
module.
4.11.11.3.1. Forwarding Log Files from a Log Server¶
If the log server needs to forward logs to another server, edit its Hiera file.
Set simp_rsyslog::forward_logs
to true
and make sure that the
log_servers
array used on the relevant node does not include itself in the
list. For example for a server using TLS:
---
simp_rsyslog::is_server: true
simp_rsyslog::forward_logs: true
rsyslog::tls_tcp_server: true
simp_options::syslog::log_servers:
- 'some-other-log-server.that.is.not.me'
simp_options::syslog::failover_log_servers:
- 'some-other-failover-server.that.is.not.me'
This will forward the server’s own logs, and all received client logs, to the specified servers.