4.9.2. Version-Specific Upgrade Instructions¶
4.9.2.1. Upgrading from SIMP-6.3.3 to SIMP-6.4.0¶
Warning
You MUST be on at least SIMP 6.3.3 to upgrade to 6.4.0 due to issues in the RPMs prior to the 6.3.3 release.
If you need to upgrade from 6.2.0 to 6.3.X, please read Upgrading from SIMP-6.2.0 to SIMP-6.3.0 with care.
As indicated by gradual updates in previous Changelogs, SIMP has been working on becoming more friendly to alternative modes of deployment and maintenance. The team has specifically been focusing on fully supporting r10k and Code Manager as methods of installation and maintenance over time.
SIMP 6.4.0 is the release where we make this happen!
We have taken care to make no technologically breaking changes and, in keeping with the version description in the General Upgrade Instructions, have not made changes that affect the API of modules provided by the supported module set.
That said, there are quite a few procedural items that need to be taken into account when upgrading to SIMP 6.4.0 so that things work generally like you are used to.
To get a full accounting of the major changes in behavior for 6.4.0, please see the SIMP Community Edition (CE) 6.4.0-0 Changelog.
Important
This upgrade has a great potential for deleting files you need. Read the instructions carefully and back up your system prior to the upgrade.
4.9.2.1.1. RPM Name Changes and the Consequences¶
4.9.2.1.1.1. simp-rsync¶
simp-rsync
has been replaced by simp-rsync-skeleton
.
Note
Due to the nature of RPM uninstallation activities, to avoid ruining any
active files in /var/simp/environments/simp/rsync
, we had to create a
final version of simp-rsync
that is generally dependency free. Nothing
requires this RPM since it is not technically required any longer, but you
MUST upgrade this RPM if you want to ensure that no files are
accidentally removed from the rsync
space on your SIMP server.
If you are not using a Puppet Environment named simp
then you can
safely remove the simp-rsync
RPM after upgrading.
4.9.2.1.1.2. FakeCA¶
simp-environment
has been renamed to simp-environment-skeleton
.
Previously, the simp-environment
RPM would place helpful FakeCA
files
into /var/simp/environments/simp/FakeCA
. These files were for convenience
and their presence, or absence, does not affect a running system in any way.
Upon upgrade, the simp-environment
RPM will be removed and you will find
that various FakeCA
files are either missing or have .rpmsave
appended
to them if they were modified configuration files.
You may run the following script as root
to restore the FakeCA
functionality:
#!/bin/sh
cd /var/simp/environments/simp
\cp -r /usr/share/simp/environment-skeleton/secondary/FakeCA .
cd FakeCA
for x in *.rpmsave; do
mv $x `basename $x .rpmsave`
done
4.9.2.1.2. Upgrade Steps¶
These steps for upgrading are for use in SIMP systems that are not currently using r10k or Code Manager.
4.9.2.1.2.1. Update OS and SIMP RPMS¶
Update the repos for the OS and SIMP and run yum update
.
The following errors/warnings are expected:
warning: /var/simp/environments/simp/FakeCA/togen saved as /var/simp/environments/simp/FakeCA/togen.rpmsave
warning: /var/simp/environments/simp/FakeCA/ca.cnf saved as /var/simp/environments/simp/FakeCA/ca.cnf.rpmsave
libsemanage.semanage_direct_remove_key: Unable to remove module simp-environment at priority 400. (No such file or directory).
/usr/sbin/semodule: Failed!
Note
The old SELinux module, simp-environment
, had to be removed
before installing the new SELinux module, simp
to avoid conflicts when
the simp
module from the simp-selinux-policy
package is installed.
Therefore, the error message above is emitted when the simp-environment
RPM tries to remove the already-removed SELinux module.
If you are using the FakeCA
in the simp
environment, follow the steps
in the FakeCA
section above to restore the needed files.
4.9.2.1.2.2. Create the Puppetfile¶
At this point in the upgrade, the SIMP-packaged Puppet modules have been
installed in /usr/share/simp/modules
and imported into local Git
repositories under /usr/share/simp/git/puppet_modules
. Now you need to
create the Puppetfile
for r10k to use to deploy modules to the
environment.
The environment name in this example is myenv
. Run the following commands
as root
to create a default Puppetfile
:
# Change to the top level puppet environment directory for your environment.
cd /etc/puppetlabs/code/environments/myenv
# Generate the main Puppetfile with local module entries
simp puppetfile generate --skeleton --local-modules myenv > Puppetfile
# Generate the Puppetfile for the SIMP modules.
simp puppetfile generate > Puppetfile.simp
chown root:puppet Puppetfile Puppetfile.simp
chmod 640 Puppetfile Puppetfile.simp
Edit the main Puppetfile:
- Make sure all the local modules you created and have in the modules
directory have
:local => true
entries. This includes thesite
module, which is no longer delivered as an RPM. - Delete the
local => true
entries for any obsolete modules that you do not need (e.g.simpcat
).
Warning
Any module not listed in the Puppetfile
will be deleted from the modules
directory.
The following shows three examples of how to include your modules in the Puppetfile
:
site
: a module that is not a Git repository and resides in themodules
directorymymodule
: a Git repository that resides in a directory on the Puppet servermysrvmod
: a Git repository on a GitLab/GitHub type server.
mod 'site',
:local => true
mod 'mymodule',
:git => 'file:///usr/share/mymodules/mymodule',
:tag => '1.1.1'
mod 'mysrvmod',
:git => 'https://gitserver.my.domain/mygitproject/mysrvmod.git',
:tag => '1.0.1'
Important
Make sure all the modules under your environments modules directory that
you created and that are not SIMP modules are listed in the Puppetfile
.
This includes the site
module, which SIMP no longer delivers.
Warning
Any module not listed in the Puppetfile
will be deleted from the
modules directory.
4.9.2.1.2.3. Deploy the Puppet Modules to the Environment¶
Run the following commands to deploy the puppet modules to the environment:
(The environment in this example is myenv
. Replace this with the name of the
environment you are updating.)
# Make sure the umask is set properly
# Set the umask and Run r10k as the puppet group to make sure the modules
# to make sure the permissions and ownership are correct on the modules
( umask 0027 && sg puppet -c '/usr/share/simp/bin/r10k puppetfile install \
--puppetfile /etc/puppetlabs/code/environments/myenv/Puppetfile \
--moduledir /etc/puppetlabs/code/environments/myenv/modules' )
4.9.2.1.2.4. Run Puppet¶
Run puppet agent -t
on all your systems in the environment to apply the update.
4.9.2.1.2.5. Other Environments¶
If you have more than one environment, you need to create a Puppetfile and deploy the modules for each environment.
4.9.2.1.3. Adding SIMP Extra Modules After The Upgrade¶
If you had modules installed from the simp-extras
package, they will have
been updated with yum update
, a local Git repository will have been created
for them under /usr/share/simp/modules
, and they will
have been included in the Puppet.simp
file when it was generated.
If, after the upgrade you decide to install a new SIMP module,
you must update the Puppetfile.simp
file and deploy the modules again.
For example if you want to install the snmp
and snmpd
modules:
yum install pupmod-puppet-snmp pupmod-simp-snmpd
# After running yum you will see the modules under
# /usr/share/simp/modules but not in your environment.
# Now update the Puppetfile.simp and deploy the environment again.
cd /etc/puppetlabs/code/environments/myenv
simp puppetfile generate > Puppetfile.simp
( umask 0027 && sg puppet -c '/usr/share/simp/bin/r10k puppetfile install \
--puppetfile /etc/puppetlabs/code/environments/myenv/Puppetfile \
--moduledir /etc/puppetlabs/code/environments/myenv/modules' )
If you add one of your own modules, manually edit the Puppetfile
and deploy
again. You do not need to run simp puppetfile generate
with the skeleton
option again.
Remember, if you do not add the module to the Puppetfile
it will get erased the next time the Puppetfile
is deployed!
4.9.2.1.4. Upgrading If R10K Is Already Being USED¶
UNDER CONSTRUCTION
4.9.2.2. Upgrading from SIMP-6.3.2 to SIMP-6.3.3¶
The 6.3.3 release of SIMP is a bugfix release focusing on a specific issue with
the updated sssd
module that may cause account lockouts on upgrade
depending on how your LDAP accounts have been maintained.
You can follow Incremental Upgrades for upgrading.
Warning
If you are upgrading from 6.2.0 to 6.3.X, please read Upgrading from SIMP-6.2.0 to SIMP-6.3.0 with care.
4.9.2.3. Upgrading from SIMP-6.3.0 to SIMP-6.3.2¶
The 6.3.2 release of SIMP is a major bugfix release.
You can follow Incremental Upgrades for upgrading.
Warning
If you are upgrading from 6.2.0 to 6.3.X, please read Upgrading from SIMP-6.2.0 to SIMP-6.3.0 with care.
4.9.2.4. Upgrading from SIMP-6.2.0 to SIMP-6.3.0¶
The main purpose of the SIMP 6.3 release is to upgrade to Puppet 5/Hiera 5. You should familiarize yourself with the Puppet 5 release notes and the Hiera 5 upgrade documents.
You can follow Incremental Upgrades for upgrading with only the exception noted below in Upgrade the Puppetserver section.
After the system is upgraded you can enable the environment layer, by giving each environment its own hiera.yaml (recommended). General instructions are given in the Updating Hiera section below.
4.9.2.4.1. Upgrade the Puppetserver¶
After you have disabled the puppet agent on the puppetserver and before you have upgraded the puppetserver to puppet 5 or updated the simp modules, make the following two edits to the puppet configuration files. If this is skipped the puppet server will not restart after it is upgraded to Puppet 5.
- File:
- /etc/puppetlabs/puppetserver/services.d/ca.cfg
- Add line:
- puppetlabs.trapperkeeper.services.watcher.filesystem-watch-service/filesystem-watch-service
- File:
- /etc/puppetlabs/puppetserver/conf.d/web-routes.conf
- Add line:
- “puppetlabs.trapperkeeper.services.metrics.metrics-service/metrics-webservice”: “/metrics”
Also note that you may see the following warnings multiple times during the upgrade:
Warning: Setting trusted_server_facts is deprecated.
(location: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/settings.rb:1169:in `issue_deprecation_warning')
Warning: Setting ca is deprecated.
(location: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/settings.rb:1169:in `issue_deprecation_warning')
This message is innocuous, but unavoidable unless you are comfortable with Disabling all Puppet deprecation warnings.
4.9.2.4.2. Restart PuppetDB¶
At this point, all packages should be updated and the puppetserver
process
has probably restarted. If you’ve been using your system with puppetdb
for a while, then you will want to restart the puppetdb
process to ensure a
successful database migration.
/opt/puppetlabs/bin/puppet resource service puppetdb ensure=stopped
/opt/puppetlabs/bin/puppet resource service puppetdb ensure=running
4.9.2.4.3. Updating Hiera¶
One of the major features of Hiera 5 is the addition of the environment layer. You can enable the Hiera environment layer with the following steps.
The new hiera.yaml
file is installed by
simp-environment-5.3.0-0.<dist>.noarch.rpm
into
/usr/share/simp/environments/simp
. This hiera.yaml file uses data
as
the directory instead of hieradata
. (If you have made changes to the hiera
hierarchy make sure to incorporate these into the file.)
For every environment, copy the new hiera.yaml to the environment and then rename the hieradata directory to data.
cp /usr/share/simp/environments/simp/hiera.yaml /etc/puppetlabs/code/environments/<environment>/
cd /etc/puppetlabs/code/environments/<environment>/
chmod 640 ./hiera.yaml
chown root:puppet ./hiera.yaml
mv hieradata data
When all the environments are converted, update the global hiera.yaml
file:
# Copy an empty hiera.yaml file to the global level.
cd /etc/puppetlabs/puppet
unlink hiera.yaml
rm hiera.yaml.simp
cp hiera.yaml.rpmnew hiera.yaml
chown root:puppet hiera.yaml
chmod 640 hiera.yaml
If the .rpmnew
file does not exist the new global hiera.yaml
file should look like:
version 5
# defaults:
# data_hash: yaml_data
# hierarchy:
# - name: Common
# data_hash: yaml_data
hierarchy: []
4.9.2.5. Upgrading from SIMP-6.1.0 to SIMP-6.2.0¶
Important
It is highly recommended that you read the information in this section in its entirety before upgrading.
4.9.2.5.1. Exclude puppet-agent from the yum update¶
A bug (SIMP-5021) was identified in SIMP 6.1.0 that will prevent the
puppetserver
service from starting after the puppet-agent
package is
upgraded. The issue is fixed in 6.2.0, but special handling is necessary during
the upgrade.
Note
- This problem affects ISO/RPM-based installations of SIMP when upgrading from 6.1.0, unless precautions are taken (detailed below).
- SIMP installations that are upgraded using r10k and Code Manager
are unlikely to be affected, unless the SIMP master’s
puppet-agent
package is updated independently. - These instructions only apply to the SIMP master—no changes are required on clients.
Before upgrading anything, add the following line to
/etc/yum.conf
:exclude=puppet-agent
Assuming you aren’t already using
/etc/yum.conf
to exclude other packages, you can add this line automatically by running:puppet resource file_line workaround path=/etc/yum.conf line='exclude=puppet-agent'
Proceed with the upgrade as outlined in Incremental Upgrades, up through the
yum update
:# (ISO Installations only:) Unpack the new ISO's RPMs into yum repositories unpack_dvd </path/to/ISO> # Make sure yum picks up the new RPMs yum clean all; yum makecache # Apply updates to the local master yum update -y
After running
yum update
, remove theexclude=puppet-agent
line from /etc/yum.conf.If you added the line using
puppet resource
, you can remove it the same way:puppet resource file_line workaround path=/etc/yum.conf line='exclude=puppet-agent' ensure=absent
Run
puppet agent -t
to upgrade the puppet-agent packageRun
puppet agent -t
again to ensure that everything runs cleanly.
Recovering if the puppetserver fails to restart
If, during the upgrade, the puppetserver
service fails to start with the
error:
Puppet::Error: Cannot determine basic system flavour
You should be able to recover by running the command:
puppet resource file /opt/puppetlabs/puppet/cache owner=puppet group=puppet
# The service should now restart
puppet resource service puppetserver ensure=running
After you have recovered in this manner, it is safe to proceed re-running the step you had been attempting when the puppetserver failed.
4.9.2.5.2. Update Kickstart Files¶
The example kickstart file
(pupclient_86_64.cfg
—see Setting up Kickstart)
was updated. Existing kickstart files that are modeled on older versions of
this file should backport these changes. New, OS-specific versions of this
file are included in the SIMP-6.2.0 ISOs, but may also be found in the
simp-core repository under
build/distributions/<OS>/<OS Major Version>/<Architecture>/DVD/ks/
.
Important
Careful examination of pupclient_86_64.cfg
is recommended, if you have
customized it for your site beyond token replacement. The latest version
contains a few subtle bug fixes, such as fixing an incorrect path that would
prevent UEFI systems from booting.
Two major changes were made:
pupclient_86_64.cfg
was updated to include instructions regarding what to change in order to accommodate UEFI boot. The associated configuration lines are commented out by default.pupclient_86_64.cfg
was updated to download and use new bootstrap service files. These files, managed bysimp::server::kickstart
, are:- A
systemd
unit file for CentOS 7 (simp_client_bootstrap.service
) or asystemv
init script for CentOS 6 (simp_client_bootstrap
). - A common bootstrap script (
bootstrap_simp_client
) used by both.
This pair of files replaces the deprecated
runpuppet
script. They are required in order to solve two timeout problems on particularly loaded systems, both of which can cause client Puppet bootstrapping to fail and require subsequent manual intervention in order to fix:- On CentOS 7,
systemd
was killingrunpuppet
, when it ran longer than 5 minutes. This was solved by using an actualsystemd
unit file (simp_client_bootstrap.service
) with a default timeout of 30 minutes. - On CentOS 7, if the DHCP lease expired in the middle of bootstrapping,
the generated Puppet configuration would erroneously use
localhost
for the client hostname. This problem was solved by setting the static hostname of the client at the beginning of the client Puppet bootstrap process.
In addition to solving these specific problems, the new bootstrap scripts use a configurable backoff algorithm in order prevent flooding a heavily-loaded Puppet master with requests. See the
simp::server::kickstart::simp_client_bootstrap
class for details.- A
4.9.2.5.3. Update dhcpd.conf¶
Changes were added to the dhcpd.conf
file that enable the DHCP server to
determine what mode, BIOS or UEFI, a system is kickstarting in and then to set
the appropriate boot loader file on the TFTP server.
On a SIMP server, the example dhcpd.conf
file is installed in
/var/simp/environments/simp/RedHat/Global/dhcpd/dhcpd.conf
via the
simp-rsync
package. This file may also be found in the
simp-rsync-skeleton repository under
environments/simp/rsync/RedHat/Global/dhcpd
.
Note
When the simp-rsync
RPM is upgraded, a message may be displayed that
indicates
warning: /var/simp/environments/simp/rsync/RedHat/Global/dhcpd/dhcpd.conf\
created as /var/simp/environments/simp/rsync/RedHat/Global/dhcpd/\
dhcpd.conf.rpmnew
Because the contents of the /var/simp/environments/simp/rsync/
directory
are pushed to remote systems, files ending with .rpmnew
are deleted from
the directory as part of the RPM deployment.
The commands below can be used to extract the new config file from the RPM,
which can then be merged with the existing dhcpd.conf
file.
Run
rpm2cpio simp-rsync-6.2.1-0.el7.noarch.rpm |cpio -ir "*dhcpd.conf"
- rpm2cpio converts the RPM file to cpio archive. In this example, the command is being run from the directory containing the RPM file; if the file is not in the current working directory, the full path to the file must be provided.
- cpio extracts files from the cpio archive. In this case, the cpio options -i, -r, and “*dhcpd.conf” direct cpio to extract one or more files from the archive, interactively rename the file, and the path and filename of files to be extracted should end with dhcpd.conf.
When prompted
rename ./var/simp/environments/simp/rsync/RedHat/Global/ dhcpd/dhcpd.conf ->
, enter the desired path and name of the file to be extracted, such as/tmp/dhcpd.conf.rpmnew
. Paths can be either absolute or relative, depending whether they start with a/
or./
.
4.9.2.5.4. Update the TFTP Root Directory¶
The default TFTP root directory was changed to /var/lib/tftpboot
to
conform to DISA STIG standards. To continue using /tftpboot
set
tftpboot::tftpboot_root_dir
in Hiera to /tftpboot
. Alternately, to use
the new directory, copy any files not managed by the rsync
module (i.e., not
stored in /var/simp/environments/<environment>/rsync/<OS>/Global/tftpboot
)
to the new directory. Make sure the permissions, including selinux context, are
correct. TFTP boot will fail to find boot files that have the incorrect selinux
context.
4.9.2.5.5. Optionally Remove OBE Logrotate Configuration Files¶
SIMP-managed logrotate
rules are now in /etc/logrotate.simp.d
instead of
/etc/logrotate.d
. The rules in /etc/logrotate.d
are still applied, but
logrotate
is configured to read the rules in /etc/logrotate.simp.d
,
first.
This change was made to ensure SIMP-managed rules take precedence over
vendor-supplied rules, because, when there are multiple rules specified for the
same file, only the first rule is applied. Any subsequent rules are discarded.
In fact, for some versions of logrotate
, a rule with a duplicate log file
is discarded in its entirety, even if only one of the managed log files is a
duplicate. This means the remaining log files specified in that discarded rule
will not be rotated!
Because the location of the SIMP-managed logrotate
rules has changed,
existing, but now OBE, SIMP rules will still reside in /etc/logrotate.d
.
Although these rules cause no issues with logrotate
, they may be confusing
to system administrators. So, you may wish to manually remove these rules.
You can easily identify OBE SIMP rules in /etc/logrotate.d
by their
This file managed by puppet.
comment lines.
4.9.2.6. Known Issues¶
4.9.2.6.1. Issues With Copying Repo Contents¶
A bug has been reported in some cases where the simp-adapter package does not appear to be properly updating existing repository spaces in all cases.
If you are getting unexpected errors, then you should run the following command and evaluate any differences. The materials in /usr/share/simp/modules are authoritative for the latest build but will not be copied over if the target files are checked into a version control system.
diff -qr /usr/share/simp/modules /etc/puppetlabs/code/environments/simp/modules
Once you have corrected any issues, you should run /opt/puppetlabs/bin/puppetserver reload and the system should function properly again.
4.9.2.7. Upgrading from SIMP-6.0.0 to SIMP-6.1.0¶
Important
It is highly recommended that you read the information in this section in its entirety.
4.9.2.7.1. Upgrade Script¶
There were several issues found during the SIMP 6.0.0 to 6.1.0 upgrade that necessitated the creation of an upgrade script that is to be run on your SIMP Puppet masters.
Note
No changes are required on your clients for the upgrade to succeed.
The upgrade script, /usr/share/simp/upgrade_scripts/upgrade_simp_6.0.0_to_6.1.0
,
will assist with the upgrade from 6.0.0
to 6.1.0
, taking into account
all of the specific issues. This script is available in the simp-utils-6.1.0
package provided by SIMP 6.1.0 or the simp-utils repository.
As always, back up your system prior to upgrading!
Note
This script assumes that you are upgrading from the SIMP RPMs!
If you have chosen some other installation method, you will need to follow the general steps outlined in the script.
To perform the upgrade, with root
permissions:
- Upgrade the
simp-utils
package, only, by executingyum update -y simp-utils
. - Run the script:
/usr/share/simp/upgrade_scripts/upgrade_simp_6.0.0_to_6.1.0
Note
This script will:
- Run a
yum -y update
. - Reinstall
simp-gpgkeys
andpupmod-simp-timezone
due to RPM issues. - Stop and uninstall the PostgresSQL 9.4 server to prevent
postgresql
upgrade issues. - Restart the
puppetserver
process. - Run
puppet agent -t
.- Some systems have shown issues with the
postgresql
upgrade during this step.
- Some systems have shown issues with the
4.9.2.7.2. Update auth.conf¶
The legacy auth.conf
, /etc/puppetlabs/puppet/auth.conf
, has been
deprecated. pupmod-simp-pupmod
will back up the legacy auth.conf
after
the upgrade.
The Puppet master’s auth.conf
is now managed by Puppet. You will need to
reproduce any custom work done to legacy auth.conf
via the new
puppet_authorization::rule
define. The stock rules are managed in
pupmod::master::simp_auth
.
4.9.2.7.3. Set up ClamAV DAT Files Updates¶
Given the wide spacing of SIMP releases, the team determined that it was
ineffective for us to maintain the simp-rsync-clamav
RPM with upstream
ClamAV DAT file updates.
From this point forward, SIMP will not ship with updated ClamAV DAT files and we highly recommend updating your DAT files from the authoritative upstream sources. See the ClamAV Virus Database FAQ for instructions on how to automatically update these files.
4.9.2.7.4. Prepare System for PostgreSQL Upgrade¶
During the Puppet-managed upgrade, from PostgreSQL 9.4 to PostgreSQL 9.6, the PostgreSQL 9.4 data is not automatically imported into the 9.6 database. If for any reason you need to retain this data, which normally is quite transitory, see Upgrading a PostgreSQL Cluster for detailed instructions.