Monitoring Red Hat 6 with SCOM 2019

by

Lets first start this post with an introduction to the topic. At several of our customers and during several speaking engagements we encountered questions about if certain versions of Linux/Unix were supported or not for newer versions of SCOM (in today’s case the most common being Red Hat 6 on SCOM 2019). The simple answer is that it is not supported and it will not be supported either. The reason is the same as for other versions not being supported before… For Microsoft products there is a 5 year mainstream support and a 5 year extended support for their products. Therefore they can often still support operating systems for agents for up to 10 years. When talking about operating systems from other vendors, Microsoft can not guarantee or support anything outside of Mainstream support (which is often the first 5 years or so). However… that does not say it will not work.
My colleague Martijn Weterings has been working on the below, and we welcome his first contribution to this blog!

Monitoring Red Hat 6 with SCOM 2019

As Microsoft lists on their web site, Red Hat 6 is not supported on SCOM 2019. There is however still a reasonable group of people that still have Red Hat 6 running for various reasons, that now have a reason not to move to SCOM 2019 yet. As one of our clients is in a similar situation, I took some time to see if this could be working anyway.

Getting the right management packs:

Examining the list of Linux related management packs; we can conclude that there is no Red Hat 6 management pack present after importing SCOM 2019’s default Linux management packs. Because of this, a remote installation through the Discovery Wizard will fail when attempted.

Discovery wizard

Fortunately, the UNIX / Linux management packs for System Center 2016 (available through this link), still provide a RHEL 6 management pack. After downloading and installing just the RHEL 6 Operating System MP, it seems to accept newer versions of its depending management packs just fine.

This would mean we should now have an agent available for our RHEL6 servers. Let’s verify this in the AgentManagement folder on the SCOM management server. This contains the installation packages used while running the Discovery Wizard;

Now, the versions are visible in the packages’ file name. While all ‘supported’ packages are of version 1.6.3-793, the RHEL6 packages are 1.6.2-343. This might be an issue later, as a newer agent might create different dependencies that are missing in older versions.

Now I could shamelessly copy/paste Kevin Holman’s article as that is all that remains, but I really would advise just to follow along the steps using his guide for SCOM 2016. It describes the required steps on creating a separate resource pool, configuring run as accounts. This should do in most cases. Thanks for that, Kevin!

Firewall:

In other cases the firewall might be blocking port 1270 used by WSMan. When this is the case, below lines can be used for making an exception, and saving it. Modify the IP / subnet according to your situation.

<em>iptables -I INPUT 1 -p tcp -s 192.168.1.0/24 --dport 1270 -m state --state NEW,ESTABLISHED -j ACCEPT</em>

<em>/sbin/service iptables save</em>

Sudoers file:

The /etc/sudoers file can be a pain to configure properly, especially for the Windows oriented audience (myself included). Sometimes the order of lines even seems to make a difference. Of course, you can choose to use the root account for both maintenance and monitoring task, but chances are your colleagues in security teams want to see all sorts of modifications to the recently applied configuration. To make it both easy and a bit more secure than that, I have modified a fresh post-installation sudoers file to Kevin Holman’s article. Use WinSCP or a similar tool to copy this over your current content

## Sudoers allows particular users to run various commands as

## the root user, without needing the root password.

##

## Examples are provided at the bottom of the file for collections

## of related commands, which can then be delegated out to particular

## users or groups.

##

## This file must be edited with the 'visudo' command.

## Host Aliases

## Groups of machines. You may prefer to use hostnames (perhaps using

## wildcards for entire domains) or IP addresses instead.

# Host_Alias FILESERVERS = fs1, fs2

# Host_Alias MAILSERVERS = smtp, smtp2

## User Aliases

## These aren't often necessary, as you can use regular groups

## (ie, from files, LDAP, NIS, etc) in this file - just use %groupname

## rather than USERALIAS

# User_Alias ADMINS = jsmith, mikem

## Command Aliases

## These are groups of related commands...

## Networking

# Cmnd_Alias NETWORKING = /sbin/route, /sbin/ifconfig, /bin/ping, /sbin/dhclient, /usr/bin/net, /sbin/iptables, /usr/bin/rfcomm, /usr/bin/wvdial, /sbin/iwconfig, /sbin/mii-tool

## Installation and management of software

# Cmnd_Alias SOFTWARE = /bin/rpm, /usr/bin/up2date, /usr/bin/yum

## Services

# Cmnd_Alias SERVICES = /sbin/service, /sbin/chkconfig

## Updating the locate database

# Cmnd_Alias LOCATE = /usr/bin/updatedb

## Storage

# Cmnd_Alias STORAGE = /sbin/fdisk, /sbin/sfdisk, /sbin/parted, /sbin/partprobe, /bin/mount, /bin/umount

## Delegating permissions

# Cmnd_Alias DELEGATING = /usr/sbin/visudo, /bin/chown, /bin/chmod, /bin/chgrp

## Processes

# Cmnd_Alias PROCESSES = /bin/nice, /bin/kill, /usr/bin/kill, /usr/bin/killall

## Drivers

# Cmnd_Alias DRIVERS = /sbin/modprobe

# Defaults specification

#

# Refuse to run if unable to disable echo on the tty.

#

Defaults !visiblepw

Defaults requiretty

Defaults:scxmaint !requiretty

Defaults:scxmon !requiretty

#

# Preserving HOME has security implications since many programs

# use it when searching for configuration files. Note that HOME

# is already set when the the env_reset option is enabled, so

# this option is only effective for configurations where either

# env_reset is disabled or HOME is present in the env_keep list.

#

Defaults always_set_home

Defaults env_reset

Defaults env_keep = "COLORS DISPLAY HOSTNAME HISTSIZE INPUTRC KDEDIR LS_COLORS"

Defaults env_keep += "MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE"

Defaults env_keep += "LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES"

Defaults env_keep += "LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE"

Defaults env_keep += "LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY"

#

# Adding HOME to env_keep may enable a user to run unrestricted

# commands via sudo.

#

# Defaults env_keep += "HOME"

Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin

## Next comes the main part: which users can run what software on

## which machines (the sudoers file can be shared between multiple

## systems).

## Syntax:

##

## user MACHINE=COMMANDS

##

## The COMMANDS section may have other options added to it.

##

## Allow root to run any commands anywhere

root ALL=(ALL) ALL

##Agent maintenance

#Agent maintenance for LINUX

#Certificate signing

scxmaint ALL=(root) NOPASSWD: /bin/sh -c cp /tmp/scx-scxmaint/scx.pem /etc/opt/microsoft/scx/ssl/scx.pem; rm -rf /tmp/scx-scxmaint; /opt/microsoft/scx/bin/tools/scxadmin -restart

scxmaint ALL=(root) NOPASSWD: /bin/sh -c cat /etc/opt/microsoft/scx/ssl/scx.pem

#RHEL

scxmaint ALL=(root) NOPASSWD: /bin/sh -c sh /tmp/scx-scxmaint/scx-1.[5-9].[0-9]-[0-9][0-9][0-9].rhel.[[\:digit\:]].x[6-8][4-6].sh --install; EC=$?; cd /tmp; rm -rf /tmp/scx-scxmaint; exit $EC

scxmaint ALL=(root) NOPASSWD: /bin/sh -c sh /tmp/scx-scxmaint/scx-1.[5-9].[0-9]-[0-9][0-9][0-9].rhel.[[\:digit\:]].x[6-8][4-6].sh --upgrade --force; EC=$?; cd /tmp; rm -rf /tmp/scx-scxmaint; exit $EC

#AIX

#scxmaint ALL=(root) NOPASSWD: /usr/bin/sh -c sh /tmp/scx-scxmaint/scx-1.[5-9].[0-9]-[0-9][0-9][0-9].aix.[[\:digit\:]].ppc.sh --install ; EC=$?; cd /tmp; rm -rf /tmp/scx-scxmaint; exit $EC

#scxmaint ALL=(root) NOPASSWD: /usr/bin/sh -c sh /tmp/scx-scxmaint/scx-1.[5-9].[0-9]-[0-9][0-9][0-9].aix.[[\:digit\:]].ppc.sh --upgrade --force ; EC=$?; cd /tmp; rm -rf /tmp/scx-scxmaint; exit $EC

##Uninstall

#Uninstall for LINUX

scxmaint ALL=(root) NOPASSWD: /bin/sh -c /opt/microsoft/scx/bin/uninstall

#Custom shell command monitoring example. Replace &lt;shell command&gt; with the correct command string

#scxmon ALL=(root) NOPASSWD: /bin/bash -c &lt;shell command&gt;

#Daemon diagnostic and restart recovery tasks (example using cron)

#scxmon ALL=(root) NOPASSWD: /bin/sh -c ps -ef | grep cron | grep -v grep

#scxmon ALL=(root) NOPASSWD: /usr/sbin/cron &amp;

##Log file monitoring

scxmon ALL=(root) NOPASSWD: /opt/microsoft/scx/bin/scxlogfilereader -p

## Allows members of the 'sys' group to run networking, software,

## service management apps and more.

# %sys ALL = NETWORKING, SOFTWARE, SERVICES, STORAGE, DELEGATING, PROCESSES, LOCATE, DRIVERS

## Allows people in group wheel to run all commands

# %wheel ALL=(ALL) ALL

## Same thing without a password

# %wheel ALL=(ALL) NOPASSWD: ALL

## Allows members of the users group to mount and unmount the

## cdrom as root

# %users ALL=/sbin/mount /mnt/cdrom, /sbin/umount /mnt/cdrom

## Allows members of the users group to shutdown this system

# %users localhost=/sbin/shutdown -h now

## Read drop-in files from /etc/sudoers.d (the # here does not mean a comment)

#includedir /etc/sudoers.d

Bonus: Not tested personally, but this might also work for AIX 6. Get the proper management packs from an earlier supported SCOM version, and remove the comment sign in front of the AIX section in the above sudoers file.

Additional Edit by Bob:
Between initial draft of this post and publishing it we also had a customer running Sun Solaris in an X86 type of version (and not a sparc version which is the only type supported by SCOM 2019). We did the same kind of trick and got the latest set of Linux/Unix packs set still supporting it (remember there are sets for 2016/1801/1807/2019), so find the one which last still supported it. Keep in mind that some things might not work now or later, especially if you plan on turning on features which are only available from higher SCOM versions.

Also we have seen cases where we needed to TEMPORARILY adjust the sudoers file to allow a bit more commands to either get an installation or an upgrade of an agent. This is due to small changes in commands and some deployment methods of sudoers files and settings seem to do something with special characters. So in that case try a:

scxmaint ALL=(root) NOPASSWD: ALL

And deploy again. You will find out quick enough if the problem was in the specifics of the commands.

 

Concluding:

After running a successful discovery, both RHEL 6 version are fully green. Both are also able to provide performance data.

Although an older version of the management pack and agent seems to work in my home lab, this is only a very limited test. It might just not go along with your current configuration. Also, keep in mind that it’s always a good idea to invest in your life cycle management. See if you can update your older Red Hat 6 servers to a newer, by SCOM supported version.

Enjoy!

Comments are closed.