Monitoring remote systems with Icinga 2

Icinga2 with web interface on Centos 6.5

First off, I'm relatively new to Linux, I'm sure there are things below that could be done better of faster. This was just for myself to get a monitoring server up and running to support me while building and improving other systems.

================= After the release ================

Install centos minimal

yum update -y
yum install wget vim -y

rpm --import http://packages.icinga.org/icinga.key

wget http://packages.icinga.org/epel/ICINGA-snapshot.repo -O /etc/yum.repos.d/ICINGA-snapshot.repo
yum makecache


yum install icinga2 -y
yum install httpd mysql-server mysql -y
yum install icinga2-ido-mysql
yum install icinga-web
yum install openssl-devel # required to do HTTPS site checks
yum install icinga2-ido-mysql icinga-idoutils-libdbi-mysql
yum install gcc # required for later plugin compilation 
yum install mailx # required for sending notifications
yum install php-mysql   #could be already there due to depencies.

chkconfig icinga2
chkconfig httpd on
chkconfig mysqld on

service mysqld start
service httpd start
service iptables stop

Icinga database

mysql -u root -p

mysql>  CREATE DATABASE icinga;

mysql>  GRANT SELECT, INSERT, UPDATE, DELETE, DROP, CREATE VIEW, INDEX, EXECUTE ON icinga.* TO 'icinga'@'localhost' IDENTIFIED BY 'icinga';

mysql> quit

 mysql -u root -p icinga < /usr/share/icinga2-ido-mysql/schema/mysql.sql

Icinga WEB database

mysql -u root -p

mysql>  CREATE DATABASE icinga_web;

mysql>  GRANT SELECT, INSERT, UPDATE, DELETE, DROP, CREATE VIEW, INDEX, EXECUTE ON icinga_web.* TO 'icinga_web'@'localhost' IDENTIFIED BY 'icinga_web';

mysql> quit

mysql -u root -p icinga_web < /usr/share/doc/icinga-web-*/schema/mysql.sql

Change the ido2db sockets type from unix to tcp

vim /etc/icinga/ido2db.cfg
socket_type=tcp
socket_name=127.0.0.1

vim /etc/icinga/idomod.cfg
output_type=tcpsocket
output=127.0.0.1

service icinga2 start
service ido2db start

Download plugins required for ping checks and more
https://www.monitoring-plugins.org/download.html

https://www.monitoring-plugins.org/doc/faq/compilation.html
https://www.monitoring-plugins.org/doc/faq/installation.html

Now the checks are installed at /usr/local/libexec/
The Icinga-web interface will point to /usr/lib64/nagios/plugins/

Depending on which directory your plugins are installed into you may need to update the global PluginDir constant in your Icinga 2 configuration. 
vim /etc/icinga2/constants.conf
change the location to /usr/local/libexec




Many of these steps are based on: http://docs.icinga.org/icinga2/latest/doc/module/icinga2/toc#!/icinga2/latest/doc/module/icinga2/chapter/getting-started#setting-up-icinga2

And not to forget

• Set a ROOT password on your Mysql installation
• Change the passwords of Icinga users (mysql, icinga)
• Create a firewall rule for port 80 and start your iptables
• Create proper SE linux policy and enable selinux

Basic actions

Adding a host

cd /etc/icinga2/conf.d/hosts
cp localhost.conf newserver.conf
vim newserver.conf
> Change the object Host name
> change the IPaddresss
> quit

chgrp icinga newserver.conf
chown icinga newserver.conf
service icinga2 reload
Wait for a bit and refresh the web interface

Correct the time shown in the icinga-web

Time comes from /etc/php.ini
Uncomment the date.timezone
eg > date.timezone = Europe/Amsterdam

Enable email notifications

Make sure your machine is allowed to make a smtp connection through the firewall to the internet.
edit /etc/postfix/main.cfg
search for mynetworks_style
create line:
mynetworks_style = host

search for relay_domains

create line: 

relay_domains=

search for relayhost
create line:
relayhost = 
If you test with a command line to send email, it's possible that it gets rejected or marked as spam. Make sure you have your SPF record, MX records and what you need setup.-- Some settings for icinga itself so it knows where to mail to 

Monitoring a httpS site

create a .conf file in /etc/icinga2/conf.d/hosts
> ripe.conf

object Host "website-ripe.net" {
  import "generic-host"
  address = "ripe.net"
  vars.os = "Linux"
  vars.sla = "24x7"
  vars.http_vhost = "ripe.net"
  vars.http_address = "195.69.144.71"
  vars.http_ssl = "1"
  vars.http_sni = "1"
  vars.http_warn_time = "5"
  vars.http_critical_time = "10"
}

object Service "httpS" {
  host_name = "wesite-ripe.net"
  check_command = "http"

}

Monitor the icinga2 log for
"critical/Application: Found error in config: reloading aborted"
which indicates errors in the conf file. The server keeps running with the working config.

Errors / Issues / FAQ

https://wiki.icinga.org/display/Dev/Icinga+Core+Debug+Config
Enable debug mode

icinga2-enable-feature debuglog

service icinga2 restart

tail -f /var/log/icinga2/debug.log

disable debug mode



Could not send command. Check if your webserver's user has correct permissions for writing to the command pipe.

At this moment the icinga service aswell as the icinga2 service need to run.

Check /usr/share/icinga-web/app/modules/Api/config/access.xml 
if the path behind <resource name="icinga_pipe">  exists. In this case (centos 6.5) it's /var/spool/icinga/cmd/icinga.cmd

ls -halt  /var/spool/icinga/cmd/
There should be a file called icinga.cmd with the owner icinga

ssl not available

yum install openssl-devel
re-make the plugins
re-install the plugins
restart the icinga services

Manual testing a check
Go to /usr/local/libexec/
sudo -u icinga ./check_http -H website.com

Some SSL check examples: https://www.monitoring-plugins.org/doc/man/check_http.html


Get SSL check and mod_security
http://www.mhaller.de/archives/145-Nagios,-mod_security-and-check_http.html

Messages about INO2db  < needs updating / more detailed information on this issue
Lot of messages
> set the connection from unix to tcp
> adjust the pipe to 127.0.0.01


Where to find the default commands for the command_check such as ping4 ping6 and http vim /usr/share/icinga2/include/command-plugins.conf