Skip to main content

Zabbix remote monitoring

Zabbix menu

Introduction

When we have a Zabbix server monitoring on-premises servers, we can use both active and passive zabbix-agents, connecting to the Zabbix server over the local network. However, for remote servers with only public IP addresses, we would need to open up the ports for Zabbix on either the Zabbix server (10051/tcp for active checks) or the monitored host (10050/tcp for passive checks). This isn't ideal of course, so it would be better to tunnel these connections over SSH. 

In the setup below, we use an SSH tunnel, initiated from the monitored host to the Zabbix server, to allow an agent on the host to communicate to the server. We only use active agents this way, so the Items and Templates configured in Zabbix should only use Zabbix agent (active).  

Server Host - Zabbix agent -> localhost:10051 
              = autossh tunnel (-L 10051:127.0.0.1:10051) = 
                          -> localhost:10051/tcp -> Zabbix server 

Setup on the Zabbix server

We will need to add a restricted user, zabbix-agent, that can only login using a certificate, and that can do nothing but setup a local forward, its shell is set to /bin/false in /etc/passwd. Its authorized_keys contains the public keys for the remote server.

$ sudo useradd zabbix-agent
$ sudo echo "ssh-rsa <<public key>> user@server" >> /home/zabbix-agent/.ssh/authorized_keys

 

Configure sshd (usually found under /etc/ssh/sshd_config) to allow our zabbix agent nothing but tunneling;

# Restrict the zabbix agent user to tunneling 10051 only
Match User zabbix-agent
        X11Forwarding no
        PermitTTY no
        AllowAgentForwarding No
        PermitOpen 127.0.0.1:10051
        ForceCommand /bin/false

 

Setup for clients (remote server)

On the server that we want to monitor, we'll need to install autossh to set up the persisted tunnel and of course the zabbix agent itself. Start by installing both packages and configuring autossh. The autossh tunnel is set to use heartbeats rather than a separate monitoring port (-M 0). If we're running on a non-standard ssh port, we can set it using -p. This example will work on Debian like distributions. 

$ sudo apt-get install zabbix-agent autossh

#Add a service description, use the ssh key, zabbix server name and port 
$ sudo vi /etc/systemd/system/autossh-zabbix-tunnel.service

[Unit]
Description=AutoSSH tunnel service for Zabbix agent
After=network.target

[Service]
Environment="AUTOSSH_GATETIME=0"
ExecStart=/usr/bin/autossh -i /home/zabbix/.ssh/id_rsa \
                                           -N -M 0 -o "ServerAliveInterval 30" \
                                           -o "ServerAliveCountMax 3" \
                                           -L 10051:127.0.0.1:10051 \ 
                                           -p <PORT> zabbix-agent@<Zabbix SERVER> 

[Install]
WantedBy=multi-user.target

#Reload the systemd daemon and start the service
$ sudo systemctl daemon-reload
$ sudo service autossh-zabbix-tunnel start
$ sudo service autossh-zabbix-tunnel status

#Test the tunnel
$ telnet localhost 10051

Once the tunnel is set up and active, we can configure the zabbix agent to connect on the local host rather than having to go over the public internet. The Zabbix-agent configuration we're looking for is the Server Active setting, since we are only allowing the agent to connect to the Zabbix server, not to be polled passively. Because of this we can prevent passive agents from spawning and to be safe, set the ip address the agent listens to to localhost only.

Edit the /etc/zabbix/zabbix_agentd.conf file and set the relevant parameters. Make sure to set the Hostname to be the same as the host that is configured in Zabbix itself. 

##### Passive checks related
Server=127.0.0.1
# ListenPort=10050
ListenIP=127.0.0.1
StartAgents=0

##### Active checks related
ServerActive=127.0.0.1
Hostname=<Hostname as set in Zabbix>

Restart the zabbix agent, sudo service zabbix-agent restart

Configure the host in Zabbix

In the Zabbix configuration, add the host, making sure to set the host name we configured in the Zabbix agent. This way, Zabbix will know what monitored server is contacting it. 

We can set the interface to ZBX, but using localhost. When the host is setup, make sure to add templates using Zabbix agent (active). Once some items have been added that use the active agent, Zabbix will request these items the first time the host contacts the Zabbix server. 

 

 

Department