Monitoring Linux logs is crucial and every DevOps engineer should know how to do it. Here’s why :
- You have real-time visual feedback about your logs : probably one of the key aspects of log monitoring, you can build meaningful visualizations (such as datatables, pies, graphs or aggregated bar charts) to give some meaning to your logs.
- You are able to aggregate information to build advanced and more complex dashboards : sometimes raw information is not enough, you may want to join it with other logs or to compare it with other logs to identify a trend. A visualization platform with expression handling lets you perform that.
- You can quickly filter for a certain term, or given a certain time period : if you are only interested in SSH logs, you can build a targeted dashboard for it.
- Logs are navigable in a quick and elegant way : I know the pain of tailing and greping your logs files endlessly. I’d rather have a platform for it.
The prerequisites for this tutorial are as follows :
- You have a Linux system with rsyslog installed. You either have a standalone machine with rsyslog, or a centralized logging system.
- You have administrator rights or you have enough rights to install new packages on your Linux system.
What does a log monitoring architecture looks like?
a – Key concepts of Linux logging
Before detailing how our log monitoring architecture looks like, let’s go back in time for a second.
Historically, Linux logging starts with syslog.
Syslog is a protocol developed in 1980 which aims at standardizing the way logs are formatted, not only for Linux, but for any system exchanging logs.
From there, syslog servers were developed and were embedded with the capability of handling syslog messages.
They rapidly evolved to functionalities such as filtering, having content routing abilities, or probably one of the key features of such servers : storing logs and rotating them.
Rsyslog was developed keeping this key functionality in mind : having a modular and customizable way to handle logs.
The modularity would be handled with modules and the customization with log templates.
In a way, rsyslog can ingest logs from many different sources and it can forward them to an even wider set of destinations. This is what we are going to use in our tutorial.
b – Building a log monitoring architecture
Here’s the final architecture that we are going to use for this tutorial.
- rsyslog : used as an advancement syslog server, rsyslog will forward logs to Logstash in the RFC 5424 format we described before.
- Logstash : part of the ELK stack, Logstash will transform logs from the syslog format to JSON. As a reminder, ElasticSearch takes JSON as an input.
- ElasticSearch : the famous search engine will store logs in a dedicated log index (logstash-*). ElasticSearch will naturally index the logs and make them available for analyzing.
- Kibana : used as an exploration and visualization platform, Kibana will host our final dashboard.
Before installing the ELK stack, you need to install Java on your computer.
To do so, run the following command:
$ sudo apt-get install default-jre
At the time of this tutorial, this instance runs the OpenJDK version 11.
ubuntu:~$ java -version
openjdk version "11.0.3" 2019-04-16
OpenJDK Runtime Environment (build 11.0.3+7-Ubuntu-1ubuntu218.04.1)
OpenJDK 64-Bit Server VM (build 11.0.3+7-Ubuntu-1ubuntu218.04.1, mixed mode, sharing)b – Adding Elastic packages to your instance
For this tutorial, I am going to use a Ubuntu machine but details will be given for Debian ones.
First, add the GPG key to your APT repository.
$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Then, you can add Elastic source to your APT source list file.
$ echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
$ cat /etc/apt/sources.list.d/elastic-7.x.list
deb https://artifacts.elastic.co/packages/7.x/apt stable main
$ sudo apt-get update
From there, you should be ready to install every tool in the ELK stack.
Let’s start with ElasticSearch.
c – Installing ElasticSearch
ElasticSearch is a search engine built by Elastic that stores data in indexes for very fast retrieval.
To install it, run the following command:
$ sudo apt-get install elasticsearch
The following command will automatically :
- Download the deb package for ElasticSearch;
- Create an elasticsearch user;
- Create an elasticsearch group;
- Automatically create a systemd service fully configured (inactive by default)
$ sudo systemctl start elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
Active: active (running) since Mon 2019-07-08 18:19:45 UTC; 2 days ago
Docs: http://www.elastic.co
In order to make sure that ElasticSearch is actually running, you can execute one of those two commands:
- Watching which applications listen on a targeted port
$ sudo lsof -i -P -n | grep LISTEN | grep 9200
java 10667 elasticsearch 212u IPv6 1159208890 0t0 TCP [::1]:9200 (LISTEN)
java 10667 elasticsearch 213u IPv6 1159208891 0t0 TCP 127.0.0.1:9200 (LISTEN)- Executing a simple ElasticSearch query
$ curl -XGET 'http://localhost:9200/_all/_search?q=*&pretty'
Your ElasticSearch instance is all set!
Now, let’s install Logstash as our log collection and filtering tool.
d – Installing Logstash
If you added Elastic packages previously, installing Logstash is as simple as executing:
$ sudo apt-get install logstash
Again, a Logstash service will be created, and you need to activate it.
$ sudo systemctl status logstash
$ sudo systemctl start logstash
By default, Logstash listens for metrics on port 9600. As we did before, list the open ports on your computer looking for that specific port.
$ sudo lsof -i -P -n | grep LISTEN | grep 9600
java 28872 logstash 79u IPv6 1160098941 0t0 TCP 127.0.0.1:9600 (LISTEN)
Great!
We only need to install Kibana for our entire setup to be complete.
e – Installing Kibana
As a reminder, Kibana is the visualization tool tailored for ElasticSearch and used to monitor our final logs.
Not very surprising, but here’s the command to install Kibana:
$ sudo apt-get install kibana
As usual, start the service and verify that it is working properly.
$ sudo systemctl start kibana
$ sudo lsof -i -P -n | grep LISTEN | grep 5601
node 7253 kibana 18u IPv4 1159451844 0t0 TCP *:5601 (LISTEN)
Kibana Web UI is available on port 5601.
Head over to http://localhost:5601 with your browser
V – Routing Linux Logs to ElasticSearch
As a reminder, we are routing logs from rsyslog to Logstash and those logs will be transferred to ElasticSearch pretty much automatically.
a – Routing from Logstash to ElasticSearch
Before routing logs from rsyslog to Logstash, it is very important that we setup log forwarding between Logstash and ElasticSearch.
To do so, we are going to create a configuration file for Logstash and tell it exactly what to do.
To create Logstash configuration files, head over to /etc/logstash/conf.d and create a logstash.conf file.
Inside, append the following content:
input {
udp {
host => "127.0.0.1"
port => 10514
codec => "json"
type => "rsyslog"
}
}
# The Filter pipeline stays empty here, no formatting is done. filter { }
# Every single log will be forwarded to ElasticSearch. If you are using another port, you should specify it here.
output {
if [type] == "rsyslog" {
elasticsearch {
hosts => [ "127.0.0.1:9200" ]
}
}
}
Note : for this tutorial, we are using the UDP input for Logstash, but if you are looking for a more reliable way to transfer your logs, you should probably use the TCP input. The format is pretty much the same, just change the UDP line to TCP.
Restart your Logstash service.
$ sudo systemctl restart logstash
To verify that everything is running correctly, issue the following command:
$ netstat -na | grep 10514
udp 0 0 127.0.0.1:10514 0.0.0.0:*
Logstash is now listening on port 10514.
b – Routing from rsyslog to Logstash
As described before, rsyslog has a set of different modules that allow it to transfer incoming logs to a wide set of destinations.
Rsyslog has the capacity to transform logs using templates. This is exactly what we are looking for as ElasticSearch expects JSON as an input, and not syslog RFC 5424 strings.
In order to forward logs in rsyslog, head over to /etc/rsyslog.d and create a new file named 70-output.conf
Inside your file, write the following content:
# This line sends all lines to defined IP address at port 10514
# using the json-template format.
*.* @127.0.0.1:10514;json-template
Now that you have log forwarding, create a 01-json-template.conf file in the same folder, and paste the following content:
template(name="json-template"
type="list") {
constant(value="{")
constant(value="\"@timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"@version\":\"1")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\",\"sysloghost\":\"") property(name="hostname")
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
constant(value="\",\"programname\":\"") property(name="programname")
constant(value="\",\"procid\":\"") property(name="procid")
constant(value="\"}\n")
}
As you probably guessed it, for every incoming message, rsyslog will interpolate log properties into a JSON formatted message, and forward it to Logstash, listening on port 10514.
Restart your rsyslog service, and verify that logs are correctly forwarded to ElasticSearch.
Note : logs will be forwarded in an index called logstash-*.
$ sudo systemctl restart rsyslog
$ curl -XGET 'http://localhost:9200/logstash-*/_search?q=*&pretty'
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 10000,
"relation": "gte"
},
"max_score": 1,
"hits": [
{
"_index": "logstash-2019.07.08-000001",
"_type": "_doc",
"_id": "GEBK1WsBQwXNQFYwP8D_",
"_score": 1,
"_source": {
"host": "127.0.0.1",
"severity": "info",
"programname": "memory_usage",
"facility": "user",
"@timestamp": "2019-07-09T05:52:21.402Z",
"sysloghost": "schkn-ubuntu",
"message": " Dload Upload Total Spent Left Speed",
"@version": "1",
"procid": "16780",
"type": "rsyslog"
}
}
]
}
}
It is time for us to build our final dashboard in Kibana.
VI – Building a Log Dashboard in Kibana
We are going to build the dashboard shown in the first part and give meaning to the data we collected.
Similarly to our article on Linux process monitoring, this part is split according to the different panels of the final dashboard, so feel free to jump to the section you are interested in.
a – A Few Words On Kibana
Head over to Kibana (on http://localhost:5601)
If it is your first time using Kibana, there is one little gotcha that I want to talk about that took me some time to understand.
In order to create a dashboard, you will need to build visualizations. Kibana has two panels for this, one called “Visualize” and another called “Dashboard”
In order to create your dashboard, you will first create every individual visualization with the Visualize panel and save them.
When all of them will be created, you will import them one by one into your final dashboard.
b – Aggregated bar chart for processes
To build your first dashboard, click on “Create new visualization” at the top right corner of Kibana. Choose a vertical bar panel.
As you can see, the bar chart provides a total count of logs per processes, in an aggregated way.
The bar chart can also be split by host if you are working with multiple hosts.
c – Pie by program name
Very similarly to what we have done before, the goal is to build a pie panel that divides the log proportions by program name.
d – Pie by severity
This panel looks exactly like the one we did before, except that it splits logs by severity.
It can be quite useful when you have a major outage on one of your systems, and you want to quickly see that the number of errors is increasing very fast.
It also provides an easy way to see your log severity summary on a given period if you are interested for example in seeing what severities occur during the night or for particular events.
e – Monitoring SSH entries
This one is a little bit special, as you can directly go in the “Discover” tab in order to build your panel.
When entering the discover tab, your “logstash-*” should be automatically selected.
From there, in the filter bar, type the following filter “programname : ssh*”.
As you can see, you now have a direct access to every log related to the SSHd service on your machine. You can for example track illegal access attempts or wrong logins.
In order for it to be accessible in the dashboard panel, click on the “Save” option, and give a name to your panel.
Now in the dashboard panel, you can click on “Add”, and choose the panel you just created.