What is Prometheus?

Prometheus is an open source monitoring tool which was developed in 2012. Its architecture is modular and comes with several readily available modules called exporters, which help you capture metrics from popular software. Prometheus is written in the Go language and stores its captured data in a time series database (A time series database is optimized for handling data points indexed by time). Prometheus offers a versatile and customizable interface allowing system administrators to heavily tune the metrics they want to measure and also enhance the accuracy of measurement.

Prometheus’s main features are:

  • a multi-dimensional data model with time series data identified by metric name and key/value pairs
  • flexible query language to leverage this dimensionality
  • no reliance on distributed storage; single server nodes are autonomous
  • time series collection happens via a pull model over HTTP
  • pushing time series is supported via an intermediary gateway
  • targets are discovered via service discovery or static configuration
  • multiple modes of graphing and dashboard creation support

Basic architecture Prometheus

The below diagram illustrates the basic architecture of the Prometheus ecosystem and where different components fit together. Note that each Prometheus server is standalone, not depending on network storage or other remote services.

With a basic understanding of Prometheus architecture, we now proceed to install and set it up on a Centos 7 system.

Step 1: Download Prometheus software from GitHub.

We’ll use wget to download Prometheus from GitHub directly onto the server.

[[email protected] prometheus]# wget "https://github.com/prometheus/prometheus/releases/download/0.15.1/prometheus-0.15.1.linux-amd64.tar.gz"
--2017-11-03 09:48:30--  https://github.com/prometheus/prometheus/releases/download/0.15.1/prometheus-0.15.1.linux-amd64.tar.gz
Resolving github.com (github.com)... 192.30.253.113, 192.30.253.112
Connecting to github.com (github.com)|192.30.253.113|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/6838921/e49fcd78-3488-11e5-8106-8935f61220fd?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20171103%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171103T134831Z&X-Amz-Expires=300&X-Amz-Signature=5d53ba71db52365395087abb1fa273ca6954b1dec3347623e9f74a63f5ca0728&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dprometheus-0.15.1.linux-amd64.tar.gz&response-content-type=application%2Foctet-stream [following]
--2017-11-03 09:48:31--  https://github-production-release-asset-2e65be.s3.amazonaws.com/6838921/e49fcd78-3488-11e5-8106-8935f61220fd?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20171103%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171103T134831Z&X-Amz-Expires=300&X-Amz-Signature=5d53ba71db52365395087abb1fa273ca6954b1dec3347623e9f74a63f5ca0728&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dprometheus-0.15.1.linux-amd64.tar.gz&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.86.43
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.86.43|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 7518280 (7.2M) [application/octet-stream]
Saving to: ‘prometheus-0.15.1.linux-amd64.tar.gz’
100%[==================================================================================================================================================================>] 7,518,280    161KB/s   in 55s
2017-11-03 09:49:29 (134 KB/s) - ‘prometheus-0.15.1.linux-amd64.tar.gz’ saved [7518280/7518280]
[[email protected] prometheus]#

Now uncompress and extract the downloaded file.

[[email protected] prometheus]# tar -zxvf prometheus-0.15.1.linux-amd64.tar.gz
prometheus
promtool
consoles/
consoles/cassandra.html
consoles/haproxy-backends.html
consoles/aws_elb.html
consoles/aws_elasticache.html
consoles/aws_redshift-cluster.html
consoles/haproxy.html
consoles/federation_template_example.txt
consoles/node.html
consoles/node-overview.html
consoles/haproxy-frontends.html
consoles/aws_redshift.html
consoles/haproxy-frontend.html
consoles/node-disk.html
consoles/node-cpu.html
consoles/haproxy-backend.html
consoles/cloudwatch.html
consoles/index.html.example
console_libraries/
console_libraries/menu.lib
console_libraries/prom.lib

This completes the installation. On typing the ls command you will observe a file named Prometheus extracted in the current working directory. This is the actual Prometheus binary. You may verify your installation of Prometheus by tying “./prometheus –version”

[[email protected] prometheus]# ls
console_libraries  consoles  prometheus  prometheus-0.15.1.linux-amd64.tar.gz  promtool
[[email protected] prometheus]# ./prometheus -version
prometheus, version 0.15.1 (branch: master, revision: 64349aa)
build user:       [email protected]
build date:       20150727-17:56:51
go version:       1.4.2

Step 2: Download the Prometheus node exporter.

As we mentioned earlier Prometheus has a modular architecture and has several modules available for use called exporters. Now we will install one such exporter referred to as the node exporter. The node exporter is an application that runs on a node and collects information at various metrics like memory, I/O and more. It also exposes an endpoint, which the Prometheus server scrapes at regular intervals and collects the metrics.

Related concept:   Review: CloudBerry Backup for Linux/Unix

We’ll now download the node exporter from GitHub.

[[email protected] prometheus]# mkdir -p node_exporter
[[email protected] prometheus]# cd node_exporter
[[email protected] node_exporter]# pwd
/root/prometheus/node_exporter
[[email protected]ix node_exporter]# wget https://github.com/prometheus/node_exporter/releases/download/0.11.0/node_exporter-0.11.0.linux-amd64.tar.gz -O ~/Downloads/node_exporter-0.11.0.linux-amd64.tar.gz
--2017-11-03 10:06:02--  https://github.com/prometheus/node_exporter/releases/download/0.11.0/node_exporter-0.11.0.linux-amd64.tar.gz
Resolving github.com (github.com)... 192.30.253.113, 192.30.253.112
Connecting to github.com (github.com)|192.30.253.113|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/9524057/ee77f366-34ba-11e5-944f-7e3f27ce85ce?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20171103%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171103T140609Z&X-Amz-Expires=300&X-Amz-Signature=01596fe7b6630ed6b6799614908fcf9eb0435d102d3945b9b5d0caa9ef73c986&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dnode_exporter-0.11.0.linux-amd64.tar.gz&response-content-type=application%2Foctet-stream [following]
--2017-11-03 10:06:09--  https://github-production-release-asset-2e65be.s3.amazonaws.com/9524057/ee77f366-34ba-11e5-944f-7e3f27ce85ce?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20171103%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171103T140609Z&X-Amz-Expires=300&X-Amz-Signature=01596fe7b6630ed6b6799614908fcf9eb0435d102d3945b9b5d0caa9ef73c986&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dnode_exporter-0.11.0.linux-amd64.tar.gz&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 54.231.72.243
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|54.231.72.243|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2846737 (2.7M) [application/octet-stream]
Saving to: ‘/root/Downloads/node_exporter-0.11.0.linux-amd64.tar.gz’
100%[==================================================================================================================================================================>] 2,846,737    145KB/s   in 20s
2017-11-03 10:06:35 (136 KB/s) - ‘/root/Downloads/node_exporter-0.11.0.linux-amd64.tar.gz’ saved [2846737/2846737]
[[email protected] node_exporter]# ls
[[email protected] node_exporter]# mv /root/Downloads/node_exporter-0.11.0.linux-amd64.tar.gz .
[[email protected] node_exporter]# pwd
/root/prometheus/node_exporter
[[email protected] node_exporter]# ls
node_exporter-0.11.0.linux-amd64.tar.gz
[[email protected] node_exporter]#

Let’s extract the file.

[[email protected] node_exporter]# tar -zxvf node_exporter-0.11.0.linux-amd64.tar.gz
node_exporter
[[email protected] node_exporter]#

Step 3: Configure node exporter as a service.

To configure node exporter as a service we’ll first create a system service file for this service.

[[email protected]~]# cat /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=root
Group=root
Type=simple
ExecStart=/root/prometheus/node_exporter/node_exporter
[Install]
WantedBy=multi-user.target

I won’t get into the detailed aspects of the service file but I’d like to mention a few points here:

The execStart variable needs to have the path of the executable which is actually the service.

The User and Group variables contain the value of the user owner and the group owner of the binary you’ve mentioned in the ExecStart variable.

If there is a discrepancy between the ownership or the binary file then the service file won’t work.

We’ll now reload system to make it aware that we’ve just added a service file.

[[email protected] ~]# systemctl daemon-reload

Now we’ll start the node exporter service and check it’s status.

[[email protected]~]# systemctl start node_exporter
[[email protected]~]# systemctl status node_exporter
node_exporter.service - Node Exporter
   Loaded: loaded (/etc/systemd/system/node_exporter.service; disabled)
   Active: active (running) since Fri 2017-11-03 11:25:24 EDT; 3s ago
 Main PID: 3078 (node_exporter)
   CGroup: /system.slice/node_exporter.service
           └─3078 /root/prometheus/node_exporter/node_exporter
Nov 03 11:25:24 still node_exporter[3078]: time="2017-11-03T11:25:24-04:00" level=info msg=" - stat" file="node_exporter.go" line=144
Nov 03 11:25:24 still node_exporter[3078]: time="2017-11-03T11:25:24-04:00" level=info msg=" - time" file="node_exporter.go" line=144
Nov 03 11:25:24 still node_exporter[3078]: time="2017-11-03T11:25:24-04:00" level=info msg=" - netstat" file="node_exporter.go" line=144
Nov 03 11:25:24 still node_exporter[3078]: time="2017-11-03T11:25:24-04:00" level=info msg=" - loadavg" file="node_exporter.go" line=144
Nov 03 11:25:24 still node_exporter[3078]: time="2017-11-03T11:25:24-04:00" level=info msg=" - filesystem" file="node_exporter.go" line=144
Nov 03 11:25:24 still node_exporter[3078]: time="2017-11-03T11:25:24-04:00" level=info msg=" - meminfo" file="node_exporter.go" line=144
Nov 03 11:25:24 still node_exporter[3078]: time="2017-11-03T11:25:24-04:00" level=info msg=" - textfile" file="node_exporter.go" line=144
Nov 03 11:25:24 still node_exporter[3078]: time="2017-11-03T11:25:24-04:00" level=info msg=" - netdev" file="node_exporter.go" line=144
Nov 03 11:25:24 still node_exporter[3078]: time="2017-11-03T11:25:24-04:00" level=info msg=" - diskstats" file="node_exporter.go" line=144
Nov 03 11:25:24 still node_exporter[3078]: time="2017-11-03T11:25:24-04:00" level=info msg="Starting node_exporter v0.11.0 at :9100" file="node_exporter.go" line=176
[[email protected] ~]#

The name of the service is same as the name of the service file minus the .service extension.

Related concept:   Review: CloudBerry Backup for Linux/Unix

With that our service for node exporter has been configured and correctly started.

If you don’t want to configure the service file you can start the node exporter directly as shown below:

[[email protected] ~]# /root/prometheus/node_exporter/node_exporter
INFO[0000] No directory specified, see --textfile.directory  file=textfile.go line=40
INFO[0000] Enabled collectors:                           file=node_exporter.go line=142
INFO[0000]  - loadavg                                    file=node_exporter.go line=144
INFO[0000]  - meminfo                                    file=node_exporter.go line=144
INFO[0000]  - time                                       file=node_exporter.go line=144
INFO[0000]  - diskstats                                  file=node_exporter.go line=144
INFO[0000]  - stat                                       file=node_exporter.go line=144
INFO[0000]  - textfile                                   file=node_exporter.go line=144
INFO[0000]  - netdev                                     file=node_exporter.go line=144
INFO[0000]  - netstat                                    file=node_exporter.go line=144
INFO[0000]  - filesystem                                 file=node_exporter.go line=144
INFO[0000] Starting node_exporter v0.11.0 at :9100       file=node_exporter.go line=176

Be sure to execute the binary with & at the end to background it because when the binary executes you don’t actually get the prompt back.

Once the node exporter starts it uses port number 9100 for communication. You can verify that with the netstat command as shown below.

[[email protected] ~]# netstat -tulpn | grep node
tcp6       0      0 :::9100                 :::*                    LISTEN      3078/node_exporter
[[email protected] ~]#

After the service has been started successfully point a web browser to the URL http://your_server_ip:9100/metrics to view the node exporters’ web interface. You should see a static web page with a lot of text on it.

Here’s a screenshot of what I saw:

Step 4: Starting the Prometheus server.

We could set up the Prometheus server also as a systemd service similar to what we did for node exporter earlier but for the sake of keeping the demonstration simple I’ll use it directly here.

Now we had to the directory where we had extracted Prometheus.

[[email protected] prometheus]# pwd
/root/prometheus
[[email protected] prometheus]# ls
console_libraries  consoles  node_exporter  prometheus  prometheus-0.15.1.linux-amd64.tar.gz  promtool

Here we’ll create quick YAML configuration file named prometheus.yml and populate it with the following content.

[[email protected] prometheus]# cat prometheus.yml
scrape_configs:
  - job_name: "node"
    scrape_interval: "10s"
    target_groups:
    - targets: ['localhost:9100']

For those of you unfamiliar with the YAML format I must point out that it’s extremely sensitive to line spacing so be careful.

In the prometheus.yml file we’ve created a scrape_configs section and defines a job called node. It includes the URL of your Node Exporter’s web interface in its array of targets. The scrape_interval is set to 10 seconds so that Prometheus scrapes the metrics once every fifteen seconds. You could name your job anything you want, but calling it “node” allows you to use the default console templates of Node Exporter.

Related concept:   Review: CloudBerry Backup for Linux/Unix

Now let’s start the Prometheus server in the background and also add nohup command so it continues to run after the current session has been terminated.

[[email protected] prometheus]# nohup ./prometheus > prometheus_linuxnix.log 2>&1 &
[1] 4156
[[email protected] prometheus]#

Let’s query the log file.

[[email protected] prometheus]# tail prometheus_linuxnix.log
nohup: ignoring input
prometheus, version 0.15.1 (branch: master, revision: 64349aa)
  build user:       [email protected]
  build date:       20150727-17:56:51
  go version:       1.4.2
time="2017-11-03T11:49:40-04:00" level=info msg="Loading configuration file prometheus.yml" file=main.go line=173
time="2017-11-03T11:49:40-04:00" level=info msg="Loading series map and head chunks..." file=storage.go line=263
time="2017-11-03T11:49:40-04:00" level=info msg="0 series loaded." file=storage.go line=268
time="2017-11-03T11:49:40-04:00" level=info msg="Starting target manager..." file=targetmanager.go line=75
time="2017-11-03T11:49:40-04:00" level=info msg="Listening on :9090" file=web.go line=186

The message “Listening on :9090” implies that the Prometheus server is now up and running.

We’ll just look for the port 9090 in the netstat output just to be sure that the port is open.

[[email protected] prometheus]# netstat -tupln | grep 9090
tcp6       0      0 :::9090                 :::*                    LISTEN      4156/./prometheus
[[email protected] prometheus]#

Now point your browser to the URL http://your_server_ip:9090 to open Prometheus homepage and you should be greeted with something similar to the following screenshot:

Step 5: View metrics scraped by Prometheus.

To verify that Prometheus is gathering data from the node exporter, click on the graph tab on the Prometheus homepage.

On this page, expand the ‘inset metric at cursor’ drop down to select a metric and view data pertaining to it and then click on execute.

Next click on the graph tab and you would be able to see a graph for that metric.

Prometheus has console templates that let you view graphs of a few commonly used metrics. These console templates are accessible only if you set the value of job_name to node in Prometheus’s configuration.

Visit http://your_server_ip:9090/consoles/node.html to access the Node Console.

From here click on localhost:9100, to view its metrics:

Conclusion on Prometheus

This concludes our illustration of a very basic Prometheus installation and setup on a Centos 7 system. One of the interesting features we found was the fact that it’s not based on a server-client architecture making it relatively easy to maneuver as compared to conventional monitoring tools. We hope that from the demonstrations provided in this article you’ve found Prometheus to be a very user-friendly and easy to set up monitoring tool and would consider giving it a try yourself.