Today, I gonna write about how to set up server monitoring with Prometheus node exporter. Just doing the setup process is quite easy and there are tons of tutorials out there on the internet. So I am going to write about why I plan to use it. How I try to explore, learn.

Why I need to use Prometheus and it concept

Normally, we are checking our server statues with like top or htop commands. Or check from dashboards of AWS EC2 or DigitalOcean base on what you use. Those are okay at the beginning, but when you got lots of servers it’s quite a problem. There is no easy way to check all the servers statues easily.

We use Laravel Forge to manage our EC2 instances (servers). Forge added metric monitoring. I haven’t tried it out yet. But, base on the blog post it’s not really what I am looking for. But, I remember learning Prometheus from one of Lynda course.

I am reading the docs and checking out tutorials and found out Prometheus is the one I am looking for. After that, I try to setup Prometheus on my newly created EC2 instance. Before we go to the installation stage, let me explain a bit of the Prometheus flow.

Prometheus is pulling the data from the applications or services which expose Prometheus readable format data. For example, you have Server A. Inside there, you installed Prometheus. And inside that server you have an application, let’s name that as A-APP. From that A-APP, it exposes the application’s metrics via HTTP endpoint /metrics and from that endpoint, Prometheus collects the metrics which we configure when running the Prometheus. I hope that makes sense.

If my explanation is not very clear you should watch the following YouTube video. It makes me confused at first, but by watching the following YouTube video. I understand the concept very clearly.

Setting up Prometheus and node_exporter

Setting up Prometheus was quite easy. I created an EC2 instance and ssh into the server and start setting up Prometheus by running the following commands

# Download the prometheus
$ curl -LO https://github.com/prometheus/prometheus/releases/download/v2.20.1/prometheus-2.20.1.linux-amd64.tar.gz

# Extract the tar
$ tar xvf prometheus-2.20.1.linux-amd64.tar.gz
# Move prometheus and promtool to /usr/local/bin
$ sudo cp prometheus-2.20.1.linux-amd64/prometheus /usr/local/bin/
$ sudo cp prometheus-2.20.1.linux-amd64/promtool /usr/local/bin/

Setting up node_exporter

After you finish installing Prometheus, you need to install the exporter. You can run the following command and set it up.

# Download the latest node_exporter
$ curl -LO https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz
# Extract the tar
$ tar xvf node_exporter-1.0.1.linux-amd64.tar.gz
# Move node_expoter under /usr/local/bin
sudo cp node_exporter-1.0.1.linux-amd64/node_exporter /usr/local/bin

When it’s done, you will need to configure node_expoter like the following.

# Create a new user (node_exporter) without creating home and give permission to run the node_expoter
$ sudo useradd --no-create-home --shell /bin/false node_exporter 
$ sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter 

# Add node_expoter configratuon
$ sudo vim /etc/systemd/system/node_exporter.service
---Start of node_exporter service---
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
---End of node_exporter service---

# Reload and start node_expoter
$ sudo systemctl daemon-reload
$ sudo systemctl start node_exporter
$ sudo systemctl enable node_exporter

# Remove downloaded files which no logner required 
$ rm -rf node_exporter-1.0.1.linux-amd64 node_exporter-1.0.1.linux-amd64.tar.gz

# Check everything is all right
$ sudo systemctl status node_exporter

Configure Prometheus to collect data from expoters

Now you need to configure Prometheus to collect data from your node exporter. In order to do that, let’s configure Prometheus like the following

# Add prometheus configratuon
$ sudo vim /etc/prometheus/prometheus.yml

---Start of prometheus configuration---
global:
  scrape_interval: 5s
scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100']
        labels:
          instance: 'node_exporters'
---End of prometheus configuration---

By checking the above code block’s config there are 2 jobs. One is Prometheus by itself and another one was by node_exporter.

The above process is setting up one machine. I mean we are collecting data from our own Prometheus server. If you want to collect information from other services and applications. You need to set up the exporter on those machines and need to collect from those services.

In my case, I want to collect data from another remote machine which are under the same VPC. So go into the server and setup node-exporter. And enabled port 9100 on the firewall. You will also need to allow the port in the AWS security group. Otherwise, you won’t be able to access it from another server. Even it’s on the same security group and VPC. I was very confused about this step and stuck for a while when I was setting up. I even asked questions on Stackoverflow.

After you finish setting up your stuff, you can update your Prometheus config and start running the Prometheus like the following.

# Create a new user (prometheus) without creating home and give permission to run the prometheus

$ sudo useradd --no-create-home --shell /bin/false prometheus 
$ sudo chown prometheus:prometheus /etc/prometheus
$ sudo chown prometheus:prometheus /var/lib/prometheus

# Add prometheus configratuon
$ sudo vim /etc/prometheus/prometheus.yml

-------------- Start -----------------
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target
-------------- End -----------------

# Reload and start prometheus
$ sudo systemctl daemon-reload
$ sudo systemctl start prometheus
$ sudo systemctl status prometheus
$ sudo systemctl enable prometheus

# Remove downloaded files which no logner required 
$ rm -rf prometheus-2.20.1.linux-amd64.tar.gz
$ rm -rf prometheus-2.20.1.linux-amd64

# Check everything is all right
$ sudo systemctl status prometheus

Currently, Prometheus is running on port 9090. If you check yourip:9090. You will see the UI like the following

Setting up Grafana

Now, let’s setup the Grafana like the following

# Install grafana
$ sudo apt-get install -y adduser libfontconfig1
$ wget https://dl.grafana.com/oss/release/grafana_7.1.0~beta2_amd64.deb
$ sudo dpkg -i grafana_7.1.0~beta2_amd64.deb

# Start grafana
$ sudo systemctl daemon-reload && sudo systemctl enable grafana-server && sudo systemctl start grafana-server

Now, Grafana is starting on port 3000. The default username and password is admin.

Now click the setting from the left side…

Click Data Sources and choose Prometheus from the list.

And fill up URL as localhost:9090 and click Save & Test. You will see “Data source is working”.

Now, it’s all ready. You can choose lots of the dashboard from the Grafana or you can build your own if you wanted. For me, I use this Dashboard. It looks like the following…

I hope you understand the whole flow. Feel free to ask questions on comments if you are not clear.