One of the most widely used tools for monitoring resources is Prometheus. Let’s see how to set it up on AWS EC2 and how the Prometheus architecture would look like.

Introduction

Prometheus software is a monitoring solution that is responsible for collecting the metrics data and gives us the ability to visualize the collected data. Still, we will be using Grafana today, as it is more focused on the data visualization part.

Prometheus works in such a way that it scrapes the specific target whose HTTP endpoint is exposed; the scraped metrics from the target are then stored in a time series database that can be queried using PromQl (Prometheus query language). When you install prometheus it allows you to generate alerts when the metrics reach the specified threshold. Imagine Prometheus in such a way that your older sister tells you to have a look at what your mom is making and give her updates every 10 minutes. The act of you checking the kitchen is similar to how Prometheus is scraping metrics from the specified targets at regular intervals. Just as reporting her back to your sister is similar to how Prometheus stores the collected data and makes it available for you, if anything goes wrong (like the soup boiling over), you alter your sister immediately; similarly, Prometheus can trigger alerts.

The metrics scraped by Prometheus can be visualised using Grafana. Grafana provides us with lots of amazing dashboards for data visualization. Data can be visualized in the form of charts and graphs, and we can also use Grafana to aggregate logs using Grafana Loki 

So, let’s see how we can use, install, and configure Prometheus to monitor our AWS EC2.

Setting up AWS EC2

To answer the question of “how to set up EC2 instance in AWS”, we will be using a single EC2 instance to set up Prometheus, Node Exporter, and Grafana. Let’s start by launching our EC2 server.

Steps by Step Guide:

 

    1. Open the Amazon EC2 console.

    1. Launch an instance.

    1. Choose Ubuntu as the AMI (Amazon Machine Image).

    1. Choose an Instance Type for this blog; we will be using t2.micro

    1. Generate a key pair.

    1. Leave the rest as default; we will be configuring it later.

    1. Next, launch instance.

If you have any trouble setting the EC2 instance, you can always refer to the gif below.

Configuring Security Rules

For running the Prometheus server, node-exporter, and Grafana server, we need to edit inbound rules so that we can access them (as we are using a single instance). To successfully install and configure Prometheus, here are the inbound rules, Follow the steps:-

 

    1. Navigate to the security sections and go to security groups.

    1. Next, click on Edit inbound rules and then Add rule. We will add rules for Prometheus, node-exporter, and Grafana.

    1. Now select the Type to Custom TCP, Port range 9090 for Prometheus, 9100 for Node Exporter, and 3000 for Grafana, CIDR block to 0.0.0.0/0, and add the description.

Then Save the rules, they must look something like this :

 

    1. Now that we have our inbound rules set up, we can now start creating our Prometheus server setup.

Go to your instance and update the package using sudo apt-get update

Installing Prometheus Server in EC2

To answer “How to install Prometheus”, let’s start by creating a new user other than the root user. And making the Prometheus directory under /etc/ and /var/lib/ directory.

#This command is to create user prometheus without the home directory
sudo useradd --no-create-home prometheus
#Making prometheus directory under etc
sudo mkdir /etc/prometheus
#Making prometheus directory under lib
sudo mkdir /var/lib/prometheus

We are using –no-create-home user because we are going to use the prometheus user for a specific purpose, i.e., for using it as a service. We do not want it to interact with the user for any kind of login or personal use. So it is better to use this than using the root user for everything.

Once our directory is set up, we need to install Prometheus. To download Prometheus, go to DOWNLOAD and copy the link.

Or just use the following command to install Prometheus 2.48.0; it will download the tar file.

# To download Prometheus 2.48.0
wget
https://github.com/prometheus/prometheus/releases/download/v2.48.0/prometheus-2.48.0.linux-amd64.tar.gz

To unzip the tar file tar xvfz prometheus-2.48.0.linux-amd64.tar.gz use the following command, and it will generate the untared file.

To check the untared, use the ls command.

You were using the bare matel; you can just go into the prometheus-2.48.0.linux-amd64 directory and run ./prometheus to start your server. You can also install it in your EC2 instance, but we will install it in our system so that we can use systemctl commands.

The following commands will be used to install the Prometheus toolkit in our system (command is explained using a comment):-

#Copy the Prometheus binary to /usr/local/bin
sudo cp prometheus-2.48.0.linux-amd64/prometheus /usr/local/bin

#Copy promtool to /usr/local/bin/
sudo cp prometheus-2.48.0.linux-amd64/promtool /usr/local/bin/

# Copy the consoles directory to /etc/prometheus
sudo cp -r prometheus-2.48.0.linux-amd64/consoles /etc/prometheus

#Copy the 'console_libraries' directory to /etc/prometheus
sudo cp -r prometheus-2.48.0.linux-amd64/console_libraries /etc/prometheus

sudo cp prometheus-2.48.0.linux-amd64/promtool /usr/local/bin/

#Remove the Prometheus archive and extracted directory
rm -rf prometheus-2.48.0.linux-amd64.tar.gz prometheus-2.48.0.linux-amd64

Now we need to add a prometheus.yml file that will have the configuration file for the prometheus.

sudo vi /etc/prometheus/prometheus.yml

Let’s add the configurations once we are in our newly created prometheus.yml.

global:
  scrape_interval: 15s
  external_labels:
    monitor: 'prometheus'
scrape_configs:
  - job_name: 'prometheus'
    static_configs:      - targets: ['localhost:9090']

Let’s see what this configuration is :

 

    • Global Configuration(global)

    • scrape_interval: This sets the interval at which Prometheus scrapes targets for metrics. In this case, it is set to 15 seconds (15s).

    • external_labels: These are labels that are applied to all scraped metrics. In this case, there is one label named monitor with the value ‘prometheus’.

    • Scrape Configurations (scrape_configs):

    • – job_name: ‘prometheus’: This is the name of the job. In this case, it is named ‘prometheus’.

    • static_configs: Defines a list of targets to scrape metrics from.

    • – targets: [‘localhost:9090‘]: This specifies that Prometheus should scrape metrics from the target, where presumably the Prometheus server itself is running.

Now that our configuration of Prometheus is done, we will have to add a service file that will define how our Prometheus service will be managed by systemd.

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]

User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Let’s understand what we are doing here:

 

    1. Unit Section ([Unit]):

       

        • Description: is a human-readable description of the service, indicating that this unit is for Prometheus.

        • Wants: Indicates a dependency on network-online.target, meaning that this service should not start until network services are available.

    1. Service Section ([Service]):

       

        • User=prometheus: Specifies the user under which the Prometheus service should run. This is likely the user you created earlier for Prometheus.

        • Group=prometheus: Specifies the group under which the Prometheus service should run.

        • Type=simple: Indicates that the service type is a simple process that does not fork.

        • ExecStart: Specifies the command to start the Prometheus service. It includes the path to the prometheus binary and various command-line options:

           

            • –config.file: Specifies the path to the Prometheus configuration file.

            • –storage.tsdb.path: Specifies the path to the directory where Prometheus should store its time-series database.

            • –web.console.templates and –web.console.libraries: Specify the paths for the web console templates and libraries.

    1. Install Section ([Install]):

       

Finally, we will change the file permission we just created:-

sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
sudo chown -R prometheus:prometheus /var/lib/prometheus

Next, is to reload system configuration, so that it could take the new configuration that we added. Then we will enable the Prometheus service to start at the boot. Lastly, we will start the service.

To check the status of the service is running or not, use sudo systemctl status prometheus.

Hooray! Prometheus service is successfully configured. Let’s check how it looks:

Just go to the new tab and enter <your-public-ip>:9090.

Yay!! We successfully configured the Prometheus service.

Conclusion

And, there you have it! We were able to successfully set up Prometheus on the EC2 instance. Just remember, this is the beginning. The world of cloud monitoring is vast and constantly evolving. There are always new metrics to track, new issues to troubleshoot, and new ways to optimise your cloud resources. We would suggest you check out our other blogs too, on the CloudZenia website, like Step-by-Step Tutorial: Monitoring AWS Resources with Prometheus and Grafana.

Loads of gratitude for reading our blog. Keep exploring and learning!

Ujjwal Mahar

Linkedin

Dec 11, 2023