Amazon Managed Service for Prometheus

 Prometheus is the most widely used monitoring service with overwhelming adoption from the community and integrations with all the major native applications technologies. Amazon Managed Service for Prometheus uses the same open-source Prometheus data model and query language that we use to monitor the performance of our containerized workloads, with the added advantage of improved scalability, availability, and security without having to manage the underlying infrastructure. 

Benefits of Amazon Managed Service for Prometheus:- 

  1. It automatically scales the ingestion, storage, and querying of operational metrics as workloads scale up and down across different availability zones in a single region
  2. It integrates with AWS security services to enable fast and secure access to data using AWS Privatelink & AWS Identity and Access Management.
  3.  Amazon Managed Service for Prometheus encrypts the data that it stores with AWS Key Management Service (AWS KMS) keys. Amazon Managed Service for Prometheus manages these keys. Data in transit is encrypted with HTTPS automatically.
  4. It uses the popular open-source Prometheus data model and PromQL query language to filter, aggregate, and alarm on metrics and quickly gain performance visibility without any code changes.

Integrate Amazon Managed Service for Prometheus with Elastic Kubernetes services

There are three stages in setting up Amazon Managed Service for Prometheus:-

  • Create Workspace
  • Ingest Prometheus Metric to the workspace
  • Query Prometheus Metric

Creating a Workspace

This is a logical space dedicated to the storage and querying of Prometheus metrics. A workspace supports fine-grained access control for authorizing its management such as update, list, describe, and delete, and the ingestion and querying of metrics. 

Note:- Make notes of the URLs displayed for Endpoint — remote write URL and Endpoint — query URL. You’ll need them when you configure your Prometheus server to remote write metrics to this workspace and when you query those metrics.

We can also create a Workspace through AWS CLI using these command.

aws amp create-workspace [ — alias my-first-workspace]

This command returns the following data:

  • WorkspaceId is the unique ID for this workspace. Make a note of this ID.
  • arn is the ARN for this workspace.

Ingest Prometheus metrics to the workspace

 Amazon Managed Service for Prometheus can collect Prometheus metrics from Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Container Service (Amazon ECS), and Amazon Elastic Kubernetes Service (Amazon EKS) environments using AWS Distro for OpenTelemetry (ADOT) or Prometheus servers as collection agents. The detailed instructions in this section are for a Prometheus server in an Amazon EKS cluster.

Requirements:- 

  • Amazon EKS cluster where the new Prometheus server will collect metrics
  • Helm Package Manager should be installed
Step 1: Add new Helm chart repositories

To add new Helm chart repositories, enter the following commands.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add kube-state-metrics https://kubernetes.github.io/kube-state-metrics
helm repo update

Step 2: Create a Prometheus namespace

Enter the following command to create a Prometheus namespace for the Prometheus server. Replace prometheus-namespace with the name that you want for this namespace.

kubectl create namespace prometheus-namespace
Step 3: Set up IAM roles for service accounts

Set up service roles for the ingestion of metrics from Amazon EKS clusters by running the below script. Create a file named createIRSA-AMPIngest.sh with the following content. Replace <my_amazon_eks_clustername> with the name of your cluster, and replace <my_prometheus_namespace> with your Prometheus namespace.

You can find the Script here in this link

https://docs.aws.amazon.com/prometheus/latest/userguide/set-up-irsa.html#:~:text=To%20set%20up%20the%20service%20role%20for%20ingestion%20into%20Amazon%20Managed%20Service%20for%20Prometheus

Step 4 :Install a new Prometheus server to send metrics to your Amazon Managed Service for Prometheus workspace

Use a text editor to create a file named my_prometheus_values_yaml with the following content.

  • Replace IAM_PROXY_PROMETHEUS_ROLE_ARN with the ARN of the amp-iamproxy-ingest-role that you created in Setup service roles for the ingestion of metrics from Amazon EKS clusters.
  • Replace WORKSPACE_ID with the ID of your Amazon Managed Service for Prometheus workspace.
  • Replace AWS_REGION with the Region of your Amazon Managed Service for Prometheus workspace.
## The following is a set of default values for prometheus server helm chart which enable remoteWrite to AMP
## For the rest of prometheus helm chart values see: https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus/values.yaml
##
serviceAccounts:
server:
name: amp-iamproxy-ingest-service-account
annotations:
eks.amazonaws.com/role-arn: ${IAM_PROXY_PROMETHEUS_ROLE_ARN}
server:
remoteWrite:
- url: https://aps-workspaces.${AWS_REGION}.amazonaws.com/workspaces/${WORKSPACE_ID}/api/v1/remote_write
sigv4:
region: ${AWS_REGION}
queue_config:
max_samples_per_send: 1000
max_shards: 200
capacity: 2500

Enter the following command to create the Prometheus server.

  • Replace prometheus-chart-name with your Prometheus release name.
  • Replace prometheus-namespace with the name of your Prometheus namespace.
helm install prometheus-chart-name prometheus-community/prometheus -n prometheus-namespace \ -f my_prometheus_values_yaml

Query your Prometheus metrics

Once the metrics are ingested into the Prometheus workspace, we can query them using a service like Grafana or you can use Amazon Managed Service for Prometheus APIs. You perform your queries using the standard Prometheus query language, PromQL. Amazon Managed Service for Prometheus supports the use of Grafana version 7.3.5 and later to query metrics in a Amazon Managed Service for Prometheus workspace. Versions 7.3.5 and later include support for AWS Signature Version 4 (SigV4) authentication.

Step 1. Set up AWS SigV4

Grafana has added a new feature to support AWS Signature Version 4 (SigV4) authentication. This feature is not enabled by default on Grafana servers. The following instructions for enabling this feature assume that you are using Helm to deploy Grafana on a Kubernetes cluster.

To enable SigV4 on your Grafana 7.3.5 or later server

  1. Create a new update file to override your Grafana configuration, and name it amp_query_override_values.yaml.
  2. Enter the following content into the file, and save the file. Replace account-id with the AWS account ID where the Grafana server is running.
serviceAccount:
name: "amp-iamproxy-query-service-account"
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::account-id:role/amp-iamproxy-query-role"
grafana.ini:
auth:
sigv4_auth_enabled: true

In that above YAML file content, amp-iamproxy-query-role is the name of the role that you will create in the next section,

Step2. Set up IAM roles for service accounts for the querying of metrics

To set up service roles for the querying of Amazon Managed Service for Prometheus metrics we can run the below script

  1. Create a file named createIRSA-AMPQuery.sh with the following content. Replace <my_amazon_eks_clustername> with the name of your cluster, and replace <my_prometheus_namespace> with your Prometheus namespace.

You can find the script in this link

https://docs.aws.amazon.com/prometheus/latest/userguide/set-up-irsa.html#:~:text=To%20set%20up%20service%20roles%20for%20the%20querying%20of%20Amazon%20Managed%20Service%20for%20Prometheus%20metrics%3B

You then need to add the Grafana service account in the conditions of the trust relationship. From a terminal window, determine the namespace and the service account name for your Grafana server. For example, you could use the following command.

kubectl get serviceaccounts -n grafana_namespace
  1. In the Amazon EKS console, open the IAM role for service accounts that is associated with the EKS cluster.
  2. Choose Edit trust relationship.
  3. Update the Condition to include the Grafana namespace and the Grafana service account name that you found in the output of the command
Step3. Upgrade the Grafana server using Helm

This step upgrades the Grafana server to use the entries that you added to the amp_query_override_values.yaml file in the previous section.

Run the following commands.

helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade --install grafana grafana/grafana -n grafana_namespace -f ./amp_query_override_values.yaml
Step4. Add the Prometheus data source in Grafana

The following steps explain how to set up the Prometheus data source in Grafana to query your Amazon Managed Service for Prometheus metrics.

To add the Prometheus data source in your Grafana server

  • Open the Grafana console.
  • Under Configurations, choose Data sources.
  • Choose Add data source

  • Choose Prometheus.
    • For the HTTP URL, specify the Endpoint — query URL displayed in the workspace details page in the Amazon Managed Service for Prometheus console. Note :- For the sake of simplicity , the network traffic for querying metrics in Amazon Managed Service for Prometheus has been done over a public internet endpoint instead of Private Endpoint)
    • In the HTTP URL that you just specified, remove the /api/v1/query string that is appended to the URL, because the Prometheus data source will automatically append it.
    • Under Auth, select the toggle for SigV4 Auth to enable it.
    • Leave the Assume Role ARN and External ID fields blank. Then for Default Region, select the Region where your Amazon Managed Service for Prometheus workspace is.
    • Choose Save & Test.
    • You should see the following message: Data source is working
    • Test a PromQL query against the new data source:
    • Choose Explore.
    • Run a sample PromQL query such as:
    prometheus_tsdb_head_series

    If everything is working you should see the data coming from Amazon Prometheus .

    Note:- I have collected this information from AWS official documentation and try to concise it for the readers. Please refer this for detailed understanding of AWS services
    https://docs.aws.amazon.com/