Setup Grafana on AWS EKS and integrate with AWS Cloudwatch

By

What is Grafana?

Grafana is open-source visualization and analytics software. It allows you to query, visualize, alert on, and explore your metrics no matter where they are stored. In plain English, it provides you with tools to turn your time-series database (TSDB) data into beautiful graphs and visualizations.

Setup Grafana on Amazon Elastic Kubernetes Service

This blog explains on how to run Grafana on Amazon Elastic Kubernetes cluster and adding cloudwatch as datasource to Grafana. Using Grafana you can simplify Kubernetes monitoring dashboards from CloudWatch metrics. Let’s look into the detailed deployment steps.

Deploying EKS cluster and nodes

We are deploying EKS and required networking resources using CloudFormation template. CloudFormation will create below resources:
  • IAM Role for EKS
  • VPC
  • Two Public Subnets and Two Private Subnets
  • Internet Gateway
  • Two NAT Gateways
  • Public Route Table and its appropriate association
  • Two Private Route Tables
  • Control Plane Security Group
  • EKS
Lets deploy the CloudFormation by cloning the repo  eks-kubernetes-grafana-cloudwatch 

Login to AWS account and the region where you want to deploy this setup and go to CloudFormation Service

Create Stack by uploading the CloudFormation template "eks_cluster_template.yaml

In Next Page, pass the parameters as per your requirement

Click Next and Go to Final Page of CloudFormation and acknowledge the IAM role creation and Create the stack.


Wait for stack to reach create complete state

EKS Cluster is created with all the required configuration. Now we will create the node group for the cluster. Before creating the node group, we need IAM role for the node group.

Go to IAM console and click on create role

Attach below managed policies which are required for EKS nodes
  • AmazonEKSWorkerNodePolicy
  • AmazonEC2ContainerRegistryReadOnly
  • AmazonEKS_CNI_Policy



Once IAM role creation for node is completed, Go to AWS EKS and click on EKS configuration and go to compute and add Node Group

Enter the Node group name and select the role which we created in previous steps

Pass the values as per your requirement on instance Type, size and scaling configuration
After node group is created, you will be seeing the nodes in Ready state

Validate the nodes from your local machine using kubectl


Now all set with Infrastructure and EKS environment, now time to deploy Kubernetes manifests - deployment, data-source and ingress service.  

Deploying Grafana and CloudWatch data source on EKS

We will start with creating an IAM role for Grafana to read the CloudWatch metrics. I have created a role and attached below managed polices. 
  • AmazonEC2ReadOnlyAccess
  • CloudWatchReadOnlyAccess

You can even granularize the permission by looking into the  Grafana official website

Trust the Grafana role with EKS node role

In this demo, Grafana will fetch the CloudWatch metric from the same account. If you want to monitor different AWS Account(s) then in the target account, create IAM role for Grafana and trust the role with EKS node role where Grafana is containerized.

Go to folder where you have clone the repo, All the Kubernetes manifests (YAML files) used in this tutorial are hosted on Github. You can clone it and use it for the setup.
  • Grafana data-source config file : CloudWatch data source is integrated with ConfigMap
  • deployment file : Grafana configuration and deployment file 
  • service file : Ingress to access the Grafana URL with load balancer URL
Step 1: Create the namespace
For this demo, I have used monitoring as a namespace
kubectl create namespace monitoring


Step 2: Create the ConfigMap

Below is the data-source yaml file for Grafana with CloudWatch data-source. Change the  "assumeRoleArn" to the your Grafana role.

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
  namespace: monitoring
data:
  cloudwatch.yaml: |-
    {
        "apiVersion": 1,
        "datasources": [
            {
                "name": "cw-data-dev",            
                "editable": true,
                "type": "cloudwatch",               
                "jsonData":
                    {
                        "authType": "arn",
                        "defaultRegion": "ap-south-1",
                        "customMetricsNamespaces": "",
                        "assumeRoleArn": "arn:aws:iam::<AWS_Account_ID>>:role/grafana_role",
                    }           
            }
        ]
    }
Run the below command to create the config map for Grafana data-source
kubectl create -f grafana-datasource-config.yaml -n monitoring

Step 3:  Create the deployment
Below is the deployment.yaml file and this Grafana deployment does not use a persistent volume. If you restart the pod all changes will be gone. Use a persistent volume if you are deploying Grafana for your project requirements. It will persist all the configs and data that Grafana uses. 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      name: grafana
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:latest
        ports:
        - name: grafana
          containerPort: 3000
        resources:
          limits:
            memory: "1Gi"
            cpu: "1000m"
          requests: 
            memory: 500M
            cpu: "500m"
        volumeMounts:
          - mountPath: /var/lib/grafana
            name: grafana-storage
          - mountPath: /etc/grafana/provisioning/datasources
            name: grafana-datasources
            readOnly: false
      volumes:
        - name: grafana-storage
          emptyDir: {}
        - name: grafana-datasources
          configMap:
              defaultMode: 420
              name: grafana-datasources
Run the below command to deploy Grafana
kubectl create -f deployment.yaml -n monitoring

Check the pods status by running the below command:
kubectl get pods-n monitoring
Step 4: Create the Service
Below is the service yaml file to access Grafana over Load balancer URL. You can also expose it to NodePort based on your requirement
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
spec:
  selector: 
    app: grafana
  type: LoadBalancer  
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3000
Run the below kubectl command to expose the Grafana URL to loadbalancer with service manifest
kubectl create -f service.yaml -n monitoring
Check the status of service by running below command
kubectl get svc -n monitoring

Validate the Load balancer status in AWS console

Now you will be able to access Grafana dashboard from Load Balancer URL
User: admin
Pass: admin

We can see, data source which we passed in ConfigMap is configured in Grafana data-source

Lets test the Grafana role configuration to check if Grafana can read the CloudWatch metrics.
We can see, data source is working with assume role ARN.

Lets, test by configuring a dashboard for EC2, There are many prebuilt Grafana dashboard available to use. One can import it and monitor the services or create their own custom dashboard depending on the use case.

You should see the dashboard immediately

Conclusion

In this blog post, I have shown you how to create the EKS cluster and nodes. Following this, I showed you how to deploy Grafana and CloudWatch data-source on Kubernetes. Hope this blog helped you in your similar use case.

Thank you for reading!

Comments

Post a Comment

Popular posts from this blog

Connect to Linux EC2 Instance if Key pair is lost after Initial Launch

By
Use Case We know that EC2 Linux instances are accessible through the private keys by default. However, SSH is allowed but you cannot use SSH password authentication to access Linux instance as it is disabled by default. So, what would happen if you lose the private key of your Linux instance? In this blog, I will walk you through the easiest ways to login when you lost the Key pair What is a key-pair file? Key-pair is a combination of a public and private key. Amazon EC2 uses public-key cryptography to encrypt and decrypt login information. Private key is only the way to get access to the instance, what if you lost the key? Is your servers lost in black hole? Thankfully nothing is lost you still can access your server, let me show you how can you solve this problem. Things that you should know before to proceed to this topic This procedure will requires s
Read more

Start or Stop services in multiple Windows EC2 Instances using AWS Systems Manager

By
Use Case You are the admin over a large amount of EC2 instances and managing different AWS accounts under Organization and you have to start/stop/disable a service for all Windows EC2 Instances in the Organization. Rather than wasting labor hours to perform this activity in each instance manually, you want to automate the process using AWS Systems Manager. Recently Windows had a flaw in Print Spooler Remote Code Execution Vulnerability and Microsoft recommended to stop until the patch is applied, in order to apply this immediately, we can make use of this design Systems Manager Systems Manager (SSM) gives you visibility and control over your AWS infrastructure. Automation, a capability of AWS Systems Manager, simplifies common maintenance and deployment tasks of Amazon Elastic Compute Cloud (Amazon EC2) instances and other AWS resources. Automation helps you to do the following: Build automations to configure and manage instances and
Read more

Concourse CI Installation and Configuration in Windows

By
Introduction to Concourse Concourse is an open source CI/CD system with integration supported to Resource types in the outside world. Concourse's principles reduce the risk of switching to and from Concourse, by encouraging practices that decouple your project from your CI's little details, and keeping all configuration in declarative files that can be checked into version control. Install and Configure Concourse in Windows Follow the below Steps to Install Installing Docker Desktop is pretty simple, download the exe file and install the docker desktop Modify Docker daemon configuration file as below
Read more

Install SSM Agent in Amazon EC2 Instance

By
What is SSM Agent AWS Systems Manager Agent is a Amazon software which can be installed on Amazon EC2 Instances. If SSM Agent is Installed in the servers then it will be easy to manage the EC2 Instances through AWS Systems manager.  By default SSM agent is pre installed for below Amazon Machine Images Amazon Linux Amazon Linux 2 Amazon Linux 2 ECS-Optimized Base AMIs macOS 10.14.x (Mojave), 10.15.x (Catalina), and 11.x (Big Sur) SUSE Linux Enterprise Server (SLES) 12 and 15 Ubuntu Server 16.04, 18.04, and 20.04 Windows Server 2008-2012 R2 AMIs published in November 2016 or later Windows Server 2016 and 2019 In this blog, I will walk you through on the ways to Install SSM Agent in the Instances which will not get SSM Agent by default. Steps to Install S
Read more

Deploy AWS infrastructure using Terraform and GitHub Actions

By
Terraform being an Infrastructure as a Code helps us to manage a lot of infrastructure for several platforms in a consistent manner.  In this blog, I will walkthrough on deploying a terraform code using  Github actions Terraform code to create S3 bucket In this blog, I will walkthrough each blocks in terraform to create an S3 bucket in AWS in automated workflow using GitHub Actions. Lets look into the Terraform files. To understand easily, I have created separate tf files for each purpose, but you can still use all in one tf file as per your convenient. You can use any name with tf extension, for better understanding I used following names. versions.tf
Read more

Automate Permission Boundary Attachment to IAM roles and Users

By
When we are managing multiple AWS Accounts in an Organization and each accounts will have their own requirement but as a centralized admin, we should be having a certain governance and compliance framework. IAM is a widely used AWS service and IAM roles/users are common used entity in AWS. As a centralized Admin, when we provide access to create IAM roles, cloud consumers in the landing zone accounts would create Users and Roles and attach Administrative policy and bypass the governance framework. In this blog, I will be briefing on restricting this scenario using IAM permission boundary and also deploying this setup through automation. AWS supports permissions boundaries for IAM entities (users or roles). A permissions boundary is an advanced feature for using a managed policy to set the maximum permissions that an identity-based policy can grant to an IAM entity. An entity's permissions boundary allows
Read more

AWS Route 53 Inbound Resolver to resolve DNS for Multi Account Organization

By
Amazon Route53 Resolver The route 53 resolvers are contactable IP addresses (endpoints) where the DNS queries from different sources can be directed. There are two types of resolvers that administrators can deploy. Inbound Resolver As the name says inbound (DNS Resolutions coming in) which will enable your resources in the on-prem network to resolve AWS resource domain names or records in a Route 53 private hosted zone by allowing your on-prem network DNS resolvers to forward queries to Route 53 Resolver. Outbound Resolver As the name says Outbound (DNS Resolutions going out) which will enable your AWS resources to resolve the domain names of your on-prem network resources using resolver rules, which would forward selected queries to the on-prem network DNS resolvers On-prem to AWS DNS Resolution for
Read more

Import Existing Resources using Terraform

By
In this blog, I will walk you through the ways to import pre-existing cloud resources in terraform. Why Import is required? There will be a scenario, where you will have a stringent timeline to setup a cloud Infrastructure and it can lead to situations where infrastructure needs to be created manually due to time pressures, emergency releases or just the fact that the infrastructure exists, and terraform was never used in the first instance. In a worst case scenario, you can lose the terraform.tfstate file. Terraform import will be essential in most of all these scenarios. Let's Import With an understanding of why Import is required, let us begin by importing a simple AWS resource S3. Firstly, we need to setup the Terraform and AWS credentials locally, we will not go into the details of installing and setting AWS credentials in
Read more

Hosting AWS VPC Interface Endpoints in Shared Model

By
What is VPC Endpoint A VPC endpoint enables private connections between your VPC and supported AWS services. Endpoints are classified as Interface and Gateways endpoints. In this blog, I will be showing how to host interface endpoints in shared model Do you have Multi AWS Accounts in Organization? When you have multi accounts structure in your Organization then creating endpoints in each accounts will increase the costs and IPs, in this blog I will explain how you can deploy endpoints in Shared Services Account and associate the member accounts VPCs with Private Hosted zone. Below is the Architectural diagram which I will be showing in detail on how to setup and also how the flow works Architecture Design AWS Services/Components required VPC Subnets Endpoints Hosted zone and Record set Transit Gateway I
Read more