Amazon S3 to DynamoDB Automation

By

Export Object from S3 to DynamoDB using AWS Lambda

In a matter of seconds and without a zero cost using serverless Lambda, allows you to automatically send object from Amazon S3 to Amazon DynamoDB. In this blog, we are going to be creating an AWS Lambda in Python to automatically process any JSON files uploaded to a S3 bucket into a DynamoDB table.

Amazon S3

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as data lakes, websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics

Amazon DynamoDB

Amazon DynamoDB is a fully managed proprietary NoSQL database service that supports key–value and document data structures and is offered by Amazon.com as part of the Amazon Web Services portfolio. Hundreds of thousands of AWS customers have chosen DynamoDB as their key-value and document database for mobile, web, gaming, ad tech, IoT, and other applications that need low-latency data access at any scale. Create a new table for your application and let DynamoDB handle the rest.

Use Case

You might have a requirement to automate whenever a new JSON file is uploaded to S3 bucket then we need to keep the json object in the DynamoDB table. JSON Object cannot directly ingest to DynamoDB table but will have a logic in lambda.

Architectural design

s3_Dynamo
Automation Workflow


Workflow

  1. User/Application put the .json Object to S3 bucket
  2. Lambda will trigger when an .json Object is created in the given S3 bucket
  3. Lambda will convert and put the data in DynamoDB table

How to Setup this WorkFlow?

  • Create the Dynamo DB table, for this blog, I have named it as "employees"
DynamoDB_table

Note: If you want to change the DynamoDB table name then the same name should be passed in Lambda

  • Create a S3 bucket for streaming your JSON objects
  • Lambda IAM Role required below permission:
  • S3 get object and DynamoDB put object permission
  • AWSLambdaBasicExecutionRole (Managed policy)
  • To configure the Trigger, there are two ways, either we can configure in S3 bucket properties by creating an event notification or you can add the Trigger as S3 in lambda console.

    image
lambdatrigger


Testing

Upload a json object in the S3 bucket, and automatically that will put the object in NoSQL DynamoDB table.

  • Uploaded below json object in S3 bucket
    { "emp_id":"4", "Name":"TestUser2", "Age":26, "Location":["USA"] }
  • From DynamoDB table, we see item has been automatically added within few secondstestDB

Conclusion

As you can see with some very basic lambda code, we were able to move the data from S3 to DynamoDB table. Refer the boto3 documentation for proper syntax when you're configuring the individual python modules like boto3. 

Thank you for reading!

Comments

Post a Comment

Popular posts from this blog

Connect to Linux EC2 Instance if Key pair is lost after Initial Launch

By
Use Case We know that EC2 Linux instances are accessible through the private keys by default. However, SSH is allowed but you cannot use SSH password authentication to access Linux instance as it is disabled by default. So, what would happen if you lose the private key of your Linux instance? In this blog, I will walk you through the easiest ways to login when you lost the Key pair What is a key-pair file? Key-pair is a combination of a public and private key. Amazon EC2 uses public-key cryptography to encrypt and decrypt login information. Private key is only the way to get access to the instance, what if you lost the key? Is your servers lost in black hole? Thankfully nothing is lost you still can access your server, let me show you how can you solve this problem. Things that you should know before to proceed to this topic This procedure will requires s
Read more

Setup Grafana on AWS EKS and integrate with AWS Cloudwatch

By
What is Grafana? Grafana is open-source visualization and analytics software. It allows you to query, visualize, alert on, and explore your metrics no matter where they are stored. In plain English, it provides you with tools to turn your time-series database (TSDB) data into beautiful graphs and visualizations. Setup Grafana on Amazon Elastic Kubernetes Service This blog explains on how to run Grafana on Amazon Elastic Kubernetes cluster and adding cloudwatch as datasource to Grafana. Using Grafana you can simplify Kubernetes monitoring dashboards from Clo
Read more

Start or Stop services in multiple Windows EC2 Instances using AWS Systems Manager

By
Use Case You are the admin over a large amount of EC2 instances and managing different AWS accounts under Organization and you have to start/stop/disable a service for all Windows EC2 Instances in the Organization. Rather than wasting labor hours to perform this activity in each instance manually, you want to automate the process using AWS Systems Manager. Recently Windows had a flaw in Print Spooler Remote Code Execution Vulnerability and Microsoft recommended to stop until the patch is applied, in order to apply this immediately, we can make use of this design Systems Manager Systems Manager (SSM) gives you visibility and control over your AWS infrastructure. Automation, a capability of AWS Systems Manager, simplifies common maintenance and deployment tasks of Amazon Elastic Compute Cloud (Amazon EC2) instances and other AWS resources. Automation helps you to do the following: Build automations to configure and manage instances and
Read more

Concourse CI Installation and Configuration in Windows

By
Introduction to Concourse Concourse is an open source CI/CD system with integration supported to Resource types in the outside world. Concourse's principles reduce the risk of switching to and from Concourse, by encouraging practices that decouple your project from your CI's little details, and keeping all configuration in declarative files that can be checked into version control. Install and Configure Concourse in Windows Follow the below Steps to Install Installing Docker Desktop is pretty simple, download the exe file and install the docker desktop Modify Docker daemon configuration file as below
Read more

Install SSM Agent in Amazon EC2 Instance

By
What is SSM Agent AWS Systems Manager Agent is a Amazon software which can be installed on Amazon EC2 Instances. If SSM Agent is Installed in the servers then it will be easy to manage the EC2 Instances through AWS Systems manager.  By default SSM agent is pre installed for below Amazon Machine Images Amazon Linux Amazon Linux 2 Amazon Linux 2 ECS-Optimized Base AMIs macOS 10.14.x (Mojave), 10.15.x (Catalina), and 11.x (Big Sur) SUSE Linux Enterprise Server (SLES) 12 and 15 Ubuntu Server 16.04, 18.04, and 20.04 Windows Server 2008-2012 R2 AMIs published in November 2016 or later Windows Server 2016 and 2019 In this blog, I will walk you through on the ways to Install SSM Agent in the Instances which will not get SSM Agent by default. Steps to Install S
Read more

Automate Permission Boundary Attachment to IAM roles and Users

By
When we are managing multiple AWS Accounts in an Organization and each accounts will have their own requirement but as a centralized admin, we should be having a certain governance and compliance framework. IAM is a widely used AWS service and IAM roles/users are common used entity in AWS. As a centralized Admin, when we provide access to create IAM roles, cloud consumers in the landing zone accounts would create Users and Roles and attach Administrative policy and bypass the governance framework. In this blog, I will be briefing on restricting this scenario using IAM permission boundary and also deploying this setup through automation. AWS supports permissions boundaries for IAM entities (users or roles). A permissions boundary is an advanced feature for using a managed policy to set the maximum permissions that an identity-based policy can grant to an IAM entity. An entity's permissions boundary allows
Read more

Deploy AWS infrastructure using Terraform and GitHub Actions

By
Terraform being an Infrastructure as a Code helps us to manage a lot of infrastructure for several platforms in a consistent manner.  In this blog, I will walkthrough on deploying a terraform code using  Github actions Terraform code to create S3 bucket In this blog, I will walkthrough each blocks in terraform to create an S3 bucket in AWS in automated workflow using GitHub Actions. Lets look into the Terraform files. To understand easily, I have created separate tf files for each purpose, but you can still use all in one tf file as per your convenient. You can use any name with tf extension, for better understanding I used following names. versions.tf
Read more

AWS Route 53 Inbound Resolver to resolve DNS for Multi Account Organization

By
Amazon Route53 Resolver The route 53 resolvers are contactable IP addresses (endpoints) where the DNS queries from different sources can be directed. There are two types of resolvers that administrators can deploy. Inbound Resolver As the name says inbound (DNS Resolutions coming in) which will enable your resources in the on-prem network to resolve AWS resource domain names or records in a Route 53 private hosted zone by allowing your on-prem network DNS resolvers to forward queries to Route 53 Resolver. Outbound Resolver As the name says Outbound (DNS Resolutions going out) which will enable your AWS resources to resolve the domain names of your on-prem network resources using resolver rules, which would forward selected queries to the on-prem network DNS resolvers On-prem to AWS DNS Resolution for
Read more

Import Existing Resources using Terraform

By
In this blog, I will walk you through the ways to import pre-existing cloud resources in terraform. Why Import is required? There will be a scenario, where you will have a stringent timeline to setup a cloud Infrastructure and it can lead to situations where infrastructure needs to be created manually due to time pressures, emergency releases or just the fact that the infrastructure exists, and terraform was never used in the first instance. In a worst case scenario, you can lose the terraform.tfstate file. Terraform import will be essential in most of all these scenarios. Let's Import With an understanding of why Import is required, let us begin by importing a simple AWS resource S3. Firstly, we need to setup the Terraform and AWS credentials locally, we will not go into the details of installing and setting AWS credentials in
Read more

Hosting AWS VPC Interface Endpoints in Shared Model

By
What is VPC Endpoint A VPC endpoint enables private connections between your VPC and supported AWS services. Endpoints are classified as Interface and Gateways endpoints. In this blog, I will be showing how to host interface endpoints in shared model Do you have Multi AWS Accounts in Organization? When you have multi accounts structure in your Organization then creating endpoints in each accounts will increase the costs and IPs, in this blog I will explain how you can deploy endpoints in Shared Services Account and associate the member accounts VPCs with Private Hosted zone. Below is the Architectural diagram which I will be showing in detail on how to setup and also how the flow works Architecture Design AWS Services/Components required VPC Subnets Endpoints Hosted zone and Record set Transit Gateway I
Read more