Data-lake on AWS.
The Problem
Addressing the Demand for Efficient Data Management: Building a Comprehensive Data Lake Solution from Diverse Input Sources, Streamlining Transformations, and Enabling Seamless SQL Querying.
The solution
Harnessing the Power of AWS: Crafting a Robust Data Lake Utilizing Lambda Functions, S3 Storage, AWS Glue, and Athena for Seamless Data Management.
Scope of work.
- Developed a series of Lambda functions for ingesting input sources.
- Constructed a suite of AWS Glue ETL Jobs for data transformation tasks.
- Established a collection of AWS Athena queries tailored for the reporting team’s needs.