Building a data warehouse AWS for reporting purposes.
The Problem
The challenge entailed centralizing data sources for multiple reports, encompassing Internal Databases, External Systems (such as event analytics systems, 3rd party databases on AWS, and those on other cloud platforms), and managing both structured and unstructured data formats.
The solution
The solution involved leveraging a Redshift database alongside appropriate IAM policies, S3 storage, Athena queries, Lambda functions, and data pipelines.
Scope of work.
- Structured a collection of Lambda functions for organization and efficiency.
- Orchestrated a suite of AWS Batch jobs to streamline processing tasks.
- Developed a custom Redshift solution as the destination for diverse input sources.
- Established a comprehensive set of IAM roles and policies to accommodate all requirements.