When data comes from multiple systems, placing it in one storage location does not make it usable on its own. Teams also need to understand its structure, find available datasets, and transform them without creating disconnected copies and processing workflows. This project explores an AWS architecture that connects storage, cataloging, and analytics in a coherent platform.
Summary
Design of an AWS data lakehouse that brings together data from multiple sources, organizes its transformation, and makes it available for analytics.
Contributions
Storage and transformation: Organization of data in Amazon S3 and implementation of ETL pipelines with AWS Glue.
Discovery and analytics: Dataset cataloging, queries over data in the lake with Amazon Athena, and integration of Amazon Redshift for analyses that require a data warehouse layer.
Infrastructure: Definition and automation of AWS resources with Terraform to make the architecture reproducible.
Results
Integrated data flow: An architecture connecting storage, transformation, cataloging, and analytics across the data lifecycle.
Two analytical approaches: Queries over lake data with Athena and a data warehouse layer with Redshift.
Infrastructure as code: Resources defined as code with Terraform.
Value
Easier data discovery: The catalog makes datasets and their structure easier to identify.
Distinct analytical uses: Athena and Redshift provide access suited to different analytical requirements.
Reproducible architecture: Terraform formalizes the required resources and supports redeployment.
