Multiple ingestion patterns
The target platform includes batch files, PostgreSQL change-data capture, REST APIs, and streaming events so ingestion design reflects how different source systems actually behave.
Flagship Data Engineering ProjectIn Active Development
Northstar Data Platform
A production-style AWS data engineering platform being built to demonstrate how I design, implement, test, operate, troubleshoot, and explain a modern data system from source generation through analytics.
The Project
Building a complete system instead of a collection of cloud exercises
Northstar is designed as an interconnected data platform rather than a series of isolated demonstrations of individual AWS services.
The project begins with synthetic operational systems, moves through multiple ingestion and processing patterns, and ultimately produces governed analytical data and business-facing models.
The emphasis is on the engineering decisions between those layers: how data is validated, how failures are handled, how processing can be rerun safely, how infrastructure is secured and reproduced, how the system is observed, and why each technology belongs in the architecture.
Synthetic Business
Northstar Outfitters
Northstar is built around a deterministic synthetic multi-channel outdoor retailer rather than a prebuilt public dataset.
Controlling the source systems makes it possible to design realistic relationships, historical behavior, edge cases, schema changes, and intentional data-quality problems that downstream engineering must actually handle.
The simulator is being expanded to model customers, products, warehouses, orders, order lines, payments, refunds, inventory, shipments, web activity, marketing activity, and other interconnected business processes.
Current Progress
Building the source systems first
- Deterministic category reference data is implemented and tested.
- Product generation is implemented with controlled category relationships and repeatable output.
- Automated pytest coverage protects simulator behavior as the source model expands.
- Additional business entities and historical behavior are being added before downstream AWS ingestion begins.
Target Architecture
Source systems to governed analytical data
This is the intended end-state data flow. Components are being implemented incrementally as the simulator and upstream source systems mature.
PostgreSQL / RDS REST APIs Event Simulator
| | |
DMS / CDC Lambda / Python Kinesis
| | |
+--------------------+--------------------+
|
v
S3 RAW
|
Glue Catalog
|
Glue / PySpark
|
+----------+----------+
| |
Quarantine S3 Curated
|
Parquet / Iceberg
|
+------------+------------+
| |
Athena Redshift
|
dbt
|
Analytical ModelsIAM, KMS, Secrets Manager, CloudWatch, orchestration, Terraform, Docker, and CI/CD form part of the surrounding target architecture for security, deployment, observability, reproducibility, and operations.
Engineering Scope
What Northstar is designed to demonstrate
Reliable data processing
Validation, transformation, deduplication, reconciliation, quarantine, lineage, testing, and observability are treated as parts of the pipeline rather than separate afterthoughts.
Analytical architecture
The target design combines S3, Parquet, Apache Iceberg, Athena, Redshift, and dbt to move from source-oriented data into reusable analytical models.
Production-style operation
Security, infrastructure as code, deployment, monitoring, failure recovery, and cost are included so the project demonstrates how a platform is operated as well as how it is built.
Build Status
Built from the source systems outward
The roadmap is organized around meaningful platform phases rather than individual services. Each layer is implemented and tested before later layers depend on it.
CompleteIn ProgressPlanned
- 01
Project foundation and development environment
CompleteRepository structure, development workflow, documentation, and local engineering environment.
- 02
Deterministic source model
CompleteCategory reference data, product generation, and automated simulator tests.
- 03
Expanded business simulation
In ProgressCustomers, orders, payments, inventory, shipments, historical behavior, and other interconnected entities.
- 04
Cloud ingestion and raw storage
PlannedBatch, API, CDC, and streaming ingestion into the AWS data platform.
- 05
Transformation, quality, and lakehouse
PlannedGlue, PySpark, validation, quarantine, reconciliation, Parquet, and Iceberg.
- 06
Warehouse and analytical modeling
PlannedAthena, Redshift, dbt, dimensional modeling, governed metrics, and business analytics.
- 07
Platform operations and automation
PlannedOrchestration, observability, IAM, security hardening, Terraform, Docker, and CI/CD.
Explore Northstar
Four views into the same platform
The overview stays intentionally high level. The sections below contain the deeper engineering evidence, design, and reasoning behind the platform.
Follow the implementation
The repository contains the simulator, automated tests, source code, engineering documentation, architecture decisions, and Git history as Northstar develops from deterministic source systems into a complete cloud data platform.