Rick Cramer
Flagship Data Engineering ProjectIn Active Development

Northstar Data Platform

A production-style AWS data engineering platform built to demonstrate how I design, build, test, operate, troubleshoot, and explain a modern data system from source to analytics.

View GitHub RepositoryAll Projects

The Problem

A portfolio project should prove engineering ability, not just list technologies.

Northstar exists to demonstrate an entire data platform rather than a collection of isolated AWS exercises.

The project covers operational data modeling, batch and streaming ingestion, incremental processing, transformation, data quality, lakehouse architecture, warehousing, orchestration, observability, security, deployment, and cost management.

The goal is not simply to show that a service was configured. The goal is to show that I can explain why it was chosen, verify that it works, diagnose failures, and defend the tradeoffs involved.

Synthetic Business

Northstar Outfitters

The platform operates on a deterministic synthetic e-commerce company selling outdoor products. The simulator will generate customers, products, warehouses, orders, order lines, web events, payments, refunds, inventory, shipments, marketing activity, and support data.

The data is intentionally imperfect. Late records, duplicates, invalid relationships, malformed values, schema changes, stockouts, refunds, shipment delays, seasonal demand, and other realistic conditions create problems the pipelines must detect and handle.

Architecture

Source systems to governed analytical data

 PostgreSQL / RDS        REST APIs         Event Simulator
        |                    |                    |
    DMS / CDC          Lambda / Python          Kinesis
        |                    |                    |
        +--------------------+--------------------+
                             |
                             v
                         S3 RAW
                             |
                       Glue Catalog
                             |
                     Glue / PySpark
                             |
                  +----------+----------+
                  |                     |
             Quarantine            S3 Curated
                                        |
                              Parquet / Iceberg
                                        |
                           +------------+------------+
                           |                         |
                        Athena                   Redshift
                                                     |
                                                    dbt
                                                     |
                                            Analytical Models
                                                     |
                                             Portfolio Data

IAM, KMS, Secrets Manager, CloudWatch, messaging, orchestration, Terraform, Docker, CI/CD, Cognito, API Gateway, and Lambda surround the core data flow to provide security, deployment, observability, and controlled access.

Engineering Scope

What Northstar is designed to prove

Multiple ingestion patterns

Batch files, PostgreSQL CDC, REST APIs, and streaming events all feed the platform through patterns appropriate to each source.

Production-style transformations

SQL, Python, Glue, and PySpark handle validation, standardization, deduplication, enrichment, quarantine, and curated data output.

Lakehouse and warehouse

S3, Parquet, Apache Iceberg, Athena, Redshift, and dbt demonstrate both lake-oriented and warehouse-oriented analytical architecture.

Operational reliability

Orchestration, idempotency, monitoring, alerting, testing, data quality, failure recovery, and cost controls are treated as core engineering requirements.

Security by design

IAM, encryption, secrets management, controlled APIs, temporary reviewer access, and infrastructure as code are built into the platform rather than added afterward.

Inspectable engineering

Architecture decisions, source code, tests, lineage, pipeline history, data-quality evidence, and engineering tradeoffs will be visible through the portfolio.

Build Status

Built incrementally, with evidence at every stage

The repository and development environment are established, and the public portfolio shell is now being built. Infrastructure and data-platform components are added only after they can be implemented, tested, observed, and explained.

  1. 01Synthetic company and historical dataset
  2. 02Dockerized simulator
  3. 03S3 data lake
  4. 04Glue Catalog and Athena
  5. 05API ingestion
  6. 06PostgreSQL and CDC
  7. 07Glue and PySpark transformations
  8. 08Data quality and quarantine
  9. 09Dimensional modeling
  10. 10Redshift warehouse
  11. 11dbt analytical layer
  12. 12Orchestration
  13. 13Streaming
  14. 14Observability
  15. 15Security hardening
  16. 16Terraform and CI/CD
  17. 17Reviewer authentication
  18. 18Live demo experience

Portfolio Evidence

The finished project will be inspectable from multiple angles

Business Analytics
Pipeline
Infrastructure
Data Quality
Lineage
Costs
Architecture Decisions
Live Demo Access

Follow the implementation

The repository contains the infrastructure, application code, engineering documentation, tests, and Git history as Northstar develops from an empty AWS account into a complete data platform.

Open Repository