Rick Cramer

Northstar Data Platform

Architecture

Northstar's architecture covers more than a list of AWS services. Infrastructure, security, deployment, observability, reproducibility, and cost are treated as parts of one operating system for the data platform.

Current Infrastructure

What exists today

Infrastructure is being added incrementally. The portfolio, simulator, and development environment already provide real engineering evidence, while the larger AWS data platform will be deployed as corresponding workloads become ready.

Next.js Portfolio

Complete

The public Northstar portfolio is built with Next.js and exported as a static site for a simple, low-maintenance deployment model.

Amazon S3 Hosting

Complete

Static portfolio assets are deployed to Amazon S3 rather than requiring an always-running application server.

CloudFront Delivery

Complete

Amazon CloudFront provides HTTPS delivery and caching for the public portfolio.

Local Python Development Environment

Complete

The simulator currently runs in a controlled local Python development environment with source control and automated tests.

Automated Test Environment

Complete

pytest validates deterministic simulator behavior, reference data, and generated entities as the source model develops.

Core AWS Data Platform

Planned

RDS, DMS, S3 data-lake layers, Glue, Kinesis, Redshift, and related services will be deployed as their upstream workloads become ready.

Target AWS Architecture

Services have specific responsibilities

These technologies represent the intended end-state architecture. They will be introduced only as their corresponding ingestion, processing, analytical, security, and operational requirements become real.

Layer

Storage

Technology

Amazon S3

Purpose

Raw, quarantine, and curated storage for source-oriented and analytical data.

Layer

Operational Database

Technology

Amazon RDS / PostgreSQL

Purpose

Transactional source data and a relational operational workload suitable for CDC ingestion.

Layer

Change Data Capture

Technology

AWS DMS

Purpose

Replicate database changes from PostgreSQL into downstream data-platform storage.

Layer

API Processing

Technology

AWS Lambda

Purpose

Run lightweight Python ingestion and event-driven application logic without maintaining idle servers.

Layer

Streaming

Technology

Amazon Kinesis

Purpose

Ingest simulated behavioral and operational events using a streaming pattern.

Layer

Catalog & Query

Technology

AWS Glue Catalog / Athena

Purpose

Register datasets and provide serverless SQL access to data stored in S3.

Layer

Distributed Processing

Technology

AWS Glue / PySpark

Purpose

Perform scalable validation, standardization, enrichment, deduplication, reconciliation, and quarantine processing.

Layer

Lakehouse

Technology

Parquet / Apache Iceberg

Purpose

Provide efficient analytical storage with schema evolution and governed table behavior above object storage.

Layer

Warehouse

Technology

Amazon Redshift

Purpose

Support warehouse-oriented analytical models and workloads that benefit from a dedicated analytical engine.

Layer

Analytical Modeling

Technology

dbt

Purpose

Centralize reusable SQL-based business logic, tests, documentation, and analytical models.

Layer

Security

Technology

IAM / KMS / Secrets Manager

Purpose

Control permissions, encryption, credentials, and service-to-service access using least-privilege design.

Layer

Observability

Technology

CloudWatch

Purpose

Capture logs, metrics, alarms, failures, and operational evidence across platform components.

Layer

Infrastructure as Code

Technology

Terraform

Purpose

Define cloud resources declaratively so infrastructure remains reproducible, reviewable, and version controlled.

Layer

Deployment

Technology

GitHub Actions / CI/CD

Purpose

Automate testing and controlled deployment of application and infrastructure changes.

Infrastructure as Code

Terraform will become the infrastructure source of truth

Northstar's target AWS environment is intended to be defined through Terraform rather than depending on manually configured console resources.

Infrastructure as code makes changes reviewable alongside application code, improves reproducibility, and reduces undocumented configuration drift.

Manual console work remains useful while learning or investigating individual services, but the production-style end state is a version-controlled infrastructure definition that explains what exists and how it is configured.

Architecture Principles

Infrastructure decisions are engineering decisions

Least privilege

Users and services should receive only the permissions required for their responsibilities rather than broad administrative access.

Reproducible infrastructure

Infrastructure should be defined through code wherever practical so environments can be recreated without undocumented console configuration.

Clear network boundaries

Public interfaces, private resources, service-to-service communication, and administrative access should be deliberately separated.

Deploy when needed

Cloud services should be introduced when a real workload is ready to use them rather than deployed early simply to make the architecture appear complete.

Observable by default

Logs, metrics, alarms, and failure information should be available as infrastructure is introduced rather than added only after problems occur.

Security as architecture

IAM, encryption, secrets management, and controlled interfaces are design decisions rather than final-stage hardening tasks.

Cost Engineering

Architecture has an economic dimension

A technically valid architecture can still be a poor design if its economics do not match the workload. Northstar therefore treats cost as another architectural constraint alongside reliability, security, and performance.

The project is intentionally avoiding unnecessary always-on AWS resources while the source simulator is still being developed. Services are deployed when they have actual work to perform.

Do not pay for idle realism

Northstar should demonstrate production architecture without keeping expensive resources running only to make the environment look enterprise-scale.

Measure before optimizing

Cost decisions should eventually be based on actual storage, execution, query, and traffic behavior rather than assumptions.

Prefer variable cost when appropriate

Usage-based and serverless services are valuable for intermittent portfolio workloads when they reduce unnecessary idle infrastructure.

Design storage for efficiency

File format, partitioning, lifecycle, and data layout affect both performance and cost, especially when services charge by data scanned or processed.

Major Cost Drivers

Different services become expensive for different reasons

Understanding a cloud service includes understanding what drives its cost and which architectural choices can control that cost.

Category

Object Storage

Technology

S3

Cost Driver

Stored data volume, requests, retrieval patterns, lifecycle policies, and data transfer.

Control Strategy

Use compact formats, sensible retention, lifecycle rules, and avoid unnecessary duplication between layers.

Category

Operational Database

Technology

RDS / PostgreSQL

Cost Driver

Instance runtime, instance class, storage, backups, and I/O.

Control Strategy

Size the environment for portfolio workloads and avoid leaving unnecessary database capacity running.

Category

CDC

Technology

AWS DMS

Cost Driver

Replication infrastructure runtime and the resources required by its source and target.

Control Strategy

Run replication infrastructure only when CDC development or demonstration work requires it.

Category

Distributed Processing

Technology

Glue / PySpark

Cost Driver

Processing duration, allocated compute, job frequency, and inefficient transformations.

Control Strategy

Use bounded jobs, columnar formats, selective processing, and tune workloads after measuring actual behavior.

Category

Streaming

Technology

Kinesis

Cost Driver

Throughput, retention, stream activity, and selected capacity mode.

Control Strategy

Generate realistic but intentionally limited event volumes appropriate for a portfolio environment.

Category

Serverless Compute

Technology

Lambda

Cost Driver

Invocation count, execution duration, memory allocation, and downstream service use.

Control Strategy

Use short-lived event-driven execution where it avoids unnecessary idle compute.

Category

Ad Hoc Analytics

Technology

Athena

Cost Driver

Amount of data scanned by queries.

Control Strategy

Use Parquet, appropriate partitioning, curated datasets, and selective queries to reduce scans.

Category

Warehouse

Technology

Redshift

Cost Driver

Compute usage, storage, workload duration, and query behavior.

Control Strategy

Introduce Redshift only when warehouse-specific workloads exist and size it for demonstration rather than enterprise-scale volume.

Category

Observability

Technology

CloudWatch

Cost Driver

Log ingestion, retention, metrics, alarms, and verbose diagnostic output.

Control Strategy

Preserve useful operational evidence without retaining unnecessary high-volume development logs indefinitely.

Cost Observability

Eventually, architecture should include actual cost evidence

As the AWS platform comes online, cost can become another observable property of Northstar rather than remaining purely architectural guidance.

Future evidence can include monthly service totals, major cost drivers, workload changes, resource-lifecycle decisions, and examples where architecture changes reduced unnecessary spend.

At that point, cost analysis may become substantial enough to justify its own page again. Until then, it belongs here as part of architecture.

What comes next

As the AWS platform is implemented, this page will add deployed resource evidence, Terraform structure, IAM relationships, network design, CI/CD behavior, operational metrics, and actual cost data from the running environment.