Solve a real architectural problem
A technology belongs in Northstar because it addresses a specific requirement, not simply because it appears frequently in job descriptions.
Northstar Data Platform
Architecture Decisions
Northstar documents important engineering decisions separately from implementation progress. Each record captures the selected approach, why it was chosen, alternatives considered, and the tradeoffs that come with the decision.
Decision Framework
Architecture is more than selecting tools
Northstar is intentionally designed so the reasoning behind the architecture remains visible. Technologies are selected in response to specific engineering requirements rather than added simply to make the stack larger.
Separate decisions from implementation status
An architectural choice may already be accepted even when the corresponding component has not yet been built.
Document alternatives
The value of an architecture decision is not only the selected technology but also understanding what other approaches were considered and why they were rejected.
Accept tradeoffs explicitly
Every architectural choice solves some problems while introducing others. Those costs should be visible rather than hidden behind a technology list.
Decision Log
Accepted architecture decisions
Implementation status is shown independently from decision status. A planned component may already have an accepted architectural direction even though its code or infrastructure has not yet been implemented.
ADR-001
Generate synthetic business data instead of using a public dataset
AcceptedImplemented
Decision
Build a deterministic Python simulator for Northstar Outfitters rather than start with a prebuilt public dataset.
Rationale
A controlled simulator makes it possible to design realistic relationships, business rules, historical behavior, edge cases, and intentional data-quality failures. It also allows the same source data to be reproduced reliably during development and testing.
Alternatives Considered
- Use an existing public retail dataset
- Generate unrelated flat files with a generic data-generation library
- Manually maintain sample CSV files
Tradeoff
Building the simulator adds significant work before the cloud pipeline begins, but produces a much more realistic and testable source environment.
ADR-002
Build the source systems before the cloud pipeline
AcceptedIn Progress
Decision
Develop and validate the synthetic operational environment before implementing downstream AWS ingestion and transformation.
Rationale
Pipeline behavior is easier to evaluate when the source systems have stable schemas, realistic relationships, deterministic output, and automated tests.
Alternatives Considered
- Build AWS infrastructure first
- Prototype each AWS service independently
- Use temporary sample files while designing the platform
Tradeoff
The AWS portion of the project appears later, but the eventual pipeline operates against a more credible source environment instead of disposable demonstration data.
ADR-003
Use AWS as the primary cloud platform
AcceptedIn Progress
Decision
Build the primary Northstar architecture around AWS services while using selected open-source tools where they provide meaningful platform experience.
Rationale
AWS provides a coherent environment for demonstrating storage, ingestion, streaming, distributed processing, warehousing, security, observability, and infrastructure automation within one production-style architecture.
Alternatives Considered
- Microsoft Azure
- Google Cloud Platform
- A cloud-neutral local-only architecture
Tradeoff
The project becomes intentionally AWS-oriented rather than cloud-neutral, but the underlying engineering concepts remain transferable to other platforms.
ADR-004
Use S3 as the central data-lake storage layer
AcceptedPlanned
Decision
Land source-oriented data in Amazon S3 and separate raw, quarantine, and curated data according to processing state.
Rationale
S3 provides durable, low-cost object storage that can support multiple ingestion patterns and downstream query and transformation services without coupling storage to a single compute engine.
Alternatives Considered
- Load all source data directly into Redshift
- Use PostgreSQL as both the operational and analytical store
- Maintain separate storage systems for each ingestion path
Tradeoff
A data lake introduces governance, file-layout, metadata, and lifecycle-management requirements that would be simpler in a database-only architecture.
ADR-005
Use Parquet for analytical file storage
AcceptedPlanned
Decision
Store transformed analytical data in columnar Parquet rather than continuing to use CSV or JSON as the primary downstream format.
Rationale
Parquet reduces storage and scan costs, preserves useful schema information, and is well suited to analytical access through tools such as Athena, Spark, and lakehouse engines.
Alternatives Considered
- CSV
- JSON
- Avro as the primary curated format
Tradeoff
Parquet is less human-readable than text formats and requires tooling to inspect directly, but it is substantially better suited to analytical workloads.
ADR-006
Use Apache Iceberg for the lakehouse table layer
AcceptedPlanned
Decision
Introduce Apache Iceberg above curated object storage where table-level lakehouse capabilities are useful.
Rationale
Iceberg provides schema evolution, partition evolution, transactional table behavior, snapshots, and a stronger abstraction over raw object-storage files.
Alternatives Considered
- Plain partitioned Parquet only
- Delta Lake
- Apache Hudi
Tradeoff
Iceberg adds another architectural layer and additional concepts to operate, so it will be introduced only after the underlying S3 and transformation layers are established.
ADR-007
Use Athena and Redshift for different analytical workloads
AcceptedPlanned
Decision
Use Athena for serverless querying of lake data and Redshift for warehouse-oriented analytical models and workloads.
Rationale
Using both demonstrates the distinction between direct querying of object-storage data and a dedicated analytical warehouse rather than treating the technologies as interchangeable.
Alternatives Considered
- Athena only
- Redshift only
- PostgreSQL as the analytical warehouse
Tradeoff
Operating two analytical query layers adds complexity and cost, so each must have a clearly defined purpose rather than duplicating the same workload.
ADR-008
Use dbt for the analytical modeling layer
AcceptedPlanned
Decision
Use dbt for warehouse-oriented transformation, reusable business logic, tests, documentation, and analytical model development.
Rationale
dbt separates analytical modeling from lower-level ingestion and distributed processing while providing a widely used framework for SQL-based transformation engineering.
Alternatives Considered
- Implement all transformations directly in Glue or PySpark
- Maintain standalone SQL scripts
- Use stored procedures for the analytical layer
Tradeoff
dbt introduces another tool into the stack, but provides clearer ownership and structure for transformations that belong in the analytical layer.
ADR-009
Use Terraform as the infrastructure source of truth
AcceptedPlanned
Decision
Define production-style AWS infrastructure through Terraform rather than relying on manually configured console resources.
Rationale
Infrastructure as code makes changes reviewable, repeatable, version controlled, and easier to reproduce while reducing undocumented configuration drift.
Alternatives Considered
- AWS Console configuration
- AWS CloudFormation
- AWS CDK
Tradeoff
Terraform adds its own state management and operational considerations, but provides valuable cross-platform infrastructure-as-code experience.
ADR-010
Containerize the simulator with Docker
AcceptedPlanned
Decision
Package the Northstar simulator and its runtime dependencies in Docker rather than relying exclusively on the developer workstation environment.
Rationale
Containerization creates a reproducible runtime boundary and makes it easier to execute the simulator consistently in local development, automated workflows, and future hosted environments.
Alternatives Considered
- Local Python virtual environment only
- Package directly for a specific AWS runtime
Tradeoff
Docker introduces container build and image-management work that is not required for the simulator itself, but strengthens reproducibility and deployment portability.
ADR-011
Keep Snowflake and Databricks outside the core architecture
AcceptedDeferred
Decision
Keep Northstar's primary implementation focused on the AWS architecture and evaluate platforms such as Snowflake and Databricks later through bounded comparison exercises.
Rationale
Adding every major modern data platform to the core project would increase scope without improving the coherence of the primary architecture.
Alternatives Considered
- Make Snowflake part of the core warehouse architecture
- Build the primary transformation platform in Databricks
- Implement multiple parallel cloud data platforms
Tradeoff
The core project demonstrates fewer vendor platforms, but development remains focused enough to reach meaningful engineering depth.
Why document this?
The reasoning matters as much as the diagram
A finished architecture diagram can show what exists, but it does not explain why the system ended up that way.
Recording decisions while Northstar develops makes the tradeoffs inspectable. It also prevents later implementation choices from looking arbitrary when several technically valid alternatives were available.
As additional questions arise around networking, orchestration, partitioning, streaming, security, cost, deployment, and observability, new decision records will be added here.