Northstar Data Platform

Pipeline

Northstar's pipeline covers more than moving data from one place to another. The design includes ingestion, transformation, automated testing, data quality, quarantine, reconciliation, lineage, and observability as parts of one end-to-end engineering system.

Current Implementation

Building reliable source data first

Northstar is being built from the source systems outward. Before introducing cloud ingestion and distributed processing, the synthetic environment needs stable relationships, predictable behavior, and automated validation.

Reference Data

Complete

Deterministic category reference data provides controlled relationships and shared business context for downstream synthetic entities.

Product Generation

Complete

A deterministic Python generator creates realistic products tied to the category hierarchy and governed by repeatable generation rules.

Automated Validation

Complete

pytest coverage validates simulator behavior, relationships, distributions, and repeatability before generated data becomes a dependency for downstream processing.

Expanded Business Simulation

In Progress

Customers, orders, payments, inventory, shipments, and other interconnected entities are being added before cloud ingestion begins.

Target Pipeline

From operational sources to analytical data

Each layer has a distinct responsibility. Original source data is preserved, quality and transformation happen explicitly, and analytical models are built only after records have moved through controlled processing stages.

Source Systems

PostgreSQL / REST APIs / Files / Event Simulator

Operational, API, file-based, and event-oriented sources provide multiple realistic ingestion patterns.

Ingestion

DMS / Lambda / Kinesis / Batch Loads

Sources enter the platform using change-data capture, API extraction, streaming, or batch processing based on their behavior.

Raw Data Lake

Amazon S3

Source-oriented records are preserved before transformation so original inputs remain inspectable and replayable.

Validation & Transformation

AWS Glue / PySpark / Python

Records are validated, standardized, deduplicated, enriched, reconciled, and routed to curated or quarantine paths.

Curated Lake

Parquet / Apache Iceberg

Validated analytical datasets are stored in optimized columnar structures with explicit schemas and governed table behavior.

Analytics

Athena / Redshift / dbt

Curated data becomes reusable analytical models, governed metrics, and business-facing outputs.

Data Quality & Testing

Quality begins before ingestion

The simulator itself is the first data-quality boundary. If source behavior cannot be reproduced and validated, later pipeline failures become significantly harder to diagnose.

Deterministic Generation

Complete

The same inputs produce reproducible source data, making failures repeatable and allowing tests to distinguish real changes from random variation.

Reference Data Validation

Complete

Category reference data is tested for structural integrity, expected codes, valid relationships, and assumptions required by downstream generators.

Product Validation

Complete

Generated products are tested for required values, valid category relationships, deterministic behavior, and expected generation rules.

Expanded Entity Validation

In Progress

Equivalent integrity and business-rule tests are being added as additional business entities enter the simulator.

Schema Validation

Verify required columns, types, formats, and structural expectations as records enter the platform.

Relationship Integrity

Detect invalid relationships, orphan records, impossible entity combinations, and broken business references.

Business Rules

Apply domain constraints such as valid quantities, financial relationships, inventory behavior, and order-state transitions.

Duplicate Detection

Identify accidental duplicate ingestion while preserving legitimate repeated business activity.

Freshness & Completeness

Measure whether expected records arrive on time and whether source populations are unexpectedly incomplete.

Reconciliation

Compare counts, totals, balances, and other control values across pipeline stages to detect silent data loss.

Failure Handling

Bad data should be visible and recoverable

Records that fail validation should not silently disappear. Northstar's target design preserves invalid records alongside the reason they failed so they can be investigated, corrected, and replayed.

                  Incoming Data
                        |
                        v
                   Validation
                    /      \
                   /        \
               Valid       Invalid
                 |            |
                 v            v
             Curated      Quarantine
               Data           |
                               v
                        Failure Reason
                               |
                               v
                     Investigate / Correct
                               |
                               v
                          Replay Data

Data Lineage

The pipeline should explain where data came from

Source-level lineage already exists through the simulator's reference relationships, generated entities, generation logic, and automated tests. As the cloud pipeline develops, that traceability will extend through transformations and analytical models.

Where did this analytical value originate?

Which source records contributed to this dataset?

Which transformations changed the data?

Which quality rules were applied?

Did any records enter quarantine?

Which downstream models depend on this dataset?

What would be affected if a source schema changed?

Can the result be reproduced from the original inputs?

Operational Sources
        |
        v
     Ingestion
        |
        v
      S3 Raw
        |
        v
Validation / Transformation
      /       \
     /         \
Curated      Quarantine
   |
   v
Parquet / Iceberg
   |
   +----------------+
   |                |
   v                v
Athena          Redshift
                    |
                    v
                   dbt
                    |
                    v
          Analytical Models
                    |
                    v
           Portfolio Outputs

Observability

A pipeline should explain its own behavior

As orchestration and AWS processing are implemented, Northstar will expose operational signals that make pipeline behavior measurable rather than requiring investigation through raw logs alone.

Execution Status

Track successful, failed, retried, and partially completed pipeline executions.

Row Counts

Compare record counts between pipeline stages to identify unexpected loss, duplication, or expansion.

Quality Failures

Measure validation failures and preserve the reasons records were rejected or quarantined.

Freshness

Track whether expected datasets and source feeds arrive within defined processing windows.

Latency

Measure how long data takes to move from source activity to downstream analytical availability.

Operational Errors

Capture logs, exceptions, retries, and other diagnostic information required to investigate failures.

Engineering Principles

Reliability is part of the pipeline design

Deterministic inputs

Repeatable source generation makes failures reproducible and gives automated tests a stable foundation.

Idempotent processing

Pipeline stages should support safe reruns without generating duplicate or inconsistent downstream results.

Explicit quality boundaries

Validation and quarantine belong inside the pipeline rather than being informal cleanup performed after analytics.

Preserve failed data

Invalid records should remain inspectable with a failure reason so they can be investigated, corrected, and replayed.

Trace transformations

Source records, processing logic, quality rules, and downstream analytical models should remain connected through lineage.

Observe execution

A production pipeline should explain what ran, what changed, what failed, how long it took, and what downstream data was affected.

What comes next

Once the simulator contains the required interconnected business entities and historical behavior, Northstar will begin moving generated data into AWS. This page will then add actual execution history, validation results, quarantine records, lineage evidence, reconciliation metrics, and operational monitoring from the running pipeline.