Testing
Dagster is a data orchestration platform built around Software-Defined Assets (SDAs) — a model where pipelines declare what data assets they produce rather than what tasks to run. This asset-centric model makes testing more natural: you test that assets produce the right data, not just that code runs. This
Testing
Recce is a dbt change review tool that answers one question: what did this PR actually change in the data? When you modify a dbt model, your CI runs dbt tests and they pass — but that only tells you the data is valid, not whether it changed. Recce builds the
Testing
Data pipelines fail in ways that pure unit tests don't catch. A transformation that works on clean test data breaks on real-world schemas with extra columns, unexpected nulls, or timezone-aware timestamps. A join that looks correct drops 40% of rows silently. This guide covers pytest patterns
Testing
PySpark code is hard to test because most engineers treat Spark jobs as scripts — a chain of transformations from input to output with no unit boundaries. The result is code where bugs only surface when you run the full job against real data, which takes 20 minutes and costs money.
Testing
AWS Glue ETL jobs are PySpark scripts that run in AWS-managed Spark clusters. Testing them properly is challenging: you need Glue-specific libraries (GlueContext, DynamicFrame), access to AWS services, and a Spark runtime. This guide covers the full testing stack: local development with the Glue Docker container, unit testing
Testing
The TaskFlow API (introduced in Airflow 2.0) transforms DAG authoring: instead of operator-heavy boilerplate, you write plain Python functions decorated with @task. This makes Airflow code significantly more testable — task logic is just a function, and functions can be unit tested. This post covers testing patterns specific to
Testing
Apache Iceberg tables support schema evolution, partition spec changes, time-travel queries, and row-level deletes — all of which are correctness-critical operations that are easy to break silently. This guide covers testing each of these operations using PyIceberg and a local catalog, verifying that column additions don't
Testing
DuckDB is an in-process analytical database that starts in milliseconds, runs SQL against Parquet/CSV files natively, and integrates seamlessly with pandas and polars. It makes an ideal test fixture for data pipelines — replace your BigQuery or Redshift dependency with DuckDB in tests, keep your production SQL mostly unchanged,
Testing
Databricks code is notoriously hard to test because notebooks mix presentation with logic, and Delta Live Tables pipelines only run inside Databricks. This guide covers extracting testable functions from notebooks, unit testing PySpark transformations with a local Spark session, testing DLT expect() constraints, and running the full suite in CI
Testing
Testing Redshift pipelines is expensive and slow when done against a live cluster. This guide covers running Redshift-compatible tests against LocalStack, validating COPY/UNLOAD workflows, asserting on distribution and sort key behavior, and integrating Great Expectations for ongoing data quality monitoring — all in a CI pipeline that costs nothing
Testing
BigQuery pipelines are notoriously hard to test because every query costs money and requires live infrastructure. This guide covers three layers of testing: bq-test-kit for Python unit tests against a local emulator, dbt's built-in test framework for data quality assertions, and direct SQL testing patterns
Testing
Soda Core is an open-source data quality framework that lets you define checks in SodaCL (Soda Checks Language) — a YAML-based DSL — and run them against your database or data lake. Unlike Great Expectations, Soda Core uses declarative YAML instead of Python code, making checks readable by data analysts