Testing
Apache Iceberg tables support schema evolution, partition spec changes, time-travel queries, and row-level deletes — all of which are correctness-critical operations that are easy to break silently. This guide covers testing each of these operations using PyIceberg and a local catalog, verifying that column additions don't break readers, partition
Testing
DuckDB is an in-process analytical database that starts in milliseconds, runs SQL against Parquet/CSV files natively, and integrates seamlessly with pandas and polars. It makes an ideal test fixture for data pipelines — replace your BigQuery or Redshift dependency with DuckDB in tests, keep your production SQL mostly unchanged, and
Testing
Databricks code is notoriously hard to test because notebooks mix presentation with logic, and Delta Live Tables pipelines only run inside Databricks. This guide covers extracting testable functions from notebooks, unit testing PySpark transformations with a local Spark session, testing DLT expect() constraints, and running the full suite in CI
Testing
Testing Redshift pipelines is expensive and slow when done against a live cluster. This guide covers running Redshift-compatible tests against LocalStack, validating COPY/UNLOAD workflows, asserting on distribution and sort key behavior, and integrating Great Expectations for ongoing data quality monitoring — all in a CI pipeline that costs nothing per
Testing
BigQuery pipelines are notoriously hard to test because every query costs money and requires live infrastructure. This guide covers three layers of testing: bq-test-kit for Python unit tests against a local emulator, dbt's built-in test framework for data quality assertions, and direct SQL testing patterns for UDFs and
Data Engineering
Spark Structured Streaming tests fall into three layers: transformation unit tests using static DataFrames, micro-batch simulation using MemoryStream for source-side logic, and full integration tests with Testcontainers-Kafka. Watermark and late-data behavior requires careful trigger and clock control that MemoryStream provides without real streaming infrastructure. Key Takeaways Test transformations with static
Data Engineering
dbt's testing model goes well beyond the four built-in tests. Generic tests written as macros are reusable across your entire project, singular tests handle complex multi-table business rules, and the dbt-expectations package brings 50+ Great Expectations-style assertions into SQL. In CI, dbt test --select state:modified+ ensures only
Data Engineering
Data pipeline QA requires a layered approach: unit tests for transformation logic, contract tests for schema agreements between systems, integration tests for end-to-end correctness, and SLA-based operational tests for latency and freshness. Idempotency and chaos testing are the two most commonly skipped layers — and the two most likely to catch
Data Engineering
Testing Apache Flink requires specialized tools at each layer: MiniClusterWithClientResource for topology-level tests, KeyedOneInputStreamOperatorTestHarness for stateful operators, and EmbeddedKafkaCluster for end-to-end integration. Event-time semantics and exactly-once guarantees demand explicit test harness control over watermarks and checkpoints. Key Takeaways Unit test operators in isolation. Use KeyedOneInputStreamOperatorTestHarness to feed elements and watermarks
Data Engineering
Great Expectations structures data quality as code: Expectations describe what data should look like, Validators apply them to DataFrames or tables, and Checkpoints bundle validation into CI-runnable steps. Data Docs turn validation results into human-readable HTML reports your team can share. Key Takeaways Write expectations from profiles, refine manually. The
Airflow
Apache Airflow DAGs are Python code. That means you can test them like any other Python code — validate structure, unit test operators, and run DAGs against test data. This guide covers the full testing spectrum for Airflow. DAG Integrity Tests The first thing to test is that your DAGs load
Testing
ETL testing verifies three things: all expected data was extracted (completeness), transformations produced correct output (accuracy), and the data was loaded without corruption or duplication (integrity). Each phase requires different testing techniques: reconciliation counts and checksums for extraction, sample-based assertions for transformation, and row count plus constraint checks for loading.