Data Engineering
dbt's testing model goes well beyond the four built-in tests. Generic tests written as macros are reusable across your entire project, singular tests handle complex multi-table business rules, and the dbt-expectations package brings 50+ Great Expectations-style assertions into SQL. In CI, dbt test --select
Data Engineering
Data pipeline QA requires a layered approach: unit tests for transformation logic, contract tests for schema agreements between systems, integration tests for end-to-end correctness, and SLA-based operational tests for latency and freshness. Idempotency and chaos testing are the two most commonly skipped layers — and the two most
Data Engineering
Testing Apache Flink requires specialized tools at each layer: MiniClusterWithClientResource for topology-level tests, KeyedOneInputStreamOperatorTestHarness for stateful operators, and EmbeddedKafkaCluster for end-to-end integration. Event-time semantics and exactly-once guarantees demand explicit test harness control over watermarks and checkpoints. Key Takeaways Unit test operators in isolation. Use KeyedOneInputStreamOperatorTestHarness
Data Engineering
Great Expectations structures data quality as code: Expectations describe what data should look like, Validators apply them to DataFrames or tables, and Checkpoints bundle validation into CI-runnable steps. Data Docs turn validation results into human-readable HTML reports your team can share. Key Takeaways Write expectations from profiles, refine
Airflow
Apache Airflow DAGs are Python code. That means you can test them like any other Python code — validate structure, unit test operators, and run DAGs against test data. This guide covers the full testing spectrum for Airflow. DAG Integrity Tests The first thing to test is that your DAGs load
Testing
ETL testing verifies three things: all expected data was extracted (completeness), transformations produced correct output (accuracy), and the data was loaded without corruption or duplication (integrity). Each phase requires different testing techniques: reconciliation counts and checksums for extraction, sample-based assertions for transformation, and row count plus constraint checks for
Testing
Testing data quality in Snowflake requires both SQL-based tests (dbt works natively with Snowflake) and Python-based data validation (Great Expectations connects via the Snowflake SQLAlchemy adapter). Run dbt tests for schema and transformation validation, Great Expectations checkpoints for statistical and pattern-based assertions, and scheduled CI pipelines to
Testing
Testing pandas code requires different patterns than testing application code. DataFrames are complex objects—you rarely want to assert on the entire DataFrame at once. Instead, assert on specific columns, row counts, data types, and statistical properties. Use pytest fixtures for reusable test data, pandas.testing for DataFrame comparisons, and
Testing
Data pipeline orchestrators (Airflow, Prefect, Dagster) each have different testing models, but the principle is the same: unit-test your business logic in isolation, integration-test the orchestration layer with real or containerized dependencies. This guide covers testing patterns for all three orchestrators plus shared practices that apply to any
Testing
Great Expectations (GX) is a Python library for defining and running data quality assertions against DataFrames, files, and databases. You define Expectations (e.g., "this column should be non-null"), group them into Expectation Suites, and run them via Checkpoints. Failed Expectations produce HTML Data Docs reports showing
Testing
dbt ships with four built-in tests (not_null, unique, accepted_values, relationships) that cover the basics. For richer assertions—value ranges, regex patterns, statistical checks—add the dbt-expectations package. When neither covers your case, write a custom generic test. This guide covers all three layers plus how to
Machine Learning
Feature stores solve the problem of feature reuse and training/serving consistency in ML systems.