Data Engineering - HelpMeTest (Page 2)

Testing

Snowflake Data Testing: dbt + Great Expectations for Data Quality

Testing data quality in Snowflake requires both SQL-based tests (dbt works natively with Snowflake) and Python-based data validation (Great Expectations connects via the Snowflake SQLAlchemy adapter). Run dbt tests for schema and transformation validation, Great Expectations checkpoints for statistical and pattern-based assertions, and scheduled CI pipelines to catch issues before

Testing

Pandas Testing Best Practices: Testing Data Transformations with pytest

Testing pandas code requires different patterns than testing application code. DataFrames are complex objects—you rarely want to assert on the entire DataFrame at once. Instead, assert on specific columns, row counts, data types, and statistical properties. Use pytest fixtures for reusable test data, pandas.testing for DataFrame comparisons, and

Testing

Data Pipeline Testing: Unit and Integration Tests for Airflow, Prefect, and Dagster

Data pipeline orchestrators (Airflow, Prefect, Dagster) each have different testing models, but the principle is the same: unit-test your business logic in isolation, integration-test the orchestration layer with real or containerized dependencies. This guide covers testing patterns for all three orchestrators plus shared practices that apply to any pipeline. Key

Testing

Great Expectations Tutorial: Data Validation and Pipeline Testing

Great Expectations (GX) is a Python library for defining and running data quality assertions against DataFrames, files, and databases. You define Expectations (e.g., "this column should be non-null"), group them into Expectation Suites, and run them via Checkpoints. Failed Expectations produce HTML Data Docs reports showing exactly

Testing

dbt Testing Guide: Schema Tests, dbt-expectations, and Custom Tests

dbt ships with four built-in tests (not_null, unique, accepted_values, relationships) that cover the basics. For richer assertions—value ranges, regex patterns, statistical checks—add the dbt-expectations package. When neither covers your case, write a custom generic test. This guide covers all three layers plus how to run tests

Machine Learning

Feature Store Testing and Data Lineage Validation

Feature stores solve the problem of feature reuse and training/serving consistency in ML systems. They also introduce a new category of bugs: features that compute correctly in isolation but produce wrong values when served at inference time due to point-in-time leakage, online/offline discrepancies, or stale data. Testing a