AWS S3 Testing Guide: Unit Testing with moto, Presigned URLs, Versioning, and Lifecycle Policies
Testing AWS S3 without hitting real buckets is straightforward with moto, a Python library that intercepts boto3 calls and simulates S3 in memory. This guide covers unit testing bucket operations, presigned URL generation, versioning, and lifecycle policies with runnable examples.
Key Takeaways
moto replaces real AWS calls in tests. Decorate your test functions with @mock_aws and boto3 behaves as if it's talking to real S3 — without credentials, costs, or network calls.
Presigned URL testing requires careful setup. moto generates valid-looking presigned URLs but they only work within the mocked context; test the URL structure and expiry parameters separately from actual download behavior.
Lifecycle policies are first-class testable objects. You can PUT a lifecycle configuration and GET it back to verify your IaC or SDK code produced exactly the right JSON structure.
Versioning tests need explicit enable calls. Bucket versioning is off by default in moto just like in real S3 — always call put_bucket_versioning before testing version-related behavior.
Use pytest fixtures to share mocked S3 state. A session-scoped fixture that creates buckets once is faster than recreating them in every test.
S3 is one of those services that looks trivially easy to test — until you actually try. Presigned URLs have expiry logic. Versioning changes how deletes work. Lifecycle policies have subtle validation rules. And hitting real AWS in CI means credentials, costs, and flaky network dependencies.
moto solves this. It's a Python library that patches boto3 at the transport layer so your code thinks it's talking to AWS but everything runs locally in memory. Here's how to use it properly.
Setting Up moto
Install the dependencies:
pip install boto3 moto[s3] pytestThe basic pattern is the @mock_aws decorator (moto 4.x+). Older tutorials use @mock_s3 — that still works but @mock_aws is the current API.
import boto3
import pytest
from moto import mock_aws
@mock_aws
def test_create_bucket():
s3 = boto3.client("s3", region_name="us-east-1")
s3.create_bucket(Bucket="my-test-bucket")
response = s3.list_buckets()
bucket_names = [b["Name"] for b in response["Buckets"]]
assert "my-test-bucket" in bucket_namesThat's it. No credentials needed. No AWS account. No cleanup — moto resets state between decorated functions.
Pytest Fixtures for S3 Tests
Decorating every test function gets repetitive. Use a pytest fixture instead:
import boto3
import pytest
from moto import mock_aws
@pytest.fixture
def s3_client():
with mock_aws():
client = boto3.client("s3", region_name="us-east-1")
yield client
@pytest.fixture
def test_bucket(s3_client):
bucket_name = "test-bucket-123"
s3_client.create_bucket(Bucket=bucket_name)
yield bucket_name
def test_upload_and_retrieve(s3_client, test_bucket):
s3_client.put_object(
Bucket=test_bucket,
Key="documents/report.pdf",
Body=b"PDF content here",
ContentType="application/pdf"
)
response = s3_client.get_object(Bucket=test_bucket, Key="documents/report.pdf")
content = response["Body"].read()
assert content == b"PDF content here"
assert response["ContentType"] == "application/pdf"
def test_object_not_found(s3_client, test_bucket):
from botocore.exceptions import ClientError
with pytest.raises(ClientError) as exc_info:
s3_client.get_object(Bucket=test_bucket, Key="nonexistent.txt")
assert exc_info.value.response["Error"]["Code"] == "NoSuchKey"The with mock_aws(): context manager in the fixture means every test that uses s3_client automatically gets a fresh, isolated mock. No state leaks between tests.
Testing Presigned URLs
Presigned URLs are generated server-side and allow time-limited access to private objects. Testing them has two parts: verifying your code generates URLs with the right parameters, and verifying the URL actually works.
import re
from datetime import timedelta
from urllib.parse import urlparse, parse_qs
def test_presigned_url_generation(s3_client, test_bucket):
# Upload an object first
s3_client.put_object(
Bucket=test_bucket,
Key="private/file.txt",
Body=b"secret content"
)
# Generate a presigned URL for 1 hour
url = s3_client.generate_presigned_url(
ClientMethod="get_object",
Params={"Bucket": test_bucket, "Key": "private/file.txt"},
ExpiresIn=3600
)
parsed = urlparse(url)
query_params = parse_qs(parsed.query)
# Verify the URL structure
assert test_bucket in parsed.netloc or test_bucket in parsed.path
assert "private/file.txt" in parsed.path
assert "X-Amz-Expires" in query_params
assert query_params["X-Amz-Expires"][0] == "3600"
assert "X-Amz-Signature" in query_params
def test_presigned_upload_url(s3_client, test_bucket):
# Generate a presigned URL for PUT (upload)
url = s3_client.generate_presigned_url(
ClientMethod="put_object",
Params={
"Bucket": test_bucket,
"Key": "uploads/user-file.jpg",
"ContentType": "image/jpeg"
},
ExpiresIn=900 # 15 minutes
)
parsed = urlparse(url)
query_params = parse_qs(parsed.query)
assert "X-Amz-Expires" in query_params
assert query_params["X-Amz-Expires"][0] == "900"
# Verify it's a PUT-type URL (different from GET)
assert "X-Amz-Signature" in query_params
def test_presigned_post(s3_client, test_bucket):
# generate_presigned_post is different from generate_presigned_url
response = s3_client.generate_presigned_post(
Bucket=test_bucket,
Key="uploads/${filename}",
Fields={"Content-Type": "image/jpeg"},
Conditions=[
{"Content-Type": "image/jpeg"},
["content-length-range", 1024, 10485760] # 1KB to 10MB
],
ExpiresIn=600
)
assert "url" in response
assert "fields" in response
assert "AWSAccessKeyId" in response["fields"]
assert "signature" in response["fields"]
assert response["fields"]["Content-Type"] == "image/jpeg"Testing Bucket Versioning
Versioning changes delete behavior significantly. With versioning enabled, deletes create a "delete marker" rather than removing the object. Your tests need to cover both modes.
def test_versioning_enabled(s3_client, test_bucket):
# Enable versioning
s3_client.put_bucket_versioning(
Bucket=test_bucket,
VersioningConfiguration={"Status": "Enabled"}
)
# Verify it's enabled
response = s3_client.get_bucket_versioning(Bucket=test_bucket)
assert response["Status"] == "Enabled"
def test_versioned_uploads_create_multiple_versions(s3_client, test_bucket):
# Enable versioning first
s3_client.put_bucket_versioning(
Bucket=test_bucket,
VersioningConfiguration={"Status": "Enabled"}
)
key = "config/settings.json"
# Upload version 1
v1 = s3_client.put_object(
Bucket=test_bucket, Key=key, Body=b'{"version": 1}'
)
version_id_1 = v1["VersionId"]
# Upload version 2 (same key)
v2 = s3_client.put_object(
Bucket=test_bucket, Key=key, Body=b'{"version": 2}'
)
version_id_2 = v2["VersionId"]
assert version_id_1 != version_id_2
# List all versions
versions_response = s3_client.list_object_versions(Bucket=test_bucket, Prefix=key)
versions = versions_response["Versions"]
assert len(versions) == 2
version_ids = [v["VersionId"] for v in versions]
assert version_id_1 in version_ids
assert version_id_2 in version_ids
def test_soft_delete_creates_delete_marker(s3_client, test_bucket):
s3_client.put_bucket_versioning(
Bucket=test_bucket,
VersioningConfiguration={"Status": "Enabled"}
)
key = "important-doc.txt"
s3_client.put_object(Bucket=test_bucket, Key=key, Body=b"important")
# Delete without specifying version ID — creates a delete marker
s3_client.delete_object(Bucket=test_bucket, Key=key)
# Object appears gone to normal GET
from botocore.exceptions import ClientError
with pytest.raises(ClientError) as exc:
s3_client.get_object(Bucket=test_bucket, Key=key)
assert exc.value.response["Error"]["Code"] == "NoSuchKey"
# But versions still exist
versions_response = s3_client.list_object_versions(Bucket=test_bucket, Prefix=key)
assert "DeleteMarkers" in versions_response
assert len(versions_response["DeleteMarkers"]) == 1
assert len(versions_response["Versions"]) == 1 # Original still thereTesting Lifecycle Policies
Lifecycle policies define automatic transitions (e.g., move to Glacier after 90 days) and expiration (delete after 365 days). Testing them means verifying the JSON structure you PUT is exactly what you expect.
def test_lifecycle_policy_creation(s3_client, test_bucket):
lifecycle_config = {
"Rules": [
{
"ID": "archive-old-logs",
"Status": "Enabled",
"Filter": {"Prefix": "logs/"},
"Transitions": [
{
"Days": 30,
"StorageClass": "STANDARD_IA"
},
{
"Days": 90,
"StorageClass": "GLACIER"
}
],
"Expiration": {"Days": 365}
}
]
}
# Apply the lifecycle config
s3_client.put_bucket_lifecycle_configuration(
Bucket=test_bucket,
LifecycleConfiguration=lifecycle_config
)
# Read it back
response = s3_client.get_bucket_lifecycle_configuration(Bucket=test_bucket)
rules = response["Rules"]
assert len(rules) == 1
rule = rules[0]
assert rule["ID"] == "archive-old-logs"
assert rule["Status"] == "Enabled"
assert rule["Filter"]["Prefix"] == "logs/"
assert len(rule["Transitions"]) == 2
assert rule["Expiration"]["Days"] == 365
def test_lifecycle_noncurrent_version_expiry(s3_client, test_bucket):
# For versioned buckets — expire old versions after N days
s3_client.put_bucket_versioning(
Bucket=test_bucket,
VersioningConfiguration={"Status": "Enabled"}
)
lifecycle_config = {
"Rules": [
{
"ID": "cleanup-old-versions",
"Status": "Enabled",
"Filter": {"Prefix": ""},
"NoncurrentVersionExpiration": {"NoncurrentDays": 30},
"AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 7}
}
]
}
s3_client.put_bucket_lifecycle_configuration(
Bucket=test_bucket,
LifecycleConfiguration=lifecycle_config
)
response = s3_client.get_bucket_lifecycle_configuration(Bucket=test_bucket)
rule = response["Rules"][0]
assert rule["NoncurrentVersionExpiration"]["NoncurrentDays"] == 30
assert rule["AbortIncompleteMultipartUpload"]["DaysAfterInitiation"] == 7Testing Your Application Code (Not Just boto3)
The real value of moto is testing your application's S3 integration, not raw boto3 calls. Here's a realistic example:
# storage.py — your application code
import boto3
from botocore.exceptions import ClientError
class FileStorage:
def __init__(self, bucket_name: str, s3_client=None):
self.bucket = bucket_name
self.s3 = s3_client or boto3.client("s3")
def upload_file(self, key: str, data: bytes, content_type: str = "application/octet-stream") -> str:
self.s3.put_object(Bucket=self.bucket, Key=key, Body=data, ContentType=content_type)
return key
def get_download_url(self, key: str, expires_in: int = 3600) -> str:
try:
self.s3.head_object(Bucket=self.bucket, Key=key)
except ClientError:
raise FileNotFoundError(f"Object {key} not found in bucket {self.bucket}")
return self.s3.generate_presigned_url(
ClientMethod="get_object",
Params={"Bucket": self.bucket, "Key": key},
ExpiresIn=expires_in
)
def delete_file(self, key: str) -> bool:
try:
self.s3.delete_object(Bucket=self.bucket, Key=key)
return True
except ClientError:
return False
# test_storage.py — tests for your application code
from moto import mock_aws
import boto3
import pytest
from storage import FileStorage
@pytest.fixture
def storage(s3_client, test_bucket):
return FileStorage(bucket_name=test_bucket, s3_client=s3_client)
def test_upload_returns_key(storage):
key = storage.upload_file("reports/q1.pdf", b"report data", "application/pdf")
assert key == "reports/q1.pdf"
def test_get_download_url_for_existing_file(storage):
storage.upload_file("file.txt", b"hello")
url = storage.get_download_url("file.txt", expires_in=300)
assert "file.txt" in url
assert "X-Amz-Expires=300" in url or "X-Amz-Expires" in url
def test_get_download_url_raises_for_missing_file(storage):
with pytest.raises(FileNotFoundError, match="missing.txt"):
storage.get_download_url("missing.txt")This approach tests your business logic in isolation. If you later switch from S3 to GCS, your FileStorage tests still define the expected behavior — only the implementation changes.
Running the Tests
# Run all S3 tests
pytest tests/test_storage.py -v
<span class="hljs-comment"># Run with coverage
pytest tests/test_storage.py --cov=storage --cov-report=term-missing
<span class="hljs-comment"># Run a specific test
pytest tests/test_storage.py::test_soft_delete_creates_delete_marker -vmoto doesn't require any environment variables, real AWS credentials, or network access. It works offline and in CI without any special setup. That's the whole point.
