Supply Chain Security

Dependabot vs Renovate: Testing Automated Dependency Update Strategies

HelpMeTest

HelpMeTest

— 6 min read
Dependabot vs Renovate: Testing Automated Dependency Update Strategies

Automated dependency updates are your first line of defense against supply chain vulnerabilities. But choosing between Dependabot and Renovate — and validating that your update pipeline actually works — requires deliberate testing. This guide walks through both tools, their testing implications, and how to verify your dependency management strategy is actually protecting you.

Why Automated Dependency Updates Are Security-Critical

The Log4Shell vulnerability (CVE-2021-44228) affected millions of applications. Organizations with automated dependency update pipelines patched within hours. Those without manual processes took days or weeks — during which attackers were actively exploiting the flaw.

Automated dependency management isn't just about convenience. It's about reducing the window between vulnerability disclosure and remediation. But automation only helps if it's correctly configured and tested.

Dependabot: GitHub's Native Solution

Dependabot is GitHub's built-in dependency update tool. It monitors your dependency files and opens PRs when updates are available.

Basic Configuration

# .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: "npm"
    directory: "/"
    schedule:
      interval: "weekly"
      day: "monday"
      time: "09:00"
    open-pull-requests-limit: 10
    reviewers:
      - "security-team"
    labels:
      - "dependencies"
      - "security"
    ignore:
      - dependency-name: "lodash"
        versions: ["4.x"]
    groups:
      dev-dependencies:
        patterns:
          - "*"
        dependency-type: "development"

  - package-ecosystem: "pip"
    directory: "/"
    schedule:
      interval: "daily"
    open-pull-requests-limit: 5
    
  - package-ecosystem: "docker"
    directory: "/"
    schedule:
      interval: "weekly"
    
  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "weekly"

Dependabot Security Alerts vs Version Updates

Dependabot has two modes:

  1. Security alerts: Triggered by GitHub Advisory Database entries. High priority, targets only vulnerable versions.
  2. Version updates: Proactive updates to latest versions. Scheduled, can generate noise.
# For security-focused teams: prioritize security alerts
updates:
  - package-ecosystem: "npm"
    directory: "/"
    schedule:
      interval: "daily"
    # Security alerts are always immediate regardless of schedule
    open-pull-requests-limit: 20  # Higher limit for security PRs

Renovate: The Flexible Alternative

Renovate is a more configurable option available as a GitHub App or self-hosted service.

// renovate.json
{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": [
    "config:recommended",
    ":dependencyDashboard",
    "security:openssf-scorecard"
  ],
  "schedule": ["before 6am on monday"],
  "prCreationHours": [9, 17],
  "timezone": "UTC",
  "labels": ["dependencies"],
  "reviewers": ["security-team"],
  "packageRules": [
    {
      "matchDepTypes": ["devDependencies"],
      "automerge": true,
      "automergeType": "pr",
      "platformAutomerge": true
    },
    {
      "matchPackagePatterns": ["*"],
      "matchUpdateTypes": ["patch"],
      "automerge": true,
      "minimumReleaseAge": "3 days"
    },
    {
      "matchPackagePatterns": ["*"],
      "matchUpdateTypes": ["minor"],
      "automerge": false,
      "groupName": "minor updates"
    },
    {
      "matchPackagePatterns": ["*"],
      "matchUpdateTypes": ["major"],
      "automerge": false,
      "labels": ["breaking-change"]
    }
  ],
  "vulnerabilityAlerts": {
    "labels": ["security"],
    "automerge": true,
    "stabilityDays": 0
  }
}

Testing Your Dependency Update Pipeline

Having a configuration file isn't enough. You need to verify your pipeline actually protects you.

Test 1: Verify PRs Are Being Created

The most basic test: check that your tool is actually opening PRs.

#!/bin/bash
# test-dependabot-activity.sh
# Requires: gh CLI authenticated

REPO="org/repo"
DAYS_TO_CHECK=7

echo "Checking for dependency update PRs in last $DAYS_TO_CHECK days..."

# Count open dependency PRs
OPEN_DEPENDENCY_PRS=$(gh pr list \
  --repo "$REPO" \
  --label "dependencies" \
  --state open \
  --json number,title,createdAt \
  --jq "map(select(.createdAt > (now - ${DAYS_TO_CHECK}*86400 | todate))) | length")

echo "Open dependency PRs: $OPEN_DEPENDENCY_PRS"

# Alert if no activity
if [ "$OPEN_DEPENDENCY_PRS" -eq 0 ]; then
  # Check if there are stale open ones
  TOTAL_OPEN=$(gh pr list \
    --repo "$REPO" \
    --label "dependencies" \
    --state open \
    --json number | jq length)
  
  if [ "$TOTAL_OPEN" -eq 0 ]; then
    echo "WARNING: No dependency update PRs found. Dependabot may be misconfigured."
    exit 1
  fi
fi

echo "PASS: Dependency updates are being generated"

Test 2: Validate Security Alert Response Time

# test_security_response_time.py
import subprocess
import json
from datetime import datetime, timedelta

def get_security_prs(repo: str, days: int = 30) -> list:
    """Get security-related dependency PRs."""
    result = subprocess.run([
        "gh", "pr", "list",
        "--repo", repo,
        "--label", "security",
        "--state", "all",
        "--json", "number,title,createdAt,mergedAt,closedAt,state",
        "--limit", "100"
    ], capture_output=True, text=True)
    
    prs = json.loads(result.stdout)
    cutoff = datetime.utcnow() - timedelta(days=days)
    
    return [
        pr for pr in prs
        if datetime.fromisoformat(pr["createdAt"].replace("Z", "+00:00")).replace(tzinfo=None) > cutoff
    ]

def calculate_mean_time_to_merge(prs: list) -> float:
    """Calculate average time from PR creation to merge in hours."""
    merge_times = []
    
    for pr in prs:
        if pr["mergedAt"]:
            created = datetime.fromisoformat(pr["createdAt"].replace("Z", ""))
            merged = datetime.fromisoformat(pr["mergedAt"].replace("Z", ""))
            hours = (merged - created).total_seconds() / 3600
            merge_times.append(hours)
    
    return sum(merge_times) / len(merge_times) if merge_times else None

def test_security_update_velocity():
    repo = "your-org/your-repo"
    prs = get_security_prs(repo)
    
    print(f"Security PRs in last 30 days: {len(prs)}")
    
    # Check merge rate
    merged = [pr for pr in prs if pr["state"] == "MERGED"]
    merge_rate = len(merged) / len(prs) if prs else 0
    
    print(f"Merge rate: {merge_rate:.1%}")
    assert merge_rate > 0.8, f"Security PR merge rate {merge_rate:.1%} below 80% threshold"
    
    # Check average time to merge
    avg_hours = calculate_mean_time_to_merge(merged)
    if avg_hours:
        print(f"Mean time to merge: {avg_hours:.1f} hours")
        assert avg_hours < 72, f"Average merge time {avg_hours:.1f}h exceeds 72h SLA"
    
    print("PASS: Security update velocity is within acceptable limits")

if __name__ == "__main__":
    test_security_update_velocity()

Test 3: Validate Automerge is Working for Safe Updates

# .github/workflows/test-automerge.yml
name: Test Automerge Configuration

on:
  pull_request:
    types: [opened, synchronize]
    
jobs:
  validate-automerge-eligible:
    if: github.actor == 'dependabot[bot]' || github.actor == 'renovate[bot]'
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
        uses: actions/checkout@v4
        
      - name: Get PR metadata
        id: pr-meta
        uses: actions/github-script@v7
        with:
          script: |
            const pr = await github.rest.pulls.get({
              owner: context.repo.owner,
              repo: context.repo.repo,
              pull_number: context.issue.number
            });
            
            const labels = pr.data.labels.map(l => l.name);
            const isDev = labels.includes('dev-dependencies');
            const isPatch = labels.includes('patch');
            const isSecurity = labels.includes('security');
            
            return { isDev, isPatch, isSecurity };
            
      - name: Validate tests pass before automerge
        run: |
          echo "Running tests for automerge candidate..."
          npm ci
          npm test
          
      - name: Log automerge eligibility
        run: |
          echo "PR is eligible for automerge: ${{ steps.pr-meta.outputs.isDev || steps.pr-meta.outputs.isPatch }}"

Test 4: Simulate a Vulnerable Dependency Introduction

# test_vulnerability_detection.py
"""
Test that your pipeline catches introduced vulnerabilities.
This test temporarily adds a known-vulnerable package to verify detection.
"""
import subprocess
import json
import time
import os

def install_vulnerable_package():
    """Install a known-vulnerable version for testing."""
    # lodash 4.17.4 has prototype pollution (CVE-2020-8203)
    subprocess.run(
        ["npm", "install", "lodash@4.17.4", "--save"],
        capture_output=True
    )

def run_vulnerability_scan() -> dict:
    """Run npm audit and return results."""
    result = subprocess.run(
        ["npm", "audit", "--json"],
        capture_output=True, text=True
    )
    return json.loads(result.stdout)

def cleanup_vulnerable_package():
    """Remove the test package."""
    subprocess.run(
        ["npm", "uninstall", "lodash"],
        capture_output=True
    )

def test_vulnerability_detection():
    """Verify that npm audit catches known vulnerabilities."""
    install_vulnerable_package()
    
    try:
        audit_results = run_vulnerability_scan()
        
        vulnerabilities = audit_results.get("vulnerabilities", {})
        lodash_vuln = vulnerabilities.get("lodash")
        
        assert lodash_vuln is not None, "lodash vulnerability not detected"
        assert lodash_vuln.get("severity") in ["high", "critical", "moderate"], \
            f"Unexpected severity: {lodash_vuln.get('severity')}"
        
        print(f"PASS: Vulnerability detected - {lodash_vuln['severity']} severity")
        
        total = audit_results.get("metadata", {}).get("vulnerabilities", {})
        print(f"Total vulnerabilities: {total}")
        
    finally:
        cleanup_vulnerable_package()

if __name__ == "__main__":
    test_vulnerability_detection()

Comparing Dependabot vs Renovate

Feature Dependabot Renovate
Setup complexity Minimal (native GitHub) Moderate (App or self-hosted)
Customization Limited Extensive
Automerge control Basic (GitHub Actions needed) Native, granular
Grouping PRs Yes (groups) Yes (packageRules)
Non-GitHub hosting GitHub only GitLab, Bitbucket, Azure DevOps
Vulnerability-only mode Yes (security alerts) Yes (vulnerabilityAlerts)
Dashboard PR No Yes (Dependency Dashboard)
Minimum release age No Yes (stabilityDays)
OSS scorecard integration No Yes
Regex versioning Limited Extensive

Building a Continuous Monitoring Workflow

# .github/workflows/dependency-security-monitor.yml
name: Dependency Security Monitor

on:
  schedule:
    - cron: '0 6 * * *'  # Daily at 6 AM UTC
  push:
    paths:
      - 'package-lock.json'
      - 'yarn.lock'
      - 'requirements.txt'
      - 'go.sum'
      - 'Cargo.lock'

jobs:
  vulnerability-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Run npm audit
        if: hashFiles('package-lock.json') != ''
        run: |
          npm ci
          npm audit --audit-level=high --json > npm-audit.json || true
          
      - name: Run OSV Scanner
        uses: google/osv-scanner-action@v1
        with:
          scan-args: |-
            --recursive
            ./
            
      - name: Run Trivy
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: 'fs'
          scan-ref: '.'
          format: 'sarif'
          output: 'trivy-results.sarif'
          severity: 'HIGH,CRITICAL'
          
      - name: Upload SARIF to GitHub Security
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: 'trivy-results.sarif'
          
      - name: Check for critical vulnerabilities
        run: |
          CRITICAL_COUNT=$(cat trivy-results.sarif | jq '[.runs[].results[] | select(.level == "error")] | length')
          if [ "$CRITICAL_COUNT" -gt "0" ]; then
            echo "FAIL: $CRITICAL_COUNT critical vulnerabilities found"
            exit 1
          fi
          echo "PASS: No critical vulnerabilities"

HelpMeTest Integration for Dependency Update Validation

After dependency updates merge, you want to verify the application still works. HelpMeTest can run your end-to-end test suite automatically when dependency PRs land.

# .github/workflows/post-dependency-update.yml
name: Post-Dependency Update Validation

on:
  pull_request:
    types: [closed]
    
jobs:
  validate-after-update:
    if: |
      github.event.pull_request.merged == true && 
      (github.event.pull_request.user.login == 'dependabot[bot]' ||
       github.event.pull_request.user.login == 'renovate[bot]')
    runs-on: ubuntu-latest
    steps:
      - name: Trigger HelpMeTest regression suite
        run: |
          curl -X POST https://api.helpmetest.com/v1/runs \
            -H "Authorization: Bearer ${{ secrets.HELPMETEST_API_KEY }}" \
            -H "Content-Type: application/json" \
            -d '{
              "suite": "regression",
              "trigger": "dependency-update",
              "pr_number": "${{ github.event.pull_request.number }}",
              "commit": "${{ github.sha }}"
            }'

HelpMeTest's plain-English test definitions mean your QA team can write tests like:

Test: Application loads after dependency update
Steps:
  1. Navigate to the home page
  2. Verify the page title is "My App"
  3. Log in with test credentials
  4. Verify the dashboard loads within 3 seconds
  5. Navigate to the settings page
  6. Verify all expected options are present

These tests run automatically after every dependency update merge, catching regressions that unit tests miss.

Best Practices Summary

For Dependabot:

  • Enable both security alerts AND version updates
  • Group dev dependencies to reduce PR noise
  • Use open-pull-requests-limit to prevent overwhelming reviewers
  • Pin GitHub Actions to SHAs for supply chain security

For Renovate:

  • Use minimumReleaseAge (3-7 days) to avoid immediately adopting broken releases
  • Enable the Dependency Dashboard for visibility
  • Automerge only patch updates and devDependencies after tests pass
  • Use packageRules to separate security updates from routine updates

For both:

  • Test that your pipeline actually creates PRs regularly
  • Measure mean time to merge for security PRs
  • Run integration tests after merges, not just unit tests
  • Monitor for "dependency confusion" attacks — see our guide on supply chain attack testing

Automated dependency management is only as good as the testing that validates each update. Configure the tooling, then verify it's working with regular, automated checks of the pipeline itself.

Read more

Start now free