Your CI/CD Pipeline Should Take Under 10 Minutes — Here's How
The 45-Minute Pipeline Problem
A client's CI pipeline took 45 minutes. Developers pushed code, went to lunch, came back, discovered a failing test, fixed it, pushed again, went to another meeting. A single feature took 3 round-trips through CI. That's 2+ hours of waiting per feature — per developer, per day.
We got it to 7 minutes. Here's every optimization.
Step 1: Measure Before You Optimize
You can't improve what you don't measure:
# GitHub Actions: export workflow timing data
gh run list --limit 50 --json databaseId,conclusion,createdAt,updatedAt \
| jq '.[] | {id: .databaseId, duration: ((.updatedAt | fromdate) - (.createdAt | fromdate)) / 60}'
# Typical findings:
# Install dependencies: 4 min (cacheable)
# Lint: 2 min (parallelizable)
# Type check: 3 min (parallelizable)
# Unit tests: 12 min (parallelizable + cacheable)
# Integration tests: 15 min (biggest bottleneck)
# Build: 6 min (cacheable)
# Deploy: 3 min (sequential, fine)
# Total: 45 minStep 2: Cache Everything
Dependencies don't change on every commit. Stop downloading them every time:
# GitHub Actions: aggressive caching
- name: Cache node_modules
uses: actions/cache@v4
with:
path: |
node_modules
~/.npm
key: node-${{ hashFiles('package-lock.json') }}
restore-keys: node-
- name: Cache Turbo
uses: actions/cache@v4
with:
path: .turbo
key: turbo-${{ github.sha }}
restore-keys: turbo-
- name: Cache test results
uses: actions/cache@v4
with:
path: .jest-cache
key: jest-${{ hashFiles('src/**/*.ts', 'src/**/*.tsx') }}Impact: Install step: 4 min → 15 seconds (cache hit). Build step: 6 min → 45 seconds (Turbo cache).
Step 3: Parallelize Independent Steps
Lint, type check, and unit tests don't depend on each other:
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/cache@v4 # ... cache config
- run: npm run lint
typecheck:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/cache@v4
- run: npm run typecheck
unit-tests:
runs-on: ubuntu-latest
strategy:
matrix:
shard: [1, 2, 3, 4] # Split tests across 4 runners
steps:
- uses: actions/checkout@v4
- uses: actions/cache@v4
- run: npm run test -- --shard=${{ matrix.shard }}/4
integration-tests:
needs: [lint, typecheck] # Run after lint/typecheck pass
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npm run test:integration
build-and-deploy:
needs: [unit-tests, integration-tests]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npm run build
- run: npm run deployImpact: Parallel lint + typecheck + unit tests: 17 min → 4 min (wall clock).
Step 4: Test Sharding
Split your test suite across multiple runners:
// jest.config.js — enable sharding
module.exports = {
// Jest built-in sharding (v29+)
// Run with: jest --shard=1/4
// Or use test file distribution
projects: [
{ displayName: "unit", testMatch: ["<rootDir>/src/**/*.test.ts"] },
{ displayName: "api", testMatch: ["<rootDir>/tests/api/**/*.test.ts"] },
],
};Impact: 12-minute test suite → 3.5 minutes across 4 shards.
Step 5: Smarter Integration Tests
Integration tests are usually the biggest bottleneck. Optimize them:
// Use test containers with pre-built images
// docker-compose.test.yml
const testDB = await new PostgreSqlContainer("postgres:16-alpine")
.withReuse() // Reuse container across test files
.withTmpFs({ "/var/lib/postgresql/data": "rw" }) // tmpfs for speed
.start();
// Run migrations once, use transactions for isolation
beforeAll(async () => {
await runMigrations(testDB.getConnectionUri());
});
beforeEach(async () => {
await db.query("BEGIN"); // Start transaction
});
afterEach(async () => {
await db.query("ROLLBACK"); // Rollback — instant cleanup
});Impact: Integration tests: 15 min → 5 min (tmpfs + transaction rollback instead of truncate).
Step 6: Only Run What Changed
Don't run the entire test suite for a README change:
# Path-based filtering
on:
pull_request:
paths:
- 'src/**'
- 'tests/**'
- 'package.json'
# Ignore: docs, README, .github/workflows (unless CI config itself changed)
# Or use affected-project detection with Turbo/Nx
- run: npx turbo run test --filter=...[origin/main]The Result
Before optimization:
Install: 4:00 → 0:15 (cached)
Lint: 2:00 → 1:30 (parallel)
Type check: 3:00 → 2:00 (parallel)
Unit tests: 12:00 → 3:30 (4 shards, parallel)
Integration tests: 15:00 → 5:00 (tmpfs + rollback)
Build: 6:00 → 0:45 (Turbo cache)
Deploy: 3:00 → 2:00 (no change needed)
Total (sequential): 45:00
Total (optimized): 6:45 (parallel execution)
The pipeline went from 45 minutes to under 7 minutes. Developer round-trips dropped from 3 per feature to 1. And the team stopped pushing directly to main "because CI takes too long" — which was the real risk all along.