Prodigy - AI Workflow Orchestration for Claude

Transform ad-hoc Claude sessions into reproducible development pipelines

Links: GitHub | Crates.io | Documentation

Overview

#

Prodigy is an AI-powered workflow orchestration tool that enables development teams to automate complex tasks using Claude AI through structured YAML workflows. It brings software engineering discipline to AI-assisted development by providing reproducible pipelines, parallel execution, comprehensive state management, and automatic error recovery.

Unlike manual Claude sessions that are ad-hoc and difficult to reproduce, Prodigy allows you to define complex development workflows as code, execute them with MapReduce-style parallelism, and track changes with full git integration.

The Problem

#

AI-assisted development faces critical limitations:

• Ad-hoc and non-reproducible - Manual Claude sessions can’t be easily repeated or refined
• Doesn’t scale - Manual sessions struggle with complex multi-step workflows or large-scale tasks
• No orchestration - No systematic way to coordinate multiple AI agents with state management
• Difficult to iterate - One-off coding sessions can’t be incrementally improved
• No error recovery - Failed tasks require manual intervention and restart from scratch

The Solution

#

Prodigy is a workflow orchestration system for Claude AI that transforms experimental AI coding into structured, maintainable processes. Key capabilities:

• Define workflows as YAML code - Version-controlled, reviewable, and reproducible pipeline definitions
• MapReduce parallel execution - Run multiple AI agents concurrently across git worktrees for large-scale tasks
• Automatic checkpoint & resume - Workflows can be interrupted and resumed from where they left off
• Dead Letter Queue (DLQ) - Failed items are automatically queued for retry after fixing root causes
• Full git integration - Automatic commits, branch management, and merge workflows
• Comprehensive error handling - Circuit breakers, exponential backoff, and automatic retry mechanisms

Key Features

#

Workflow Types

#

Standard Workflows: Sequential command execution with conditional logic, error handling, and output capture:

- shell: "cargo build"
- shell: "cargo test"
  on_failure:
    claude: "/fix-failing-tests"
- shell: "cargo clippy"

MapReduce Workflows: Parallel processing across git worktrees with map/reduce phases for large-scale tasks:

mode: mapreduce
setup:
  - shell: "debtmap analyze . --output items.json"
map:
  input: "items.json"
  json_path: "$.items[*]"
  agent_template:
    - claude: "/process '${item}'"
  max_parallel: 10
reduce:
  - claude: "/summarize ${map.results}"

Command Types

#

Core Commands:

• shell: - Execute shell commands with output capture and conditional execution
• claude: - Invoke Claude Code commands with arguments and commit requirements

Advanced Commands:

• goal_seek: - Iteratively refine code until validation thresholds are met (e.g., “achieve 90% test coverage”)
• foreach: - Iterate over lists with parallel execution support
• validate: - Validate implementation completeness with automatic retry
• write_file: - Write content to files with format validation (JSON, YAML, text)
• analyze: - Run analysis handlers for coverage, complexity metrics, etc.

Error Handling & Recovery

#

Automatic Retry:

• Exponential backoff with configurable strategies (fixed, linear, exponential, fibonacci)
• Circuit breaker pattern to prevent cascading failures
• Maximum failure thresholds with graceful degradation

Dead Letter Queue (DLQ):

• Failed work items automatically queued for review
• Retry failed items after fixing root causes: prodigy dlq retry <job_id>
• Preserves correlation IDs and failure context

Checkpoint & Resume:

• Automatic checkpoints enable resumption after interruption
• Work item state recovery (in-progress items reset to pending)
• Partial results preserved across resume operations

Parallel Processing

#

MapReduce Architecture:

• Process work items in parallel across multiple git worktrees
• Global storage in ~/.prodigy/ for cross-worktree event aggregation
• Configurable parallelism with max_parallel setting
• Event tracking for debugging and monitoring

Performance Tuning:

• Configure parallelism based on task characteristics (CPU-bound, I/O-bound, memory-intensive)
• Agent timeout configuration per workflow
• Circuit breaker tuning for flaky operations

Git Integration

#

Worktree Isolation:

• Each parallel agent runs in isolated git worktree
• Prevents conflicts between concurrent operations
• Automatic cleanup after workflow completion

Merge Workflows:

• Custom validation and testing before merging changes
• Automatic conflict resolution with Claude
• Integration with main branch workflows

Configuration & Customization

#

Environment Management:

• Global environment variables with env section
• Secret variables (masked in logs) with secrets section
• Environment files (.env) loading
• Environment profiles for switching contexts (development, production, etc.)
• Step-level environment overrides

Variable System:

• Dynamic value capture from command outputs
• Multiple capture formats (string, json, lines, number, boolean)
• Variable interpolation throughout workflows
• Metadata capture (exit codes, success status, duration)

Flexible Configuration:

• Project-level (.prodigy/config.yml)
• User-level (~/.prodigy/config.yml)
• System-level configuration
• Template support with imports for reusability

Technical Highlights

#

• Language: Rust (for reliability, performance, and memory safety)
• Integration: Claude AI API via Claude Code
• State Management: Git worktree isolation with global storage architecture
• Architecture: Workflow engine with checkpoint system and event tracking
• Parallel Execution: MapReduce pattern across isolated worktrees
• Storage: Global storage in ~/.prodigy/ for events, DLQ, and checkpoints

Real-World Use Cases

#

Code Review at Scale

#

Review all open pull requests in parallel:

mode: mapreduce
setup:
  - shell: "gh pr list --json number,title --limit 100 > prs.json"
map:
  input: "prs.json"
  json_path: "$[*]"
  agent_template:
    - shell: "gh pr checkout ${item.number}"
    - claude: "/review-pr ${item.number}"
  max_parallel: 10

Multi-File Refactoring

#

Refactor common patterns across many files:

mode: mapreduce
setup:
  - shell: "rg -l 'unwrap\\(\\)' src/ --json > files.json"
map:
  input: "files.json"
  json_path: "$[*]"
  agent_template:
    - claude: "/refactor-unwrap ${item.path}"
    - shell: "cargo test"
      on_failure:
        claude: "/fix-tests"
  max_parallel: 4

Documentation Drift Analysis

#

Analyze and fix documentation for multiple chapters:

mode: mapreduce
setup:
  - shell: "ls book/src/*.md | jq -R -s 'split(\"\\n\")' > chapters.json"
map:
  input: "chapters.json"
  json_path: "$[*]"
  agent_template:
    - claude: "/analyze-drift ${item}"
    - claude: "/fix-drift ${item}"
  max_parallel: 8

Test Suite Parallelization

#

Run test suites in parallel with automatic DLQ handling:

mode: mapreduce
setup:
  - shell: "cargo test --list --format json > tests.json"
map:
  input: "tests.json"
  json_path: "$[*]"
  agent_template:
    - shell: "cargo test ${item.name}"
  max_parallel: 16
  error_policy:
    on_item_failure: dlq
    continue_on_failure: true

Installation

#

# Via Cargo
cargo install prodigy

# Or build from source
git clone https://github.com/iepathos/prodigy
cd prodigy
cargo install --path .

Quick Start

#

Create a simple workflow in workflow.yml:

- shell: "cargo build"
- shell: "cargo test"
  on_failure:
    claude: "/fix-failing-tests"
- shell: "cargo clippy"

Run it:

prodigy run workflow.yml

Commands

#

Workflow Execution:

• prodigy run <workflow.yml> - Execute a workflow
• prodigy run --profile <name> - Run with environment profile

Job Management:

• prodigy resume-job <job_id> - Resume interrupted MapReduce job
• prodigy events <job_id> - View job events
• prodigy events <job_id> --follow - Stream events in real-time

DLQ Operations:

• prodigy dlq list <job_id> - List failed items
• prodigy dlq retry <job_id> - Retry failed items
• prodigy dlq show <job_id> <item_id> - Show item details
• prodigy dlq clear <job_id> - Clear DLQ after manual fixes

Impact

#

Prodigy transforms one-off AI coding sessions into structured, maintainable, and repeatable development processes. Teams can now:

• Automate repetitive tasks - Code reviews, refactoring, testing at scale
• Recover from failures - Automatic retry with DLQ for systematic error resolution
• Work in parallel - Process hundreds of files/tasks concurrently
• Iterate and improve - Version-controlled workflows that can be refined over time
• Apply engineering rigor - Bring the same discipline to AI-assisted development as traditional software engineering

Links

#

• GitHub: github.com/iepathos/prodigy
• Documentation: iepathos.github.io/prodigy
• Crates.io: Available via cargo install prodigy

Integration with Other Tools

#

Prodigy integrates seamlessly with other development tools:

• Debtmap Integration - Use debtmap to identify refactoring candidates, then automate fixes with prodigy MapReduce workflows
• GitHub CLI - Automate PR reviews, issue management, and code reviews at scale
• Coverage Tools - Integrate cargo tarpaulin or other coverage tools in validation workflows
• CI/CD Pipelines - Run prodigy workflows as part of automated testing and deployment pipelines

Project Status

#

Active development, production-ready, available on crates.io.

Key Features Implemented:

• Standard and MapReduce workflow execution
• Checkpoint & resume functionality
• Dead Letter Queue (DLQ) with retry
• Git worktree isolation
• Circuit breaker and retry mechanisms
• Event tracking and monitoring
• Environment management and variable system