Automating Documentation Maintenance with Prodigy: A Real-World Case Study

The Documentation Drift Problem

#

Every developer knows the pain: you ship a new feature, refactor your code, update configuration options—and your documentation slowly falls out of sync. What starts as a minor discrepancy becomes a maze of outdated examples, incorrect defaults, and missing features that frustrate users and waste maintainer time.

For Debtmap, a Rust-based technical debt analyzer, this problem was particularly acute. I was iterating quickly so the docs I’d have one day were wrong a week or a few days later. Keeping documentation aligned with a rapidly evolving codebase felt like a Sisyphean task.

The solution? I built an automated workflow using Prodigy that detects and fixes documentation drift. Using it for helping me maintain documentation in Debtmap Documentation It let me shift from a basic README.md to 27 chapter comprehensive book chapters covering everything from CLI reference to advanced architectural analysis.

What is Prodigy?

#

Prodigy is an AI-powered workflow automation system that orchestrates complex multi-step tasks using a map-reduce architecture. Think of it as CI/CD for AI agents: you define workflows as declarative YAML files and Claude commands, and Prodigy handles parallel execution, error recovery, merging, and validation.

The key innovation is agent-based map-reduce: instead of just running shell commands, each work item gets its own AI agent that can analyze code, understand context, make decisions, and apply fixes—all while maintaining clean git history and handling failures gracefully.

The Workflow Architecture

#

Here’s the high-level structure of the documentation drift detection workflow:

name: prodigy-book-docs-drift-detection
mode: mapreduce

# Setup: Analyze codebase and detect gaps
setup:
  - Analyze codebase for features, commands, and config options
  - Detect documentation gaps and create missing chapters

# Map: Process each chapter in parallel
map:
  input: "workflows/data/prodigy-chapters.json"
  json_path: "$.chapters[*]"

  agent_template:
    - Analyze chapter for drift against current implementation
    - Fix detected drift issues

  max_parallel: 3

# Reduce: Validate and merge
reduce:
  - Build the book to ensure all chapters compile
  - Clean up temporary analysis files

Three-Phase Approach

#

1. Setup Phase: Feature Discovery

Before we can detect drift, we need to know what the current codebase actually supports. The workflow:

• Scans the codebase for workflow features, command types, and configuration options
• Extracts actual implementation details (not just what the docs claim)
• Identifies completely missing chapters for undocumented features
• Creates a feature inventory at .prodigy/book-analysis/features.json

This phase uses custom Claude Code slash commands like /prodigy-analyze-features-for-book that encapsulate domain-specific analysis logic.

2. Map Phase: Parallel Drift Analysis

Each of the 27 chapters gets its own dedicated AI agent that:

1. Analyzes drift - Compares chapter content against the feature inventory

   • Detects outdated examples
   • Finds incorrect default values
   • Identifies missing features
   • Checks configuration option accuracy

2. Fixes drift - Applies corrections while preserving style

   • Updates code examples
   • Corrects configuration defaults
   • Adds missing sections
   • Maintains existing prose quality

The agents run in parallel (3 at a time to balance throughput and resource usage), each working in isolated git worktrees to prevent conflicts.

3. Reduce Phase: Validation

After all chapters are fixed:

• Rebuild the book with mdbook build to catch broken links
• Validate cross-chapter consistency
• Merge all changes back to the main branch
• Clean up temporary analysis artifacts

If the book build fails, an agent automatically diagnoses and fixes the issues.

Real Results:

#

I just ran this workflow on Debtmap’s documentation. Here’s what happened:

Execution Metrics

#

• 27 chapters analyzed and fixed
• 122 drift analysis reports generated (avg 4.5 per chapter)
• 94 fix commits applied
• 100% completion rate - zero permanent failures
• Book built successfully - all chapters compile together

Example: The Configuration Chapter

#

One representative fix (commit 9ad9445a):

Fixed 10 drift issues (0 critical, 6 medium, 4 low)

Key updates:
- Updated role coverage weights (pure_logic: 1.0, io_wrapper: 0.5)
- Added documentation for debug role multiplier
- Added deprecation warnings for obsolete threshold fields
- Fixed max_total_debt_score default (1000 → 10000)
- Added classification configuration section
- Documented advanced configs (orchestration_adjustment, etc.)

The agent didn’t just find surface-level issues—it caught semantic drift like a 10x error in a default value that would have confused users.

Example: The Analysis Guide Chapter

#

Another fix (commit c0e1fe9e):

Fixed 9 drift issues (all low severity)

Key updates:
- Corrected debt type count (24 → 25)
- Added est_branches field with formula explanation
- Clarified minimal_count vs legacy risk categories
- Enhanced unified JSON format documentation

These are the kinds of subtle discrepancies that accumulate over time and degrade documentation quality.

The Multi-Pass Refinement Strategy

#

You might notice something interesting: 122 analyses for 27 chapters means an average of 4.5 analysis passes per chapter. This isn’t a bug—it’s a feature.

The workflow uses an iterative refinement approach:

1. First pass: Broad drift detection
2. Second pass: Verify fixes and catch edge cases
3. Subsequent passes: Refine until no drift remains

Each pass generates a detailed drift report that guides the next round of fixes. This ensures comprehensive coverage without requiring perfect first-pass detection.

The workflow stops when drift severity drops below thresholds or all issues are resolved—whichever comes first.

Error Handling and Recovery

#

Real-world workflows encounter failures. Prodigy handles this with:

error_policy:
  on_item_failure: dlq        # Dead letter queue for failures
  continue_on_failure: true    # Don't stop the whole workflow
  max_failures: 2              # Reasonable fault tolerance
  error_collection: aggregate  # Collect all errors for analysis

In this run: zero items reached the DLQ. All chapters were successfully analyzed and fixed, demonstrating the robustness of the agent-based approach.

When issues do occur, agents can often self-recover. For example, if a book build fails due to broken links, the reduce phase automatically spawns an agent to diagnose and fix the build errors.

The Git History Tells the Story

#

Each step leaves a clear audit trail:

07541ed7 Merge branch 'prodigy-session-654c3786-e04d-44ce-bfc0-600817dbb9ea'
d013e30e Merge agent agent-mapreduce-20251030_202918_agent_3-item_3
c0e1fe9e docs: fix Debtmap book chapter 'Analysis Guide'
32a43c20 analysis: drift report for Debtmap book chapter 'Analysis Guide'

The pattern is consistent:

1. Analysis commit with drift report
2. Fix commit with detailed changes
3. Agent merge commits for parallel work
4. Final session merge

This creates a reviewable, bisectable history where every change is justified by a corresponding analysis.

Key Learnings

#

1. Feature Inventory is Critical

#

You can’t detect drift without ground truth. Investing in comprehensive codebase analysis upfront pays dividends in detection accuracy.

2. Iterative Refinement Beats One-Shot Analysis

#

Multi-pass refinement caught issues that a single analysis missed. The extra compute cost is worth it for thoroughness.

3. Parallel Processing Scales

#

Processing 27 chapters serially would have taken hours. With max_parallel: 3, the entire workflow completed in a reasonable timeframe while staying within resource constraints.

4. Agent Autonomy Reduces Toil

#

Instead of reviewing 122 drift reports manually, I let agents fix the issues and reviewed the git diffs. This inverted the effort: AI does the tedious work, humans do the strategic review.

5. Validate Everything

#

The book build step caught integration issues that per-chapter analysis missed. Always include cross-cutting validation in your reduce phase.

Practical Benefits

#

Since implementing this workflow:

1. Confidence in documentation - I can ship code changes knowing docs will be updated automatically
2. Reduced maintenance burden - No more manual doc review marathons
3. Faster releases - Documentation updates don’t block releases
4. Better user experience - Examples actually work, defaults are correct
5. Continuous improvement - Run the workflow on every release or schedule it weekly

How to Adapt This for Your Project

#

The workflow is generalizable. Here’s how to adapt it:

1. Define Your Chapters

#

Create a JSON inventory of your documentation:

{
  "chapters": [
    {
      "id": "getting-started",
      "title": "Getting Started",
      "file": "docs/getting-started.md",
      "topics": ["Installation", "Quick start", "First example"],
      "validation": "Check installation commands are current"
    }
  ]
}

2. Build Feature Extraction

#

Write a slash command (or script) that extracts your project’s current state:

• CLI commands and options
• Configuration schema
• API endpoints
• Supported features

3. Create Drift Detection Logic

#

The /prodigy-analyze-book-chapter-drift command compares chapter content against the feature inventory. You can implement this as:

• Custom slash command with domain logic
• Generic AI agent with clear instructions
• Combination of static analysis + AI refinement

4. Define Fix Standards

#

Give your agents clear guidelines:

• Preserve existing writing style
• Update only factual inaccuracies
• Add missing sections for new features
• Flag breaking changes for human review

5. Add Validation

#

Your reduce phase should validate:

• Documentation builds successfully
• Cross-references are valid
• Code examples compile/run
• No broken links

Future Enhancements

#

This workflow is already production-ready, but there’s room for improvement:

• Proactive gap detection: Alert when new code is committed without corresponding docs
• Version-specific docs: Generate documentation for multiple versions automatically
• Interactive examples: Validate that code examples actually execute correctly
• Screenshot updates: Detect when UI screenshots are outdated
• Multi-language support: Maintain docs in multiple languages with translation drift detection

Conclusion

#

Documentation drift is a systemic problem that can’t be solved by individual discipline—it requires systemic automation. By treating documentation as code and applying AI-powered map-reduce workflows, we can maintain high-quality, accurate documentation without manual toil.

The Prodigy workflow I’ve described isn’t theoretical—it’s running in production on Debtmap and Prodigy right now, keeping chapters synchronized with fast-moving codebases. The same approach can work for your project.

The era of stale documentation is over. Automate it, validate it, and ship with confidence.

Resources

#

• Debtmap: Learn more about Debtmap | GitHub | Crates.io
• Prodigy: Learn more about Prodigy | GitHub | Crates.io
• Workflow file: book-docs-drift.yml
• Example drift analysis: Commit 32a43c20

Get Started

#

Want to automate your documentation maintenance? Check out the Prodigy project page to learn how to set up AI-powered workflows for your team.

Interested in technical debt analysis? Explore Debtmap to see how entropy-based complexity metrics can help you prioritize refactoring.

About the Author

#

I’m Glen Baker, building Debtmap to help teams tackle technical debt systematically, and Prodigy to automate the kind of repetitive, high-stakes work that developers shouldn’t have to do manually. If you’re interested in AI-powered development workflows, read more on my blog or get in touch.

This blog post was written after a successful production run of the automated documentation workflow, demonstrating that the system works as advertised—dogfooding at its finest.