Multi-Provider Extraction

When accuracy is critical, running extraction multiple times and comparing results can catch errors that a single pass might miss. This guide covers consensus voting strategies for high-stakes document processing.

Prerequisites

Node.js 18+
API keys for multiple providers (or OpenRouter for unified access)
A document to process

When to Use Consensus

Consensus voting is valuable when:

Processing high-value documents (financial, legal, medical)
Extraction errors have significant business impact
You need confidence scores for extracted values
Auditing requirements demand verification

Trade-off: Consensus increases cost and latency proportionally to the number of runs.

Basic Consensus Configuration

Enable consensus on the extract node:

import { createFlow, extract } from '@doclo/flows';
import { createVLMProvider } from '@doclo/providers-llm';

const vlmProvider = createVLMProvider({
  provider: 'google',
  model: 'google/gemini-2.5-flash',
  apiKey: process.env.OPENROUTER_API_KEY!,
  via: 'openrouter'
});

const flow = createFlow()
  .step('extract', extract({
    provider: vlmProvider,
    schema: invoiceSchema,
    consensus: {
      runs: 3,              // Run extraction 3 times
      strategy: 'majority', // Use majority voting
      threshold: 0.6        // Require 60% agreement
    }
  }))
  .build();

Voting Strategies

Majority (Default)

Each field is voted on independently. The value with the most votes wins:

consensus: {
  runs: 3,
  strategy: 'majority',
  threshold: 0.5  // More than half must agree
}

Example with 3 runs:

Run 1: { total: 1250.00 }
Run 2: { total: 1250.00 }
Run 3: { total: 1250.50 }

Result: { total: 1250.00 } (2/3 agreement)

Unanimous

All runs must agree for a value to be accepted:

consensus: {
  runs: 3,
  strategy: 'unanimous'
}

If any run disagrees, that field is marked as uncertain or null.

Weighted

Assign different weights to different providers or runs:

consensus: {
  runs: 3,
  strategy: 'weighted',
  weights: [0.5, 0.3, 0.2]  // First run weighted highest
}

Useful when you have a primary provider and backup providers.

Multi-Provider Consensus

Run extraction across different AI providers for diverse perspectives:

import { createFlow, extract } from '@doclo/flows';
import { createVLMProvider } from '@doclo/providers-llm';

// Create providers from different vendors
const geminiProvider = createVLMProvider({
  provider: 'google',
  model: 'google/gemini-2.5-flash',
  apiKey: process.env.OPENROUTER_API_KEY!,
  via: 'openrouter'
});

const claudeProvider = createVLMProvider({
  provider: 'anthropic',
  model: 'anthropic/claude-sonnet-4.5',
  apiKey: process.env.OPENROUTER_API_KEY!,
  via: 'openrouter'
});

const gpt4Provider = createVLMProvider({
  provider: 'openai',
  model: 'openai/gpt-4.1',
  apiKey: process.env.OPENROUTER_API_KEY!,
  via: 'openrouter'
});

// Run extraction with each provider
const flow = createFlow()
  .step('extract', extract({
    providers: [geminiProvider, claudeProvider, gpt4Provider],
    schema: invoiceSchema,
    consensus: {
      strategy: 'majority',
      threshold: 0.66  // 2 of 3 must agree
    }
  }))
  .build();

Agreement Thresholds

Set minimum agreement levels:

consensus: {
  runs: 5,
  strategy: 'majority',
  threshold: 0.8  // 80% must agree (4 of 5 runs)
}

Threshold	3 runs	5 runs	Description
0.5	2/3	3/5	Simple majority
0.66	2/3	4/5	Two-thirds majority
0.8	3/3	4/5	Strong consensus
1.0	3/3	5/5	Unanimous

Fields that don’t meet the threshold can be:

Set to null
Flagged for human review
Returned with a confidence score

Accessing Consensus Results

The output includes agreement information:

const result = await flow.run({ base64 });

console.log('Extracted data:', result.output.data);
console.log('Agreement scores:', result.output.agreement);

// Agreement per field
// {
//   invoiceNumber: { value: 'INV-001', agreement: 1.0, votes: 3 },
//   total: { value: 1250.00, agreement: 0.66, votes: 2 },
//   date: { value: '2024-01-15', agreement: 0.33, votes: 1, uncertain: true }
// }

Observability Hooks

Monitor consensus execution:

const flow = createFlow({
  observability: {
    onConsensusStart: (ctx) => {
      console.log(`Starting ${ctx.runsPlanned} consensus runs`);
      console.log(`Strategy: ${ctx.strategy}`);
    },
    onConsensusRunComplete: (ctx) => {
      console.log(`Run ${ctx.runIndex + 1} complete`);
      console.log(`  Provider: ${ctx.provider}`);
      console.log(`  Output: ${JSON.stringify(ctx.output)}`);
    },
    onConsensusComplete: (ctx) => {
      console.log('Consensus complete');
      console.log(`  Agreement: ${(ctx.agreement * 100).toFixed(0)}%`);
      console.log(`  Total cost: $${ctx.totalCost.toFixed(4)}`);
    }
  }
});

Complete Example: Financial Document Validation

Extract and validate financial data with high confidence:

import { createFlow, parse, extract } from '@doclo/flows';
import { createVLMProvider } from '@doclo/providers-llm';
import { createOCRProvider } from '@doclo/providers-datalab';
import { readFileSync } from 'fs';

// Types
interface FinancialExtraction {
  documentType: string;
  amount: number;
  currency: string;
  date: string;
  referenceNumber: string;
  parties: {
    payer: string;
    payee: string;
  };
}

// Schema
const financialSchema = {
  type: 'object',
  properties: {
    documentType: {
      type: 'string',
      enum: ['invoice', 'receipt', 'wire_transfer', 'check'],
      description: 'Type of financial document'
    },
    amount: {
      type: 'number',
      description: 'Transaction amount with full decimal precision'
    },
    currency: {
      type: 'string',
      description: '3-letter ISO currency code'
    },
    date: {
      type: 'string',
      description: 'Transaction date in YYYY-MM-DD format'
    },
    referenceNumber: {
      type: 'string',
      description: 'Reference, invoice, or transaction number'
    },
    parties: {
      type: 'object',
      properties: {
        payer: { type: 'string', description: 'Entity making payment' },
        payee: { type: 'string', description: 'Entity receiving payment' }
      },
      required: ['payer', 'payee']
    }
  },
  required: ['amount', 'currency', 'documentType']
};

// Providers
const ocrProvider = createOCRProvider({
  endpoint: 'https://www.datalab.to/api/v1/marker',
  apiKey: process.env.DATALAB_API_KEY!
});

const geminiProvider = createVLMProvider({
  provider: 'google',
  model: 'google/gemini-2.5-pro',
  apiKey: process.env.OPENROUTER_API_KEY!,
  via: 'openrouter'
});

const claudeProvider = createVLMProvider({
  provider: 'anthropic',
  model: 'anthropic/claude-sonnet-4.5',
  apiKey: process.env.OPENROUTER_API_KEY!,
  via: 'openrouter'
});

const gptProvider = createVLMProvider({
  provider: 'openai',
  model: 'openai/gpt-4.1',
  apiKey: process.env.OPENROUTER_API_KEY!,
  via: 'openrouter'
});

// Build flow with multi-provider consensus
const financialFlow = createFlow({
  observability: {
    onConsensusComplete: (ctx) => {
      console.log(`Consensus: ${(ctx.agreement * 100).toFixed(0)}% agreement`);

      // Flag low-agreement extractions for review
      if (ctx.agreement < 0.8) {
        console.warn('Low agreement - flagging for human review');
      }
    }
  }
})
  .step('parse', parse({ provider: ocrProvider }))
  .step('extract', extract<FinancialExtraction>({
    providers: [geminiProvider, claudeProvider, gptProvider],
    schema: financialSchema,
    inputMode: 'ir+source',
    consensus: {
      strategy: 'majority',
      threshold: 0.66
    },
    additionalInstructions: `
      - Amount must have exact decimal precision (e.g., 1234.56)
      - Currency must be a valid 3-letter ISO code
      - Date must be in YYYY-MM-DD format
      - Reference number should include any prefixes (INV-, REF-, etc.)
    `
  }))
  .build();

// Process document
async function validateFinancialDocument(filePath: string) {
  const fileBuffer = readFileSync(filePath);
  const base64 = `data:application/pdf;base64,${fileBuffer.toString('base64')}`;

  const result = await financialFlow.run({ base64 });

  console.log('\n--- Financial Document Validation ---');
  console.log('Type:', result.output.documentType);
  console.log('Amount:', result.output.currency, result.output.amount);
  console.log('Date:', result.output.date);
  console.log('Reference:', result.output.referenceNumber);
  console.log('Payer:', result.output.parties?.payer);
  console.log('Payee:', result.output.parties?.payee);

  console.log('\n--- Confidence ---');
  if (result.output.agreement) {
    for (const [field, info] of Object.entries(result.output.agreement)) {
      const status = info.uncertain ? '⚠️' : '✓';
      console.log(`${status} ${field}: ${(info.agreement * 100).toFixed(0)}%`);
    }
  }

  console.log('\n--- Cost ---');
  console.log(`Total: $${result.aggregated.totalCostUSD.toFixed(4)}`);

  return result.output;
}

validateFinancialDocument('./financial-doc.pdf').catch(console.error);

Cost Optimization Strategies

Consensus increases costs. Here are strategies to manage this:

Tiered Consensus

Use single extraction for low-value documents, consensus for high-value:

async function processDocument(doc: Document) {
  const estimatedValue = await estimateDocumentValue(doc);

  if (estimatedValue > 10000) {
    // High-value: 3-provider consensus
    return await highValueFlow.run(doc);
  } else if (estimatedValue > 1000) {
    // Medium-value: 2-run consensus
    return await mediumValueFlow.run(doc);
  } else {
    // Low-value: single extraction
    return await basicFlow.run(doc);
  }
}

Selective Field Consensus

Run consensus only on critical fields:

// First pass: extract all fields
const initialResult = await basicFlow.run({ base64 });

// Second pass: validate critical fields only
const criticalSchema = {
  type: 'object',
  properties: {
    amount: { type: 'number' },
    accountNumber: { type: 'string' }
  }
};

const validationResult = await createFlow()
  .step('validate', extract({
    providers: [provider1, provider2, provider3],
    schema: criticalSchema,
    consensus: { runs: 3, strategy: 'unanimous' }
  }))
  .build()
  .run({ base64 });

// Merge results
const finalResult = {
  ...initialResult.output,
  ...validationResult.output
};

Conditional Re-extraction

Only run consensus when initial extraction has low confidence:

const result = await flow.run({ base64 });

// Check for fields that might need validation
const uncertainFields = Object.entries(result.output.agreement || {})
  .filter(([_, info]) => info.agreement < 0.8)
  .map(([field]) => field);

if (uncertainFields.length > 0) {
  console.log('Re-validating uncertain fields:', uncertainFields);

  // Create schema for just uncertain fields
  const validationSchema = createSchemaForFields(uncertainFields);

  const validation = await consensusFlow.run({ base64, schema: validationSchema });
  // Merge validated fields back
}

Handling Disagreement

When providers disagree, you have options:

Return Null for Uncertain Fields

consensus: {
  runs: 3,
  strategy: 'majority',
  threshold: 0.66,
  onDisagreement: 'null'  // Set uncertain fields to null
}

Flag for Human Review

const result = await flow.run({ base64 });

const needsReview = Object.entries(result.output.agreement || {})
  .filter(([_, info]) => info.uncertain)
  .map(([field]) => field);

if (needsReview.length > 0) {
  await queueForHumanReview(result, needsReview);
}

Return All Candidate Values

consensus: {
  runs: 3,
  strategy: 'majority',
  threshold: 0.66,
  returnCandidates: true  // Include all extracted values
}

// Result includes candidates
// {
//   amount: {
//     value: 1250.00,
//     candidates: [1250.00, 1250.00, 1250.50],
//     agreement: 0.66
//   }
// }

Using Doclo Cloud with Consensus

Configure consensus for cloud-based flows:

import { DocloClient } from '@doclo/client';

const client = new DocloClient({
  apiKey: process.env.DOCLO_API_KEY!
});

// Run a pre-configured consensus flow
const result = await client.flows.run<FinancialExtraction>('financial-validation', {
  input: {
    document: { base64, filename: 'invoice.pdf', mimeType: 'application/pdf' }
  },
  wait: true,
  timeout: 120000  // Longer timeout for multi-run consensus
});

console.log('Extracted:', result.output);
console.log('Agreement:', result.output.agreement);

Next Steps

Consensus Voting

Consensus configuration reference

Providers

Configure different providers

Extract Invoice

Basic extraction guide

Error Recovery

Handle provider failures

Getting Started

Concepts

SDK

Doclo Cloud

Guides

Resources

Multi-Provider Extraction

Prerequisites

When to Use Consensus

Basic Consensus Configuration

Voting Strategies

Majority (Default)

Unanimous

Weighted

Multi-Provider Consensus

Agreement Thresholds

Accessing Consensus Results

Observability Hooks

Complete Example: Financial Document Validation

Cost Optimization Strategies

Tiered Consensus

Selective Field Consensus

Conditional Re-extraction

Handling Disagreement

Return Null for Uncertain Fields

Flag for Human Review

Return All Candidate Values

Using Doclo Cloud with Consensus

Next Steps

Consensus Voting

Providers

Extract Invoice

Error Recovery

​Prerequisites

​When to Use Consensus

​Basic Consensus Configuration

​Voting Strategies

​Majority (Default)

​Unanimous

​Weighted

​Multi-Provider Consensus

​Agreement Thresholds

​Accessing Consensus Results

​Observability Hooks

​Complete Example: Financial Document Validation

​Cost Optimization Strategies

​Tiered Consensus

​Selective Field Consensus

​Conditional Re-extraction

​Handling Disagreement

​Return Null for Uncertain Fields

​Flag for Human Review

​Return All Candidate Values

​Using Doclo Cloud with Consensus

​Next Steps

Consensus Voting

Providers

Extract Invoice

Error Recovery

Prerequisites

When to Use Consensus

Basic Consensus Configuration

Voting Strategies

Majority (Default)

Unanimous

Weighted

Multi-Provider Consensus

Agreement Thresholds

Accessing Consensus Results

Observability Hooks

Complete Example: Financial Document Validation

Cost Optimization Strategies

Tiered Consensus

Selective Field Consensus

Conditional Re-extraction

Handling Disagreement

Return Null for Uncertain Fields

Flag for Human Review

Return All Candidate Values

Using Doclo Cloud with Consensus

Next Steps