Automate schema generation for intelligent document processing

This article introduces a multi-document discovery feature for the IDP Accelerator that automates schema generation for intelligent document processing without manual classification.

• Automatically clusters unknown documents by type using visual embeddings and k-means clustering
• Generates JSON schemas ready for IDP Accelerator using Strands Agents and Amazon Bedrock LLMs
• Uses Cohere Embed v4 for visual embeddings capturing layout and formatting cues
• Silhouette score determines optimal number of document types (k value from 2-20)
• Agents strategically sample documents across clusters to generate comprehensive schemas
• Schema analysis step reviews outputs for overlaps and inconsistencies before human review
• Benchmarking on OCR dataset achieved perfect clustering accuracy across 9 document types
• Step-by-step guide provided for running discovery jobs in IDP Accelerator Console
• Best practice: use single-page PDFs; review quality report before finalizing schemas

The solution eliminates the chicken-and-egg problem of needing schemas before document processing by automating discovery and schema generation from unlabeled document collections.