From RAG to fabric: Lessons learned from building real-world RAGs at GenAIIC – Part 2

This article discusses how to build retrieval-augmented generation (RAG) systems that integrate heterogeneous data sources, including structured data (like tables) and multimodal data (text and images).

Specifically, the article covers:

• Using routers to direct queries to the appropriate data source and processing pipeline
• Prompting large language models (LLMs) to generate code for analyzing structured tabular data
• Two approaches for multimodal RAG systems:
  • Using multimodal embedding models to embed images and text into a shared vector space
  • Using multimodal LLMs to generate image captions, then embedding the captions for retrieval
• Comparative evaluation of the two multimodal approaches based on speed, cost, detail, customization, and input latency
• Conclusion highlighting the importance of breaking down the problem into modular components and leveraging the strengths of LLMs for different components