Multimodal Retrieval-Augmented Document Intelligence for Text, Table and Visual Question Answering

A multimodal retrieval-augmented document intelligence system for text, table and visual question answering.

A multimodal retrieval-augmented document intelligence system for text, table and visual question answering. As a reference project it shows how a system like this is put together in practice, which parts matter and how they are tested, so that a developer can learn from it and build on it.

It suits learners who want to work with language models and text or speech: preparing text data, using or fine-tuning a model, evaluating it and building an interface around it.

The main parts are retrieval over an indexed document collection that grounds the language model's answers, fusion of more than one type of input, text or speech data preparation, model training, fine-tuning or prompting, and evaluation with task-appropriate metrics. The reference build uses LangChain, FAISS, Python, Hugging Face Transformers, and spaCy, with a clear project structure, setup instructions and sample data so that it can be run and extended.