Efficient and Interpretable Multimodal Document Understanding: A Survey of Layout Transformers, Vision–Language Models, Multi-Agent Frameworks, and Conditional Computation
DOI:
https://doi.org/10.33003/fjs-2026-1019-5746Keywords:
multimodal document understanding, document AI, layout transformers, vision–language models, multi-agent systems, conditional computation, dynamic routing, retrieval-augmented generationAbstract
Multimodal document understanding—the joint interpretation of textual content, visual appearance, and spatial layout in documents such as forms, receipts, contracts, scientific papers, and long reports—has advanced rapidly, moving from layout-aware transformers to large vision–language models (LVLMs) and, most recently, to retrieval-augmented and multi-agent frameworks. This progress, however, has been driven overwhelmingly by accuracy, and it has come at a steep and largely unexamined cost in computation, latency, and interpretability. Dense-by-default attention, exhaustive multi-agent pipelines, and opaque internal state have produced systems that are accurate on benchmarks yet difficult to deploy at scale and difficult to trust in regulated domains. This survey organises the field around that tension. We first establish a taxonomy that separates architectural paradigms (layout transformers, OCR-free generative models, LVLMs, and structural/graph reasoning) from efficiency and orchestration mechanisms (retrieval augmentation, multi-agent collaboration, conditional computation, and cost-aware routing). We then review each paradigm in depth, tracing how the field's persistent problems—architectural fragmentation across siloed encoders, weak explicit structural reasoning, uniform-density execution in multi-agent systems, and an interpretability deficit—have shaped successive generations of models. We give particular attention to an emerging synthesis in which conditional computation, long established for tokens and experts in mixture-of-experts models, is transferred to the orchestration of document agents, decoupling a system's capability from its per-query cost. We consolidate the benchmark and evaluation landscape, including the strict-versus-fuzzy metric problem on long-document question answering, and we survey interpretability and Green-AI considerations that are increasingly treated as first-class objectives rather than afterthoughts. We conclude with open challenges, arguing...
References
Abootorabi, M. M., et al. (2025). Scaling beyond context: A survey of multimodal retrieval-augmented generation for document understanding (arXiv:2510.15253). arXiv. https://doi.org/10.48550/arXiv.2510.15253
Appalaraju, S., Tang, Y., Dong, Q., Sankaran, S., Zhang, Y., & Manmatha, R. (2023). DocFormerv2: Local features for document understanding. Proceedings of the AAAI Conference on Artificial Intelligence, 38(2), 949–957. https://doi.org/10.1609/aaai.v38i2.27828
Bengio, Y., Léonard, N., & Courville, A. (2013). Estimating or propagating gradients through stochastic neurons for conditional computation (arXiv:1308.3432). arXiv. https://doi.org/10.48550/arXiv.1308.3432
Chen, L., Zaharia, M., & Zou, J. (2023). FrugalGPT: How to use large language models while reducing cost and improving performance (arXiv:2305.05176). arXiv. https://doi.org/10.48550/arXiv.2305.05176
Chen, Y., et al. (2025). Talk to right specialists: Iterative routing in multi-agent systems for question answering (arXiv:2501.07813). arXiv. https://doi.org/10.48550/arXiv.2501.07813
Chen, Z., et al. (2024). InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 24115–24126. https://doi.org/10.1109/CVPR52733.2024.02283
Deng, C., et al. (2025). LayoutmLongDocURL: A comprehensive multimodal long document benchmark integrating understanding, reasoning, and locating. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics. https://doi.org/10.48550/arXiv.2412.18424
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, 4171–4186. https://doi.org/10.18653/v1/N19-1423
Ding, Y., Luo, S., Dai, Y., Jiang, Y., Li, Z., Sun, Q., Martin, G., Liu, W., & Peng, Y. (2025). A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends. ArXiv. https://arxiv.org/abs/2507.09861
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ArXiv. https://arxiv.org/abs/2010.11929
Fabian, D., et al. (2025). RAGentA: Multi-agent retrieval-augmented generation for attributed question answering (arXiv:2506.16988). arXiv. https://doi.org/10.48550/arXiv.2506.16988
Faysse, M., et al. (2025). ColPali: Efficient document retrieval with vision language models. International Conference on Learning Representations. https://doi.org/10.48550/arXiv.2407.01449
Fedus, W., Zoph, B., & Shazeer, N. (2022). Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120), 1–39. https://doi.org/10.48550/arXiv.2101.03961
Fu, P., Guan, T., Wang, Z., Guo, Z., Duan, C., Sun, H., Chen, B., Ma, J., Jiang, Q., Zhou, K., & Luo, J. (2025). Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review. ArXiv. https://arxiv.org/abs/2502.16586
Han, S., et al. (2025). MDocAgent: A multi-modal multi-agent framework for document understanding (arXiv:2503.13964). arXiv. https://doi.org/10.48550/arXiv.2503.13964
Harley, A. W., Ufkes, A., & Derpanis, K. G. (2015). Evaluation of deep convolutional nets for document image classification and retrieval (RVL-CDIP). Proceedings of the 2015 International Conference on Document Analysis and Recognition, 991–995. https://doi.org/10.1109/ICDAR.2015.7333910
Hu, A., et al. (2024). mPLUG-DocOwl 1.5: Unified structure learning for OCR-free document understanding. Findings of the Association for Computational Linguistics: EMNLP 2024, 3089–3103. https://doi.org/10.18653/v1/2024.findings-emnlp.175
Huang, Y., Lv, T., Cui, L., Lu, Y., & Wei, F. (2022). LayoutLMv3: Pre-training for document AI with unified text and image masking. Proceedings of the 30th ACM International Conference on Multimedia, 4083–4091. https://doi.org/10.1145/3503161.3548112
Huang, Z., Chen, K., He, J., Bai, X., Karatzas, D., Lu, S., & Jawahar, C. V. (2019). ICDAR2019 competition on scanned receipt OCR and information extraction (SROIE). Proceedings of the 2019 International Conference on Document Analysis and Recognition, 1516–1520. https://doi.org/10.1109/ICDAR.2019.00244
Huo, J., Yan, Y., Hu, B., Yue, Y., & Hu, X. (2024). MMNeuron: Discovering neuron-level domain-specific interpretation in multimodal large language models. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 4467–4482. https://doi.org/10.18653/v1/2024.emnlp-main.387
Jaume, G., Ekenel, H. K., & Thiran, J.-P. (2019). FUNSD: A dataset for form understanding in noisy scanned documents. Proceedings of the 2019 International Conference on Document Analysis and Recognition Workshops, 2, 1–6. https://doi.org/10.1109/ICDARW.2019.10020
Khattab, O., & Zaharia, M. (2020). ColBERT: Efficient and effective passage search via contextualized late interaction over BERT. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, 39–48. https://doi.org/10.1145/3397271.3401075
Kim, G., et al. (2022). OCR-free document understanding transformer (Donut). European Conference on Computer Vision, 493–517. https://doi.org/10.1007/978-3-031-19815-1_29
Lee, C.-Y., et al. (2022). FormNet: Structural encoding beyond sequential modeling in form document information extraction. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 1, 3735–3754. https://doi.org/10.18653/v1/2022.acl-long.260
Lee, K., et al. (2023). Pix2Struct: Screenshot parsing as pretraining for visual language understanding. International Conference on Machine Learning, 18832–18843. https://doi.org/10.48550/arXiv.2210.03347
Lepikhin, D., et al. (2021). GShard: Scaling giant models with conditional computation and automatic sharding. International Conference on Learning Representations. https://doi.org/10.48550/arXiv.2006.16668
Lewis, P., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33, 9459–9474. https://doi.org/10.48550/arXiv.2005.11401
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023). Visual instruction tuning (LLaVA). Advances in Neural Information Processing Systems, 36, 34892–34916. https://doi.org/10.48550/arXiv.2304.08485
Liu, J., et al. (2024). A survey on inference optimization techniques for mixture of experts models (arXiv:2412.14219). arXiv. https://doi.org/10.48550/arXiv.2412.14219
Liu, Y., et al. (2024). TextMonkey: An OCR-free large multimodal model for understanding document (arXiv:2403.04473). https://doi.org/10.48550/arXiv.2403.04473
Luo, C., Shen, Y., Zhu, Z., Zheng, Q., Yu, Z., & Yao, C. (2024). LayoutLLM: Layout instruction tuning with large language models for document understanding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15638–15647. https://doi.org/10.1109/CVPR52733.2024.01480
Ma, Y., et al. (2024). MMLongBench-Doc: Benchmarking long-context document understanding with visualizations. Advances in Neural Information Processing Systems, 37. https://doi.org/10.48550/arXiv.2407.01523
Mathew, M., Karatzas, D., & Jawahar, C. V. (2021). DocVQA: A dataset for VQA on document images. Proceedings of the 2021 IEEE/CVF Winter Conference on Applications of Computer Vision, 2199–2208. https://doi.org/10.1109/WACV48630.2021.00171
Ogar, A. O., Abah, J., Suleiman, M. A., Akande, O. N., & Muhammed, F. O. (2026). Optimising domain-specific neuron activation for efficient multimodal language understanding in cloud AI systems. International Journal of Computer Information Systems and Industrial Management Applications, 18(5s), 816–831.
Ong, I., et al. (2024). RouteLLM: Learning to route LLMs with preference data (arXiv:2406.18665). arXiv. https://doi.org/10.48550/arXiv.2406.18665
OpenAI. (2024). GPT-4o system card. https://openai.com/index/gpt-4o-system-card/
Radford, A., et al. (2021). Learning transferable visual models from natural language supervision (CLIP). International Conference on Machine Learning, 8748–8763. https://doi.org/10.48550/arXiv.2103.00020
Ratnamogan, P., Tang, J., Lam, L., Vanhuffel, W., & Caspani, F. (2023). Information redundancy and biases in public document information extraction benchmarks (arXiv:2304.14936). arXiv. https://doi.org/10.48550/arXiv.2304.14936
Schwartz, R., Dodge, J., Smith, N. A., & Etzioni, O. (2020). Green AI. Communications of the ACM, 63(12), 54–63. https://doi.org/10.1145/3381831
Shazeer, N., et al. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. International Conference on Learning Representations. https://doi.org/10.48550/arXiv.1701.06538
Sun, Z., et al. (2025). CODA: Coordinating the cerebrum and cerebellum for a dual-brain computer use agent with decoupled reinforcement learning (arXiv:2508.20096). arXiv. https://doi.org/10.48550/arXiv.2508.20096
Van Landeghem, J., et al. (2023). Document understanding dataset and evaluation (DUDE). Proceedings of the IEEE/CVF International Conference on Computer Vision, 17616–17627. https://doi.org/10.1109/ICCV51070.2023.01789
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998–6008. https://doi.org/10.48550/arXiv.1706.03762
Wang, D., et al. (2024). DocLLM: A layout-aware generative language model for multimodal document understanding (arXiv:2401.00908). https://doi.org/10.48550/arXiv.2401.00908
Wang, P., et al. (2024). Qwen2-VL: Enhancing vision-language model's perception of the world at any resolution (arXiv:2409.12191). https://doi.org/10.48550/arXiv.2409.12191
Wang, W., et al. (2022). mmLayout: Multi-grained multimodal transformer for document understanding. Proceedings of the 30th ACM International Conference on Multimedia, 4242–4251. https://doi.org/10.1145/3503161.3548406
Wang, Y., et al. (2025). Structured attention matters to multimodal LLMs in document understanding (arXiv:2506.21600). arXiv. https://doi.org/10.48550/arXiv.2506.21600
Xu, Y., et al. (2021). LayoutLMv2: Multi-modal pre-training for visually-rich document understanding. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, 1, 2579–2591. https://doi.org/10.18653/v1/2021.acl-long.201
Xu, Y., Li, M., Cui, L., Huang, S., Wei, F., & Zhou, M. (2020). LayoutLM: Pre-training of text and layout for document image understanding. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 1192–1200. https://doi.org/10.1145/3394486.3403172
Yang, Q., et al. (2024). Routing experts: Learning to route dynamic experts in multi-modal large language models (arXiv:2407.14093). arXiv. https://doi.org/10.48550/arXiv.2407.14093
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., & Chen, E. (2024). A survey on multimodal large language models. National Science Review, 11(5), Article nwae403. https://doi.org/10.1093/nsr/nwae403
Zhong, X., Tang, J., & Jimeno-Yepes, A. (2019). PubLayNet: Largest dataset ever for document layout analysis. Proceedings of the 2019 International Conference on Document Analysis and Recognition, 1015–1022. https://doi.org/10.1109/ICDAR.2019.00166
Downloads
Published
Issue
Section
Categories
License
Copyright (c) 2026 Goni Mahmud Mustapha, Nurudeen Mahmud Ibrahim, Aliyu Sulieman Muhammed, Austin Olom Ogar

This work is licensed under a Creative Commons Attribution 4.0 International License.