A General Multimodal Probing Framework for Data Fusion and Prediction with Frozen Large Language Models: Evidence from Multimodal EHR Data

The main objective of this paper is to propose a general framework for prediction based on different sources of multimodal data in the healthcare domain. We evaluated whether frozen medical large language model (LLM) representations can serve as a shared embedding space for multimodal primary diagno