AI & Machine Learning Engineer

Abstract—Medical Large Language Models (LLMs) require specialized domain knowledge that is often distributed across multiple fine-tuned models in modern healthcare ecosystems. In IoT-enabled healthcare environments, where edge devices and distributed clinical systems generate specialized models across different institutions, parameter-space integration (model merging) offers a privacy-preserving and bandwidth-efficient ap- proach to consolidating knowledge without requiring centralized retraining or data sharing. This paper presents a systematic evaluation of six parameter-space merging techniques applied to two architecturally compatible medical LLMs derived from the Mistral-7B base model. We introduce a novel hierarchical method that combines selective Optimal Transport (OT) alignment for attention layers with cosine similarity-weighted interpolation, designed to address permutation variance while minimizing computational overhead for edge deployment scenarios. Our study evaluates Task Arithmetic, Linear Averaging, DARE-TIES, DELLA, Breadcrumbs, and our Hierarchical approach across five medical benchmarks. Results demonstrate that architec- turally compatible models benefit significantly from simple aver- aging methods, with Task Arithmetic achieving 45.80% accuracy on MedQA, outperforming complex pruning-based approaches. These findings offer critical insights for the deployment of distributed medical AI in resource-constrained IoT environments, where computational efficiency and model compatibility are paramount. Our work suggests that with compatible models, simple averaging can yield strong performance with minimal computational cost, offering a pragmatic path for federated medical AI systems and edge-based healthcare applications. Index Terms—Model Merging, Medical AI, Large Language Models, Parameter Efficiency, Domain Adaptation, Healthcare Informatics, Biomedical Computing, IoT Healthcare

Abstract—The recent advancements in large language models (LLMs) transformed the way the machines understand and gen- erate human language at the cost of huge computational power and time. This research proposes an approach that allows the collaboration of pre-trained LLMs by weighted linear merging of their attention and feedforward layer features thereby saves the compute power and time by eliminating the need of training of the merged models. Firstly, the proposed framework chooses a base model, llemma 7b, followed by a creation of merged model 1, Llama-2-tulu-7b by sequentially combining Llama-2 with tulu-2-dpo-7b using a weighted linear merging of their selected layers, and a merged model 2, Llama-2-tulu-7b-lemma, that sequentially combines merged model 1 with lemma model (second merge) as its preparatory step for evaluation. Then these models are validated against fourteen widely recognized Natural Language Processing (NLP )benchmark datasets to assess the efficacy of the merged models that are created using the proposed approach. Experimental results demonstrate that the integrated model, Llama-2-tulu-7b, exhibits significant improvements by outperforming baseline models across diverse tasks, reducing perplexity by 0.78 and boosting accuracy by 0.0542 on tasks like LAMBADA Cloze and OpenBookQA. Though the results strongly support the viability of proposed parameter merging strategies,an 18% performance drop in the MathQA dataset during hierarchical merging alarms the need of development of flexible, context-aware merging strategies by considering model incompatibility and interference in specialized domains. In concise, this research definitely acts as a stepping stone for the development of robust and adaptable collaborative LLMs and also opens up new research avenues in context-aware merging of specialized LLMs.