KL Grading-Based Knee Abnormality Classification Using Multi-Modal Deep Learning: KOMMCF – A Fusion of X-Ray Imaging and Clinical Biomarkers

I.INTRODUCTION

Like other organs, bones are critical to structure and function. Excessive force causes bones to change size, become stronger, weaker, or even break. Unlike most other organs, bones can grow back and heal without disfiguring. Human babies have about 270 bones, according to the National Library of Medicine. Many are fused together during development to become the 206 to 213 bones in an adult skeleton.

One of the most common musculoskeletal ailments distressing millions of people worldwide, knee osteoarthritis (KOA) affects daily mobility and quality of life. KOA is the fourth utmost common reason of disability worldwide and is characterized by joint space narrowing, subchondral osteosclerosis (hardening beneath cartilage), osteophyte formation (bone spurs), and cartilage breakdown. KOA is most common in adulthood and causes chronic pain and functional disability. Initial and precise diagnosis is important for operative intervention, discomfort management, and slowing the disease process. [1–4].

Clinicians primarily diagnose KOA through X-ray imaging via the KL grading system that classifies severity into 5 grades – from Grade Ranking 0 (normal) to 4 (severe KOA). Early detection of KOA, especially Grades 0, 1, and 2, is critical for timely treatment to prevent irreversible joint damage. However, early discovery of KOA is challenging due to understated radiographic deviations that may be problematic to detect. Manual diagnosis by experienced radiologists is standard but prone to variability and error. Automated Deep Learning (DL) models have been proposed to improve KOA diagnosis accuracy and standardization. [5–8].

Several DL algorithms such as XGBoost, CNN, VGG, ResNet, RNN, and MobileNet have been explored for KOA classification and have demonstrated high precision in feature mining and anomaly detection from medical images. However, integrating clinical biomarkers such as inflammatory markers, metabolic indices, and biomechanical factors along with imaging modalities such as X-rays, MRI, and CT scans provides an additional holistic approach to disease characterization and prognosis. [9].

Most existing studies focus on unimodal approaches [10]. In this paper, we propose an innovative cross-modal learning framework that fuses heterogeneous medical data (KOMMCF: Knee Osteoarthritis Multi-Modal Classification Framework). The model is trained using X-ray images and 14 clinical biomarkers (Table I) to predict Kellgren–Lawrence (KL) grades, which provide a standardized way to quantify KOA severity. These biomarkers are derived from the osteoarthritis initiative (OAI) clinical assessment protocol and represent semi-quantitative measures of cartilage degeneration, osteophyte formation, and subchondral bone changes. After pre-processing the clinical data, we had total 32 biomarker features because the model used 14 bilateral differences + 14 patient-normalized biomarkers + 4 clinical/structural features, not just the 14 raw biomarkers because these raw measurements alone were insufficient to capture the complex and heterogeneous progression patterns of KOA. Rather than introducing redundant information, the expanded representation explicitly encodes inter-patient normalization, intra-patient asymmetry, and clinically meaningful interactions, which are critical for robust ordinal learning. This helped the proposed model capture the transitions between severity levels, especially between consecutive KL grades, which aided improve ordinal agreement and severity deviation error rates.

Table I. Biomarker definitions (OAI-derived)

By making the most of the complementarities in multi-modal data, the projected system expects to expand the reliability of diagnoses, enhance the identification of the ailment in the early phases, and support doctors to better manage patients with osteoarthritis of the knee.

The rest of the paper is systematized as follows. Section II comprehends the review of the pertinent literature. Section III discusses the methodology, which includes the dataset, proposed framework, algorithm, and results. Section IV concludes the paper by deliberating the limitations and possible guidelines for impending research.

II.Literature Review

In the research of Kohn et al. [11], deep learning methods such as CNNs and RNNs prove extremely adept at interpreting medical images and identifying abnormalities. Recent studies have explored the use of DL techniques for predicting abnormalities within the knee using different imaging modalities. The KL technique remains widely used despite its limitations. The grading technique by KL has been useful but only as an adjunct when combined with a comprehensive clinical examination.

The work of Kokkotis et al. [12] have investigated machine learning algorithms including Bayesian methods, Decision Trees, SVM, and ANN to diagnose and forecast KOA. They highlighted that incorporating image, genomic, and clinical information could improve diagnostic accuracy.

A fusion tactic was also proposed by Mahum et al. in [13], which combined handcrafted features and CNNs for the diagnosis of KOA from X-ray images. This approach improved the accuracy of the diagnosis compared to the use of a CNN alone. However, the approach was unimodal and did not use any clinical biomarkers.

Researchers Kim et al. [14] studied the effect of combining patient information with imaging facts on the prediction of KOA severity using DL. The study carried out two experiments: one using simply imaging data and the other combining imaging and clinical data. A CNN model with six SE-ResNet was utilized to build the modules. The findings revealed that integrating patient info improved the AUC scores for each KL grade, with AUCs of 0.91, 0.80, 0.69, 0.86, and 0.96 for the imaging-only model and 0.97, 0.85, 0.75, 0.86, and 0.95 for the model that included both imaging and clinical data. KL grade 2 has been regarded as the most difficult to anticipate, but overall performance was improved.

The authors of this paper, Liu et al. [15], introduced a novel approach that was an integrated multi-modal learning method (MMLM) for detecting early-stage knee OA. In that they combined clinical, imaging, and demographic data to enhance categorization. To integrate clinical and imaging data, the approach utilized ResNet50 and XGBoost for feature extraction and optimized the fusion through L1-norm regularization. When tested on the OAI dataset, MMLM outperformed models that use only imaging or mixed data, with 83.45% accuracy, 79.27% precision, and 84.83% F1-score. The findings showed that MMLM considerably improves OA classification accuracy and interpretability.

Norman et al. [16] proposed an automated Deep Learning (DL) framework to classify the severity of knee osteoarthritis using the Kellgren–Lawrence (KL) grading system. First, A U-Net architecture was first used to precisely localize the left and right knee joints, enabling automated and consistent extraction of regions of interest. The localized knee images from multiple models were then used to train an ensemble model to predict osteoarthritis severity across the KL grades. Sensitivities of the ensemble were 83.7% for no OA, 70.2% for mild, 68.9% for moderate, and 86.0% for severe OA. The specificities were 86.1% for no OA, 83.8% for mild, 97.1% for moderate, and 99.1% for severe OA. The method was believed to be high performance for moderate and severe OA. However, the sensitivity was poor for mild and early-stage OA which presents the challenge of relying on radiography alone to determine the initial stages of the disease.

The study by Teh et al. [17] focused on the usage of multi-modal neural networks to diagnose KOA by mixing organized demographic data with unstructured imaging data. Three deep learning unimodal models – DIKO, InceptionV3, and EfficientNetv2 – were converted into multi-modal models by combining picture features with demographic information. Contrary to predictions, the results demonstrate that unimodal models (InceptionV3 and DIKO) outperform multi-modal models, with accuracies of 0.75 and 0.67 for 3-class severity classification. According to the findings, unimodal models may be sufficient for diagnosing knee OA, contradicting the idea that multi-modal models usually perform better.

In the study, author Slimane [18] projected an innovative multi-modal tactic for predicting KOA development by integrating biomarkers, clinical, and imaging data. He demonstrated accurate knee abnormality progression using deep learning and ensemble models and suggested using multi-modal data fusion to improve prediction of knee OA progression, providing cutting-edge information to drive personalized care and optimize clinical outcomes, which leveraged combined data modalities and outperformed single-source approaches by achieving an AUC improvement from 79% to 82% and AP from 68% to 72%.

The researchers Castagno et al. [19] built an autoML tool to predict the swift accelerations of knee OA. The autoML representations include medical, biological, and MRI and X-ray cross-sectional data. The OAI provided the datasets for training, validation, and external validation. The models predict OA progression for multi-class (pain/radiographic) and binary outcomes. The results show that simplified models that utilized only clinical factors performed adequately, achieving an AUC-PRC of 0.727 for the multi-class and 0.764 for the binary predictions. Multi-class models worked best for early-stage OA, whereas binary models were more steadfast for those 60 and younger.

Wang et al. [20] examined how complicated it is to train multi-modal classification networks. They discussed issues regarding imbalance of modalities, optimization, and cross-modal feature misalignment. In their findings, they discussed balanced sampling and improved fusion as useful and suggested that design is an important factor to consider for effective and stable multi-modal training.

Althnian et al. [21] demonstrated that smaller datasets resulted in a model that over-fitted and lacked generalization. Conversely, larger datasets improved model robustness and accuracy. This indicates the importance of having large volume, quality datasets that pertain to the medical field in order to derive dependable AI models for use in healthcare.

Using radiographic images from the OAI dataset, Mohammed et al. [22] assessed the automated classification of KOA smearing various pretrained deep neural network architectures. They assessed six different models (VGG16, VGG19, ResNet101, MobileNetV2, InceptionResNetV2, and DenseNet121) in a binary, three-class, and five-class cataloguing for the KL grading system. They acknowledged ResNet101 as best model with classification accurateness of 83%.

Lee et al. [23] developed a model based on a no-code AI platform (DEEP:PHI) with a ResNet101 model. The model was applied to a dataset from OAI to evaluate the knee radiographs and estimate the severity of osteoarthritis based on the first five levels of the KL grade. The model corroboration for the KL grade outcome reported that the model attained an average accurateness of 80.2% in the KL grade. The model was able to accurately grade moderate to severe osteoarthritis, but the accuracy was poor in early KL grades.

The work of distinguishing the initial stage of KL osteoarthritis from moderate and severe stages of osteoarthritis was the aim of the research work by Choi et al. (2025) [24]. A new AI approach to the KL grade system of KOA severity classification was the basis of the research work. This work compared the efficiency of various models. A total of 15,000 images of the knee X-rays were segmented based on 0, 1, 2, 3, and 4 grades of KL. Each of the models of DL algorithms (DenseNet201, ResNet101, and EfficientNetV2) was trained using the 15,000 images, and the efficiency of the DenseNet201 model reached 73% accuracy.

Architectural advancements such as EfficientNetV2, developed by Tan and Le [25], enhance computational efficiency by providing more compact models that enable faster training while maintaining high accuracy.

Summary of literature review is highlighted in Table II.

Table II. Literature review summary for year 2020 to 2025

III.METHODOLOGY

Convolutional and recurrent neural networks have also gained popularity in the analysis of KOA. The recent standard in the assessment of KOA is the KL grade, despite its limitations. For the purpose of improving the accuracy of diagnosis, previous researchers have also considered the use of a combination of radiographic images and clinical parameters. They have demonstrated that the use of patient-specific parameters, such as age, gender, BMI, and biochemical parameters, improves the accuracy of classification, especially in the early stage.

The proposed KOMMCF has been designed to overcome the foremost challenges in the traditional diagnosis of KOA. The existing diagnosis technique is mainly based on X-ray images, which have less sensitivity towards the minute changes that occur in the early stage of KOA and lead to delayed diagnosis. In addition, the KL grading discrepancies and imbalance in the severity levels can negatively impact the diagnosis. Finally, the vagueness in the KL grading system, especially in the neighboring grades, can negatively impact the system’s effectiveness in the decision-making process for the treatment of KOA. KOMMCF overcame these challenges through the inclusion of multi-modal data and readjusting the KL grading system to accurately capture the clinically important differences in severity.

IV.CONCLUSION AND FUTURE WORK

KOMMCF is an innovative and efficient model for the classification of knee OA, based on the combination of X-ray image and clinical and biomarker information, the fusion of multi-modal features, and the use of an ordinal distance-aware loss function, while using patient-wise Group K-Fold cross-validation to avoid data leakage. Our study gave high priority to the three-class severity classification model (Normal, Early, and Moderate-Severe) with test accuracy of 90.58%, corresponding to the decision-making criteria used in the medical field. Additionally, the efficiency and generality of the proposed model were validated through its adaptation to the two-class (98.73% accuracy) and five-class (81.34% accuracy) classification problems. Moreover, the results of the McNemar’s test validated the model statistically, and they showed that the proposed model is significantly better compared to the conventional models of ResNet50, VGG16, and MobileNetV2 for all severity levels (p < 0.001). Lastly, the use of the Grad-CAM visualization technique enhanced the model interpretability. It ensures the interpretability of the results of the model by identifying the regions of the X-ray image with high importance to the classification process, and this increases the clinical validity and applicability of the model to real medical problems.

Some limitations remain. The model misclassified doubtful/mild (KL 1&2) cases, and further improvements are needed to discriminate early OA cases. As the model had difficulties with doubtful/mild OA cases, future work will combine higher resolution imaging techniques such as MRI scans with X-rays to detect understated structural variations in early OA. Advanced self-supervised learning methods and contrastive learning should be explored to improve feature discrimination between mild and moderate OA cases. We aim to develop this framework into a real-time clinical decision support system to assist radiologists in high-volume settings, reducing manual diagnostic errors and streamlining patient treatment planning.