Abstract
Purpose
To externally validate an artificial intelligence (AI) tool for radiographic knee osteoarthritis severity classification on a clinical dataset.
Method
This retrospective, consecutive patient sample, external validation study used weight-bearing, non-fixed-flexion posterior-anterior knee radiographs from a clinical production PACS. The index test was ordinal Kellgren-Lawrence grading by an AI tool, two musculoskeletal radiology consultants, two reporting technologists, and two resident radiologists. Grading was repeated by all readers after at least four weeks. Reference test was the consensus of the two consultants. The primary outcome was quadratic weighted kappa. Secondary outcomes were ordinal weighted accuracy, multiclass accuracy and F1-score.
AI tool
The AI tool under investigation in this study is the product RBkneeTM v.2.1 (Radiobotics, Copenhagen, Denmark). The AI algorithms inherent in the product are based on convolutional neural networks (CNN) and have been trained on medical images from a large multicenter cohort from the USA. The product integrates the AI algorithms into a radiology environment via a PACS integration based on DICOM communication. Subject to availability of a graphical processing unit (GPU) the processing time of a bilateral knee PA projection radiograph can vary between approximately 8–60 s. RBknee is commercially available in a CE marked version (v.2.1 MDR 2017/745 class IIa) and an FDA cleared version (v.1 class II).
Results
50 consecutive patients between September 24, 2019 and October 22, 2019 were retrospectively included (3 excluded) totaling 99 knees (1 excluded). Quadratic weighted kappa for the AI tool and the consultant consensus was 0.88 CI95% (0.82–0.92). Agreement between the consultants was 0.89 CI95% (0.85–0.93). Intra-rater agreements for the consultants were 0.96 CI95% (0.94–0.98) and 0.94 CI95% (0.91–0.96) respectively. For the AI tool it was 1 CI95% (1–1). For the AI tool, ordinal weighted accuracy was 97.8% CI95% (96.9–98.6 %). Average multiclass accuracy and F1-score were 84% (83/99) CI95% (77–91%) and 0.67 CI95% (0.51–0.81).
Conclusion
The AI tool achieved the same good-to-excellent agreement with the radiology consultant consensus for radiographic knee osteoarthritis severity classification as the consultants did with each other.
Open access
This research is open access.
