ORUK-VA-0004

Section 1 - Basic information about you and your application:

Title of research project
Automated Implant Identification using plain X-ray Images using Artificial Intelligence

Duration
24

Start date
01/11/2023

Profession
Orthopaedic surgeon

Your current job title/position
Senior Clinical Fellow

Have you previously received funding from Orthopaedic Research UK or Versus Arthritis
No

Please provide project ref number for previous funding from Orthopaedic Research UK


Please provide project ref number for previous funding from Versus Arthritis


Did you attend the AI in Orthopaedic Conference organised by Orthopaedic Research UK in 2022?
Yes

Did you attend the Microsoft AI training courses organised by Orthopaedic Research UK?
Yes

What other AI training courses have you previously attended?
NA

Are you an early-career researcher (ECR)? (definition of ECR)
yes

Section 2 - Lay summary

Lay summary:

In the UK, one in twenty-five people has an implanted medical device [1]. Over time, these devices may wear and fail to operate properly. For example, for 85 to 90% of people with total knee replacements, the implants used will last about 10 to 15 years.

There are several reasons for the failure of implants such as infection, instability, stiffness, wear and loosening, and fractures. In such cases, a revision surgery is required.

In 2021, there were a combined 162,000 primary total hip and knee replacement surgeries performed in the UK [2]. By 2030, UK demand for primary total knee arthroplasties is estimated to grow by 117% respectively from a 2012 baseline, and primary total hip replacements are predicted to increase 134% [3]. Although the clinical success rate at 10 years exceeds 90%, late failure remains a problem that can result in revision surgery. The demand for revision procedures in the UK is also projected to grow, by 332% and 31% for knee and hip revision surgeries respectively [3]. There are similar growth rates in the USA [3] and other territories.

When surgeons are planning for revision surgery, they need to order the correct spare part from the right company. However, in many cases it is unclear which device a patient has implanted. This uncertainty can delay treatment, and cause harm to patients.  To address this challenge, we propose reliable, fast-decision-making AI-enabled implant identifier software. Our software is designed to assist surgeons by quickly identifying the implant make and model, saving manual identification hours, allowing cost savings for hospitals and patients, and, notably, enabling surgeons to provide faster patient care.

 

Section 3 - Purpose of research

Purpose of research:

Research question: Our fundamental research question is “Is it possible to automate the process of identifying orthopaedic implants?”. To answer this question, we will utilise artificial intelligence (a deep neural network). Alongside, we will publish a complete website and mobile app to host this service.

Objectives: In this project, we will build on our existing work, scaling up the database and improving the AI. We will also implement the deployment of the system.

Objective 1: Curate an open source implant database of over 30,000 images. This will form an essential data resource (data lake) for further AI-based studies.

Objective 2: Develop an open source AI algorithm to identify implants using the database.

Objective 3: Deploy the algorithm in a cloud-hosted environment, allowing users to automatically search for implants

Direct Benefits

1. Reduce the manual labour of surgeons (average of 30 minutes to find implants from atlases, emailing vendors etc.) to less than 1 minute with automated identification. The upload of the x-ray containing the implant will only be required.
2. Unplanned and urgent revision arthroplasty procedures can be safely undertaken without the need to contact hospitals in different countries for international patients.
3. There will be significant cost savings for both patients & insurance providers in privately funded healthcare and hospitals in public funded health systems.
4. International patients can be at ease and not be constrained due to unavailability of patient notes. The current national joint registry is burdened with constant identification requests. This will exponentially bring down all requests in terms of identification.

Indirect Benefits

1. Setting up an implant database will allow future studies in both domains of orthopaedics, cardiac implant analysis, and use of AI in medical imaging.
2. AI models developed can be repurposed for other medical image computing research.

Section 4 - Background to investigation

Background to investigation:

There have been several studies using AI to identify implants using X-ray images. Borjali et al. [4] designed a fully automatic and interpretable approach to identify total hip replacement implants using a deep convolutional neural network (CNN) and achieved 100% accuracy for 3 commonly used designs. Karnuta et al. [5] validated and externally tested a deep-learning algorithm for total knee arthroplasty (TKA) to classify on 9 implant models with data from 4 referral sites validating on 682 radiographs across 424 patients and achieved 99% accuracy in the external-testing data set of 74 radiographs. Yi et al. [6] collected 237 anteroposterior (AP) knee radiographs with equal proportions of native knees, TKA, and unicompartmental knee arthroplasty (UKA) and 274 AP knee radiographs with equal proportions of 2 TKA models. A ResNet-18 DNN achieved optimal sensitivity and specificity of 100% each. Kang et al. [7] collected 170 X-ray images of hip implants from 29 models from five hospitals and from internet searches. Performance of these automated methods is typically on par with trained surgeons and radiologists, and superior to general practitioners. However, many of these systems use limited training data and/or only attempt to recognize small numbers of implant types, e.g. [5-7], or confine themselves to validate on one joint [8].

Based on the literature, there remains multiple limitations to be addressed:

1. There is no benchmark dataset available to holistically evaluate the multiple studies which have taken place. This makes it difficult to compare studies.
2. The majority of studies focus on single centre data. Therefore, generalisability of models remains poorly characterised.
3. Studies cover only one or two types of implant. There is extensive scope to test newer, and better performing AI models with more variety of implants.
4. There is an inherent dependency to utilise a large number of images per class to achieve strong performance. However, in practice, it may be hard to achieve this. Hence it is important to develop deep learning models which can work on a small number of collected images.
5. The majority of the studies fail to open-source their code, and hence are not reproducible. This is a major hindrance towards confidence levels of clinically adopting studies.

Section 5 - Plan of investigation

Plan of investigation:

Our project will consist of five work-packages (WPs), described below.

WP1: Data curation and data lake creation (18 months, Lead applicant/Clinical Fellow/Co-applicant)

Tasks: we will expand the existing database for uncollected models in collaboration with partner institutions (NHS UK and international: US, EU, India). This activity will take place in two main periods at the start and at the 12-month point in preparation for training the AI. This will involve:

Data curation activities: de-identification, data cleaning, and labelling

Quality Control (QC): a senior orthopedic registrar, one surgical trainee experienced implant surgery will be used for QC of the ingested data.

Collection of specifications from manufacturers, e.g. Exatech (USA), Smith and Nephew (UK), Stryker (USA), and Zimmer (USA).

Deliverables: A curated database of over 100-300 implant types over several joints, with approximately 100 examples of each implant model.

Risk and contingency: Insufficient numbers of exemplar X-rays for certain implant models. We have already established collaborative agreements with hospitals in the UK, US, and India, and collected over 13K+ radiographs of 93 manufactures consisting of 353 designs. We will prioritise and select models based on cost-benefit analysis. Collaborative agreements with manufacturers are already in place (Braun, Smith and Nephew; Stryker, and Zimmer Biomet). We have budgeted consumable costs to cover data acquisition costs and we will seek to widen our data search with new data-sharing agreements in the UK and abroad.

WP2: AI Algorithm development and deployment (12 months, PDRA/Co-Lead)

Tasks: Training/validation of algorithms, interface design, software deployment, user testing.  The curated data generated by WP1 will be used in tranches to train and validate the new version of the automatic deep neural network algorithms for accuracy, precision, and recall AI model architecture design: use of conditional neural networks which adapt their weights based on apriori information (e.g. joint) and meta learning approaches for quick adaptation to cases where there are few images of an implant.

Hyperparameter tuning, validation, and testing

Iterative improvement based on feedback from clinicians

Deliverables: A validated, implant identification system: v1 to identify at least 30 common UK implant types in the knee; v2 to accurately identify up to 100 implant types in several joints; v3 to identify 100-300 common and rare implant types and be extensible.

Risks and contingencies: (1) Competitor systems: we have seen a rise in the last few years research papers on AI-based implant identification systems and commercial applications (e.g. www.implantidentifer.app, USA). If necessary, we will approach potential commercial partners and make collaborative agreements (WP4). A key asset of this project will be a well-curated database (WP1); (2) Poor uptake of the system because of poor interfaces; inaccuracy of results; accessibility; slow response; limited database of searchable implants. We plan to work with users (technicians, clinical staff, surgeons) to rapidly address concerns to improve system efficacy and user acceptance (WP3).

WP3: App Development and software Testing (20 months, Clinical Research Nurse/Co-Lead)

Tasks: Each deployment phase will be followed by user feedback, validation, and impact analysis activity

On-site survey and focus group meetings to take place with stakeholders at clinics

Analysis of data and system usage logs to evaluate cost-benefit

Deliverables: A user requirements analysis and a user-friendly system, addressing stakeholder needs. A cost-benefit analysis of utility.

 

Section 6 - Research environment and resources

Research environment and resources:

We formed a team of clinical advisor (orthopaedic surgeon), AI team experts, project manager, software engineers, data supervisors, and data curators.
The AI team experts continuously update the curated database and work on improving the existing AI models accuracy by selecting the parameters of the AI model.
The software engineers team and project manager would work on developing the RestAPI, data encryption, secured website, user friendly design management, software architecture, and deployment.
The data curators keep on working on adding new image entries to the existing database, and data supervisor works on checking the data labelling. The advisor would keenly watch the entire team and clarify the queries for clinical information when needed.
We secured a UKRI Innovation Funding research grant for this project in 2022. We have a research lab setup in UK and in India.

Section 7: Research impact

Who will benefit from this research?

1. Surgeons:  Reduce the manual labour in identifying the implant model (average of 30 minutes to find implants from atlases, emailing vendors etc.) to less than 1 minute with automated identification. The upload of the x-ray containing the implant will only be required.
2. Hospitals/Healthcare centers: Unplanned and urgent revision arthroplasty procedures can be safely undertaken without the need to contact hospitals in different countries for international patients.
3. Patients: There will be significant cost savings for both patients & insurance providers in privately funded healthcare and hospitals in public funded health systems.
4. International patients: International patients can be at ease and not be constrained due to unavailability of patient notes.

How can your research be translated in real-life?

Currently, orthopedic surgeons planning requires an implant model of primary surgery. In most cases, primary surgery notes are unavailable, which is the gold standard method for identifying the make and model of an implant. For such cases, our software would help the surgeons identify implant models, thereby reducing the manual hours for the ortho surgeons. Thus, it improves the overall success rate for revision surgery.

How will your research be beneficial for Orthopaedic Research UK, Versus Arthritis and their purpose?

We plan to deploy a global product usage that will help all orthopaedics surgeons worldwide.

Section 8: Outreach and engagement

Outreach and engagement

The targeted outcome is to identify 50 most common hip and knee implants and the 20 most common shoulder, elbow, wrist, foot and ankle implants. Alongside, the focus is also in terms of minimising the number of images required to build the AI model and maximising on the utilisation of both AP and Lateral views. Finally, our objective is to provide surgeons a clear roadmap and allow them to rely on the software for implant identification purposes.
Automatic identification of implants can lead to better inventory stock management, and better use of surgical and technical staff’s time in contrast to spending hours correctly identifying implants makes and models. We expect this to be invaluable in every joint replacement hospital, particularly those situated in areas with large migrant populations who may have been previously operated for a Total Knee Arthroplasty/Total Hip Arthroplasty with foreign unfamiliar implants.
The project will allow patients to easily transfer to separate hospitals, significantly reduce the costs for both hospitals and patients, especially in the case of unplanned surgeries. The project will also lead to an implant database which can help in further advancement of the field.

Section 9: Research budget

Requested funding from Orthopaedic Research UK and Versus Arthritis

University fees (if any)
£0

Salary
£61000

Consumables
£12500

Publications
£21500

Conference attendance
£5000

Other items
£

Total 'requested fund'
£100000



Other items




Other secured funds

Internal funding
£0

Partner (University)
£0

Partner (Commercial)
£0

Partner (Charity)
£0

Other sources
£0

Total 'other funds'
£0



List all the 'other sources' and explain how their funds are used to cover the costs of your research.

Section 10: Intellectual property and testing on animal

Is there an IP linked to this research?
No

Who owns and maintains this patent?


Does your research include procedures to be carried out on animals in the UK under the Animals (Scientific Procedures) Act?
No

If yes, have the following necessary approvals been given by:

The Home office(in relation to personal, project and establishment licences)?


Animal Welfare and Ethical Review Body?


Does your research involve the use of animals or animal tissue outside the UK?
No

Does the proposed research involve a protected species? (If yes, state which)



Does the proposed research involve genetically modified animals?


Include details of sample size calculations and statistical advice sought. Please use the ARRIVE guidelines when designing and describing your experiments.


There should be sufficient information to allow for a robust review of any applications involving animals. Further guidance is available from the National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC3Rs), including an online experimental design assistant to guide researchers through the design of animal experiments.

Please provide details of any moderate or severe procedures


Why is animal use necessary, are there any other possible approaches?


Why is the species/model to be used the most appropriate?