ORUK-VA-0007

Section 1 - Basic information about you and your application:

Title of research project
Developing a knowledge base and investigating whether a deep learning large language model can be used to predict outcome following total hip and knee joint replacement surgery through analysis of patient related documentation.

Duration
24

Start date
01/01/2024

Profession
Academic scientist

Your current job title/position
Lecturer

Have you previously received funding from Orthopaedic Research UK or Versus Arthritis
No

Please provide project ref number for previous funding from Orthopaedic Research UK


Please provide project ref number for previous funding from Versus Arthritis


Did you attend the AI in Orthopaedic Conference organised by Orthopaedic Research UK in 2022?
No

Did you attend the Microsoft AI training courses organised by Orthopaedic Research UK?
No

What other AI training courses have you previously attended?
Dr Diptesh Kanojia (DK) is an expert in developing and deploying natural language processing (NLP) solutions. He obtained his joint PhD from IIT Bombay, India and Monash University, Australia. During his PhD, his work on the automatic detection of cognates was awarded an honourable mention among the best papers at the annual European chapter of Association for Computational Linguistics (EACL) conference in 2021. His current research interest in multilinguality has introduced novel multilingual datasets for detection of aggression/hate, quality estimation and automatic post-editing in different NLP areas. His prior research on Wordnets, i.e., dictionary-like knowledge graphs with connected word meanings, provides background and expertise for knowledge base construction, a primary task in this project. He publishes regularly at annual conferences like Association for Computational Linguistics (ACL), North American chapter of ACL (NAACL), Empirical Methods for Natural Language Processing (EMNLP), and Association for Advancement of Artificial Intelligence (AAAI).

Are you an early-career researcher (ECR)? (definition of ECR)
yes

Section 2 - Lay summary

Lay summary:

This project proposes the construction of a framework for information extraction from multiple unstructured data sources such as typed text, scanned documents, and radiographic / video images within a hospital. Unstructured data is a very common problem affecting most hospitals within the United Kingdom and worldwide.

Being able to extract useful information should allow us to improve workflow, efficiency and safety within the hospital. It may also predict the likely outcome following hip and knee joint replacement surgery.

Current methods of extracting information use keyword-based search, however this is unreliable since different words are often used to express the same meaning. We believe that a machine learning semantic searching (searching on meaning) methodology instead of keyword searching, would be a far better method to search and extract meaningful information from a knowledge base created from the various unstructured data sources around the hospital.

The first phase of the project will involve setting up a computer system and using freely available software to develop this database on the computer. Various data sources within the hospital will be defined and a methodology to input data into the knowledge base developed. These will be implemented and tested.

The second phase of the project will involve applying the machine learning process using a large language model to determine whether patient outcomes can be predicted from clinic letters that are typed.

The work will be carried through a partnership between South West London Elective Orthopaedic Centre (one of the largest joint replacement hospitals in Europe) and The University of Surrey, which has the largest machine learning department in the United Kingdom. Members of the public and patients will be invited to a meeting to discuss the project to seek and consider their opinions on making this an acceptable technology.

Section 3 - Purpose of research

Purpose of research:

Aims:

1. Identify all unstructured data sources within the orthopaedic hospital.
2. To setup a knowledge base that will be able to store this information in a structured format with the hospital (including clinic letters, operation notes and pre-assessment information)
3. Define sustainable methodologies to extract information from various sources within the hospital for input into the knowledge base.
4. Set up a machine learning large language model to establish whether the outcome scores following total hip and knee replacement surgery can be predicted through analysis of pre and post-operative clinic letters.

Objectives:

1. Develop and set-up a knowledge base.
2. Establish sustainable methodologies for extracting information from various sources within the hospital.
3. Perform experiments with large language model to predict outcomes following surgery using information derived from patient clinic letters, before and after total hip and knee replacement surgery.

Deliverables:

1. List of data sources within the orthopaedic hospital
2. Methodology for setting up a knowledge base.
3. Sustainable methods (standard operating procedure) for procuring information from the various data sources for input into the knowledge base.
4. A trained large language model able to predict the outcome (Oxford score) based on the clinic letter information. The AUC, Precision, Recall and F1-score will be reported.

Section 4 - Background to investigation

Background to investigation:

The overarching goal for the project is to enable research using state-of-the-art AI-enabled solutions, and thus support operational efficiencies for orthopaedic surgeons and clinicians at SWLEOC; and develop novel algorithms to facilitate detection of emergent patterns from a large orthopaedic dataset, i.e., a knowledge base.

Towards this, the project will construct an orthopaedic knowledge base (KB) that builds structured information from heterogeneous unstructured data sources such as surgical planning and operative notes as well as referral letters. Addressing the challenge of extraction of relevant information from pre- and post-operative data, with information scattered across multiple data sources, the project will develop novel knee orthopaedic entity recognition and extraction methods, using hybrid unsupervised and supervised AI methods.

The training datasets for the supervised methods will be co-created with the SWLEOC surgeons, infusing their surgical domain knowledge and machine learning experience into the training process. The creation of an orthopaedic KB will enable machine-enabled tagging of pre/post operative records and form the precursor to new operational functionalities such as thematically-relevant and context-sensitive outcomes prediction. 

The project will demonstrate such functionality by implementing a proof-of-concept demonstrator of a post-surgery outcomes predictor from the constructed knowledge base. The developed methods will utilise vectors indexed from the knowledge base and apply recent advances in text embeddings and large language models to showcase retrieval of relevant data, such as those from patient clinic letters, before and after total hip and knee replacement surgery to predict the Oxford Score.

The project plans to utilise publicly available large language models. The aim is to first, design prompts and information extraction methodologies which can help the creation of a knowledge base. Next, a large language model will be fine tuned towards the prediction tasks required of this project. A representative architecture, developed by DKA, of an adapted pre-trained language model is shown in the image attached. DKA trained this model using the NASH-MTL method which has been shown to be a  robust method for fine tuning a Large Language Model, that is proposed for use in this project. 

Team experience includes:

DKA has past experience in curation of datasets [1,2] and use of pre-trained language models for detecting specific patterns [3]. His recent work also utilizes multi-task learning to efficiently solve both sentence-level and word-level quality estimation problems simultaneously [4]. 

SD has past experience in semantic modelling and search [5-6]. Her recent work focusses on researching ML algorithms to automate metadata extraction [7] and uplift [8] for longitudinal social science and biomedical questionnaires.

VA is an experienced orthopaedic surgeon specialising in hip and knee surgery. He has previously set-up and is working on a successful project that has predicted failing hip replacements several years before clinical and radiographic features become evident. He is co-organising the ORUK / AI national meeting.

YS is a principle investigator of the Sketch X laboratory (computer vision related research)

FN devopls novel optimistaion algorithms for deep neural networks – world.

 

[1] Nazia Nafis, Diptesh Kanojia, Naveen Saini, and Rudra Murthy. 2023. Towards Safer Communities: Detecting Aggression and Offensive Language in Code-Mixed Tweets to Combat Cyberbullying. In The 7th Workshop on Online Abuse and Harms (WOAH), pages 29–41, Toronto, Canada. Association for Computational Linguistics.

[2] Chrysoula Zerva, Frédéric Blain, Ricardo Rei, Piyawat Lertvittayakumjorn, José G. C. de Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Constantin Orăsan, Marina Fomicheva, André F. T. Martins, and Lucia Specia. 2022. Findings of the WMT 2022 Shared Task on Quality Estimation. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 69–99, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

[3] Rudra Murthy, Pallab Bhattacharjee, Rahul Sharnagat, Jyotsana Khatri, Diptesh Kanojia, and Pushpak Bhattacharyya. 2022. HiNER: A large Hindi Named Entity Recognition Dataset. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4467–4476, Marseille, France. European Language Resources Association.

[4] Sourabh Deoghare, Paramveer Choudhary, Diptesh Kanojia, Tharindu Ranasinghe, Pushpak Bhattacharyya, and Constantin Orăsan. 2023. A Multi-task Learning Framework for Quality Estimation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9191–9205, Toronto, Canada. Association for Computational Linguistics.

[5] L Alkhariji, S De, O Rana, C Perera, Semantics-based privacy by design for Internet of Things applications, Future Generation Computer Systems 138, 280-295, 2023.

[6] N Gligoric, S Krco, L Hakola, K Vehmas, S De, et al. Smarttags: IoT product passport for circular economy based on printed sensors and unique item-level identifiers, Sensors 19 (3), 586, 2019.

[7] S De et al., Engineering a machine learning pipeline for automating metadata extraction from longitudinal survey questionnaires, IASSIST Quarterly 46 (1), 2022.

[8] V Sharifian-Attar, S De, S Jabbari, J Li, H Moss, J Johnson, Analysing Longitudinal Social Science Questionnaires: Topic modelling with BERT-based Embeddings, IEEE International Conference on Big Data, Osaka, Japan, 2022.

Section 5 - Plan of investigation

Plan of investigation:

Goal 1.  Identify data sources and develop methodologies to create a knowledge base

1. Aim 1 2 weeks – Identifying data sources and understand what is in the data source (Bluespier, clinic letters)

Define how data is to be stored in the knowledge base (model and schema) [Deliverable 1]

1. 6 weeks: Define headings
2. 2 weeks: Organise a focus group to agree model, headings
3. 1 weeks: (Patient and public involvement) patient forum to review the schema
4. 1 weeks: Final review meeting with stakeholders
5. 6 weeks: Obtain permission and IT access
6. Aim 2 3 weeks: Set up server and SQL; linux.
7. Aim 3 3 months Set up data import pipeline from data sources: [Deliverable 2]
   1. Define Bluespier to SQL server link.
      1. Meeting with Bluespier – to provide data access / development of an API.
      2. Automate as much as possible; limit bottle necks.
   2. 2 weeks: Import data into knowledge base
   3. 3 months: Scraping data from unstructured data
   4. 1 month: Check pipeline working.
   5. Add dummy case to Bluespier and confirm all queried data transfer to knowledge base.
8. 1 month: Report outcomes [Deliverable 3]

Goal 2. Fine tuning large language model for predicting outcomes

1. Aim 4 1 month : Set up Machine Learning Large Language Model (LLM); Bloom or Llama v2.
2. 3 months:
   1. Appropriate consents obtained from patients
   2. Ensure knowledge base has Oxford outcomes score, date of joint replacement and all Bluespier letters before and after surgery for a defined patient population.
3. 3 months: Train LLM: Use notes to add vocabulary to the model.
4. 3 months: Test LLM: correlation
5. Investigate factors to improve model
   1. Fine tune the model
   2. Create other tasks around the data can help algorithm to better predict Oxford scores. eg. There is another numerical value – other Oxford scores.
6. 1 month: Report outcomes [Deliverable 4]
7. 1 month: Prepare publication and presentations.

Opportunities

1. Development of a knowledge base for clinical use and further research.
2. Establish common API standards for other hospitals you set-up similar facilities
3. Predict patient outcomes on a personalised basis for clinicians to be able to better tailor treatment on an individual basis.
4. To provide a system that will eventually automate the process of handling electronic referrals, triage patients for sub-specialities and automatic allocation of patients to consultants, automate theatre scheduling for surgical procedures. The benefit of these will be to improved efficiency, patient safety, reduce cost and free up clinicians to focus on patient specific care.

 

Risks

1. Ethical approval. This is mitigated by the experience that our unit already has in obtaining ethics, setting up and successfully working on a project that has confirmed that failing total hip replacements can be predicted using machine learning.

 

2. Developing APIs for data extraction from various hospital sources may be costly. Data can already be exported from most of these (Bluespier and other data sources) and this is therefore unlikely to pose a significant problem.

 

3. Large language model may show an inability to predict outcomes based on clinic letters. This is unlikely because our unit is one of the largest in the United Kingdom. Furthermore, we will also include other data sources and fine-tuning techniques during development to improve the algorithm.

 

Impact of research

1. Evidence based predictions will inform patients decision making.
2. Enable patients to make better informed decisions
3. Efficient workflow using AI will enable clinicians to focus on more time on clinical care.
4. Cost effectiveness, reduced labour, increased accuracy, better speed, patient matched delivery of treatment options. Improved efficiency.
5. Enhanced patient experience – better predictability of outcome, less dissatisfaction post-operatively. Empowered decision making (with better information).
6. Joined up working / Integrated care pathways.
7. Better communication
8. Improved patient safety through reducing errors related to miscommunication
9. Template for other models of AI development

Section 6 - Research environment and resources

Research environment and resources:

The research will be conducted as a collaboration between one of Europe’s largest joint replacement centres, and one of the top Artificial Intelligence centres in Europe.

The orthopaedic unit is a high volume GIRFT (Getting It Right the First Time) accredited elective orthopaedic centre which performs approximately 3000 hip and knee joint replacements annually. Several high-quality data systems are currently used for the management of patients. The hospital has already developed a system to predict failing total hip replacements using radiographs.

The University of Surrey hosts Nature Inspired Computing and Engineering (NICE) research group with researchers whose specialist fields natural language processing (large language models), trustworthy artificial intelligence, and computer vision. Their knowledge and expertise cover all the domains required for successful installation of hardware, software, development of methodologies to extract data into a knowledge base and for its analysis using a large language model.

A high-specification computer with a graphics processing unit will be purchased (2 x A6000 NVIDIA GPU 48GB, i9 11th Gen 192GB RAM, 1 TB + 8 TB storage). Work in kind will be provided by the Supervisors. We believe that this collaboration is very likely to be successful and will allow SWLEOC and University of Surrey to form a long-standing relationship for AI-based research towards better healthcare.

Section 7: Research impact

Who will benefit from this research?

The increasing burden of osteoarthritis due to an increasing elderly population and younger people receiving hip and knee replacements means that there is an important need to provide patient specific treatment based on predicted outcomes. It helps to improve the efficiency and safety of patient care by improving processes within the hospital.

This will benefit not only patients, but also their careers, relatives, employers. It will also benefit NHS staff and the organisation by reducing costs, improving reputation through the provision of consistent high quality care pathways.

The application of semantic search on a knowledge base derived from multiple data sources within a hospital is novel. Undertaking this project in a high volume, GIRFT accredited orthopaedic unit will provide the best environment to develop this new tool to benefit society in the UK and beyond. Clearly, this project has massive potential to improve patient care all over the world.

How can your research be translated in real-life?

As stated, our aim is for ML language model to predict outcomes following surgery. We envisage this tool will be used universally to assess all patients who may require a surgical procedure to predict outcomes and tailor treatment on an individual basis.

To achieve this ambition, the results this study will form the basis of the next stage of the project which will evaluate the clinical effectiveness of the machine learning algorithm in predicting outcomes following surgery  under a prospective study.

We are also hopeful that the testing platform implemented in this project will help future surgeons, in their patient-centred decision making regarding how they match implant and procedure choice with an individual patient.

How will your research be beneficial for Orthopaedic Research UK, Versus Arthritis and their purpose?

Development of the knowledge base will facilitate the development of many other machine learning projects and provide a source of trustworthy data for research.

We believe that applying the large language model will demonstrate the utility of the knowledge base in helping to predict outcomes following surgical intervention more accurately. This will lead to better patient outcomes, improvements for family and carers and improved work conditions within the NHS. It is hoped that this will lead to a commercial product that can be rolled out to every hospital within the NHS.

ML and AI are at the forefront of contemporary research, both throughout medicine and orthopaedics. We would hope that if we are successful in our research and our ambition is realised, that ORUK would be very pleased to be associated with it fulfilling their own ambitions for research.

Section 8: Outreach and engagement

Outreach and engagement

Besides communicating through standard orthopaedic journals to orthopaedic surgeons and radiologists, information regarding the project will be disseminated to the public using public / patient meeting groups and the press. Our publications and presentations will be shared through twitter and LinkedIn to reach colleagues and patients. The team will devise a strategic approach for the dissemination of the information relating to this project, to the public. This will include a 1-day workshop aiming to attract 200 attendees, presented by experts using a lay language and open to the discussion with the public. We want to ensure that patients, public and staff working in healthcare find these new technologies acceptable for use in modern medicine.

The support provided by ORUK will be willingly acknowledged.

Section 9: Research budget

Requested funding from Orthopaedic Research UK and Versus Arthritis

University fees (if any)
£33866

Salary
£29764

Consumables
£30000

Publications
£3000

Conference attendance
£3000

Other items
£

Total 'requested fund'
£99630



Other items




Other secured funds

Internal funding
£0

Partner (University)
£0

Partner (Commercial)
£0

Partner (Charity)
£0

Other sources
£0

Total 'other funds'
£0



List all the 'other sources' and explain how their funds are used to cover the costs of your research.

Section 10: Intellectual property and testing on animal

Is there an IP linked to this research?
No

Who owns and maintains this patent?


Does your research include procedures to be carried out on animals in the UK under the Animals (Scientific Procedures) Act?
No

If yes, have the following necessary approvals been given by:

The Home office(in relation to personal, project and establishment licences)?


Animal Welfare and Ethical Review Body?


Does your research involve the use of animals or animal tissue outside the UK?
No

Does the proposed research involve a protected species? (If yes, state which)



Does the proposed research involve genetically modified animals?


Include details of sample size calculations and statistical advice sought. Please use the ARRIVE guidelines when designing and describing your experiments.


There should be sufficient information to allow for a robust review of any applications involving animals. Further guidance is available from the National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC3Rs), including an online experimental design assistant to guide researchers through the design of animal experiments.

Please provide details of any moderate or severe procedures


Why is animal use necessary, are there any other possible approaches?


Why is the species/model to be used the most appropriate?