This is a summary of my PhD thesis "Designing an Algorithmically Enhanced NHS: Towards a Conceputal Model for the Successful Implementation of Algorithmic Clinical Decision Support Software in the National Health Service" which was accepted without corrections 01/11/2023.
Abstract
Established in 1948, the National Health Service (NHS) has lasted 75 years. It is, however, under considerable strain: facing chronic staff shortages; record numbers of emergency attendances; an ambulance wait-time crisis; and more. Increasingly, policymakers are of the view that the solution to these problems is to rely more heavily on one of the NHS’s greatest resources: its data. It is hoped that by combining the NHS’s data riches with the latest techniques in artificial intelligence (AI), that the means to make the NHS more effective, more efficient, and more consistent, can be identified and acted upon via the implementation of Algorithmic Clinical Decision Support Software (ACDSS). Yet, getting this implementation right will be both technically and ethically difficult. It will require a careful re-design of the NHS’s information infrastructure to ensure the implementation of ACDSS results in intended positive emergence (benefits), and not unintended negative emergence (harms and risks). This then is the purpose of my thesis. I seek to help policymakers with this re-design process by answering the research question ‘What are the information infrastructure requirements for the successful implementation of ACDSS in the NHS?’. I adopt a mixed-methods, theory-informed, and interpretive approach, and weave the results into a narrative policy synthesis. I start with an analysis of why current attempts to implement ACDSS into the NHS’s information infrastructure are failing and what needs to change to increase the chances of success; anticipate what might happen if these changes are not made; identify the exact requirements for bringing forth the changes; explain why the likelihood of these requirements being met by current policy is limited; and conclude by explaining how the likelihood of policy meeting the identified requirements can be increased by designing the ACDSS’s supporting information infrastructure around the core concepts of ‘utility, usability, efficacy, and trustworthiness’.
Chapter 1: Complex systems that cannot be controlled must be designed
In this introductory and conceptual chapter, I argue that the NHS must change its modus operandi to cope with 21st century challenges. I stress that the implementation of ACDSS may help achieve this change in operation, by enabling the NHS to become a national ‘learning healthcare system’ (henceforth ACDSS-enabled NHLS). However, I go on to explain that whilst there is enormous potential in this strategy, the NHS is not yet able to benefit from the significant potential of ACDSS, in part because it is not yet known by NHS policymakers how to manage the various technical, ethical, social, and regulatory complexities to ensure successful implementation. I end the first half of the chapter by noting that it was my desire to help NHS policymakers overcome this hurdle, and so benefit from the potential benefits of ACDSS whilst avoiding the potential risks (so-called the ‘dual advantage’), that motivated me to write this thesis, as reflected in the motivating question:
Motivating Question: How can the NHS be enabled to capitalise on the dual advantage of ACDSS?
In the second half of the chapter, I futher explain that motivated by this desire to help the NHS capitalise on the dual advantage of ACDSS, and inspired by the theory of systems engineering which argues that it is the design of the underpinning architecture (or information infrastructure) that is responsible for determining the success of a desired system (in this instance the desired ACDSS-enabled NLHS), the aim for my thesis became:
Aim: To design the information infrastructure that will enable the NHS to capitalise on the dual advantage of ACDSS.
Then drawing upon the theory of the logic of design as a conceptual logic of requirements developed by my supervisor, Professor Luciano Floridi, which sees the elicitation of requirements (scope, function, features, purpose) as the cornerstone of design, I introduce my overarching research question:
ORQ: What are the information infrastructure* requirements for the successful implementation of ACDSS in the NHS?
*Hardware and software; clinical content (raw data); human computer interface; people; workflow and communication; internal organisational policies; external rules, regulations, and pressures; and system measurements and monitoring requirements (see here)
Finally, I explain that I chose to base this question on the ‘theory of the logic of design as a conceptual logic of requirements’ because Government technology implementation projects (including NHS technology projects) often fail when a typical command-and-control or planned approach to implementation is taken. I reason, therefore, that if I were to adopt a similarly planned and inflexible approach to the identification of the necessary information infrastructure requirements for ACDSS implementation, I too may fail to help the NHS achieve success. In contrast, I reason that if I were to adopt a more creative, more ontological, more flexible, and more contextually aware, design-based approach – specifically intended to overcome the pitfalls of planned implementation approaches – I may be more able to help the NHS achieve success.
Chapter 2: Identifying the known knowns and known unkowns of ACDSS implementation
In this chapter, for the dual purpose of (a) developing a more precise definition of 'success'; and (b) clarifying the scope of the thesis, I provide a brief overview of the existing literature on 'factors associated with technology implementation success.' Drawing from existing literature on change management in the NHS; technology adoption (in general and in the NHS specifically); acceptance and adoption of AI in healthcare; implementation of AI into healthcare systems; and evaluation of indiviudal ACDSS products, I was able to produce a list of 'success factors' which, when themed (see below), gave rise to the following definition of successful implementation:
Successful implementation: technically feasible, socially acceptable, ethically justifiable, and legally compliant
Once this has been established, I combine the new definition of success with the above definition of 'information infrastructure' to expand the overarching research question as follows:
ORQ: What are the hardware and software; clinical content (raw data); human computer interface; people; workflow and communication; internal organisational policies; external rules, regulations, and pressures; and system measurements and monitoring requirements for the technically feasible, socially acceptable, ethically justifiable, and legally compliant implementation of ACDSS in the NHS?
Finally, I use this expanded definition to clarify the scope of the thesis as below:
Chapter 3: How to design NHS information infrastructure for the 21st century
In this methodology chapter, I explain that the fully defined research question led me to position my thesis within the Science and Technology Studies (STS) paradigm. More sepcifically, I explain that I decided this was the most appropriate paradigm because STS research focuses on understanding how the development, deployment, use (i.e., implementation), and consequences of specific technologies is influenced by historical, social, cultural, and contextual factors. In other words, I note that STS research covers all the components within the definition of successful (technical, ethical, social, and legal), and its multidisciplinary approach is well-suited to the broad definition of information infrastructure.
Next, I stress that it was important to establish this positioning because it helped to guide my selection of research methods. Specifically, I explain that I chose to adopt a mixed-methods, theory-informed, ethnographic, and interpretive methodology.
Finally, I explain that to guide the analysis process, and slowly - but sequentially - build up an answer to the overarching research question, I broke the overarching research question down into a series of sub-research questions, with one question aligned to one phase of the logic of design as shown below:
Chapter 4: ACDS and the NHS: It's (more than) complicated
In this first empirical chapter, I answer RQA(i)
RQA(i): Why is the current NHS information infrastructure design resulting in ACDSS implementation failure? What changes are required to increase the likelihood of implementation success?
I base the analysis on the ‘Non-Adoption, Abandonment, Scale-up, Spread, and Sustainability’ (NASSS) Framework (Greenhalgh et al., 2017a), as this is framework is well known to policymakers and used frequently as an internal tool by the Department of Health and Social Care. I use the analysis to argue that the current NHS information infrastructure design is fraught with disorganised complexity. I note that disorganised complexity is associated with a higher likelihood of the system experiencing negative unintended emergence, and a lower likelihood of the system experiencing positive intended emergence following ACDSS implementation. Therefore, current ACDSS implementation efforts are failing.
To make this argument clearer, I compare the level of complexity present in each of the NASSS Framework domains for traditional passive clinical decision support software (i.e., expert systems or computerised decision-trees) and ACDSS as below.
Finally, I conclude that to reduce the likelihood of implementation failure and increase the likelihood of implementation success, the NHS’s information infrastructure must be re-designed to make inherent information infrastrucutral complexity better organised.
Chapter 5: Be careful what you wish for
In this second empirical chapter, I answer RQA(ii)
RQA(ii): What would be the consequences if these [information infrastructure] changes were not made, or if the wrong changes were made?
I do this by developing a detailed description of three ‘anticipated’ scenarios that – if not proactively mitigated by the careful re-design of the NHS’s information infrastructure – could result in a fundamental re-engineering of the NHS and its underpinning values. The three scenarios are:
Scenario A: The patient is displaced from the centre of care because ACDSS may change what counts as valid knowledge about the body, what counts as good patient behaviour, & undermine the meta right not to know.
Scenario B: The fundamentals of care are disrupted because ACDSS may change power dynamics, devalue the ethics of care, & challenge established mechanisms of accountability.
Scenario C: The NHS ceases to be ‘for all’ because ACDSS may bake in bias and amplify the effects of the inverse care law (here the inverse data law).
I conclude that if the NHS wants to ensure successful implementation of ACDSS, the requirements for designing-out harms must be identified as well as the requirements for achieving positive outcomes. I explain that this elicitation of requirements is a key step in the process of making complexity more organised.
Chapter 6: Mind the gap
In this third empirical chapter, I answer RQB(i):
RQB(ii):What are the ideal NHS information infrastructure requirements to enable the successful implementation of ACDSS? How likely is it that these requirements will be met by current policy?
I do this by identifying the ideal vision (benefit-enhancing) in the following categories (derived from the ITPOSMO Framework (Heeks, 2002)): Information, Technology, Process, Objectives and Values, Staffing and Skills, and Management systems and structures:
Information requirement: epistemic certainty so that clinicians are confident ACDSS outputs are valid, valuable, and constructed to improve their capacity to make shared decisions that will ultimately lead to better outcomes for their patients.
Technology requirement: robust information exchange so that ACDSS facilitates the trustworthy and caring exchange of information between patients to (a) ACDSS developers; (b) policymakers and regulators responsible for determining the standards of care; and (c) clinicians
Process requirement: validated outcomes so that ACDSS enables clinicians to uphold and enhance the tenets of evidence-based medicine, to ensure all care in the NHS is delivered to an appropriate standard following the best evidence to deliver the best possible health outcome
Objectives & values requirement: actively protected values so that ACDSS not only upholds, but actively protects the core values of the NHS from all trade-offs or dilutions
Knowledge requirement: autonomous staff so that the autonomy of clinical staff over their decisions regarding their patients is maintained as is their confidence to not question their ‘right’ to this autonomy even when using ACDSS.
Management systems and structures requirement: meaningful accountability so that ACDSS implementation is supported by a proportionate governance framework that guarantees ACDSS developers, deployers, and users are meaningfully accountable to patients and the public.
Within each of these headline ‘vision requirements’ I identify ideal delivery (risk-mitigating) information infrastructure requirements i.e., what must the information infrastructure deliver in order to provide the system with the vision requirements withotu introducing new harms:
Epistemic certainty: consistent data quality, sufficient data quality, reliable data interpretability
Robust information exchange: UX focused EHR design, privacy enhancing data access, seamless system integration, protection from vendor lock-in
Validated outcomes: clearly stated clinical objectives, mindful model development, rigorous technical validation, rigorous clinical evaluatio, careful local calibration, ongoing impact evaluation
Actively protected values: commimtment to collective provision, commitment to patient centricity, commitment to quality care
Autonomous staff: protection of clincian epistemic authority, valued informatics workforce, skilled leadership,
Meaningful accountability: fit for purpose IG, regulated medical devices, clinician and patient protection, auditability
I then examine whether, given current policy developments, it is likely that these requirements will be met and, therefore, whether it is likely that the NHS will be able to capitalise on the dual advantage of ACDSS.
I conclude that there is a significant sociotechnical gap between the ideal requirements and the requirements covered by current policy and, therefore, it is currently unlikely that the NHS will be able to achieve successful ACDSS implementation unless this gap is closed (see below).
Chapter 7: Putting ACDSS in the right frame
In this fourth empirical chapter I answer RQB(ii): What underpinning assumptions and contextual factors might limit the likelihood of the ideal requirements being met by current policy?
RQB(ii): What underpinning assumptions and contextual factors might limit the likelihood of the ideal requirements being met by current policy?
I explain that answering this question is necessary because the first step to closing the sociotechnical gap identified in the previous chapter is understanding why it exists.
I go on to explain that the gap exists because the ideal information infrastructure requirements and the requirements covered by policy are underpinned by different assumptions. The requirements covered by policy are underpinned by assumptions of technical determinism, first order change, rationality, the innovation-stifling power of regulation. The ideal requirements are underpinned by assumptions of sociotechnical complexity, second order change, non-rationality, and the innovation-promoting power of regulation.
Next, I explain that the two sets of assumptions are so different because they belong to different communities. The ideal requirements are influenced by the scientific epistemic community (comprised primarily of clinicians, social scientists, and research software engineers). The requirements covered by policy are influenced by the technical epistemic community (comprised primarily of third-party private technology and ACDSS developers). The two communities do not agree on the function of the desired ACDSS-enabled NLHS and so they do not agree on the form requirements of the supporting information infrastructure.
I conclude that to close the sociotechnical gap, therefore, it will first be necessary to identify a unifying function that meets the needs of all stakeholders (including members of both communities) and negates the negative influence of the technical epistemic community’s assumptions on the development of policy at source. Unless this function is identified, it is possible that an inforamtion infrastructure will be designed that supports a function that the end-beneficiaries (patients and clinicians) do not want. Were this to be the case, implementation may be technically feasible and legally compliant but it is unlikely to be socially acceptable or ethically justifiable.
Chapter 8: A recipe for success
In this final empirical chapter, I answer RQC:
RQC: What is the unifying function of the desired ACDSS-enabled NLHS? How might policy successfully operationalise the ideal requirements to meet this function?
I answer this question by developing the conceptual model for the successful implementation of ACDSS into the NHS. Inspired by the middle-out approach to systems engineering, I do this in stages. Working from the top down, I first identify the conceptual function requirements that can negate the influence of the technical epistemic community’s assumptions on the development of policy at source. This involves first identifying a unifying function for the ACDSS-enabled NLHS.
To negate the negative influence of the technical epistemic community's assumptions on the design of the supporting NHS information infrastructure, said function must be:
Relatively narrow in scope - positioning ACDSS as one part of a much large 'whole' and avoiding the impression that quantity of information matters more than quality;
Focused on getting the basics right; grounded in the theory of evidence-based medicine (tied closely to use cases for which there is a clear, robust, evidence-base) and based on the principle of primum non nocere (recognising taht ACDSS is interventional);
Sufficiently flexible and adaptable to allow for contextualisation and incremental expansion;
Centred on 'recommendations' and 'second opinions' rather than 'decisions'; and
Intended to support (a) the competentence of clinciians - particualrly their role as learned intermediaries, and (b) shared decision-making.
For clinicians, patients, and system managers, acceptable functions of the desired ACDSS-enabled NLHS shaped by these rules are:
Clinician acceptable function: Improving the utility and usability of relevant NHS infroamtion at the point of care.
Patient acceptable function: Improving the trustworthines s of information used to inform decisions about care.
System manager acceptable function: Improving the evidence base (effectivenes of information) used to inform public health and service planning decisions.
From these functions, a unifying acceptable function can be identied as follows:
ACDSS-enabled NLHS: Maximise the utility, usability, efficacy, and trustworthiness of existing NHS information for the purpose of meeting clearly identifiable information needs.
This function gives rise to a set of function requirements utility, usabiltiy, efficacy, and trustworthiness. I then mapped to the success requirements (technically feasible, socially acceptable, ethically justifiable, legally compliant) and the vision requirements. (1) utility and usability, are aligned most closely with the information, and technology requirements, and so with the success criteria technical feasibility, and social acceptability; (2) trustworthiness is aligned most closely with the objectives and values, and skills and knowledge requirements, and so with the success criteria ethical justifiability; and (3) efficacy is aligned most closely with the process, and management systems and structures requirements, and so with the success criteria legal compliance.
Then, working from the bottom up, I identify the form requirements that can operationalise the ideal vision (benefit-enhancing) and delivery (risk-mitigating) requirements in keeping with the function requirements. In so doing I illustrate how function maps to form via concept.
Technically feasible (<> Usability <> Robust Information Exchange <> user friendly EHR, privacy enhancing data access, seamless system integration, protection from vendor lock-in): user-centred EHR design templates, model-to-data approaches to development, interoperability standards, audit and feedback tools, modular system design, local customisability.
Socially acceptable (<>Utility <> Epistemic certainty <> consistent data quality, sufficient data quantity, reliable data interpreability): valued data work, ACDSS ready datasets, mandatory explainability, clinician-led demand.
Ethically justifiable (<>trustworthiness <> protected values & autonomous staff <> commitment to collective provision, commitment to patient centricity, commitment to quality care, data literate senior leaders, valued analytics workforce, clinician epistemic authority): value pluralism, mandatory PPIE, recorded qualitative data, provably trustworthy relationships, buddy system, clincial not clerifcal analytical staff, continuing professional development for all staff, protection of the clinician right to override.
Legally compliant (<>efficacy <> validated outcomes & meaningful accountability <> celarly stated outcomes, mindful model development, rigorous clinical validation, rigorous clinical evaluation, careful local calibration, ongoing impact evaluation, fit for purpose IG, regulated medical devices, clinician and patient protection, auditability): health technology assessment and national screening committee review of new population-level ACDSS, guidelines on model and feature selection, centralised in-silico testing of ACDSS, centralised in-socio testing of ACDSS, roving data science team for local implementation, oversight of integration and use, simplified ownership and controllership of NHS data, classification of ACDSS as SaMD, clear regulation regarding liability, discrimination, and consumer protection, distributed responsibility and clear sanctions for bad actors, public logs of all transactions.
Finally, I combine the results of these two analyses with the analyses from Chapters Four, Five, and Six to produce a hierarchical conceptual model for the successful implementation of ACDSS into the NHS, that can be read as follows:
To become a Learning Healthcare System, the NHS requires the implementation of ACDSS. This in turn requires:
Supporting information infrastructure that ensures the implementation of ACDSS is technically feasible, ethically justifiable, socially acceptable, and legally compliant. This in turn requires:
An information infrastructure design that is intended to support the function “maximise the utility, usability, efficacy, and trustworthiness of existing NHS information for the purpose of meeting clearly identifiable information needs”. This in turn requires:
An information infrastructure design intended to provide the system with epistemic certainty, robust information exchange, validated outcomes, protected values, autonomous staff, and meaningful accountability. This in turn requires:
An information infrastructure design intended to deliver consistently good data quality; sufficient data quantity; reliable data interpretability; UX focused EHR design; privacy enhancing data access; seamless system integration; protection from vendor lock-in; clearly stated clinical objectives; mindful model development; rigorous technical validation; rigorous clinical evaluation; careful local calibration; ongoing impact evaluation; commitment to collective provision; commitment to patient centricity; commitment to quality care; protection of clinician epistemic authority; valued informatics workforce; da-literate leaders; fit for purpose Information Governance; regulated medical devices; clinician and patient protection; and auditability. This in turn requires:
Policy that is focused on EHR design templates; model-to-data ACDSS development approaches; interoperability standards; audit and feedback; modularity and local customisability; data-work; NHS ACDSS-ready data assets; clinicians as originators of demand; value pluralism; patient and public involvement and engagement; qualitative health information; provably trustworthy relationships; a technical buddy system; the status of NHS analysts as clinical not clerical; the clinician’s right to override; HTA and NSC review of ACDSS; model development and feature selection guidance; ‘in silico’ and ‘in socio’ model testing; roving data science teams; oversight of ACDSS integration and use’ data governance simplification and clarification; reglation of ACDSS as SaMD; process accountability regarding liability, discrimination, and consumer protection; and aduitability.
Chapter 9: An algorithmically enhanced NHS: conclusions, contributions, and next steps
Finally, in this concluding chapter, I surmise that whilst the existence of this conceptual model for the successful implementation of ACDSS into the NHS, represents a not-insignificant step forwards towards the aim of enabling the NHS to capitalise on the dual advantage of ACDSS, there are limitations that must be acknowledged, and future work will be required to overcome these limitations. Specifically:
The final model is still comprised only of necessary, not sufficient requirements, but sufficient requirements will be need to increase the chances of successful ACDSS implementation;
The final model remains at the middle-level of abstraction and is, therefore, still open to interpretation;
Design is an iterative process and the final model has not been iterated following independent evaluation and feedback, and may therefore be overly influenced by my own personal experiences and positionality;
The final model does not address whether an ACDSS-enabled NLHS will genuinely help the NHS deal with its pressing resource challenges, or whether it will simply divert resources away from more useful or valuable policy interventions; and
There is evidence to suggest that the NHS may not have the capacity, or the willing, to tackle change of this at this time.
I note that to overcome these limitations future work will be needed to identify the sufficient requirements following robust stakeholder engagement to subject the model to external independent scrutiny, iteration, and validation; to combine the mode with the results of research at a lower level of abstraction, for example research focused on identifying best practice for specific algorithm design; and to pilot the model to assess its overall chances of success.
Final Words
My thesis started with the knowledge that the NHS is struggling to cope following the UK’s exit from the European Union and the impact of the COVID-19 crisis and that, in enabling the NHS to become a NLHS, the implementation of ACDSS might help the NHS overcome these challenges if only it was known how to close the ACDSS implementation gap. It ended with the knowledge that to close the implementation gap a supporting information infrastructure characterised by a high-degree of organised complexity and designed to meet the ACDSS-enabled NLHS function of “maximising the utility, usability, efficacy, and trustworthiness of existing NHS information for specific information needs” is required. This new knowledge represents a not-insignificant step forwards towards the goal of enabling the NHS to capitalise on the dual advantage of ACDSS, and perhaps a small step towards enabling healthcare in general to benefit from the many potential advantages of AI. However, balancing the contributions of this thesis with appropriate recognition of the limitations, and ensuring the outlined next steps (future research involving the identification of sufficient requirements, further stakeholder engagement, piloting, and iterating) happen, is vital. This is vital, not only because honesty about limitations and needed further work will help improve the quality of the research, but also because the future of the NHS is too important not to insist every single proposed solution, recommendation, or decision is viewed from all possible angles and subject to intense scrutiny. We cannot move fast and break things, when the things we might break are people’s lives.The NHS might never have been in a deeper crisis, but its central premise of “comprehensive care, free at the point of need, for all” is undeniably worth fighting for (Abbasi, 2023). Hopefully, this thesis has made a small contribution to this fight. Hopefully others will feel equally motivated to contribute more. Hopefully the NHS will continue to exist for the next 75 years.