Value and Acceptability of a Novel Machine Learning Technology for Heart Failure Readmission Reduction: Qualitative Analysis of Clinical Roles and Workflows

Abstract

Background: Despite widespread adoption and demonstrated value in a range of industries, machine learning predictive algorithms are yet to be routinely used in frontline medical care. Significant health system and industry-based resources are allocated towards validating and refining predictive algorithms for a range of applications to ensure accuracy and reliability. For these algorithms to be useful and useable, further work is required to understand how and why they might fit into, and augment existing clinical workflows.

Objective: This qualitative study assessed the value and usability of a novel machine learning technology to predict and explain the risk of 30-day hospital readmission in patients with heart failure (HF). It involved exploring opportunities for integration of the technology within existing clinical workflows, and investigating key roles that use current readmission risk scores and may use future scores.

Methods: Semi-structured interviews (n=27) and targeted observations (n=3) were carried out with key stakeholders, including physicians, nurses, hospital administration, and non-clinical support staff. Participants were recruited from cardiology and general medicine units at an academic medical center within the Partners HealthCare system. Data was analyzed via inductive thematic and workflow analysis. Findings were validated via member checking across limited key roles (n=3).

Results: Results highlighted a number of factors that were deemed necessary by staff for successful integration of a risk prediction tool into existing clinical workflow. These included, but were not limited to the following. Staff clearly stated that any new tool must be easily accessible from within the electronic health record, which dictates the majority of existing clinical workflow. Staff emphasized that information should be consistently accurate and that any display must be digestible efficiently, intuitively and quickly (ie, within <5 seconds). Additionally, staff discussed that outputs of the risk prediction tool must match their clinical intuition, experience and interactions with the patient. To be truly valuable, the tool must also provide added value over and above these factors: some staff indicated that provision of role-specific and actionable next steps based on the system output would provide novel value to their daily work. Using these considerations, a number of role groups were identified as potentially able to derive value from the proposed risk prediction tool, including case managers, attending RNs, responding clinicians, hospital administration staff, nursing directors and attending physicians. Acceptability and value varied by role, specialization and clinical context. For example, cardiology-trained clinicians reported feeling well-versed in providing good clinical care and minimizing preventable readmissions, and thus saw less value in the tool. General medicine staff, however, indicated that a HF-specific tool may be impractical for their day-to-day work given the range of clinical presentations seen by them.

Conclusions: Findings resonate with existing literature around successful implementation and adoption of technologies in health care. Frontline clinicians are incredibly discerning around proposed changes to their existing workflow. Many HF readmission risk tools and initiatives have been trialled with mixed success; frontline staff demonstrated fatigue around piloting new initiatives. However, given the right conditions, staff reported some perceived value in machine learning-based tools to improve their daily work.

Keywords