The Evolution of Clinical Knowledge During COVID-19: Towards a Global Learning Health System

Summary

Objectives: We examine the knowledge ecosystem of COVID-19, focusing on clinical knowledge and the role of health informatics as enabling technology. We argue for commitment to the model of a global learning health system to facilitate rapid knowledge translation supporting health care decision making in the face of emerging diseases.

Methods and Results: We frame the evolution of knowledge in the COVID-19 crisis in terms of learning theory, and present a view of what has occurred during the pandemic to rapidly derive and share knowledge as an (underdeveloped) instance of a global learning health system. We identify the key role of information technologies for electronic data capture and data sharing, computational modelling, evidence synthesis, and knowledge dissemination. We further highlight gaps in the system and barriers to full realisation of an efficient and effective global learning health system.

Conclusions: The need for a global knowledge ecosystem supporting rapid learning from clinical practice has become more apparent than ever during the COVID-19 pandemic. Continued effort to realise the vision of a global learning health system, including establishing effective approaches to data governance and ethics to support the system, is imperative to enable continuous improvement in our clinical care.

Keywords

Medical informatics - knowledge representation - information storage and retrieval - pandemic response - learning health system

1 Introduction

The emergence of the global pandemic of coronavirus COVID-19 dominated much of the health informatics and medical research landscape during 2020. Hence it is appropriate that this end-of-year review of recent developments in medical knowledge management focuses on the pandemic.

The pandemic has highlighted the clear need for informatics to support management and synthesis of health information at the global scale and pace in the face of a rapidly spreading infection. However, it has also highlighted the presence of severe limitations in our ability to share, integrate, and analyse data at this scale.

To address those limitations, we propose that the model of a “global learning health system” (gLHS) can be deployed. The concept is of a learning health system (LHS) [[1] [2]], expanded to a global scale but with the singular focus on the viral disease. Indeed, the international effort to quickly gather and share knowledge for clinical diagnosis, management, and treatment of COVID-19 can be seen as an exemplar of a gLHS, albeit not yet fully realised or effective.

The key elements of a LHS, including the core information cycles that characterise it, were observed throughout the interactions of the global scientific community. Information flowed from practice (what was being done on the ground to manage COVID-19 patients) to data (what was captured about those patients and their clinical characteristics or response to interventions) to knowledge (about disease characteristics and trends, what care approaches worked, and what did not, based on analysis and modeling of the data) and rapid implementation for practice again. Furthermore, as required by a LHS, information technology infrastructure played a critical role in enabling these information flows.

In the wake of the previous SARS and Ebola virus epidemics, it was already argued that unified frameworks supporting clinical and biological data integration were critical to support evidence generation in a pandemic [[3]] and that information technology was needed for knowledge management [[4]]. Broader adoption of electronic health records has facilitated evidence generation, including in observational studies and in traditional randomised controlled trials, through their use to identify eligible patients, support data collection, and monitor outcomes [[5]]. Electronic sharing of patient-level data from trials facilitates re-analysis of outcomes and fosters reproducibility and trust in findings [[6]]. But, our public health and clinical responses during COVID-19 demanded much more sophisticated strategies for rapid information synthesis and knowledge management than was available, which we argue an effective gLHS would facilitate.

The rapid spread of COVID-19 internationally and its immediate impact on the global economy has led to much more widespread appreciation of the need to coordinate pandemic research, including substantially increased scientific globalism [[7]] – international research collaboration – and sharing of patient-level clinical data [[6]]. However, despite a few shining examples of rapid deployment of multi-site clinical trials [[8] [9] [10] [11]], clinical research for COVID-19 has been highly fragmented [[12]], with a particular dearth of meaningful evidence in the area of non-drug interventions with public policy impacts [[13]].

This survey will show that although there is still work to be done, the pandemic has illustrated that many of the elements required for a global learning health system are in place, critically including the human motivation to achieve it.

The framework of the gLHS that we present is not only a useful way of characterising how knowledge evolved during the pandemic under the strong impetus to support knowledge-informed clinical care and public health response to COVID-19, but also provides an architecture for a system that we can invest in to ensure robust knowledge evolution for ongoing global health needs into the future.

2 A Global Learning COVID-19 System

In this section, we introduce the framework for ‘learning’ that we will adopt in our analysis of the evolution of knowledge during the first year of the COVID-19 crisis (Section 2.1). We then present a model of the collective activity toward learning about COVID-19, viewed as a global-scale learning health system (Section 2.2), and illustrating the elements of the gLHS that emerged as observed from the literature.

2.1 Framework for Conceptualising Learning in a Crisis

To frame the evolution of knowledge in the COVID-19 crisis, we follow a recent proposal by Tovstiga and Tovstiga (2020) to adopt a classical four-quadrant ‘conscious-competence’ conceptual framework from learning theory [[14]]. The model is presented in [Figure 1], illustrating the stages of a learning trajectory, from unconscious ignorance of lack of knowledge, to deeply embedded knowledge.

1. Quadrant 1: Zone of uncertainty, including lack of clarity about a topic

2. Quadrant 2: Zone of learning, where the value of knowledge on the topic is recognised and sought. Questions play a crucial role in this zone;

3. Quadrant 3: Zone of actionable knowledge, where learning is consolidated and integrated with existing knowledge;

4. Quadrant 4: Zone of embedded understanding, enabling intuitive action. Knowledge in this zone is often not fully recognised.

Fig. 1 The ‘conscious-competence’ matrix of learning, based on a model originally attributed to Broadwell (1969) [[83]]; adapted from [[14]].

While typically applied in the context of an individual learner, the Tovstigas argue that the framework effectively reflects the general knowledge evolution process in the context of the COVID-19 crisis. They further suggest that it is useful for structuring and understanding the learning trajectory with respect to the crisis, demonstrating the various phases through an analysis of information about COVID-19 communicated through news reports.

Their analysis identifies data analytics and scientific knowledge sharing as key drivers of the learning trajectory in COVID-19, citing efforts such as the World Health Organization's creation of a global COVID-19 clinical information platform based on a standard case report form[1] requesting data on specific detailed clinical and demographic parameters on COVID-19 positive patients. This underscores the important role of data standards in supporting the learning needed to manage the pandemic. These are also key elements of the gLHS and highlights the relevance of the model.

2.2 Modelling the COVID-19 Knowledge Ecosystem Through the Learning Health System

Building on this framework for conceptualising learning, we propose that the core model of a LHS [[1] [2]] can be applied to characterise the rapid evolution of knowledge that has occurred during the COVID-19 pandemic. In this model, data and analytics over clinical practice data from patient care drives learning of new knowledge that can be implemented to improve clinical practice, leading to continuous improvement. A number of critical knowledge management and information technology elements can be identified as key enablers of this learning process, supporting the significant human efforts that catalysed and provided the appropriate socio-technical conditions for the learning cycle.

Our proposed model for the COVID-19 gLHS is shown in [Figure 2]. The impetus for the learning process in COVID-19 arose from a knowledge gap, the gap between purposeful action grounded in knowledge (Q3) that exists for routine clinical care and the uncertainty surrounding diagnosis and management of the novel virus in affected patients (Q1). This then triggered a broad effort to gather data to fill the gap, primarily taking advantage of data collected through electronic health record systems. Data collection and integration, facilitated through electronic data sharing, then enabled learning (Q2), actioned through observational analysis, clinical trials, predictive modelling, and other research leveraging data, with the objective of turning data into information. Publications summarising these studies served as the key vehicle for sharing research results, but challenges in finding and interpreting papers – particularly in the face of a flurry of research activity – resulted in remaining uncertainty in the knowledge that existed (Q4). Translation of new knowledge into practice (Q3) required evidence synthesis approaches such as systematic reviews, critically involving searching (information retrieval), screening, appraisal, and meta-analysis of research publications. Key conclusions were rapidly shared via actively maintained, living guidelines [[15]] and platforms [[16]] or tools [[17]] for making available clinical decision support knowledge artifacts.

Fig. 2 Abstraction of the structure of the global learning system in place for COVID-19. Knowledge management activities are overlaid onto the Learning Health System model (core LHS cycle figure adapted from [[84]]), and related to the learning framework of ‘conscious-competence’ presented in [Figure 1].

2.2.1 Electronic Data Capture and Data Sharing

Electronic health records (EHRs) are a key resource in the learning health system, as they provide the data that is used to drive learning from practice. For COVID-19, EHRs were analysed to characterise early cases of the infection in Wuhan, China [[18] [19]], to provide important information related to the efficacy of symptom-based screening [[20]], and to collect data on patients prospectively after enrolment in a trial [[21]]. In the UK, the OpenSafely Platform[2] was used to identify factors associated with COVID-19 deaths through analysis of the primary care records of over 17 million patients [[22]], facilitated through the use of a single EHR system (TPP SystemOne) by general practice surgeries covering approximately 40% of the UK population. Similarly, a study of risk factors associated with death due to COVID-19 [[23]] was made possible by the use of a single integrated EHR system across many sites, and the Quick COVID-19 Severity Index was developed with data from a single health system with nine Emergency Departments [[24]]. A highly-cited study demonstrating lack of effectiveness of hydroxychloroquine treatment was conducted using data extracted directly from the New York-Presbyterian / Columbia University Irving Medical Center EHR [[25]].

Leveraging distributed EHRs across national and international boundaries through collaborative consortia and clinical data networks [[26]], several large-scale studies were undertaken to characterise COVID-19 patients in relation to similar disease groups [[27]], to understand the trajectory of the disease [[28]], and to study interaction of the disease with patient medications [[29]]. Such studies were enabled through the adoption of common data models to harmonise data, prominently the Observational Health Data Sciences and Informatics (OHDSI) Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM) [[30]]; partners in the OHDSI network contribute statistical results based on federated querying of local data sources represented with the CDM.

The Columbia Open Health Data for COVID-19 Research (COHD-COVID) data set [[31]] takes a different approach, also based on the OMOP CDM. They make publicly available prevalence data for conditions, drugs, procedures and their co-occurrences, calculated from their EHR, thereby sharing aggregate counts rather than patient-level data. Coupled with slight perturbation and suppression of rare concepts, this eliminates privacy concerns while still enabling comparative data analysis.

Absent adoption of common data models and data sharing platforms for EHR data, research across multiple sites required manual extraction of defined clinical data elements from EHRs submitted to a central study team. This was done in the UK for a pediatric study related to COVID-19 [[32]] and for a study of factors associated with coronavirus death in the US [[33]]. In China, targeted national data reporting to a central research team supported observational studies [[34] [35]]. Data harmonisation and cleaning in these cases relied on expert review of data submissions. For example, to support the development of a machine learning-based predictive model for COVID-19 associated mortality based on EHR data from five New York hospitals, expert mappings and data harmonisation were performed by a multi-disciplinary team of clinicians [[36]].

Other approaches to collecting electronic clinical data were also utilised, including deployment of clinical natural language processing to rapidly identify and characterise patients relevant to COVID clinical questions [[37]]. Registries were established to collect disease-specific data items utilising case report forms (CRFs). The most sophisticated of these used electronic CRFs mapped to common data standards and submitted to a central database, such as the VIRUS-COVID-19 registry[3] [[38] [39]] based on REDCap [[40]], which coordinated data entered in over twenty sites. Electronic surveys to gather data directly from patients[4] were introduced to allow for collection of data from non-hospital settings [[41]].

Despite these successful demonstrations of the use of EHR and registry data to study COVID-19, a number of challenges remain, summarised effectively in Madhavan et al. [[26]]. EHRs are not primarily designed to support coordinated research and public health response, and their use in this context placed substantial strain on informatics and data science teams at hospitals. Even basic conversion of spreadsheet-based systems for research data collection to formal database structures can prove a challenge where local differences exist in how fields are interpreted or used [[4]]. Infrastructure for individual-level storage and exchange of data for research purposes is required, as well as commitment to common data models, terminologies, and data interchange standards [[42]]. Rapid adaptation of standard clinical vocabularies such as ICD, LOINC, and SNOMED-CT to include relevant new vocabulary is needed [[43] [44]]. Governance and ethics factors must also be crucially addressed. The National COVID Cohort Collaborative[5] (N3C) in the US aims to address many of these points, utilising the OMOP CDM to bring together data from disparate sources, and aiming to facilitate record-level analysis of COVID-19 patients (and matched controls) in a secure environment [[45]]. Tremendous progress has been made in the rush to address the pandemic, but there is still much work to be done before truly international-scale data can be efficiently and effectively brought together.

2.2.2 Data Analytics and Modelling

With the availability of large-scale and complex data about COVID-19 came the need to analyse and model it. Advanced computational methods including machine learning, natural language processing and other artificial intelligence (AI) methods can play key roles [[46]], and indeed significantly contributed to detecting the COVID-19 outbreak, diagnosing the disease, and predicting outcomes [[47]]. Models are critical to inform decision making [[48]], supporting prediction and simulation of outcomes under varying conditions or patient characteristics.

Several of the EHR-based studies cited above utilise machine learning over clinical variables [21] [36]], while more traditional statistical or epidemiological modelling is typically employed for observational studies. Imaging analysis models have also been adapted to COVID-19 from models for related diseases such as pneumonia [[49]], facilitated by public sharing of COVID-19 images with the AI community[6] [7] [8] [[50]].

The challenges faced in building sufficiently large data sets has meant that the modelling of COVID-19 has resulted in high risk of bias and poor external validation [[51] [52] [53]]. Additionally, the inherent nature of observational EHR-based studies, lacking controlled cohort selection, may lead to unreliable results due to confounding [[5]] and a risk of case contamination due to ambiguous cohort definitions [[54]].

2.2.3 COVID-19 Information Retrieval and Synthesis

The amount of COVID-19 research output has been remarkable; based on the LitCovid index of this research at the US National Library of Medicine[9] [[55]], over 75,000 COVID-19-related publications were added to the PubMed literature repository between January and late November 2020, at a steady pace of approximately 2000 articles per week (see [Figure 3]). A review of the literature focused on clinical presentation and management of COVID-19 as of June 15, 2020, prioritised for general medicine readers, identified over 100 relevant articles [[56]]. Over 7000 COVID-19 clinical trials are registered in the World Health Organization's International Clinical Trials Registry Platform[10], which plays a key role in identifying research gaps [[57]]. Nearly 3000 systematic reviews related to COVID-19 are currently catalogued in the Living OVerview of Evidence (L·OVE) platform[11] [[58]].

Fig. 3 Weekly publications in 2020 related to COVID-19, as indexed in the LitCovid collection of PubMed [[55]].

This scientific knowledge is also broadly accessible; over 75% of this research is available in open access publications, an unprecedented proportion, more than double the rate for publications generally during 2015-2019 and for other topics in 2019-2020 [[7]].

However, the accumulation of this body of evidence about one disease in such a short time is also overwhelming. A key challenge in COVID-19 knowledge management lay in navigating this massive quantity of research evidence to support diagnosis, treatment, and public policies, as well as molecular information about the virus. The sheer volume of the research – published in natural language texts that must be read and interpreted – requires significant effort to translate into knowledge. Studies must be synthesised and evaluated, and broader conclusions drawn from comparing multiple studies examining the same question.

Therefore, many systems based on information retrieval or text mining were created in response to this challenge, including our COVID-SEE Scientific Evidence Explorer system [[59]]; more are reviewed in [[60] [61] [62]]. An important resource in these efforts was the COVID-19 Open Research Dataset (CORD-19) which compiled a significant collection of literature for both COVID-19 and related coronaviruses into a single, downloadable resource [[63]].

Leveraging such tools, community-based approaches to collect, curate, and model knowledge rapidly emerged for COVID-19. Groups began working together to review the literature and build living evidence guidelines that were updated as new information was made available [[15]]. Utilising systematic review automation technologies, complete reviews could be undertaken in a matter of weeks [[64]].

However, in the rapidly changing information space of COVID-19, the rush to explore and share research outcomes also resulted in poor study designs, poor research reporting, and lack of coordination and redundancy in research activities [[13]]. Coupled with the data biases noted above, this creates new problems – wasted effort, increased review and quality appraisal work, and uncertainty about key diagnostic, prognostic, and treatment decisions. The gLHS, effectively implemented, could provide the coordination and feedback mechanisms needed to address these problems.

2.2.4 Knowledge Dissemination

Knowledge has been recognised as strategically important for managing pandemics [[48]] and it plays a central role in our learning-based model. As knowledge is acquired through learning, it must be shared in order to have impact. While publications serve a key role in disseminating knowledge, alone they are insufficient and ambiguous to guide practice. Social media have been used effectively for knowledge dissemination during COVID-19 [[65]], but this focuses on transferring knowledge between individuals.

Knowledge management implemented through information technology can improve information sharing and coordination [[4]]. Several key elements for knowledge management in pandemics have been identified [[66]]:

• Shared knowledge spaces utilising consistent vocabulary.

• Formal representations of knowledge.

• Enabling reusable knowledge.

• Empowering human collaboration through knowledge sharing.

All of these elements were adopted to one degree or another during the COVID-19 pandemic, through the scientific globalism that emerged; in consortia like the N3C [[45]], working groups like those organised through the Research Data Alliance [[42]], and informal data and clinical networks.

Knowledge has been disseminated through numerous mechanisms, including online platforms such as the Australian National COVID-19 Clinical Taskforce[12], Registry sites [[39]] and through clinical decision support tools such as the Magic Evidence Ecosystem Foundation (MAGIC) MAGICApp [[17]].

The push for formal representation of knowledge, including computational and executable models that can be integrated into health information systems to enable application of knowledge to practice [[67] [68]], has gained momentum during COVID-19. The COVID-19 Disease Map project [[69] [70]] captured and made available molecular interaction information for the SARS-CoV-2 virus, based on manual curation, supported by weekly videoconferences. Knowledge graphs are also being used to support representation and integration of the variety of biomedical data related to COVID-19 [[71]], including via text mining [[72]]. Through the adoption of standardised ontology identifiers, comparable data in different resources can be linked together for analysis, and combined in different ways for different tasks.

Much of this work has been based on automated or semi-automated analysis of literature. Direct generation of computable evidence from structured clinical trial registries has also been proposed [[73] [74]], which would shortcut the need for literature-based synthesis.

3 Discussion

Friedman and colleagues have stated [[75]]:

“A national-scale LHS will have to be understood and designed as such a cyber-social ecosystem: a large-scale, decentralized, human-intensive, cyber-catalyzed and cyber-supported information processing system. The system as a whole—not just the digital infrastructure, but also networks of people and institutions—will have to be understood not just as users of a technological infrastructure, but also as parts of the information system itself.”

Extending this to a global-scale LHS demands an even broader view of the relevant ecosystem. The scale is larger, even more decentralised, and crosses a more diverse set of legal jurisdictions. It is apparent that the people and institutions play a critical part of the information system, to make possible the required data sharing – including tackling legal barriers and leading ethical discussions around data sharing – as well as by supporting effective communication of knowledge.

It has been observed that we entered the pandemic without a functioning LHS [[76]]. The authors ask [[76]]:

“We have the motivation. We have the vision. We have the technology. We have a roadmap. What are the barriers?”

answering:

“The issue is culture. We need to treat medical data as a public good.”

They further point to the ethics framework of Faden et al. [[77]] that identifies the dual obligations of health professionals to learn and implement, and patients to participate in the learning system by contributing their data.

Initiatives such as the US N3C are making important strides towards realising a gLHS. We do appear to have the motivation, the vision, and the technology. A recent review of the use of digital technologies during COVID-19 highlights how far we have come in leveraging technology for the pandemic response [[78]]. What is required to achieve an ongoing gLHS is a commitment to the vision, coupled with rigorous data governance and legal and regulatory frameworks that safeguard patient privacy while supporting the learning knowledge ecosystem.

4 Conclusions

The model we have proposed is strongly aligned with the Agency for Healthcare Research and Quality evidence-based Care Transformation Support (ACTS) Knowledge Ecosystem initiative referred to as the ‘ACTS COVID-19 Evidence to Guidance to Action Collaborative’[13], which aims to continually enhance patient care throughout the pandemic, as the evidence base evolves. This Collaborative emphasises development of digital infrastructure to support the Knowledge Ecosystem, a cycle of Action-Data-Evidence-Guidance that mirrors the LHS cycle. It has further been active in developing groups such as COKA, the COVID-19 Knowledge Accelerator Initiative[14] [[74]], a response focused on COVID-19 to the call by Dunn and Bourgeois [[73]] to aim for computable knowledge synthesis and representation, through the use of standards such as EBMonFHIR[15] [[79]] and CPGonFHIR[16], or rule formalisms for computational clinical guideline specification [[80]].

The vision pursued in these initiatives is still under active development, and has required a vast community of clinicians, researchers, informaticians, developers, industry and government representatives, and beyond coming together with the common objective of addressing the technical, policy or legal, and cultural hurdles to enable more effective management of the COVID-19 pandemic. It has been argued that infrastructure is currently sorely lacking in most public health organisations to realise this vision effectively or efficiently [[81]]. There are still many unanswered questions about how to overcome bias and determine causality through real-world data [[24] [82]]. As we have shown, many of the learning and knowledge sharing activities in the context of the pandemic have been limited to very human-intensive approaches.

However, the core gLHS framework is in place, technology has been harnessed in many ways to share data and knowledge at a pace that arguably outstripped the spread of the virus, the requirements for information technology systems to support data and knowledge exchange are increasingly being clarified, and the initial steps toward achieving the vision have been made. This is entirely thanks to a tremendous response by the scientific community with a shared objective of improving outcomes for patients. Successful examples of large-scale, truly international data sharing and research collaborations now exist [[28]]. Both the need for and the value of continued work towards a healthcare system enabled through data and information technologies – a system that can be achieved through the gLHS – are now obvious.

Continued efforts towards achieving a robust gLHS are important, not only to allow us to respond to this pandemic and to prepare us to respond to the next pandemic, but to support continuous improvements in how we care for human health. We now know that we can do this.

No conflict of interest has been declared by the author(s).

Acknowledgements

KV is supported by numerous grants from the Australian Research Council and the National Health and Medical Research Council of Australia. This review primarily relates to her work with the Industrial Transformation Training Centre in Cognitive Computing for Medical Technologies, grant IC170100030 from the Australian Research Council, and the Centre of Research Excellence in Digital Health, grant 1134919 from the Australian National Health and Medical Research Council. The author would also like to thank Dr Brian Hur and the anonymous reviewers for feedback on previous versions of the manuscript.

¹ https://www.who.int/publications/i/item/WHO-2019-nCoV-Clinical_CRF-2020.4

² https://opensafely.org

³ https://www.sccm.org/Research/Research/Discovery-Research-Network/VIRUS-COVID-19-Registry

⁴ http://covidhealthquest.com/

⁵ https://covid.cd2h.org/n3c

⁶ https://github.com/ieee8023/covid-chestxray-dataset

⁷ https://coronacases.org/

⁸ http://medicalsegmentation.com/covid19/

⁹ https://www.ncbi.nlm.nih.gov/research/coronavirus/

¹⁰ https://www.who.int/clinical-trials-registry-platform

¹¹ https://iloveevidence.com/

¹² https://covid19evidence.net.au/

¹³ https://covid-acts.ahrq.gov/

¹⁴ https://www.gps.health/covid19_knowledge_accelerator.html

¹⁵ https://confluence.hl7.org/display/CDS/EBMonFHIR

¹⁶ http://build.fhir.org/ig/HL7/cqf-recommendations/

• References

• 1 Friedman CP, Wong AK, Blumenthal D. Achieving a Nationwide Learning Health System. Science Translational Medicine 2010; 2: 57cm29-57cm29 DOI: 10.1126/scitranslmed.3001456.
• 2 Grossman C, Powers B, McGinnis JM. Digital infrastructure for the learning health system: The foundation for continuous improvement in health and health care workshop series summary. National Academies Press Washington; DC: 2011: 311
• 3 Rojek AM, Horby PW. Modernising epidemic science: enabling patient-centred research during epidemics. BMC Med. 2016; 14: 212 DOI: 10.1186/s12916-016-0760-x.
• 4 Devadoss PR, Shan Ling P, Singh S. Managing knowledge integration in a national health-care crisis: lessons learned from combating SARS in Singapore. IEEE Transactions on Information Technology in Biomedicine 2005; 9: 266-75 DOI: 10.1109/TITB.2005.847160.
• 5 Collins R, Bowman L, Landray M, Peto R. The Magic of Randomization versus the Myth of Real-World Evidence. New England Journal of Medicine 2020; 382: 674-8 DOI: 10.1056/NEJMsb1901642.
• 6 Ewers M, Ioannidis JPA, Plesnila N. Access to data from clinical trials in the COVID-19 crisis: open, flexible, and time-sensitive. Journal of Clinical Epidemiology 2020; DOI: 10.1016/j.jclinepi.2020.10.008.
• 7 Lee JJ, Haupt JP. Scientific globalism during a global crisis: research collaboration and open access publications on COVID-19. High Educ (Dordr) 2020; 1-18 DOI: 10.1007/s10734-020-00589-0.
• 8 The RECOVERY Collaborative Group. Dexamethasone in Hospitalized Patients with Covid-19 ’ Preliminary Report. New England Journal of Medicine 2020; DOI: 10.1056/NEJMoa2021436.
• 9 The RECOVERY Collaborative Group. Effect of Hydroxychloroquine in Hospitalized Patients with Covid-19. New England Journal of Medicine. 2020; 383: 2030-40 DOI: 10.1056/NEJMoa2022926.
• 10 Horby PW, Mafham M, Bell JL, Linsell L, Staplin N, Emberson J. et al. Lopinavir-ritonavir in patients admitted to hospital with COVID-19 (RECOVERY): a randomised, controlled, open-label, platform trial. The Lancet 2020; 396: 1345-52 DOI: 10.1016/S0140-6736(20)32013-4.
• 11 Pan H, Peto R, Karim QA, Alejandria M, Henao-Restrepo AM, García CH. et al. Repurposed antiviral drugs for COVID-19 -interim WHO SOLIDARITY trial results. medRxiv . 2020:2020.10.15.20209817 DOI: 10.1101/2020.10.15.20209817.
• 12 Tikkinen KAO, Malekzadeh R, Schlegel M, Rutanen J, Glasziou P. COVID-19 clinical trials: learning from exceptions in the research chaos. Nature Medicine 2020; 26: 1671-2 DOI: 10.1038/s41591-020-1077-z.
• 13 Glasziou PP, Sanders S, Hoffmann T. Waste in COVID-19 research. BMJ 2020; 369: m1847 DOI: 10.1136/bmj.m1847.
• 14 Tovstiga N, Tovstiga G. COVID-19: a knowledge and learning perspective. Knowledge Management Research & Practice. 2020; 1-6 DOI: 10.1080/14778238.2020.1806749.
• 15 Tendal B, Vogel JP, McDonald S, Norris S, Cumpston M, White H. et al. Weekly updates of national living evidence-based guidelines: Methods for the Australian living guidelines for care of people with COVID-19. J Clin Epidemiol 2020; 131: 11-21 DOI: 10.1016/j.jclinepi.2020.11.005.
• 16 Lomotan EA, Meadows G, Michaels M, Michel JJ, Miller K. To Share is Human! Advancing Evidence into Practice through a National Repository of Interoperable Clinical Decision Support. Appl Clin Inform 2020; 11: 112-21 DOI: 10.1055/s-0040-1701253.
• 17 Rochwerg B, Agoritsas T, Lamontagne F, Leo Y-S, Macdonald H, Agarwal A. et al. A living WHO guideline on drugs for covid-19. BMJ 2020; 370: m3379 DOI: 10.1136/bmj.m3379.
• 18 Wang D, Hu B, Hu C, Zhu F, Liu X, Zhang J. et al. Clinical Characteristics of 138 Hospitalized Patients With 2019 Novel Coronavirus-Infected Pneumonia in Wuhan, China. JAMA 2020; 323: 1061-9 DOI: 10.1001/jama.2020.1585.
• 19 Huang C, Wang Y, Li X, Ren L, Zhao J, Hu Y. et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. The Lancet 2020; 395: 497-506 DOI: 10.1016/S0140-6736(20)30183-5.
• 20 Callahan A, Steinberg E, Fries JA, Gombar S, Patel B, Corbin CK. et al. Estimating the efficacy of symptom-based screening for COVID-19. npj Digital Medicine 2020; 3: 95 DOI: 10.1038/s41746-020-0300-0.
• 21 Burdick H, Lam C, Mataraso S, Siefkas A, Braden G, Dellinger RP. et al. Prediction of respiratory decompensation in Covid-19 patients using machine learning: The READY trial. Comput Biol Med. 2020; 124: 103949 DOI: 10.1016/j.compbiomed.2020.103949.
• 22 Williamson EJ, Walker AJ, Bhaskaran K, Bacon S, Bates C, Morton CE. et al. Factors associated with COVID-19-related death using OpenSAFELY. Nature 2020; 584: 430-6 DOI: 10.1038/s41586-020-2521-4.
• 23 Tartof SY, Qian L, Hong V, Wei R, Nadjafi RF, Fischer H. et al. Obesity and Mortality Among Patients Diagnosed With COVID-19: Results From an Integrated Health Care Organization. Annals of internal medicine 2020; 173: 773-81 DOI: 10.7326/M20-3742.
• 24 Haimovich AD, Ravindra NG, Stoytchev S, Young HP, Wilson FP, van Dijk D. et al. Development and Validation of the Quick COVID-19 Severity Index: A Prognostic Tool for Early Clinical Decompensation. Annals of Emergency Medicine 2020; 76: 442-53 DOI: 10.1016/j.annemergmed.2020.07.022.
• 25 Geleris J, Sun Y, Platt J, Zucker J, Baldwin M, Hripcsak G. et al. Observational Study of Hydroxychloroquine in Hospitalized Patients with Covid-19. New England Journal of Medicine 2020; 382: 2411-8 DOI: 10.1056/NEJMoa2012410.
• 26 Madhavan S, Bastarache L, Brown JS, Butte A, Dorr D, Embi PJ. et al. Use of Electronic Health Records to Support a Public Health Response to the COVID-19 Pandemic in the United States: A Perspective from Fifteen Academic Medical Centers. Journal of the American Medical Informatics Association 2020; DOI: 10.1093/jamia/ocaa287.
• 27 Burn E, You SC, Sena AG, Kostka K, Abedtash H, Abrahão MTF. et al. Deep phenotyping of 34,128 adult patients hospitalised with COVID-19 in an international network study. Nature Communications 2020; 11: 5009 DOI: 10.1038/s41467-020-18849-z.
• 28 Brat GA, Weber GM, Gehlenborg N, Avillach P, Palmer NP, Chiovato L. et al. International electronic health record-derived COVID-19 clinical course profiles: the 4CE consortium. npj Digital Medicine 2020; 3: 109 DOI: 10.1038/s41746-020-00308-0.
• 29 Morales DR, Conover MM, You SC, Pratt N, Kostka K, Duarte Salles T. et al. Renin-angiotensin system blockers and susceptibility to COVID-19: a multinational open science cohort study. medRxiv . 2020:2020.06.11.20125849. DOI: 10.1101/2020.06.11.20125849.
• 30 Hripcsak G, Shang N, Peissig PL, Rasmussen LV, Liu C, Benoit B. et al. Facilitating phenotype transfer using a common data model. Journal of Biomedical Informatics 2019; 96: 103253 DOI: 10.1016/j.jbi.2019.103253.
• 31 Lee J, Kim JH, Liu C, Hripcsak G, Ta C, Weng C. COHD-COVID: Columbia Open Health Data for COVID-19 Research. medRxiv . 2020:2020.11.17.20232983. DOI: 10.1101/2020.11.17.20232983.
• 32 Deep A, Upadhyay G, du Pré P, Lillie J, Pan D, Mudalige N. et al. Acute Kidney Injury in Pediatric Inflammatory Multisystem Syndrome Temporally Associated With Severe Acute Respiratory Syndrome Coronavirus-2 Pandemic: Experience From PICUs Across United Kingdom. Crit Care Med. 2020; 48: 1809-18 DOI: 10.1097/ccm.0000000000004662.
• 33 Gupta S, Hayek SS, Wang W, Chan L, Mathews KS, Melamed ML. et al. Factors Associated With Death in Critically Ill Patients With Coronavirus Disease 2019 in the US. JAMA Internal Medicine 2020; 180: 1436-46 DOI: 10.1001/jamainternmed.2020.3596.
• 34 Guan W-J, Ni Z-Y, Hu Y, Liang W-H, Ou C-Q, He J-X. et al. Clinical Characteristics of Coronavirus Disease 2019 in China. N Eng J Med. 2020; 382: 1708-20 DOI: 10.1056/NEJMoa2002032.
• 35 Liang W, Liang H, Ou L, Chen B, Chen A, Li C. et al. Development and Validation of a Clinical Risk Score to Predict the Occurrence of Critical Illness in Hospitalized Patients With COVID-19. JAMA Internal Medicine 2020; 180: 1081-9 DOI: 10.1001/jamainternmed.2020.2033.
• 36 Vaid A, Somani S, Russak AJ, De Freitas JK, Chaudhry FF, Paranjpe I. et al. Machine Learning to Predict Mortality and Critical Events in a Cohort of Patients With COVID-19 in New York City: Model Development and Validation. J Med Internet Res. 2020; 22: e24018 DOI: 10.2196/24018.
• 37 Neuraz A, Lerner I, Digan W, Paris N, Tsopra R, Rogier A. et al. Natural Language Processing for Rapid Response to Emergent Diseases: Case Study of Calcium Channel Blockers and Hypertension in the COVID-19 Pandemic. J Med Internet Res. 2020; 22: e20773 DOI: 10.2196/20773.
• 38 Walkey AJ, Kumar VK, Harhay MO, Bolesta S, Bansal V, Gajic O. et al. The Viral Infection and Respiratory Illness Universal Study (VIRUS): An International Registry of Coronavirus 2019-Related Critical Illness. Crit Care Explor 2020; 2: e0113 DOI: 10.1097/cce.0000000000000113.
• 39 Walkey AJ, Sheldrick RC, Kashyap R, Kumar VK, Boman K, Bolesta S. et al. Guiding Principles for the Conduct of Observational Critical Care Research for Coronavirus Disease 2019 Pandemics and Beyond: The Society of Critical Care Medicine Discovery Viral Infection and Respiratory Illness Universal Study Registry. Crit Care Med. 2020; 48: e1038-e44 DOI: 10.1097/ccm.0000000000004572.
• 40 Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)’a metadata-driven methodology and workflow process for providing translational research informatics support. Journal of biomedical informatics 2009; 42: 377-81
• 41 Verma SS, Chung WK, Dudek S, Williamson JL, Verma A, Robinson S. et al. Research on COVID-19 through patient-reported data: a survey for observational studies in the COVID-19 pandemic. Journal of Clinical and Translational Science 2020; 1-5 DOI: 10.1017/cts.2020.509.
• 42 Research Data Alliance COVID-19 Working Group. RDA COVID-19 Recommendations and Guidelines on Data Sharing. Zenodo 2020
• 43 Leaman R, Lu Z. editors. A Comprehensive Dictionary and Term Variation Analysis for COVID-19 and SARS-CoV-22020 dec; Online: Association for Computational Linguistics.
• 44 Vorisek CN, Klopfenstein SAI, Sass J, Lehne M, Schmidt CO, Thun S. Evaluating Suitability of SNOMED CT in Structured Searches for COVID-19 Studies. Stud Health Technol Inform. 2021; 281: 88-92 DOI: 10.3233/shti210126.
• 45 Haendel MA, Chute CG, Gersing K. The National COVID Cohort Collaborative (N3C): Rationale, Design, Infrastructure, and Deployment. Journal of the American Medical Informatics Association 2020; DOI: 10.1093/jamia/ocaa196.
• 46 Moore JH, Barnett I, Boland MR, Chen Y, Demiris G, Gonzalez-Hernandez G. et al. Ideas for how informaticians can get involved with COVID-19 research. BioData Mining 2020; 13: 3 DOI: 10.1186/s13040-020-00213-y.
• 47 Scott IA, Coiera EW. Can AI help in the fight against COVID-19?. Medical Journal of Australia 2020; 213: 439-41.e2 DOI: 10.5694/mja2.50821.
• 48 Ammirato S, Linzalone R, Felicetti AM. Knowledge management in pandemics. A critical literature review. Knowledge Management Research & Practice. 2020; 1-12 DOI: 10.1080/14778238.2020.1801364.
• 49 Shi F, Wang J, Shi J, Wu Z, Wang Q, Tang Z. et al. Review of Artificial Intelligence Techniques in Imaging Data Acquisition, Segmentation and Diagnosis for COVID-19. IEEE Reviews in Biomedical Engineering 2020; 01 DOI: 10.1109/RBME.2020.2987975.
• 50 Cohen JP, Morrison P, Dao L, Roth K, Duong TQ, Ghassemi M. Covid-19 image data collection: Prospective predictions are the future. arXiv preprint arXiv:200611988. 2020
• 51 Reps JM, Kim C, Williams RD, Markus AF, Yang C, Salles TD. et al. Can we trust the prediction model? Demonstrating the importance of external validation by investigating the COVID-19 Vulnerability (C-19) Index across an international network of observational healthcare datasets. medRxiv 2020:2020.06.15.20130328 DOI: 10.1101/2020.06.15.20130328.
• 52 Barish M, Bolourani S, Lau LF, Shah S, Zanos TP. External validation demonstrates limited clinical utility of the interpretable mortality prediction model for patients with COVID-19. Nature Machine Intelligence 2020; DOI: 10.1038/s42256-020-00254-2.
• 53 Wynants L, Van Calster B, Collins GS, Riley RD, Heinze G, Schuit E. et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ 2020; 369: m1328 DOI: 10.1136/bmj.m1328.
• 54 Wang L, Schnall J, Small A, Hubbard RA, Moore JH, Damrauer SM. et al. Case contamination in electronic health records based case-control studies. Biometrics 2020; 1-11 DOI: 10.1111/biom.13264.
• 55 Chen Q, Allot A, Lu Z. Keep up with the latest coronavirus research. Nature 2020; 579: 193
• 56 Wiersinga WJ, Rhodes A, Cheng AC, Peacock SJ, Prescott HC. Pathophysiology, Transmission, Diagnosis, and Treatment of Coronavirus Disease 2019 (COVID-19): A Review. JAMA 2020; 324: 782-93 DOI: 10.1001/jama.2020.12839.
• 57 Boutron I, Chaimani A, Meerpohl JJ, Hróbjartsson A, Devane D, Rada G. et al. The COVID-NMA Project: Building an Evidence Ecosystem for the COVID-19 Pandemic. Ann Intern Med. 2020; 173: 1015-7 DOI: 10.7326/m20-5261.
• 58 Rada G, Verdugo-Paiva F, Ávila C, Morel-Marambio M, Bravo-Jeria R, Pesce F. et al. Evidence synthesis relevant to COVID-19: a protocol for multiple systematic reviews and overviews of systematic reviews. Medwave 2020; 20: e7868 DOI: 10.5867/medwave.2020.03.7867.
• 59 Verspoor K, Suster S, Otmakhova Y, Mendis S, Zhai Z, Fang B. et al. Brief Description of COVID-SEE: The Scientific Evidence Explorer for COVID-19 Related Research. 2021; 559-64
• 60 Wang LL, Lo K. Text mining approaches for dealing with the rapidly expanding literature on COVID-19. Briefings in Bioinformatics 2020; DOI: 10.1093/bib/bbaa296.
• 61 Verspoor K, Cohen KB, Dredze M, Ferrara E, May J, Munro R. et al., editors. Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020. ACL2020; 2020: Association for Computational Linguistics
• 62 Verspoor K, Cohen KB, Conway M, de Bruijn B, Dredze M, Mihalcea R. et al., editors Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020. EMNLP2020; 2020: Association for Computational Linguistics
• 63 Wang LL, Lo K, Chandrasekhar Y, Reas R, Yang J, Burdick D. et al. CORD-19: The COVID-19 Open Research Dataset. Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020: Association for Computational Linguistics 2020
• 64 Clark J, Glasziou P, Del Mar C, Bannach-Brown A, Stehlik P, Scott AM. A full systematic review was completed in 2 weeks using automation tools: a case study. Journal of Clinical Epidemiology 2020; 121: 81-90 DOI: 10.1016/j.jclinepi.2020.01.008.
• 65 Chan AKM, Nickson CP, Rudolph JW, Lee A, Joynt GM. Social media for rapid knowledge dissemination: early experience from the COVID-19 pandemic. Anaesthesia 2020; 75: 1579-82 DOI: 10.1111/anae.15057.
• 66 Dorasamy M, Raman M, Kaliannan M. Knowledge management systems in support of disasters management: A two decade review. Technological Forecasting and Social Change 2013; 80: 1834-53 DOI: 10.1016/j.techfore.2012.12.008.
• 67 Richesson RL, Bray BE, Dymek C, Greenes RA, McIntosh LD, Middleton B. et al. Summary of second annual MCBK public meeting: Mobilizing Computable Biomedical Knowledge’A movement to accelerate translation of knowledge into action. Learning Health Systems 2020; 4: e10222 DOI: 10.1002/lrh2.10222.
• 68 Kawamoto K, Hongsermeier T, Wright A, Lewis J, Bell DS, Middleton B. Key principles for a national clinical decision support knowledge sharing framework: synthesis of insights from leading subject matter experts. J Am Med Inform Assoc. 2013; 20: 199-207 DOI: 10.1136/amiajnl-2012-000887.
• 69 Ostaszewski M, Mazein A, Gillespie ME, Kuperstein I, Niarakis A, Hermjakob H. et al. COVID-19 Disease Map, building a computational repository of SARS-CoV-2 virus-host interaction mechanisms. Scientific Data 2020; 7: 136 DOI: 10.1038/s41597-020-0477-8.
• 70 COVID-19 Disease Map. 2020 [17 December 2020]. Available from: 10.17881/covid19-disease-map
• 71 Reese JT, Unni D, Callahan TJ, Cappelletti L, Ravanmehr V, Carbon S. et al. KG-COVID-19: A Framework to Produce Customized Knowledge Graphs for COVID-19 Response. Patterns DOI: 10.1016/j.patter.2020.100155.
• 72 Wang Q, Li M, Wang X, Parulian N, Han G, Ma J. et al. COVID-19 Literature Knowledge Graph Construction and Drug Repurposing Report Generation. arXiv:2007.00576; 2020
• 73 Dunn AG, Bourgeois FT. Is it time for computable evidence synthesis?. Journal of the American Medical Informatics Association. 2020; 27: 972-5 DOI: 10.1093/jamia/ocaa035.
• 74 Alper BS, Richardson JE, Lehmann HP, Subbian V. It is time for computable evidence synthesis: The COVID-19 Knowledge Accelerator initiative. Journal of the American Medical Informatics Association 2020; 27: 1338-9 DOI: 10.1093/jamia/ocaa114.
• 75 Friedman C, Rubin J, Brown J, Buntin M, Corn M, Etheredge L. et al. Toward a science of learning systems: a research agenda for the high-functioning Learning Health System. Journal of the American Medical Informatics Association 2014; 22: 43-50 DOI: 10.1136/amiajnl-2014-002977.
• 76 Surma V, Kudchadkar S, Bembea M, Fackler JC. The Critical Care Learning Healthcare System: Time to Walk the Walk. Crit Care Med. 2020; 48: 1907-9 DOI: 10.1097/ccm.0000000000004700.
• 77 Faden RR, Kass NE, Goodman SN, Pronovost P, Tunis S, Beauchamp TL. An ethics framework for a learning health care system: a departure from traditional research ethics and clinical ethics. Hastings Cent Rep. 2013; Spec No:S16-27. DOI: 10.1002/hast.134.
• 78 Budd J, Miller BS, Manning EM, Lampos V, Zhuang M, Edelstein M. et al. Digital technologies in the public-health response to COVID-19. Nature Medicine 2020; 26: 1183-92 DOI: 10.1038/s41591-020-1011-4.
• 79 Alper B, Mayer M, Shahin K, Richardson J, Schilling L, Tristan M. et al. Achieving evidence interoperability in the computer age: setting evidence on FHIR. BMJ Evidence-Based Medicine 2019; 24: A15-A DOI: 10.1136/bmjebm-2019-EBMLive.28.
• 80 Iglesias N, Juarez JM, Campos M. Comprehensive analysis of rule formalisms to represent clinical guidelines: Selection criteria and case study on antibiotic clinical guidelines. Artif Intell Med. 2020; 103: 101741 DOI: 10.1016/j.artmed.2019.101741.
• 81 Foraker RE, Lai AM, Kannampallil TG, Woeltje KF, Trolard AM, Payne PRO. Transmission dynamics: Data sharing in the COVID-19 era. Learn Health Syst. 2020; e10235 DOI: 10.1002/lrh2.10235.
• 82 Schulz WL, Kvedar JC, Krumholz HM. Agile analytics to support rapid knowledge pipelines. npj Digital Medicine 2020; 3: 108 DOI: 10.1038/s41746-020-00309-z.
• 83 Broadwell MM. Teaching For Learning (XVI.). The Gospel Guardian 1969; 1-3a
• 84 Flynn AJ, Friedman CP, Boisvert P, Landis-Lewis Z, Lagoze C. The Knowledge Object Reference Ontology (KORO): A formalism to support management and sharing of computable biomedical knowledge for learning health systems. Learning Health Systems 2018; 2: e10054 DOI: 10.1002/lrh2.10054.

Publication History

Article published online:
03 September 2021