Introduction: Co-production of evidence syntheses has the potential to facilitate translation of research findings into policy and practice. However, few studies have explored the process of implementing co-produced evidence. This gap limits our understanding of how, and to what extent, co-production promotes knowledge translation. In this study, we used an implementation science lens to explore the factors that would motivate and shape interest-holders (including policymakers, health service representatives and consumers) to participate in implementing the Best Practice Principles in partnering with consumers (BPP) in hospitals in Melbourne, Australia. The BPP were developed as part of a co-produced Cochrane qualitative evidence synthesis exploring consumers’ and health providers’ experiences and perceptions of partnering. Methods: This exploratory, qualitative study was informed by cooperative inquiry and normalisation process theory (NPT). A six-member panel, including researchers, policy makers and consumers, guided data collection and analysis. Data collection involved semi-structured interviews with eleven participants (including consumer engagement leads, consumer representatives and a policymaker) about how to implement the BPP in hospitals in Melbourne, Australia. Interviews were analysed using framework analysis. Results: Interview participants reported the BPP were relevant to practice, consumer-centred, practical and flexible. There were several additional factors that could impact their uptake into practice. These included integration of the BPP into government policies and guidelines, evidence of the cost/benefit of BPP implementation, endorsement from health service leadership, involvement of consumers throughout the implementation process, a structured implementation and flexible measurement of implementation success. Conclusion: This exploratory study suggested that the BPP, a tool developed through a co-produced Cochrane qualitative evidence synthesis, promoted knowledge translation. Other factors at the macro- (political and economic), meso- (systems and organisations) and micro- (individual) levels could influence the implementation’s success. Implications for Cochrane authors aiming to optimise the knowledge translation of their review results are discussed.

A document by Ryan PW Kenny. Click on the document to view its contents.

A document by Shaun Treweek. Click on the document to view its contents.

Introduction Priority screening has the potential to reduce the number of records that need to be annotated in systematic literature reviews. So‐called technology‐assisted reviews (TAR) use machine‐learning with prior include/exclude annotations to continuously rank unseen records by their predicted rele‐ vance to find relevant records earlier. In this article, we present a systematic evaluation of methods to determine when it is safe to stop screening when using prioritisation. Methods We implement an open‐source evaluation framework that features a novel method to generate rankings and simulate priority screening processes for 86 real‐world datasets. We use these sim‐ ulations to evaluate 12 statistical or rule‐based (heuristic) stopping methods, testing a range of hyper‐parameters for each. Results The work‐saving potential and performance of stopping criteria heavily rely on ‘good’ rankings, which is typically not achieved by a single ranking algorithm across the entire screening process. Our evaluation shows that the existing stopping methods either fail to reliably stop without missing relevant records or fail to utilise the full potential work‐savings. Only one method reliably met the set recall target but stops conservatively. Conclusions Many digital evidence synthesis tools provide priority screening features that are already used in many research projects. However, the theoretical work‐savings demonstrated in retrospective sim‐ ulations of prioritisation can only be unlocked with safe and reproducible stopping criteria. Our results highlight the need for improved stopping methods and guidelines on how to responsibly use priority screening. We also urge screening platforms to provide indicators and authors to transpar‐ently report metrics when automating (parts of) their synthesis.

A document by Jennifer Petkovic. Click on the document to view its contents.

A document by Kevin M. Kallmes. Click on the document to view its contents.

IntroductionNovel automation or Artificial Intelligence (AI) applications for information retrieval (AI search tools) have the potential to expedite evidence synthesis tasks, such as study identification or search strategy development.1,2 These AI search tools, including generative AI Large Language Model (LLM) chatbots, require performance testing and validation to inform implementation recommendations for Canada’s Drug Agency (CDA-AMC) and other evidence synthesis producers.There is a long history of automation in the information sciences3 and we recognize AI as an umbrella term for technology tools that perform tasks that would ordinarily require biological brainpower to accomplish.4 Our definition of AI search tools is deliberately broad to include successive generations of technologies for information retrieval and to recognize the potential value of older, developing, emerging, or novel tools.In an earlier project phase, Research Information Services (RIS) at CDA-AMC evaluated 51 AI search tools using our evaluation instrument.5 In this successive phase, RIS evaluated the performance of 3 top-ranked tools:Lens.org (“The Lens”) https://www.lens.org,SpiderCite https://sr-accelerator.com/\#/spiderciteMicrosoft Copilot https://www.microsoft.com/en-ca/microsoft-copilotOur objective in testing these tools was to determine if they should replace or supplement current RIS information retrieval methods given the estimated contribution of eligible and unique studies, and the resources needed to conduct searches and screen the results. As these tools use different automation technologies to perform different search tasks, our investigation also aimed to compare their distinct strengths and weaknesses.

Abstract Introduction: Information retrieval is essential for evidence synthesis in health and care research, but developing search strategies is labour-intensive and time-consuming. Automating these processes is highly desirable, though it’s unclear if Information Specialists (IS) are willing to adopt artificial intelligence (AI) methodologies or how they currently use them. In January 2025, the NIHR Innovation Observatory and NIHR Methodology Incubator co-sponsored the inaugural CORE Information Retrieval Forum, where attendees discussed AI’s role in information retrieval. Methods: The CORE Information Retrieval Forum hosted a Knowledge Café, titled ”Building Our Community and Expanding Our Horizons”. Participation was voluntary and attendees could choose one of six event themed discussion tables including AI. To support each discussion, a QR code linking to a virtual collaboration tool (Padlet; padlet.com) and a poster in the exhibition space were available throughout the day for attendees to contribute to. Results: The CORE Information Retrieval Forum was attended by 131 IS from nine different types of organisations and ten countries. Among the six discussion points available in the Knowledge Café, the AI table was the most popular, receiving the highest number of contributions (n=49). Following the Forum, contributions to the AI topic were categorized into four themes: Critical Perception (n=21), Current Uses (n=19), Training Wants/Needs (n=7), and Specific Tools (n=2). Conclusions: While there are critical perspectives on the integration of AI in the IS space, this is not due to a reluctance to adapt and adopt but from a need for structure, education, training, ethical guidance, and systems to support the responsible use and transparency of AI. There is significant interest in automating repetitive and time-consuming tasks, but a lack of tools currently used by attendees. More work needs to be done to identify currently available tools and their potential to complement the work conducted by IS.

A document by Kristen L. Scotti. Click on the document to view its contents.

Background: Elicit AI aims to simplify and accelerate the systematic review process without compromising accuracy. However, research on Elicit’s performance is limited. Objectives: To determine whether Elicit AI is a viable tool for systematic literature searches. Methods: We compared the included studies in four systematic reviews to those identified searching with Elicit. We calculated sensitivity, precision and observed patterns in Elicit’s performance. Results: Elicit had an average of 39.6% precision (26.7% - 46.2%) which was higher than the 7.55% average of the original reviews (0.65% - 14.7%). However, the sensitivity of Elicit was poor, averaging 37.9% (25.5% - 69.2%) compared to 93.5% (87.2% - 98.0%) in the original reviews. Elicit also identified some included studies not identified by the original searches. Discussion: At the time of this evaluation, Elicit did not search with high enough sensitivity to replace traditional literature searching. However, the high precision of searching in Elicit could prove useful for preliminary searches, and the unique studies identified mean that Elicit can be used by researchers as a useful adjunct. Conclusion: Whilst Elicit searches are currently not sensitive enough to replace traditional searching, Elicit is continually improving, and further evaluations should be undertaken as new developments take place.

Title:

A document by Petek Eylul Taneri. Click on the document to view its contents.

\receivedDD MMMM YYYY \acceptedDD MMMM YYYY

A document by Bronwen Merner. Click on the document to view its contents.

Background Over ten years since the first qualitative evidence synthesis (QES) was published in the Cochrane Library, QES and mixed-methods reviews (MMR) with a qualitative component have become increasingly common and influential in healthcare research and policy development. The quality of such reviews and the completeness with which they are reported is therefore of paramount importance. Aim This review aimed to assess the reporting quality of published QESs and MMRs with a qualitative component in the Cochrane Library. Methods All published QESs and MMRs were identified from the Cochrane Library. A bespoke framework developed by key international experts based on the Effective Practice and Organisation of Care (EPOC), Enhancing Transparency in Reporting the Synthesis of Qualitative Research (ENTREQ) and meta-ethnography reporting guidance (eMERGe) was used to code the quality of reporting of QESs and MMRs. Results Thirty-one reviews were identified, including 11 MMRs. The reporting quality of the QESs and MMRs published by Cochrane varied considerably. Based on the criteria within our framework, just over a quarter (8, 26%) of the reviews achieved a score of at least 80%, 18 (58%) required fuller detail in their reporting (scoring between 65%-79%) and 5 (16%) achieved a score of less than 65%. Conclusion This assessment offers important insights into the reporting practices prevalent in these review types and underscores the need for ongoing surveillance. The variability in reporting quality within QESs and MMRs reinforces the need to develop Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) specifically for QES.

Background Living systematic reviews (LSRs) should provide up-to-date evidence for priority questions where the evidence maybe uncertain and fast-moving. LSRs featured prominently during COVID-19 and formed part of Cochrane’s response to the pandemic. We conducted a mixed-methods study to describe the characteristics of Cochrane’s COVID-19 living reviews, determine the currency of the included evidence, and evaluate authors’ experiences in conducting and publishing these reviews. Methods We identified living reviews of COVID-19 from the Cochrane Database of Systematic Reviews and extracted data on the number of versions published and publication timelines. We assessed the currency of evidence by comparing studies included in the reviews against a comprehensive list of studies maintained for the Australian living guidelines for COVID-19. The qualitative component involved semi-structured interviews with review authors to identify the barriers and enablers to conducting, reporting and publishing living reviews. Findings Cochrane published 25 COVID-19 living systematic reviews. Half of these reviews had not been updated when assessed in June 2023 and only four had been updated more than once. A total of 118 studies were included in the living reviews. We estimated that an additional 119 studies were available and potentially relevant for inclusion. Interviews with six authors indicated that publication timelines were reduced by editorial delays, loss of funding, waning commitment, and the burden of screening search results. An inability to communicate the living status of reviews in the Cochrane Library was a common frustration for many authors. Although authors felt the conclusions of their reviews were still current, only one living review communicated its update status and made new evidence accessible after the review was published. Conclusions Maintaining and communicating the currency of Cochrane’s COVID-19 living systematic reviews was not feasible for many author teams because of author-side, editorial and platform barriers.

Empowering the Future of Evidence-Based Healthcare: The Cochrane Early Career Professionals Network

BACKGROUND Historically, peer reviewing has focused on the importance of research questions/hypotheses, appropriateness of research methods, risk of bias, and quality of writing. Until recently, the issues related to trustworthiness - including but not limited to plagiarism and fraud - have been largely neglected because of lack of awareness and lack of adequate tools/training. We set out to identify all relevant papers that have tackled the issue of trustworthiness assessment to identify key domains that have been suggested as an integral part of any such assessment. METHODS We searched the literature for publications of tools, checklists or methods used or proposed for the assessment of trustworthiness of randomised trials. Data items (questions) were extracted from the included publications and transcribed on Excel including the domain of assessment as described in the original publication. Both authors then independently assessed each data item to see if the original domain(s) could be re-categorised in 5 domains (governance, plausibility, plagiarism, reporting, statistics). RESULTS From the 41 publications we extracted a total of 284 questions and framed 77 summary questions grouped in 5 domains: governance (13 questions); plausibility (16 questions); plagiarism (4 questions), reporting (28 questions and statistics (16 questions). CONCLUSION The proposed menu of domains and questions should encourage peer reviewers, editors, systematic reviewers and developers of guidelines to engage in a more formal trustworthiness assessment. Methodologists should aim to to identify the domains and questions that should be considered mandatory, those that are optional depending on the resources available, and those that could be discarded because of lack of discriminatory power.