\documentclass{article}
\usepackage[affil-it]{authblk}
\usepackage{graphicx}
\usepackage[space]{grffile}
\usepackage{latexsym}
\usepackage{textcomp}
\usepackage{longtable}
\usepackage{tabulary}
\usepackage{booktabs,array,multirow}
\usepackage{amsfonts,amsmath,amssymb}
\providecommand\citet{\cite}
\providecommand\citep{\cite}
\providecommand\citealt{\cite}
\usepackage{url}
\usepackage{hyperref}
\hypersetup{colorlinks=false,pdfborder={0 0 0}}
\usepackage{etoolbox}
\makeatletter
\patchcmd\@combinedblfloats{\box\@outputbox}{\unvbox\@outputbox}{}{%
  \errmessage{\noexpand\@combinedblfloats could not be patched}%
}%
\makeatother
% You can conditionalize code for latexml or normal latex using this.
\newif\iflatexml\latexmlfalse
\AtBeginDocument{\DeclareGraphicsExtensions{.pdf,.PDF,.eps,.EPS,.png,.PNG,.tif,.TIF,.jpg,.JPG,.jpeg,.JPEG}}

\usepackage[utf8]{inputenc}
\usepackage[english]{babel}




\begin{document}

\title{Publication bias evaluations are not routinely conducted in clinical oncology systematic reviews}


\author{David Herrmann}
\affil{Affiliation not available}

  
\author{Matt Vassar}
\affil{Affiliation not available}
  
\author{Jonathan Holmes}
\affil{Affiliation not available}
  
\author{Sarah Khan}
\affil{Affiliation not available}
  
\author{Chelsea Koller}
\affil{Affiliation not available}
  



\date{\today}

\maketitle




Research in context

Evidence before this study:
Publication bias has been widely researched with regard to being a threat to the validity of systematic reviews. It is described in depth in the Cochrane handbook and numerous other studies. Onishi et al. found that publication bias was under reported amongst high impact factor journals across all fields of medicine, which raises the question of how it compares to journals within clinical specialties.  

Added value of this study:
Our study gives valuable insight to the extent of which publication bias is assessed by systematic reviews in clinical oncology. It is also one of the first articles of this sample size to show the importance publication bias has on effect sizes for unevaluated systematic reviews.

Implications of all the available evidence:
Our study in combination with previous efforts calls to attention the problem of publication bias in systematic reviews. It also shows deficiencies in searching grey literature and clinical trials registries to discover unpublished studies.
  
  
  
  

\section{Introduction}


Publication bias (PB) can result when the sample of studies that authors consider for systematic review (SR) is not representative of the population of completed studies, with a likely over-representation of published works with statistically significant outcomes. The consequence is an inaccurate, likely exaggerated, estimate of summary effects. Statistically significant results are found with greater frequency than studies containing null or negative outcomes. This occurs for many reasons, including the fact that such studies are more likely to be published sooner, published in high impact factor journals, published in English, and published at all when compared with studies that report non-significant results \cite{11451790}. However, the use of more sophisticated search methods can minimize the degree of PB \cite{Rothstein_2005} \cite{17443631}. In the largest PB study to date, 1,106 meta-analyses from the Cochrane Database of Systematic Reviews found evidence of PB despite the high quality of these reviews. PB had decreased in recent years, suggesting greater effectiveness of measures used to reduce bias \cite{25988604}. Ryder et al. examined the uptake of  methods to reduce PB and how these approaches had improved over time. Examples include the use of MEDLINE, Cochrane Library, CINAHL, and checking references, which all increased from 1996 to 2006. There was also a 20\% increase in searches for non-English language articles and a 26\% increase in the search for unpublished studies from this time period.
  
  
Many approaches have been developed to assess PB. The most commonly used methods include funnel plots, Egger's test, Begg's test, trim-and-fill method, fail-safe N, and modeling \cite{20800992}. These methods take varying approaches. For instance, evaluating the asymmetry of funnel plots provides a clear visual representation of bias when placing effect estimates on the horizontal axis and measure of study size on the vertical. Egger's test is used to quantify the asymmetry of the funnel plots. The trim-and-fill method corrects for funnel plot asymmetry by imputing the necessary studies to achieve a symmetrical plot \cite{10877304}. The fail-safe N method is used to calculate the number of additional "negative" studies that would render the summary effect to a null value, but is not recommended because the estimate of fail-safe N is highly dependent on the mean intervention effect assumed for the unpublished studies \cite{iyengar1988selection}.
  
Publication bias is a potentially serious threat to the validity of SRs \cite{Rothstein_2007}. Inclusion of a PB assessment is an integral part of the SR process and is a requirement of commonly adopted reporting guidelines, such as the Quality of Reporting of Meta-analysis (QUOROM) \cite{10584742}, Meta-analysis of Observational Studies in Epidemiology (MOOSE) \cite{10789670}, and Preferred Reporting Items for Systematic reviews and Meta-analysis (PRISMA) \cite{19621070}. However, despite the acceptance of PB as a vital component of the SR process, it is significantly underreported. A 2007 study of 300 SRs revealed that only 23.1\% reported an evaluation of PB \cite{17388659}. Ryder found that although the number of Cochrane reviews assessing for PB using a funnel plot had increased from 6\% in 1996 to 26\% in 2006, there was still vast room for improvement. More recently, Onishi and Furukawa conducted a study based on the top ten impact factor journals and found that about 69\% did, in fact, examine PB \cite{25194857}. However, this study was based only on articles from high impact factor journals published in 2011 and 2012, and questions remain about the frequency of PB in clinical specialties, like oncology, which may represent different methodological standards. 


For this study, we examined oncology because it is a discipline where PB has not been sufficiently studied. We evaluated how often PB was assessed in clinical oncology, what methods were used, how often PB was found from these assessments, whether authors adhered to reporting guidelines requiring assessment of PB, and if search methods were conducted to limit the likelihood of PB. In addition, we evaluated PB for the SRs that did not perform an assessment themselves to examine the frequency with which SRs with PB went unreported. 

  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  

\section{Methods }


\subsection{Search Criteria}


A PubMed search of six oncology journals from 2007 to 2015 was conducted using the following search string: (((((("Journal of clinical oncology : official journal of the American Society of Clinical Oncology"[Journal] OR "Nature reviews. Cancer"[Journal]) OR "Cancer research"[Journal]) OR "The Lancet. Oncology"[Journal]) OR "Clinical cancer research : an official journal of the American Association for Cancer Research"[Journal]) OR "Cancer cell"[Journal]) AND ("2007/01/01"[PDAT] : "2015/12/31"[PDAT]) AND "humans"[MeSH Terms]) AND (((meta-analysis[Title/Abstract] OR meta-analysis[Publication Type]) OR systematic review[Title/Abstract]) AND ("2007/01/01"[PDAT] : "2015/12/31"[PDAT]) AND "humans"[MeSH Terms]) AND (("2007/01/01"[PDAT] : "2015/12/31"[PDAT]) AND "humans"[MeSH Terms]). This search strategy was adapted from a previously established method that is sensitive to identifying SRs and meta-analyses \cite{15619601}. Journals were selected based on 2014 h5 index scores from Google Scholar metrics in the oncology sub-category. Searches were conducted on May 18 and May 26, 2015.

\subsection{Review Selection and Data Extraction}


We used Covidence (covidence.org) to initially screen articles based on the title and abstract. To qualify as a SR, articles had to summarize evidence across multiple studies and provide information on the search strategy, such as search terms, databases, and inclusion/exclusion criteria. Meta-analyses were classified as quantitative syntheses of results across multiple studies \cite{25194857}. All articles that did not meet the requirements for classification as a SR or meta-analysis were excluded. Two screeners independently categorized all articles for inclusion, and a follow up meeting was held to discuss differences in screening. All disagreements were settled by consensus. Full-text versions of included articles were obtained via EndNote. 
    
To ensure the accuracy of the coding process, an abstraction manual was developed and piloted prior to training coders. A training session was conducted to familiarize coders with the process, and subsets of studies from the screening process were jointly coded as a group. After training, each coder was given three new articles to code independently. These data were analyzed for inter-rater agreement by calculating Cohen's kappa statistic. As inter-rater agreement was high (k = 0.86; agreement = 91 percent), each coder was assigned an equal subset of articles to code. Coded elements for this study included a) the article title, b) authors, c) year published, d) journal name, e) whether PB was discussed, f) whether PB was assessed, g) what method was used for assessment, h) whether a funnel plot was published as a figure in the SR, i) whether PB, if evaluated, was present, j) number of studies, k) which guidelines were followed, l) whether foreign languages were searched, m) whether a hand search of references was performed, n) whether grey literature was searched, and o) whether clinical trials registries were searched. After the initial coding process, validation checks were conducted such that each coded element was verified by the other coder. After these checks were performed, coders met to discuss disagreements and settle them by consensus. Analysis of the final data was conducted using STATA 13.1. Data from this study are publicly available at Figshare (http://dx.doi.org/10.6084/m9.figshare.1493020). 
    
\subsection{Assessment of unreported publication bias}


After initial coding was completed, additional analyses were performed to evaluate PB among those SRs failing to report such an assessment. Comprehensive Meta-Analysis 3 (CMA) was used to perform the analysis. PB was assessed only from SRs that met the following criteria: PB was not originally assessed in the SR, data from greater than 10 primary studies were reported, and data from primary studies were listed in the SR in a usable form. A minimum of 10 studies was selected because with fewer studies the power of the statistical tests is too low to distinguish chance from asymmetry \cite{Higgins_2008}. If SRs contained multiple meta-analyses, then they were individually evaluated. We used the trim-and-fill \cite{10877304} and Egger's regression \cite{9310563} methods because they are among the most commonly reported statistical methods for detection of PB in SRs\cite{20181324}. Trim-and-fill adjusts for the potential effect that non-published studies might have had on the measured outcome \cite{10877304}. Egger's regression determines if PB is statistically significant. For analyses, the point estimates from the data were calculated by CMA and then compared with the point estimates presented in the article to ensure accuracy of our entered data.A fixed or random effects model was selected based in the model used in the original SR, but if the choice of model was not provided, we analyzed the results using both methods and chose the one that produced an aggregate effect size that most closely matched the SR's point estimate. The direction of the bias (left or right), number of studies trimmed, point estimate, confidence intervals, adjusted point estimate, and adjusted confidence intervals were all recorded from the trim-and-fill method for each meta-analysis. The t-value and intercept were recorded from the Egger's regression calculations, and p values were considered significant at p < 0.05.


  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  

\section{Results}


The Pubmed search resulted in 337 articles from four journals. After screening titles and abstracts, 79 were excluded because they were not SRs or meta-analyses. An additional 76 articles were excluded after full text screening. Two articles could not be retrieved after multiple attempts. A total of 182 articles were included in this study (Figure 1).

We found that 40\% of SRs (73/182) discussed PB and about 29\% of SRs (52/182) included an assessment of PB. Across journals, \textit{Clinical Cancer Research} had the largest percentage of SRs that discussed PB (53\%, 9/17) and evaluated (41\%, 7/17). The \textit{Journal of Clinical Oncology} and \textit{The Lancet Oncology} both have similar rates (Figure 2).  Of the SRs that did assess PB most used multiple methods (67\%, 35/52). Funnel plots were the most common assessments (73\%, 38/52), followed by Egger's test (58\%, 30/52), and Begg and Mazumdar's test (42\%, 22/52), trim-and-fill (15\%, 8/52), fail-safe-N (6\%, 3/52), and Gleser and Olkin's method (6\%, 3/52). Of the 52 studies that performed an assessment of PB, 40 found no evidence of PB, 10 did contain PB, and 2 had inconclusive results. From this subset of 52 studies, 10 included a funnel plot as a figure in the article. Adherence to PRISMA guidelines was reported in 38 SRs, MOOSE in 10 reviews, and QUOROM in 10. The percent of SRs that actually followed the "guidelines" when assessing PB was 45\% (17/38) for PRISMA, 70\% (7/10) for MOOSE, and 50\% (5/10) for QUOROM.

We looked for evidence of mitigating PB by examining search strategies of SR authors. Over half (52\%, 94/182) included a hand search of the references of included articles. Conference abstracts were the most common form of grey literature searched (27\%, 49/182) followed by clinical trials registries (8\%, 15/182). The most common registry searched was clinicaltrials.gov (67\%, 10/15) followed by WHO International Trials Registry (13\%, 2/15). Thirty percent (55/182) of articles expanded their search to including articles in foreign languages, 42\% (77/182) did not, and 27\% (50/182) did not mention language in the search strategy.   

Thirteen SRs containing 26 meta-analyses were selected to assess PB (Table 1). These SRs did not report a formal PB evaluation, so we re-analyzed the data from each SR to determine whether PB might be present. We could not analyze Valsecchi (2011) due to incompatible confidence intervals in CMA. Of the remaining 25 meta-analyses, the trim-and-fill method found some evidence for PB in 20 of them (80\%). The number of imputed studies from each meta-analysis ranged from 0 to 7 with a mean of 2.5 (Figure 3). A random effects model was used to calculate the effect size in 48\% (12/25) and a fixed model was used in the remaining 52\% (13/25) of the meta-analyses. Egger's regression found that two meta-analyses (7\%, 2/25) had statistically significant PB at p < 0.05. Eighteen meta-analyses (72\%, 18/25) had some degree of PB according to the trim-and-fill method, but not a statistically significant amount according to Egger's regression. The mean absolute percent difference between adjusted and observed point estimates was 8.4\% and the median was 4.1\%, with a range between 0\% and 44\%.











  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  

\section{Discussion}
 
 
Our results show PB assessment among SRs in oncology journals could be greatly improved. Despite being recognized as an important evaluation, 71\% (130/182) of the articles included in our study did not include an assessment. The 29\% (52/182) of articles that did assess PB was much lower than the 69\% (80/116) found by Onishi in high impact factor journals, but is similar to the 23\% (68/294) that Moher found in a search across all fields in 2004 and the 30\% (91/300) found by Parekh-Burke in different types of SRs in 2006. Our study, along with previous studies, points to the same conclusion that authors of SRs need to assess PB to strengthen the quality of their reviews. Our research calls into question the effectiveness of reporting guidelines. For example, although assessing PB is a principle part of the PRISMA guidelines, over half of the articles that reported use of the guidelines (55\%, 21/38) did not not conduct this evaluation. Discussion and assessment of PB did not vary significantly between oncology journals; however, very few SRs or meta-analyses were found in\textit{ Cancer Cell}, \textit{Cancer Research}, or \textit{Nature Reviews Cancer}. 

The methods used for assessing PB were also similar to Parekh-Burke's study, with the funnel plot being the most commonly used method followed by Egger's regression and Begg's test. Funnel plots were most commonly used in combination with Egger's regression or Begg's test so statistical significance could be determined. 

Of the studies that conducted a PB assessment, only 19\% (10/52) found PB to exist in their study. This is lower than Onishi's finding (26\%, 7/36). One possible reason for the PB found in these SRs is a deficiency in the use of comprehensive search strategies. Only 30\% (55/182) searched foreign languages, 52\% (94/182) hand searched references, 32\% (58/182) searched grey literature, and 8\% (15/182) searched trials registries. These values are comparable with Parekh-Burke's study, which found that 34\% (105/300) of SRs searched for grey literature and 39\% (116/300) searched foreign languages. We recommend that SRs search grey literature and trials registries to find unpublished studies and minimize bias.

We evaluated PB across SRs that failed to conduct this analysis. We found only a few studies with statistically significant PB. Using Egger's test, we found only two meta-analyses that showed evidence for PB (8\%, 2/25) compared with Onishi who found that 19\% (7/36) contained PB. The trim-and-fill method identified a much higher rate of PB (80\%, 20/25). This is because the trim-and fill method corrects funnel plot asymmetry. \cite{Vavken_2011} 

Future research is needed to evaluate PB in other clinical specialties to see how it compares with oncology. Aside from Souza's study in reproductive health \cite{Souza_2007} and Onishi's in general medicine, no studies of significant sample size exist to shed light on this issue. There is also a need to look at how clinical trials registries can be used to find unpublished data for mitigating PB. Sinnett (in press) found that very few SRs had reported searching a clinical trials registry in neurology (citation needed). Jones et al. arrived at this same conclusion from examining high impact factor general medical journals \cite{25348628}. Keil et al. also assessed the use of registry searches in emergency medicine and found discouraging results \cite{25447560}.  

In summary, we found that there exists an underutilization of PB assessments by SRs in clinical oncology. When PB was assessed, the most common method was a funnel plot complemented with Egger's regression or Begg's test. Using these assessments, PB was a routine finding in the SRs. We also found other areas of weakness such as adhering to reported guidelines and searching the grey literature, including trials registries. Finally, we found that the magnitude of effect sizes generally decreased when conducting our independent assessments of PB among SRs in our sample that did not evaluate for it. These results should inform future SR methodologies and create an awareness that PB should be routinely reported among SRs in oncology.\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/Flow-diagram/Flow-diagram}
\caption{{Figure 1. PRISMA flow diagram%
}}
\end{center}
\end{figure}\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/PB-assessment-by-journal/PB-assessment-by-journal}
\caption{{Figure 2. Publication bias discussion and assessment by journal. \textit{Cancer Research} not included because it contained only one systematic review. This one review discussed PB, but did not evaluate it.%
}}
\end{center}
\end{figure}\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/CMA-Results/CMA-Results}
\caption{{Figure 3. Comprehensive Meta-Analysis results of studies trimmed by trim-and-fill method shown on graph. Results of Egger's regression test denoted by coloring.%
}}
\end{center}
\end{figure}\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/CMA-table/CMA-table}
\caption{{Table 1. Publication bias results from Comprehensive Meta-Analysis%
}}
\end{center}
\end{figure}

\selectlanguage{english}
\FloatBarrier
\bibliographystyle{plain}
\bibliography{bibliography/converted_to_latex.bib%
}

\end{CJK}\end{document}

