Discussion
The key results are: (1) there are differences in consultant neurologist diagnoses and investigative approach in the general neurology outpatient setting; (2) whilst differences in diagnosis are generally minor, a significant proportion have potentially important management implications; (3) greater diagnostic certainty is associated with higher diagnostic agreement and a lower likelihood of arranging investigations; and (4) individualized metrics can be produced representing different facets of clinical practice. Consideration could be given to developing this system of peer comparison as a tool to facilitate reflective practice at appraisal.
The case-mix encountered in this study by a newly qualified consultant neurologist, working in a district general hospital, appears representative of typical outpatient practice and current UK curriculum content, with headache, sensorimotor disorders, blackouts and tremor common problems. This differentiates our study from most others published to date, in which unselected cases have been studied infrequently. Previous literature on diagnostic consensus and investigative approach has focused on highly specific issues. One study1 assessed classification of dysarthria by 72 neurologists from patients’ speech recordings and found low agreement (Cohen’s kappa 0.16-0.32). However, a different study6on clinical assessment of stroke found good inter-observer agreement in diagnosis (91/98 cases, Cohen’s kappa=0.77). These conflicting data suggest that disease group may be an important factor, and this was supported by our study, in which balance disorders exhibited lowest diagnostic agreement, followed by headache, sensory/motor disorders, disorders of consciousness and tremor. Possible reasons for varying inter-rater agreement between disease groups include greater heterogeneity of disorders, fewer formalized diagnostic protocols, or more overlap between different disorders with similar phenotypes. For example, overlap in the clinical features of migraine and tension headaches exists, with 58.4% of definite migraine patients having tension-type symptoms and 68.1% of tension-headache patients reporting migraine-type symptoms in one study7. Furthermore, in this context, the primary aim of a consultant neurologist may be to rule out secondary headaches. Despite not always agreeing on the primary headache disorder, there were only three cases in which one consultant suspected a secondary headache disorder in contradiction to the others and none were found when investigations were performed, implying agreement in decisions with implications for patient safety. Previous research has also highlighted the difficulty in diagnosing balance disorders; the sensation can be difficult to define, with numerous possible causes, overlapping phenotypes and frequently inconclusive investigations8. Previous studies9have found that a firm diagnosis may not be attainable in up to 20% of cases.
Relationships between diagnostic confidence and experience have been investigated in previous studies. In medically unexplained symptoms, greater clinician experience was associated with increased confidence in diagnosis amongst a mixed group of neurologists, cardiologists, gastroenterologists and rheumatologists4. A comparison of US and UK neurologists10 using a similar case-based methodology to our study, found that more experienced neurologists ordered fewer investigations. This study had a far greater number of participants (705), but far fewer number of cases (3) than ours, so the two studies provide complementary data on this issue. In contrast, the GALATEA trial, an observational study into practice variability in Alzheimer’s and cerebrovascular disease, found no relationship between experience and investigative decisions across 107 physicians2. Whilst we lacked statistical power to investigate relationships between experience and diagnostic confidence in our study, we were able to evaluate associations between diagnostic certainty, diagnostic peer-agreement and investigative approach, which are novel data, and we consider may have value as an appraisal tool. All UK clinicians undertake annual review and 5-yearly revalidation procedures to monitor and maintain competencies. Individualized metrics allow comparison of diagnostic consensus with peers, and could capture more complex aspects of clinical practice. For example, the strength of positive association between diagnostic certainty and diagnostic agreement may reflect aspects of diagnostic judgement, and associations between diagnostic certainty and investigative approach may indicate an individual’s reliance on their clinical judgement, approach to resource usage and management of uncertainty. Our data are supported by a previous qualitative study11 of general practitioners that found diagnostic uncertainty to be a positive determinant for ordering tests in 22 physicians surveyed. Neurologists in our study were sometimes uncertain. There is literature on teaching medical students the management of uncertainty12 in an age when perhaps patients, and potentially doctors too, may trust in technology more than clinical judgement. Diagnostic uncertainty can result in over-investigation13, which carries a risk of incidental findings that can be detrimental. Management of uncertainty may become especially important at transition to consultant level, with increased responsibility, and has not traditionally been part of medical school curricula. Further research into the influence of uncertainty on clinicians’ decision-making may help develop tools to facilitate and support recognition, management, discussion and documentation of this normal aspect of clinical practice. This may be especially important in the current era of Covid-19, when face-to-face contacts and access to investigations are limited, potentially increasing diagnostic uncertainty.
A limitation of our study was that follow-up data, to confirm veracity of the diagnoses made, was often not available, and we recognize that diagnostic peer-agreement is not synonymous with correct diagnosis. However, this limitation accurately reflects the real-world general neurology clinic setting in which the study was conducted; many diagnoses in neurology are clinical, follow-up and investigations are not always necessary or conclusive and, conversely, a strength of our study is the external validity of investigating unselected consecutive cases. A further limitation is that, whilst our methodology was designed to simulate real-life practice, the raters not present in clinic with the patients did not have the benefit of taking the history and performing neurological examination themselves, and may have picked up additional signs, or non-verbal cues and clues, which can be important for diagnosis14. This potential bias between theoretical and actual practice could be investigated by two or more raters assessing the same patients and performing a reciprocal analysis, with the factor of interest management differences between consultants that met the patient or theoretically answered the questionnaire.
Despite these limitations, we propose that this methodology has potential as an appraisal tool, with the advantages of producing quantitative data derived from a directly relevant clinical setting and enabling anonymized peer comparison of complex aspects of practice. Lack of standardized assessments15, time burden16 and subjectivity of current assessment15 have been identified as problems with current appraisal tools. Only 43% of physicians reported changing their practice in response to their appraisal in one study17, with another finding that 43% of physicians felt that patient safety had not been improved by revalidation18. New approaches based on real-world practice might help address some of these areas of need, although would have to be adapted to be feasible in a non-research setting. Collating larger datasets on diagnostic and investigative approaches may also help plan neurological services by helping direct resources appropriately and enabling feedback loops to be developed between resource providers and clinical practitioners.
In summary, there is variability in consultant neurologist practice, which may influence management and can be captured by metrics reflecting individual approaches. With refinement and consideration of the limitations listed above, the methodology applied in this study may be developed into new tools to facilitate reflection and appraisal, and maintain the current high standards of neurological practice.