Synopsis  

Documenting and improving quality has become a hot topic in the world of endoscopy. Key issues are what to measure, how to do it, and why.

 Introduction   

No one—practitioner, payer, or patient—can doubt the importance of ensuring that interventional procedures are performed well. Professional organizations representing endoscopy around the world have published numerous recommendations and guidelines on various aspects of this quality agenda [1,2]. These include methods for training, evaluation of competence, principles for the granting of privileges to perform procedures, and methods for recredentialing (revalidation). The mirror of excellence is accountability. There is increasing interest in defining the metrics and mechanisms for documenting levels of endoscopic performance, which will allow appropriate comparisons and benchmarking.

The proliferation of thoughtful publications on this subject has had relatively little effect so far in the complex and diffuse world of endoscopic practice. Guidelines are not mandatory, and peer pressure has limited influence. Training program directors have little objective data with which to judge the competence of their graduates, and credentialing bodies have even less. In addition, there is (as yet) no consensus on what level of performance is really acceptable. In the United States, many of the routine procedures are performed by doctors (general internists and surgeons) who have not undergone formal gastroenterology training. These endoscopists may see the recommendations of the gastroenterology and endoscopy societies as being too stringent and also self-serving. Finally, most routine procedures are performed in offices and free-standing endoscopy centers and are not subject to the (albeit flawed) credentialing process in hospitals.

 Achieving competence—the goal of training   

Endoscopists can be categorized broadly into four levels of performance—from beginner/learner through states of competence, proficiency, to excellence (Fig. 1). Competence defines the level of performance at which independent practice is justified, i.e. the completion of initial training. The key issues are:

• What needs to be learned to become competent?
  
• What level of performance is needed for 'competence'?
  
• Who makes that determination?
  
• How is competence assessed and documented?
  

The appropriate application of endoscopy requires cognitive knowledge, technical skills, and some humanity. Until recently, it was assumed that a standard period of gastroenterology training (e.g. GI Fellowship in the United States, or a Specialist Registrar position in the UK) would get trainee endoscopists to a reasonable level of competence. However, this is clearly not always the case, even for standard procedures such as colonoscopy. The situation for advanced procedures (e.g. ERCP and EUS) is even more unsatisfactory.

 What experience is necessary in training? The fallacy of numbers   

In the late 1980s, professional societies began to recommend minimum numbers of cases during training. Initially, the American Society for Gastrointestinal Endoscopy (ASGE) suggested that endoscopists might be competent after approximately 100 upper endoscopies, 100 colonoscopies, and 100 ERCPs (provided that these included the relevant common therapeutic applications) [1]. However, these numbers were not based on data, and there was never a clear definition of what should 'count' as a case; how much of the procedure must the trainee do? The reliance on numbers was softened somewhat by the inclusion of the word 'threshold' in later ASGE recommendations. This was intended to indicate that trainees could ask their Training Program Directors whether they were indeed competent only after they reached that threshold number of cases. However, this subtlety was widely ignored. More important, several studies showed clearly that these numbers were too low. This was brought into bright focus by the seminal study at Duke University which showed that ERCP trainees were only barely reaching 80% competence after participating in 180–200 procedures [3]. This study and others led to an escalation of the numbers recommended by the ASGE (EGD 130, colon 140, ERCP 200), and by other professional organizations [1]. The Australian Gastroenterology Society proposed the most rigorous test for ERCP, indicating that trainees need to have completed 200 ERCP procedures, unassisted. The fact that professional societies representing surgeons and internists initially recommended much lower numbers [1] was confusing and contentious.

Some of these issues of competence and numbers are amplified in the chapter on 'Training and credentialing in gastrointestinal endoscopy' by Jonathan Cohen (http://www.gastrohep.com/ebooks/ebook.asp?book=1405120762&id=11).

It is now obvious that mere numbers are poor markers for expertise. Would we submit ourselves to a procedure by one of our graduates if the sole determinant of competence was the number of procedures that he/she had experienced (or just observed)? We need meaningful objective outcome data.

 Beyond numbers: tools to measure competence   

In order to measure endoscopic performance, it is necessary to dissect out the essential elements of the procedures, and then to develop a list of the essential 'competencies', covering the domains of knowledge, technical skill, and attitude. Some competencies can be assessed by standard written tests, others by documenting the achievement of technical milestones. Computer simulation may well become an important part of the assessment process. Evaluating attitudes, appropriate application of knowledge (judgement), and technical skills require proctoring of real cases, so-called DOPS (direct observation of procedural skills). The Joint Advisory Group (JAG) in Britain, which represents all parties interested in endoscopy, has recently defined processes for assessing competence of trainees [4]. This group is also actively pursuing a competence test, at least for colonoscopy, for established endoscopists who have not been through a competency test as a trainee. This 'driving test' has been stimulated by the upcoming introduction of a national bowel cancer screening program.

 What level of competence is good enough? How is it recognized?   

When we have the tools to measure performance, we then will have to decide on the 'pass mark' for competence, and to agree who is in a position to make that determination. Should it be the professional organizations (who may be seen as self-serving, and often fail to find a consensus), the payers (who come in many guises), or the consumers? All have different agendas.

Another issue is how endoscopic competence should be certified. An endoscopy diploma (procedure specific) would seem to be the obvious answer, but there is no move towards that concept in the United States. The JAG issues certificates of competence in Britain that are recognized by physicians and nurses. To date the surgical colleges in Britain have not acknowledged the legitimacy of JAG certification.

 Endoscopic performance beyond training   

Once a trainee is deemed 'competent', a process is required to ensure that acceptable performance is maintained. Hopefully, skills will increase progressively in practice. Competent endoscopists should become proficient, and some will become true experts (Fig. 1). Performance is influenced by the extent and quality of prior training, by case volume, by the availability of mentoring by senior colleagues, and by continuing education activities. An important but unanswered question is how many procedures need to be done on an annual basis to ensure continuing adequate performance (let alone to enhance it). Numbers in this context are important not only to maintain expertise but also to provide sample sizes sufficient to detect areas of concern within a reasonable time frame. For example, adverse endoscopic event rates are so low that it could take many years for variance to be recognized and remedial action taken.

Making individual endoscopists aware of their skill levels compared to their colleagues is a strong stimulus to improve performance [5]. This 'benchmarking' can be done only if there is a continuous and reliable measurement process in place, and a central system for analysis and feedback.

 Issues in measuring endoscopic performance   

Most published data on the outcomes of endoscopy come from expert centers, or from motivated collaborative groups [1,2,6], and concentrate on technical 'success rates' and complications. For colonoscopy, attention has been focused mainly on cecal intubation rates. Studies from experts and major multicenter research studies suggest that completion rates often exceed 95% [7,8]. However, the non-experts rarely publish their data, so that it is difficult to know much about overall standards in the community. A database audit of almost 20 000 colonoscopies from seven hospitals in the United States showed that only 54% of 108 endoscopists reached the cecum in more than 90% of cases [9], and even lower rates have been reported [10]. However, it is obvious that the goal of colonoscopy is not merely to reach the cecum, but rather to examine the whole mucosa and to detect/manage all lesions with acceptable levels of comfort and safety. A good example of more meaningful quality data comes from a community study which correlated polyp detection rates with withdrawal durations [11]. Recent comparative studies with CT colonography have shown that colonoscopy is not 'as good as gold'[12]. There are very few data on the complications of colonoscopy in community practice, and even fewer of patient acceptability. Two British studies, one of unselected practice and the other of a screening pilot, showed wide variation in perforation rates of from 1 : 769 to 1 : 4000 [5,13]. Several more widespread audits are underway in Britain (http://www.healthcarecommission.org.uk).

These general surveys are of interest, but we need to embrace a new paradigm—the collection of performance data for and by individual endoscopists.

 The report card agenda   

The American Society for Gastrointestinal Endoscopy (ASGE) recommended in 2000 that all endoscopists should keep track of their practice [14], but there is little evidence that many have done so. A key issue is what parameters to measure. The most comprehensive review of endoscopy performance metrics was produced by a joint working party of the ASGE and ACG [7]. An attempt to provide slightly simpler metrics was published recently [15], and other working parties are in progress. Some data points are obvious (e.g. annual procedure numbers, case mix, certain technical endpoints), and can be incorporated easily into a report card (Fig. 2). These should be available to any interested parties, whether payers, credentialing bodies, patients, and even lawyers [16].

Skeptics of this approach put forward several arguments. They are concerned about the quality of the data, which may not be verifiable independently, and that the need to report outcomes may stimulate interventionists to avoid the most difficult and risky cases. Perhaps the most powerful point is that there is little evidence that their use in other fields (for example in cardiac surgery) has yet influenced the choices of patients or indeed payers. The final problem is that this exercise will be time consuming and expensive. Some of these concerns will fade as all endoscopy reporting becomes electronic. The performance data can be generated automatically at the time of the procedure, and easily uploaded into systems for benchmarking. Furthermore, such systems allow adjustment for case mix.

The argument for collecting and sharing data remains strong. If we do not collect the data, others will do it for us, and we will have little control over its relevance or quality. Secondly, documentation of our experience will provide some legal protection to individual endoscopists, and indeed, to those who credential them. Perhaps the most persuasive argument is that endoscopists with report cards will have a practice advantage in the future. Patients (and payers) will increasingly ask their providers for objective data on their expertise, and may well go elsewhere if it is not forthcoming or reassuring. Furthermore, there is a strong tide running in the United States for 'pay for performance', in other words tying reimbursement to outcomes. It really is time for action to supersede discussion. Professionals proud of their practice should set an example and start documenting their practice and outcomes. We should wear our data as badges of quality.

 Benchmarking   

If enough endoscopists collect their data, and agree to share them (albeit anonymously) in a voluntary 'quality network', it will be possible to compare practice and outcomes, and to develop benchmarks of performance. When such data are fed back to participants, they act as a powerful incentive for self-improvement [5]. Benchmarks can help to define and to raise standards. An important question is who will pay for the necessary infrastructure.

 The quality of endoscopy units   

The quality of an endoscopy experience depends on the environment in which it is performed, as well as on the individual practitioner, in the same way that the experience in a restaurant depends on more than the individual chef (or server). Thus it is logical to extend the concept of report cards to document the structure and function of endoscopy units. This concept has been embraced in the United Kingdom with the development of a 'global rating scale (GRS)' for endoscopy units (http://www.grs.nhs.uk). Initially this was simply a self-administered questionnaire on aspects of practice and process. It is now a web-based reporting tool backed up with specific measures of processes, data, and performance. The GRS has been accepted as the service standard for accreditation of endoscopy units participating in the national bowel cancer screening programme (http://www.bcsp.nhs.uk). Furthermore, it has been underpinned with a knowledge management system to support quality improvement.

 Conclusion   

Everyone is (or should be) interested in improving the performance of endoscopy, and its documentation. We are convinced that this agenda can be advanced effectively and easily by the widespread acceptance of ongoing collection and sharing of quality data, for example, the use of report cards for endoscopists, and for endoscopy units. In most environments this will be a voluntary exercise initially, but there will be increasing peer pressure to participate. Aggregation of the data in a voluntary 'quality network' will allow benchmarking, and stimulate improvement. It is the right thing to do.

 Outstanding issues and future trends   

Numerous professional bodies and groups in several countries are now addressing the key issues of endoscopy performance measurement and enhancement. Soon there will be a consensus on the basic metrics, and methods for collecting and sharing the data, a process which will be greatly facilitated by electronic reporting. The resistance to 'report cards' will be overcome as their practice advantages are better appreciated, and as our patients learn to ask for the data. It remains to be seen who will provide the infrastructure needed to support this agenda, and how exactly the data will impact the quality of endoscopy in different practice environments.

 References  

 1 Cohen, J. (2004). Endoscopic training and credentialing. In: Advanced endoscopy, e-book/annual (ed. Cotton PB), http://www.gastrohep.com

 2 Freeman, ML. Training and competence in gastrointestinal endoscopy. Rev Gastroenterol Disord 2001; 1: 73–86. PubMed

 3 Jowell, PS, Baillie, J, Branch, MS, Affronti, MD, Browning, CL & Bute, BP. Quantitative assessment of procedural competence: a prospective study of training in endoscopic retrograde cholangiopancreatography. Ann Intern Med 1996; 125: 983–8. PubMed

 4 http://www.thejag.org.uk

 5 Ball, JE, Osbourne, J, Jowett, S, Pellen, M & Welfare, MR. Quality improvement program to achieve acceptable colonoscopy completion rates: prospective before and after study. BMJ 2004; 329: 665–7. PubMed

 6 Cotton, PB & Leung, JWC (2005). Advanced digestive endoscopy: ERCP. Blackwell Publishing, Oxford.

 7 Rex, DK, Bond, JH, Winawer, S, Levin, TR, Burt, RW & Johnson, DA et al. Quality in the technical performance of colonoscopy and the continuous quality improvement process for colonoscopy: recommendations of the U.S. Multi-Society Task Force on Colorectal Cancer. Am J Gastroenterol 2002; 97: 1296–305. PubMed CrossRef

 8 Cotton, PB, Durkalski, VL, Pineau, BC, Palesch, Y, Mauldin, PD & Hoffman, B et al. Computer tomographic colonography (virtual colonoscopy): a multicenter comparison with standard colonoscopy for detection of colorectal neoplasia lesions. JAMA 2004; 291: 1713–19. PubMed

 9 Cotton, PB, Connor, P, McGee, D, Jowell, P, Nickl, N & Schutz, S et al. Colonoscopy: practice variation among 69 hospital-based endoscopists. Gastrointest Endosc 2003; 57: 352–7. PubMed

10 Bowles, CJ, Leicester, R, Romaya, C, Swarbrick, E, Williams, CB & Epstein, O. A prospective study of colonoscopy practice in the UK today: are we adequately prepared for national colorectal cancer screening tomorrow? Gut 2004; 53: 277–83. PubMed

11 Barclay, RI, Vicari, JJ, Johanson, JF & Greenlaw, RI. Variation in adenoma detection rates and colonoscopic withdrawal times during screening colonoscopy. Gastrointest Endosc 2005; 6: AB107.

12 Lieberman, D. Colonoscopy: as good as gold? Ann Int Med 2004; 141: 401–3. PubMed

13 UK Colorectal Cancer Screening Pilot Group. Results of the first round of a demonstration pilot of screening for colorectal cancer in the United Kingdom. BMJ 2004; 329: 133.

14 ASGE. Quality and outcomes assessment in gastrointestinal endoscopy. Gastrointest Endosc 2000; 52: 827–30.

15 Cotton, PB, Hawes, RH, Barkun, A, Ginsberg, GG, Amman, S & Cohen, J et al. Excellence in endoscopy: toward practical metrics. Gastrointest Endosc 2006; 63: 286–91. PubMed

16 Cotton, PB. How many times have you done this procedure, doctor? Am J Gastrenterol 2002; 97: 522–3.

Copyright © Blackwell Publishing, 2006