|
|
|
|
|
|
||
|
|
||||
|
|
||||
|
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Pediatric Neuro-Oncology |
Royal Marsden Hospital, Sutton, Surrey, SM2 5PT, UK (D.R.H., U.A.H); and Paediatric Brain Tumour Program, Hospital for Sick Children, Toronto, ON M5G 1X8, Canada (E.B.)
1 Address correspondence to Darren R. Hargrave, Paediatric Oncology Unit, Royal Marsden Hospital, Downs Road, Sutton, Surrey SM2 5PT, U.K. (darren.hargrave{at}rmh.nhs.uk).
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionReferences |
|---|
Key Words: health information • Internet • neuro-oncology • pediatrics • readability
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Although a few studies have attempted to objectively rate the quality of health information in oncology (Bichakjian et al., 2002; Biermann et al., 1999), no studies have looked at Web-based health information for children or the field of neuro-oncology. Therefore, we critically evaluated health information on the Internet, from a patient or family perspective, by examining the content and readability of pediatric neuro-oncology information.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Only the first 30 results from each search engine were used for initial evaluation, as previous searches had found that results lower down the relevancy lists were often duplications of earlier results. It was also presumed that lay searchers would be unlikely to scroll through pages and pages of results. The first 30 listed Web sites on each search engine were then visited and initially assessed for suitability to be rated. Web sites did not necessarily provide evaluable information, as the search term was sometimes found in isolation on a Web page, for example, in an online dictionary or a pathology slide of a tumor.
Web sites were broadly categorized as institutional (e.g., government, hospital, or university), commercial (e.g., sponsored site or private medical site), charitable (e.g., recognized cancer society), support (e.g., personal Web page or patient support group), or alternative medicine (e.g., nonorthodox medicine). The languages available in addition to English on individual Web sites were recorded.
|
Each site was rated independently by the authors, two of whom (D.R.H. and E.B.) are pediatric neuro-oncologists, the third (U.A.H.) being a lay searcher who had no prior knowledge of the subject area and was a competent but nonexpert Internet user. An average of the combined scores was used in the final assessment.
The DISCERN tool does not provide a mechanism for assessing in detail the accuracy of the information provided. Therefore, we adapted the Checklist Rating System Instrument, as described by Bichakjian et al. (2002), to record key neuro-oncology information and record its accuracy for the content topics shown in Fig. 2. For this element of the Web site evaluation, a prior knowledge of the subject was required, so only the pediatric neuro-oncologists (D.R.H. and E.B.) performed this assessment. The knowledge base for the assessment of accuracy came from standard texts such as Principles and Practice of Pediatric Oncology (Pizzo and Poplack, 2002), plus relevant papers in the field, including a series of state-of-the-art review articles published by recognized experts in pediatric neuro-oncology (Bouffet and Foreman, 1999; Freeman and Perilongo, 1999; Gajjar et al., 1999; Hayward, 1999; Reddy and Packer, 1999).
|
Statistical Methods
Simple descriptive statistics were used to analyze the data. Simple linear regression and Pearson's correlation coefficient were used to assess the relationship between the DISCERN and the Checklist Rating System Instrument scores and Web site variables. The interobserver variability for both rating methods was measured by using either weighted or unweighted kappa statistics. The SPSS software statistics package (SPSS, Chicago, Ill.) was used in the analysis.
Readability of Content
We used the inbuilt readability statistics in the word processor software program Microsoft Word 2003 (Microsoft Corporation, Redmond, Wash.) as a simple and reproducible measure of the readability of Web site content. The top-five-ranking Web sites for each tumor type, as scored by the DISCERN tool, were cut and pasted into the word processor, and all free text was scored by using the Flesch Reading Ease score and the Flesch-Kincaid Grade Level score. The Flesch Reading Ease score rates text on a 100-point scale. The higher the score, the easier it is to understand.
It is suggested that to achieve "plain English" a score should be around 65 (D'Alessandro et al., 2001; Klingbeil et al., 1995). The Flesch-Kincaid Grade Level score rates text on a U.S. school-grade level (i.e., a score of 8.0 means that an eighth-grader can understand the text). It has been suggested that health information should aim for a grade level of 6.0 to 8.0 (D'Alessandro et al., 2001; Davis et al., 1994).
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
A total of 114 Web sites (17 brain stem glioma, 20 craniopharyngioma, 29 ependymoma, 25 low-grade glioma, and 23 medulloblastoma) were fully assessed by two authors using both rating tools. Of the 114 sites assessed, the sources were 46% institutional, 35% commercial, 15% charitable, 2% support group, and 2% alternative medicine. Only two languages (Spanish and German) were offered in addition to the search language (English), and of the 11 multilingual Web sites, nine were based on the NCI Web pages, which provide English and Spanish translation.
Interobserver Rating Statistics
A weighted kappa statistic was applied to all the scores for each of the 15 DISCERN questions for all 114 Web sites rated by two authors. The weighted kappa statistics for the DISCERN instrument were 0.54 (95% confidence interval [CI],2 0.46–0.62) for raters E.B. versus U.A.H., 0.58 (95% CI, 0.5–0.66) for D.R.H. versus U.A.H., and 0.65 (95% CI, 0.56–0.74) for E.B. versus D.R.H. This represents an acceptable (although not substantial) agreement for a scaled instrument. A nonweighted kappa of 0.88 for D.R.H. versus E.B. was generated for the Checklist Rating System Instrument, representing an acceptable agreement for interobserver ratings.
Rating Instrument Scores
For each type of tumor, the Web sites were ranked both according to their total DISCERN score and according to the number of content topics that were present and accurate according to the Checklist Rating System Instrument. The Pearson correlation between the rankings/scores of the two instruments that were used was statistically significant, r = 0.57 (95% CI, 0.42–0.69), with a two-tailed P of less than 0.0001. This result suggests good correlation between the two methods, and the content-rating results are therefore displayed according to the DISCERN instrument only because this was our primary assessment tool.
By using the total DISCERN score, from the 15 questions, the Web sites have been grouped into categories of excellent (63–75), good (51–62), fair (39–50), poor (27–38), and very poor (15–26) in content, and this is displayed per tumor type in Fig. 3 and by Web site category in Fig. 4. Web sites were categorized by DISCERN score as excellent (4%), good (7%), fair (29%), poor (39%), and very poor (21%). The mean overall DISCERN score was 34.5 (poor). We attempted subgroup analysis to ascertain any relationships between ratings scores and Web site types. This analysis involved small sample sizes, which may affect the results.
|
|
|
The mean rating scores from all 114 Web sites for each of the 15 DISCERN questions are listed in Fig. 1. The median score for all questions combined was only 2 out of 5. No question had a mean score of 4 or above. The lowest scoring questions related to source of information, alternative sources of information, areas of uncertainty, no treatment, quality of life, choices of treatment, and shared decision making.
The Checklist Rating System Instrument showed that overall only 54% of the content topics selected by the authors as important were included in the Web sites (see Fig. 2). Again, there were no significant differences between Web site scores on the basis of tumor type. The content topic areas best covered were definition of tumor type, signs and symptoms, and types of treatment (all >70% present). However, the topics of etiology, tumor staging, prognosis, late effects, management of recurrence, and tumor-specific research were found in less than 40% of Web sites. There were one or more inaccuracies in 5% of the 114 Web sites evaluated, and the number of inaccuracies varied from 0 to 14 (median, 0; mean, 5) per Web site or 0 to 8% (median, 0%; mean, 3%) per topic. Most inaccuracies were found in the content topics of treatment (surgery, radiotherapy, and chemotherapy) and prognosis.
Readability Scores
The median Flesch Reading Score was 20.4 (mean, 27.7; range, 2.3–72.2) for the 25 top-ranking Web sites (five sites regarding the five tumor types) as rated by DISCERN. The recommended score for plain English is 60–70; only two sites achieved this, both from the same hospital and designed for adolescents and children. The median for the Flesch-Kincaid Grade Level was 12 (mean, 11.8; range, 6.8–12), and 64% of Web sites were grade 12. The version of this scoring system as used in the word-processing software described provides only U.S. education grades up to grade 12 (and not college/postgraduate level) and therefore will tend to underrepresent the difficulty of most sites. The recommended level for health literature for adults is suggested as U.S. grades 6–8; again, only the two sites already mentioned achieved this. There was excellent correlation between the two scoring systems (P < 0.0001).
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
has the potential to speed the transformation of the patient–physician relationship from that of physician authority ministering advice and treatment... to that of shared decision making between patient and physician
equitable access to information, imbalance between patient health literacy and the information provided, extreme variability in the quality of the content (AMA, 2006)
Previous studies have found that an imbalance between patient health literacy and the information provided certainly exists on the Internet, with the majority of such sites written at a reading level that is too complex for most adults in developed countries (Berland et al., 2001; D'Alessandro et al., 2001; Graber et al., 1999). This study confirms these previous results, with the Web sites demonstrating an average Flesch Reading Ease score of less than 30 (very difficult) and a U.S. Kincaid Grade Level of 12+ (high school graduate level or higher). This is considerably more difficult than the recommended level for health literature aimed at adults. Unfortunately, the sites with the best content often had the highest reading levels, perhaps negating their usefulness. When one considers that our patients range from young children to adolescents, who may also wish to look at information about their own disease, these readability statistics are of great concern because they effectively exclude the very population that suffers from the tumors they discuss.
There is an extreme variability in the quality of the content of health information in pediatric neuro-oncology on the Internet. The results of our ratings assessment indicate that very few sites achieve high standards by the criteria of the DISCERN tool. Only 11% of sites rated as excellent or good, whereas 60% were rated as poor or very poor. These poor ratings were mainly the result of not providing sufficient details to enable a health information consumer to judge the validity of the information, for example, no details about contributors or lack of referencing. This is a vital part of good practice in providing health information to the public, and this data should always be available. The DISCERN tool also places major significance on the availability of information with respect to shared decision making in health care, such as alternative treatment options and effects on quality of life. Although it is understandable and appropriate that some Web sites may specifically avoid providing information that may be controversial, debatable, or potentially upsetting (e.g., prognosis, treatment of relapse, or specific late effects of therapy), it is useful to direct the viewer to appropriate sources for this information, that is, a more comprehensive Web site, such as the National Cancer Institute or, of course, the patient's own health care team.
A major worry about medical information on the Internet is inaccuracy. However, the results of our Checklist Rating System Instrument were reassuring in documenting only a few inaccuracies (mean, 5%). These, though, appeared to be clustered in a few sites with a high number of inaccuracies, which meant that any viewer of these particular Web sites could gain contradictory information from the patient's own health care team that could lead to significant distress. Some of the inaccuracies were simply due to poor presentation, for example, "30% of brain stem gliomas are cured" (no explanation to distinguish the rarer low-grade focal tumors with a good prognosis from the more common diffuse intrinsic pontine glioma with a dismal outcome). More worrying were two examples of Web sites with obvious agendas, which were antiorthodox medicine. These included statements such as "Oncologists must be stopped from using children for experimentation" and "Be certain to undertake biopsy only if it is absolutely essential. Don't rely on the word of orthodox doctors alone; consult also with progressive or holistic physicians." Despite these examples, other Web sites described the role of complimentary or alternative therapies in relation to orthodox treatment in a balanced and well-informed manner.
Only two sites consistently rated as good or excellent across all five tumor types. The U.S. government site for the National Cancer Institute, which contained comprehensive, well-referenced, and updated information, which was provided in English and Spanish (an adaptation of this site by the University of Bonn also translated this into German). Second was the sponsored site eMedicine, which also provided comprehensive, well-referenced, and updated information by invited experts in the field and also usually provided more than one opinion for each tumor. Both sites provide two levels of information: one version for health professionals and one for patients.
There was no correlation between the type of Web site and either quality of information as rated by the DISCERN tool or the number of inaccuracies found. This may provide some reassurance as pertains to the potential for commercial interests to influence content, although this remains a possible concern.
A possible criticism of this study is that we evaluated all Web sites providing tumor health information, whether or not health consumers were their stated audience. This obviously means that some sites may be judged unfairly when compared to those where providing patient information is the primary aim. However, the study aimed to replicate the situation faced by a patient or family member using a search engine to access more knowledge about a brain tumor, and as stated above, it is unlikely that they will encounter only such dedicated sites. For this reason, we believe there needs to be a shared responsibility of health information consumers and Web site developers.
The general public must be made aware that the Internet is largely unregulated, that information may not be reliable, and that they must be cautious in viewing content not specified for the purpose of patient information. Equally, all those who provide information on the Web must be aware that it may be viewed by those other than their intended audience and that it is wise to state clearly the aim and limitations of any health information–containing Web site.
There has been an extensive debate about whether health information on the Internet should be controlled, rated, or validated in some way and about the possible means of how this could be achieved (Gagliardi and Jadad, 2002; Jadad and Gagliardi, 1998; Kim et al., 1999; Purcell et al., 2002; Shepperd and Charnock, 2002). We chose to use a validated ratings instrument as our primary tool, which was designed mainly to enable health consumers to judge health information. The DISCERN tool provided acceptable interobserver agreement for a graded instrument and also correlated with our second tool, the Checklist Rating System Instrument. However, the time it takes to learn how to use and apply the tool makes the authors doubt that this would be widely accepted by the general public. This means that other methods would need to be employed (e.g., simple consumer guidelines, voluntary codes of conduct, filtering tools, or third-party rating schemes) if some degree of external control is deemed necessary (Durkin, 2000; Edgar et al., 2002; Eysenbach et al., 2000; Wilson, 2002). Alternatively, one can choose to believe that members of the general public are quite capable of distinguishing between a well-designed, clear, evidence-based Web site and a substandard one. However, this means that high-quality, well-maintained Web sites must be available, and this is a major task and an expensive exercise. Ideally, this would be undertaken by national and international cancer groups and funded directly by government as part of public health care.
We decided not to assess the other less tangible elements of the Internet structure such as the architecture of sites, the use of links, and multimedia use (e.g., audio, animation, and video). Although a potentially major advantage over other media types, these elements are difficult to assess objectively (and are not accessible for all because they often require a high-speed Internet connection), but we did subjectively feel that there was a lack of imagination on the sites assessed as compared to Web sites providing information in other areas of health, which use audio, animation, and video footage to help explain complex information at a simpler level, especially engaging for children and adolescents.
Patients with CNS tumors and their families deserve the best possible information available 24 h a day. The Internet can provide this and augment other traditional sources of information in an imaginative and novel way. Health care professionals have already begun to harness this unique tool for the good of their patients, and we hope that the initial enthusiasm will lead to a new and improved generation of Web sites.
|
Footnotes |
|---|
Received for publication July 11, 2005. Accepted for publication November 21, 2005.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
AMA. American Medical Association (2006) Medical and Health Information Sites on the Internet. Available at www.ama-assn.org/ama/pub/category/1905.html (last updated January 2006).
Berland, G.K., Elliott, M.N., Morales, L.S., Algazy, J.I., Kravitz, R.L., Broder, M.S., Kanouse, D.E., Munoz, J.A., Puyol, J.A., Lara, M., Watkins, K.E., Yang, H., and McGlynn, E.A. (2001) Health information on the Internet: Accessibility, quality, and readability in English and Spanish. JAMA 285, 2612-2621.
Bichakjian, C.K., Schwartz, J.L., Wang, T.S., Hall, J.M., Johnson, T.M., and Biermann, J.S. (2002) Melanoma information on the Internet: Often incomplete—a public health opportunity? J. Clin. Oncol. 20, 134-141.
Biermann, J.S., Golladay, G.J., Greenfield, M.L., and Baker, L.H. (1999) Evaluation of cancer information on the Internet. Cancer 86, 381-390.[CrossRef][ISI][Medline]
Bouffet, E., and Foreman, N. 1999, Chemotherapy for intracranial ependymomas. Childs Nerv. Syst. 15, 563-570.[CrossRef][ISI][Medline]
Carlsson, M. (2000) Cancer patients seeking information from sources outside the health care system. Support. Care Cancer 8, 453-457.[ISI][Medline]
Charnock, D., Shepperd, S., Needham, G., and Gann, R. (1999) DISCERN: An instrument for judging the quality of written consumer health information on treatment choices. J. Epidemiol. Community Health 53, 105-111.[Abstract]
Chen, X., and Siu, L.L. (2001) Impact of the media and the internet on oncology: Survey of cancer patients and oncologists in Canada, J. Clin. Oncol. 19, 4291-4297.
Crocco, A.G., Villasis-Keever, M., and Jadad, A.R. (2002) Analysis of cases of harm associated with use of health information on the internet. JAMA 287, 2869-2871.
D'Alessandro, D.M., Kingsley, P., and Johnson-West, J. (2001) The readability of pediatric patient education materials on the World Wide Web. Arch. Pediatr. Adolesc. Med. 155, 807-812.
Davis, T.C., Mayeaux, E.J., Fredrickson, D., Bocchini, J.A., Jr., Jackson, R.H., and Murphy, P.W. (1994) Reading ability of parents compared with reading level of pediatric patient education materials. Pediatrics 93, 460-468.
Durkin, C. (2000) Accessing the Internet. New tools for researching cancer questions on the Internet. Natl. Netw. 25, 4-5.[Medline]
Edgar, L., Greenberg, A., and Remmer, J. (2002) Providing internet lessons to oncology patients and family members: A shared project. Psychooncology 11, 439-446.[CrossRef][Medline]
Eysenbach, G., Yihune, G., Lampe, K., Cross, P., and Brickley, D. (2000) Quality management, certification and rating of health information on the Net with MedCERTAIN: Using a medPICS/RDF/XML metadata structure for implementing eHealth ethics and creating trust globally. J. Med. Internet Res. 2 (suppl.), article e1.
Freeman, C.R., and Perilongo, G. (1999) Chemotherapy for brain stem gliomas. Childs Nerv. Syst. 15, 545-553.[CrossRef][ISI][Medline]
Gagliardi, A., and Jadad, A.R. (2002) Examination of instruments used to rate quality of health information on the internet: Chronicle of a voyage with an unclear destination. BMJ 324, 569-573.
Gajjar, A., Kuhl, J., Epelman, S., Bailey, C., and Allen, J. (1999) Chemotherapy of medulloblastoma. Childs Nerv. Syst. 15, 554-562.[CrossRef][ISI][Medline]
Graber, M.A., Roller, C.M., and Kaeble, B. (1999) Readability levels of patient education material on the World Wide Web. J. Fam. Pract. 48, 58-61.[ISI][Medline]
Hayward, R. (1999) The present and future management of childhood craniopharyngioma. Childs Nerv. Syst. 15, 764-769.[CrossRef][ISI][Medline]
Helft, P.R., Hlubocky, F., and Daugherty, C.K. (2003) American oncologists' views of internet use by cancer patients: A mail survey of American Society of Clinical Oncology members, J. Clin. Oncol. 21, 942-947.
Hellawell, G.O., Turner, K.J., Le Monnier, K.J., and Brewster, S.F. (2000) Urology and the Internet: An evaluation of Internet use by urology patients and of information available on urological topics. BJU Int. 86, 191-194.[CrossRef][ISI][Medline]
Houts, P.S., Rusenas, I., Simmonds, M.A., and Hufford, D.L. (1991) Information needs of families of cancer patients: A literature review and recommendations. J. Cancer Educ. 6, 255-261.[Medline]
Jadad, A.R., and Gagliardi, A. (1998) Rating health information on the Internet: Navigating to knowledge or to Babel? JAMA 279, 611-614.
Jenkins, V., Fallowfield, L., and Saul, J. (2001) Information needs of patients with cancer: Results from a large study in UK cancer centres. Br. J. Cancer 84, 48-51.[CrossRef][ISI][Medline]
Kim, P., Eng, T.R., Deering, M.J., and Maxfield, A. (1999) Published criteria for evaluating health related web sites: Review. BMJ 318, 647-649.
Klingbeil, C., Speece, M.W., and Schubiner, H. (1995) Readability of pediatric patient education materials. Current perspectives on an old problem. Clin. Pediatr. (Phila.) 34, 96-102.
Levenson, P.M., Pfefferbaum, B., Silberberg, Y., and Copeland, D.R. (1981) Sources of information about cancer as perceived by adolescent patients, parents, and physicians. Patient Couns. Health Educ. 3, 71-76.[ISI][Medline]
Levenson, P.M., Pfefferbaum, B., and Overall, J.E. (1983) A factor analysis of the informational concerns of adolescent cancer patients. Child Health Care 11, 148-153.
Leydon, G.M., Boulton, M., Moynihan, C., Jones, A., Mossman, J., Boudioni, M., and McPherson, K. (2000) Cancer patients' information needs and information seeking behaviour: In depth interview study. BMJ 320, 909-913.
Meredith, C., Symonds, P., Webster, L., Lamont, D., Pyper, E., Gillis, C.R., and Fallowfield, L. (1996) Information needs of cancer patients in west Scotland: Cross sectional survey of patients' views. BMJ 313, 724-726.
Pautler, S.E., Tan, J.K., Dugas, G.R., Pus, N., Ferri, M., Hardie, W.R., and Chin, J.L. (2001) Use of the internet for self-education by patients with prostate cancer. Urology 57, 230-233.[ISI][Medline]
Pizzo, P.A., and Poplack, D.G. (2002) Principles and Practice of Pediatric Oncology, 4th ed. Philadelphia: Lippincott Williams & Wilkins.
Purcell, G.P., Wilson, P., and Delamothe, T. (2002) The quality of health information on the internet. BMJ 324, 557-558.
Reddy, A.T., and Packer, R.J. (1999) Chemotherapy for low-grade gliomas. Childs Nerv. Syst. 15, 506-513.[CrossRef][ISI][Medline]
Roberts, J. (1999) Are healthcare professionals really prepared to deal with information-empowered citizens? Stud. Health Technol. Inform. 68, 523-527.[Medline]
Roberts, J.M., and Copeland, K.L. (2001) Clinical websites are currently dangerous to health. Int. J. Med. Inf. 62, 181-187.[CrossRef][ISI][Medline]
Shepperd, S., and Charnock, D. (2002) Against internet exceptionalism. BMJ 324, 556-557.
Vordermark, D., Kolbl, O., and Flentje, M. (2000) The Internet as a source of medical information. Investigation in a mixed cohort of radiotherapy patients. Strahlenther. Onkol. 176, 532-535.[CrossRef][Medline]
Wilson, P. (2002) How to find the good and avoid the bad or ugly: A short guide to tools for rating quality of health information on the internet. BMJ 324, 598-602.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
