Radiography (2005) 11, 55e59 REVIEW ARTICLE Critiquing a research article Gill Marshall) Division of Medical Imaging Sciences, St. Martin’s College, Lancaster LA1 3JD, UK Accepted 16 September 2004 Available online 1 December 2004 KEYWORDS Critical reading; Peer review; Literature; Journal review Abstract This article explores certain concepts relating to critiquing research papers. These include considering the peer review process for publication, demonstrating the need for critiquing, providing a way to carefully evaluate research papers and exploring the role of impact factors.
Whilst all these features are considered in this article, the focus is on presenting a systematic and comprehensive way of critiquing research papers. The information provided should be of use to the many radiographers, associated health professionals and undergraduate and postgraduate students embarking on research projects. ? 2004 The College of Radiographers. Published by Elsevier Ltd. All rights reserved. Why critique peer reviewed research articles? The peer review process is integral to the functioning of all scienti? c journals and plays a pivotal role in the publication of new scienti? material. 1 The ”invisible hand” of peer review is what is claimed to maintain the quality of refereed i. e. peer reviewed, journal literature. 2 The publication of a research article in a peer reviewed journal may thus appear to be a measure of its worth. 3 However, the process of peer review has attracted its share of criticisms from academics over the years4 with one author going as far as to say ”those that review essays for inclusion in scholarly journals know what they are supposed to do. Their ) Tel. : C44 1524 384639; fax: C44 1524 384591. E-mail address: g. [email protected] ac. uk unction is to take innovative and challenging work by young scholars and ? nd reasons to reject it”. 5 Furthermore reviewers need not necessarily have expert knowledge of the subject matter they review3 as even experts have gaps in their knowledge. 6 Peer review is notoriously unreliable and subject to bias and con? ict of interest. Publication bias, the tendency of editors and reviewers to accept manuscripts submitted by investigators based on the strength and direction of their own research ? ndings,7 means that what is published may not be representative of the research in an area which may mislead the reader.
Consequently, publication bias can reduce the intellectual value of the research. The problems associated with the peer review process seem dif? cult to overcome, as even training peer reviewers does not increase the quality of their reviews to a level of editorial signi? cance or in a way that is maintained long term. 8 1078-8174/$ – see front matter ? 2004 The College of Radiographers. Published by Elsevier Ltd. All rights reserved. doi:10. 1016/j. radi. 2004. 09. 001 56 G. Marshall consistent with the ”Guide for Authors” of the speci? c journal.
These key words should encapsulate the main topics of the research and should allow the article to be accessed when searching the literature using key words as search terms. Critical evaluation of research articles This is necessary so that you do not take what you read at face value but consider the work with a critical mind in order that you can decide on the value of the article. This empowers you to decide whether to change your practice based on what you have read9 or whether the work is a worthwhile study to base future research around. Critical evaluation is de? ed as ”a systematic way of considering the truthfulness of a piece of research, the results and how relevant and applicable they are”. 10 How to critically evaluate research articles is a topic addressed by a plethora of books on research methodology11e13 and by various articles. 6,9,14,16 Set out below is a way of systematically critiquing such articles in a structured way. This is the method for critiquing the literature taught to undergraduate and postgraduate students in this institution. It should be remembered that a critique will often be positive and should not be seen as just negative.
If negative, the implications of any weaknesses in the study, need to be considered. 9 In the context of this paper, a research article is a written published report of original research presented in a peer reviewed journal, to allow it to be judged in the context of the body of knowledge. The article will allow an assessment by readers of what observations were made, how the research was conducted and its intellectual value or ”so what” factor, for example an article describing a method of recording blood pressure to six decimal places may be scienti? cally robust but have little application to practice.
An article should be considered under the following headings. Introduction This usually contains (i) evidence of a literature review, (ii) background information to the study to orientate the reader to the problem, (iii) the hypothesis or aims of the study and (iv) the rationale for the study. These elements should be logically presented and well written. 6,14 A literature review should be present that is relevant and recent, unless the article has a historical focus. Older articles acknowledged as seminal works in the area should be cited. It should contain few if any secondary sources but should con? e itself to a review of primary sources. 14 It should be comprehensive and even handed in its selection of both theoretical and research information on the topic, and should be presented in an objective way. The literature review should be critical in its appraisal of other works, rather than merely descriptive of them. To assess how comprehensive and balanced the literature review is, a literature search can be done by the reader to ensure the breadth of the literature cited and that, in instances where there are con? icting opinions, that they are represented.
It is clearly tempting for authors to supply only background literature that supports their own premise or research ? ndings. From the literature reviewed and thus the background information provided, a rationale for the current work should evolve, justifying the need for the current work, for instance, to explore an uninvestigated gap in the literature. 14 The purpose or aim of the study and the research hypothesis, if provided, should be de? ned so that the research problem can be clearly identi? ed. The research objectives by which the aim will be achieved are also commonly stated in the introduction. Title
The title may be better judged after reading the article. 6,9 It should precisely and concisely re? ect the content of the work, but does not necessarily give an indication of the quality of the article. Whilst it should not contain jargon or buzz words that are not directly relevant, the title should stimulate the interest of readers and encourage them to read it. 6 Materials and methods The materials and methods form the precise recipe for the research so that another worker could exactly replicate the study elsewhere, usually to allow disproof of ? ndings. This section should include, for example, precise technical speci? ations of equipment used, procedure utilised, selection criteria, sample size, response rate and statistics used. The justi? cation of why such Key words These are drawn not from the title but from the body of the work. Three to six is a common number of key words, but the number presented should be Critiquing the literature a recipe was used is the methodology i. e. the study of the method, which explains the rationale for the research method used including aspects such as sample size selected, exclusion criteria and statistics used. The design of the research must have this justi? ation to show that the study is capable of achieving its aims14 e. g. the use of a postal questionnaire sent a month after an MRI examination would be a dubious way to assess the anxiety provoked prior to an MRI scan. Things to consider when reading this section are: What sort of sampling technique and sample size was used? What proportion of an eligible sample participated? Were all eligible groups sampled e. g. was the questionnaire, if used, only provided in English? Can the results be generalised to a wider population? What are the strengths and weaknesses of the study? Are there any threats to the study’s validity and reliability? If so did the researcher attempt to control these? Are there any obvious biases or confounding variables introduced e. g. when comparing patient’s preferences for two techniques such as an endoscopy and barium meal, were the tests undertaken by different operators? Was the trial, if used, the stronger randomised control trial (where participants have an equal chance of being in the experimental or control group) or the weaker case controlled trial (where patients with a particular condition are ”matched” with controls)? 2 Was the power of the study calculated? 6 This refers to the ability of the research design to detect existing relationships among variables. 13 It will determine how likely it is that a relationship may be missed and is particularly important in interpreting null results. The number of participants needed in a study to ensure that relationships are not missed may not have been realistic, due to other constraints such as time or funding, leading in effect to a pilot or feasibility study being undertaken. Common sense has a bearing here. If a condition is uncommon e. . aspergillosis, a rare pulmonary fungal infection, it may be judged reasonable to image 20 patients with the condition but not so reasonable to image only 20 patients with lung cancer, a much more prevalent condition. 57 Results The data presented should not be raw but should be scienti? cally analysed to present representative and relevant values, that the ”average” reader of the journal in which the paper is published can easily assimilate. 14 If an unfamiliar test is used the values it generates should be presented along with a normal range of values.
The results should be sequenced appropriately and a decision should be presented by the author as to whether the aims and hypotheses of the study were met by the results. Graphs and tables of the data, if provided, should promote clarity. They should have a title or legend, a key and labelled axes. It should be possible to understand them without referring to the text. 6 Clearly, the way the results were analysed will depend on whether the research was qualitative or quantitative. Points to consider include: Are there any major omissions? E. g. not all of the sample is represented in the results. Are percentages used to disguise small sample sizes? Are the data generated consistent with the data collected? Statistical tests, if used, should be named but not described. Consider whether the appropriate statistics were used depending on whether statistically differences or correlations were sought. Were the data gathered interval/ratio data (the strongest data achieved by the use of a calibrated scale e. g. density readings from a densitometer)? Ordinal (where the data have a clear order but not from a calibrated scale e. g. strongly agree, agree etc. from a Likert scale). Nominal (the least robust data which categorise but do not rank data e. g. a list of radiographers, radiologists and nurses working in a particular work area). Most statistics used by researchers are parametric, a term which classi? es a group of tests including the one way analysis of variance (ANOVA) and the paired and unpaired t-tests. If parametric tests are used you need to check that the data are: Approximately normally distributed, Derived from interval or ratio scales, The variances of the data are similar. 15 58 Non-parametric tests are used for ordinal or nominal data e. g. he Wilcoxon matched pairs test and the chi-squared test. This group of tests requires few assumptions to be met, regarding the underlying population distributions. 11 Findings which are negative are just as relevant to the body of knowledge but are harder to get published, which is an example of publication bias. 8 G. Marshall References Different journals have different requirements for presentation of references. 17 The ”Guide for Authors” of a speci? c journal will state their requirements. In general the references should follow a consistent format and correlate with the citations in the text, be up to date, comprehensive and relevant.
There should not be the excessive use of secondary sources. 6,14 A secondary source is when the author refers to an account of a study prepared by someone other than by the original researcher. 13 Discussion The discussion is a commentary on the research ? ndings and should show an insight into their meaning and signi? cance. It should not repeat the results or introduce new ones. 6 It should demonstrate that the aims and objectives of the research have been met. The discussion should present all the relationships demonstrated by the results and state the extent to which these ? dings can be generalised. If there are any exceptional results or correlation failures these should be explained. The discussion should embed the current ? ndings in the context of previous research work and theoretical concepts. 14 Any limitations of the work and problems with the design of the research and methods should be acknowledged, as should the effect of any biases on the results. The reader should consider whether the author’s interpretation of the results follows from the results presented and whether it is the only possible interpretation.
If not, does the author present a balanced discussion? 6 Abstract This is presented ? rst but is written last by the author/s. It is often structured e. g. purpose, method, results, conclusion. It is always concise, around 300 words and should not contain so much information that reading the article is redundant. 14 It is in essence the ”nub” of the work and it does not usually contain references. It allows readers to judge its appropriateness to their research needs. 17 Impact factors When critiquing an article it is worth considering the impact factor of the journal in which it is published.
The Institute for Scienti? c Information produces the Journal Citation ReportsO (JCRO). This provides a qualitative tool to rank, evaluate, categorise and compare journals. The impact factor is one of these tools; it is a measure of the frequency with which the ”average article” in a journal has been cited in a particular year or period. Impact factors are the ratio between citations and recent citable articles published. They are dynamic factors which alter year on year and are published annually. Impact factors are thus valuable in academic evaluation.
They provide a gross approximation of the prestige of the journal in which the article is published and can be studied by accessing ! http://jcrweb. com/ jcr_summaryO, which will generate a list of these factors for various journals. If your institution does not subscribe to this web resource an internet search for ”impact factors” will take you to a variety of sites where similar information can be accessed free of charge. The higher the value of the impact factor the more prestigious the journal. Factors for Radiology journals currently range between 0. 3 and 6. 2. 8 This measure must be used Conclusions The conclusions should be clearly stated and can only be valid if the study was reliable, valid and the sample size representative. Reliability is the degree of consistency or dependability with which the instrument measures the attribute it is designed to measure and validity is the degree to which the instrument measures what it is intended to measure. 13,16 The extent to which the sample size represents the population is a factor in assessing the validity of a study i. e. the extent to which the results can be generalised to other samples or situations. 1,13 The conclusions often give rise to recommendations for future practice and, or further research. The conclusions should not over-claim and they should be based on the results. These should be feasible and the reader should make a judgement as to whether it is reasonable to make these on the basis of one study. 6 Critiquing the literature with some caution as the amount of review or other types of articles published in a journal, variations between disciplines and item by item impact make it a less than absolute measure of the academic prestige of a journal.
It is nevertheless a useful factor to consider in critiquing peer reviewed articles. 19 59 2. The invisible hand of peer review. !http://www. exploit-lib. org/issue5/peer-review/O Exploit interactive ; issue 5 [accessed 9. 7. 04]. 3. Elwood M. Critical appraisal of epidemiological studies and clinical trials. 2nd ed. Oxford: Oxford University Press; 1998. p. 46e7. 4. Roberts P. Scholarly publishing, peer review and the internet. !http://www.? rstmonday. dk/issues/issues4_4/ proberts/O [accessed 9. 7. 04]. 5. Readings B. Caught in the net: notes from the electronic underground. Surfaces 1994;4(1):9e10. 6.
Drummond D. Reviewing a research article. Br J Occup Ther 1996;59(2):84e6. 7. Dickersin K. How important is publication bias? A synthesis of available data. AIDS Educ Prev 1997;9(Suppl. 1):15e21. 8. Schroter S, Black N, Evans S, Carpenter J, Godless F, Smith R. Effects of training on the quality of peer review: a randomized control trial. BMJ 2004;328(7441):657e8. 9. McCaughan D. Developing critical appraisal skills. Prof Nurse 1999;4(12):843e7. 10. Bury T, Mead J. Evidence based health care: a practical guide for therapists. 1st ed. Oxford: Butterworth Heinemann; 1998. p. 26e42. 11. Polgar S, Thomas S.
Introduction to research in the health sciences. 3rd ed. Melbourne: Churchill Livingstone; 1995. p. 243e306. 12. Greenhalgh T. How to read a paper. 2nd ed. Graphycems: Navarra BMJ books; 2001. p. 45e52. 13. Polit D, Hungler B. Nursing research: principles and methods. 6th ed. Philadelphia: Lippincott; 1991. p. 623e42. 14. Giuffre M. Critiquing a research article. J Perianesth Nurs 1998;13(2):104e8. 15. Banks S. Analysis of quantitative research data: part 2. Prof Nurse 1999;14(10):710e3. 16. Giuffre M. Reading research critically: assessing the reliability and validity of the research instrumentation e part 1.
J Post Anesth Nurs 1995;10:107e12. 17. Hendry C, Farley A. Reviewing the literature: a guide for students. Nurs Stand 1998;12(44):46e8. 18. !http://jcrweb. com/jcr_summaryO [accessed 9. 7. 04]. 19. !http://www. isinet. com/essays/journalcitationreports/7. html/O [accessed 9. 7. 04]. Summary The above information has demonstrated the need to read literature, even that published in peer reviewed journals, with a critical mind. It has provided a systematic framework with which to do this, allowing the reader to appreciate both the strengths and weaknesses of the work.
This should empower the reader to assess the value of the work and thus judge how much credence be given to it, in in? uencing future practice or research activities. Acknowledgements I would like to thank my colleagues Nick Jones and Dr. David Manning for their constructive criticism during the preparation of this article. References 1. Levine AM, Heckman JD, Hensinger RN. The art and science of reviewing manuscripts for orthopaedic journals: part 1 de? ning the review. Instr Course Lect 2004;53:679e88.