Why should content analysis researchers care about intercoder reliability?
[TOP] It is widely acknowledged that intercoder reliability is a critical component of content analysis, and that although it does not insure validity, when it is not established properly, the data and interpretations of the data can not be considered valid. As Neuendorf (2002) notes, “given that a goal of content analysis is to identify and record relatively objective (or at least intersubjective) characteristics of messages, reliability is paramount. Without the establishment of reliability, content analysis measures are useless” (p. 141). Kolbe and Burnett (1991) write that “interjudge reliability is often perceived as the standard measure of research quality. High levels of disagreement among judges suggest weaknesses in research methods, including the possibility of poor operational definitions, categories, and judge training” (p. 248). A distinction is often made between the coding of manifest content, information “on the surface,” and latent content under these surface elements.