Iota Reliability Concept of the Second Generation. Measures for Content Analysis Done by Humans or Artificial Intelligences

Abstract

In educational settings, analyzing textual data via content analysis is a popular research method. The data is a valuable source of information as it offers deep insights into learning and learning outcomes. In practice, it can be used to improve classroom diagnostics and instruction. Nowadays, technology such as learning analytics can be used for the same cause. For both purposes, reliable research instruments are needed. Content analysis, often the measure of choice, is required to meet quality criteria such as objectivity, reliability and validity. However, some of the reliability measures most frequently used have lately been discussed controversially, indicating that there is room for improvement. The first generation of the Iota concept caters to the idea of improved reliability measures for content analysis done by humans or artificial intelligences. In this book, the authors introduce a refined measure: The Iota concept of the second generation. In contrast to pre-existing measures, second generation Iota can for example a) provide insights into the reliability of every single category of a scale and how a coding scheme may produce bias, b) provide rules of thumb for evaluating content analysis and c) provide possibilities for data replication and error-corrected data. This book is structured as a guide for researchers that want to learn more about the mechanics and details of the Iota concept or use it as the reliability measure of choice in their research.