RELIABILITY ANALYSIS

Overview

Researchers must demonstrate instruments are reliable since without reliability, research results using the instrument are not replicable, and replicability is fundamental to the scientific method. Reliability is the correlation of an item, scale, or instrument with a hypothetical one which truly measures what it is supposed to. Since the true instrument is not available, reliability is estimated in one of four ways:

" Internal consistency: Estimation based on the correlation among the variables comprising the set (typically, Cronbach's alpha)

" Split-half reliability: Estimation based on the correlation of two equivalent forms of the scale (typically, the Spearman-Brown coefficient)

" Test-retest reliability: Estimation based on the correlation between two (or more) administrations of the same item, scale, or instrument for different times, locations, or populations, when the two administrations do not differ on other relevant variables (typically, the Spearman Brown coefficient)

" Inter-rater reliability: Estimation based on the correlation of scores between/among two or more raters who rate the same item, scale, or instrument (typically, intraclass correlation, of which there are six types discussed below).

These four reliability estimation methods are not necessarily mutually exclusive, nor need they lead to the same results. All reliability coefficients are forms of correlation coefficients, but there are multiple types discussed below, representing different meanings of reliability and more than one might be used in single research setting.

The full content is now available from Statistical Associates Publishers. Click here.

Below is the unformatted table of contents.

Table of Contents Overview 6 Key Concepts and Terms 7 Scores 7 Number of scale items 7 Models 7 SPSS 7 SAS 8 Triangulation 8 Calibration 8 Internal consistency reliability 9 Cronbach's alpha 9 Overview 9 Interpretation 9 Cut-off criteria 9 Formula 9 Number of items 10 Cronbach's alpha in SPSS 10 Example 1 10 Alpha if deleted 10 Item-total correlation 11 R-squared 11 Negative alphas 11 KR2 11 Example 2 12 Standardized item alpha 12 Cronbach's alpha in SAS 13 SAS syntax 13 SAS output 13 Other internal consistency reliability measures 14 Ordinal reliability alpha 14 Raykov's reliability rho (?) 15 Armor's reliability theta 15 Spearman's reliability rho 15 Split-half reliability 16 Split-half reliability in SPSS 16 Overview 16 SPSS menu selections 16 Spearman-Brown split-half reliability coefficient 17 Guttman split-half reliability coefficient 19 Guttman's lower bounds (lambda 1-6) 19 Split-half reliability in SAS 20 Odd-Even Reliability 20 Overview 20 SPSS 21 Test-retest reliability 22 Overview 22 Inter-rater reliability 23 Overview 23 Cohen's kappa 23 Kappa in SPSS 23 Example 24 Interpretation 24 Weighted Kappa 25 Intraclass correlation (ICC) 25 Example 25 Sample size: ICC vs. Pearson r 26 Data setup 26 Interpretation 27 Obtaining ICC in SPSS 28 Single versus average measures 28 ICC Models 29 ICC use in other contexts 31 Assumptions 31 Additivity 31 Independence 32 Uncorrelated error 32 Consistent coding 33 Random assignment of subjects 33 Equivalency of forms 33 Equal variances 33 Similar difficulty of items 34 Same assumptions as for correlation 34 Frequently Asked Questions 34 How is reliability related to validity? 34 How is Cronbach's alpha related to factor analysis? 34 How is reliability related to attenuation in correlation? 35 How should a negative reliability coefficient be interpreted? 36 What is Cochran's Q test of equality of proportions for dichotomous items? 36 What is the derivation of intraclass correlation coefficients? 37 What are Method 1 and Method 2 in the SPSS RELIABILITY module? 38 Bibliography 38