Assess The Validity of Data Part III

Data Validity Assessment Criteria

Assess The Validity of Data, Ferketich, Figueredo, & Knapp, 1991; Lowe & Ryan-Wenger, 1992,Other Criteria for Assessing Quantitative Measures.

    Another approach to construct validation uses a statistical procedure known as factor analysis. Factor analysis is a method for identifying clusters of related variables. Each cluster, called a factor, represents a relatively unitary attribute. The procedure is used to identify and group together different items measuring an underlying attribute. 

    In effect, factor analysis constitutes another means of looking at the convergent and discriminant validity of a large set of items. Indeed, a procedure known as confirmatory factor nalysis is sometimes used as a method for analyzing MTMM data (Ferketich, Figueredo, & Knapp, 1991; Lowe & Ryan-Wenger, 1992). 

    Construct validation is the most important type of validity for a quantitative instrument. Instrument developers should use one or more of the techniques described here in their effort to assess the instrument's worth. Interpretation of validity like reliability, validity is not an all-or-nothing characteristic of an instrument. An instrument does not possess or lack validity; it is a question of degree. 

    An instrument's validity is not proved, established, or verified but rather is supported to a greater or lesser extent by evidence. Strictly speaking, researchers do not validate an instrument but rather an application of it. A measure of anxiety may be valid for esurgical patients on the day of an operation but may not be valid for nursing students on the day of a test. 

    Of course, some instruments may be valid for a wide range of uses with different types of samples, but each use requires new supporting evidence. The more evidence that can be gathered that an instrument is measuring what it is supposed to be measuring, the more confidence researchers will have in its validity.

Other Criteria for Assessing Quantitative Measures

    Reliability and validity are the two most important criteria for evaluating quantitative instruments. High reliability and validity are a necessary, although not sufficient, condition for good quantitative research. Researchers sometimes need to consider other qualities of an instrument, as discussed in this section. 

    Sensitivity and specificity Sensitivity and specificity are criteria that are important in evaluating instruments designed as screening instruments or diagnostic aids. For example, a researcher might develop a new scale to measure risk of osteoporosis. Such screening/diagnostic instruments could be self-report, observational, or biophysiologic measures. 

    Sensitivity is the ability of an instrument to identify a “case correctly,” that is, to screen in or diagnose a condition correctly. An instrument's sensitivity is its rate of yielding "true positives." Specificity is the instrument's ability to identify noncases correctly, that is, to screen out those without the condition correctly. 

    Specificity is an instrument's rate of yielding "true negatives." To determine an instrument's sensitivity and specificity, researchers need a reliable and valid criterion of "caseness" against which scores on the instrument can be assessed. There is, unfortunately, a trade off between the sensitivity and specificity of an instrument. When sensitivity is increased to include more true positives, the number of true negatives declines. 

    Therefore, a critical task is to develop the appropriate cutoff point, that is, the score value used to distinguish cases and noncases. To determine the best cutoff point, researchers often use what is called a receiver operating characteristic curve (ROC curve). 

    To construct an ROC curve, the sensitivity of an instrument (ie, the rate of correctly identifying a case vis-à-vis a criterion) is plotted against the false-positive rate (ie, the rate of incorrectly diagnosing someone as a case, which is the inverse of its specificity) over a range of different cutoff points. 

    The cutoff point that yields the best balance between sensitivity and specificity can then be determined. The optimum cutoff is at or near the shoulder of the ROC curve. The example at the end of this chapter illustrates the use of ROC curves. Fletcher, Fletcher, and Wagner (1996) is a good source for further information about these procedures.

Efficiency

    Instruments of comparable reliability and validity may differ in their efficiency. A depression scale that requires 10 minutes of people's time is efficient compared with a depression scale that requires 30 minutes to complete. One aspect of efficiency is the number of items incorporated into an instrument. 

    Long instruments tend to be more reliable than shorter ones. There is, however, a point of diminishing returns. As an example, consider a 40-item scale to measure social support that has an internal consistency reliability of .94. Using the Spearman-Brown formula, we can estimate how reliable the scale would be with only 30 items:

where

k is the factor by which the instrument is being incremented or decreased; in this case, k 30 40 .75

r ¹ = reliability estimate for shorter (longer) scale

    As this calculation shows, a 25% reduction in the instrument's length resulted in a negligible decrease in reliability, from .94 to .92. Most likely researchers would sacrifice a modest amount of reliability in exchange for reducing subjects' response burden and data collection costs. Efficiency is more characteristic of certain types of data collection procedures than others. 

    In self-reports, closed-ended questions are more efficient than open-ended ones. Self-report scales tend to be less time-consuming than projective instruments for a comparable amount of information. Of course, a researcher may decide that other advantages (such as depth of information) offset a certain degree of inefficiency. Other things being equal, however, it is desirable to select as efficient an instrument as possible.

    Other criteria A few remaining qualities that sometimes are considered in assessing a quantitative instrument can be noted. 

    Most of the following six criteria are actually aspects of the reliability and validity issues: 

    1. Comprehensibility. Subjects and researchers should be able to comprehend the behaviors required to secure accurate and valid measures. 

   2. Precision. An instrument should discriminate between people with different amounts of an attribute as precisely as possible. 

    3. Speediness. For most instruments, researchers should allow adequate time to obtain complete measurements without rushing the measuring process.

    4. Range. The instrument should be capable of achieving a meaningful measure from the smallest expected value of the variable to the largest. 

    5. Linearity. A researcher normally strives to construct measures that are equally accurate and sensitive over the entire range of values. 

    6. Reactivity. The instrument should, as far as possible, avoid affecting the attribute being measured.