Instruments and Barriers of Evaluation In Nursing Education
What are Evaluation Instruments
In the selection, revision, or construction of evaluation instruments, nurse educators must consider some key points. possible, an evaluation Whenever should be conducted using existing instruments. This is because instrument development not only requires considerable expertise, time, and resources but also requires testing to be sure the instrument, whether it is in the form of a questionnaire or a type of equipment, demonstrates reliability and validity before it is used for collecting data.
This testing can take several months to several years to determine the usefulness of the tool (Osborne, Elsworth, & Whitfield, 2007). The initial step in instrument selection is to conduct a literature search for evaluations similar to the evaluation that is being planned. A helpful place to begin is with the same journals listed earlier in this chapter. Instruments that have been used in more than one study should be given preference over an instrument =developed for a single use.
Instruments for Reliability and Validity of Instruments
This is because instruments used multiple times generally have been more thoroughly tested for reliability and validity. Once potential instruments have been identified, each instrument must be carefully critiqued to determine whether it is, in fact, appropriate for the evaluation planned.
First, the instrument must measure the performance being evaluated exactly as that performance has been operationally defined for the evaluation. For example, if satisfaction with a continuing education program is operationally defined to include a score of 80% or higher on five specific program components (such as educator responsiveness to questions, relevance of content, and so on), then the instrument selected to measure participant satisfaction with the program must include exactly those five components and must be able to be scored in percentages.
Second, an appropriate instrument should have documented evidence of its reliability and validity with individuals who are as closely matched as possible with the people from whom data will be collected. For example, when evaluating the ability of older adult patients to complete activities of daily living, one would not want to use an instrument developed for evaluating the ability of young orthopedic patients to complete routine activities.
Similarities in reading level and visual acuity also should exist if the instrument being evaluated is a questionnaire or scale that participants will complete themselves. In addition, existing instruments being considered for selection must be affordable, must be feasible for use in the location planned for conducting data collection, and should require minimal training on the part of data collectors.
A cognitive test is the evaluation instrument most likely to require modification from an existing tool or development of an entirely new instrument. The primary reason for constructing such a test is that it must be consistent with the content covered during the educational program or activity.
The intent of a cognitive test is to be comprehensive and relevant and to fairly test the learner's knowledge of content. Using a test blueprint is one of the most helpful methods for ensuring comprehensiveness and relevance of test questions. A blueprint enables the evaluator to be certain that the test covers each area of instructional content and that content areas emphasized during instruction are similarly emphasized during testing.
Multiple questionnaires, scales, and other types of instruments exist to measure patient and healthcare provider characteristics, such as traits, perceptions, beliefs, attitudes, activity levels, conflicts, communication skills, and relationships. For example, many measurement tools are already available and have been tested for how well and how consistently they perform in collecting data related to patient education. Redman (2003) describes many of these useful evaluation instruments.
Monsivais and Reynolds (2003) describe how to evaluate patient education materials using an instrument to measure effectiveness for patient teaching and learning. Also, a patient education materials assessment tool (PEMAT) is an instrument recently developed to evaluate whether patients are able to understand and take action on information available in print and audiovisual formats that are used so frequently for patient education (Shoemaker, Wolf, & Brach, 2013).
One helpful step in instrument selection for evaluation of patient education is to review materials used to provide that education. Did the educator use a step-by-step checklist to teach the patient how to flush a urinary catheter? That same checklist might be used to observe whether the patient remembers each step during return demonstration.
A checklist for both teaching and evaluation almost guarantees that the nurse is measuring the performance being evaluated exactly as that performance was operationally defined prior to instruction and evaluation.
Barriers to Evaluation in Nursing Education
If evaluation is so important to healthcare education, why is evaluation often an afterthought or even overlooked entirely? The reasons given for not conducting evaluations are many and varied but rarely, if ever, are impossible to overcome.
Barriers to evaluation must first be identified and understood; then, the nurse educator must design and conduct the evaluation in a way that minimizes or eliminates as many identified barriers as possible. Barriers to conducting an evaluation can be classified into three broad categories:
1. Lack of clarity
2. Lack of ability
3. Fear of punishment or loss of self esteem
Lack of Clarity
If the focus for evaluation is unclear, unstated, or not well defined, then undertaking an evaluation is difficult if its purpose or what will be done with the results is unknown. Often evaluations are attempted to determine the quality of an educational program or activity, yet quality is not defined beyond some vague sense of goodness.
What is “goodness” and from whose perspective will it be determined? Who or what will demonstrate evidence of goodness? What will happen if goodness is or is not obvious? Inability to answer these or similar questions creates a significant barrier to conducting an evaluation even the most seasoned evaluator.
Barriers in this category have the greatest potential for successful resolution because the best solution for lack of clarity is to provide clarity. Recall that evaluation focus includes five components:
(1) audience
(2) purpose
(3) questions
(4) scope
(5) resources
To overcome a potential lack of clarity, the nurse must educator identify all five components and make them available to those conducting the evaluation.For example, clear identification of who constitutes the primary audience is very important. This is because it is from the perspective of the primary audience that terms such as “quality” should be defined and operationalized.
Although the results of the evaluation provide the information on which decisions will be made, the primary audience makes those decisions. Also, a clearly stated purpose is as important as knowing who the audience is because the purpose explains why the evaluation is being conducted.
For example, if the purpose for teaching a patient about his medications is so that he can be independent in self care at home, then the nurse must evaluate both the knowledge and physical ability of the patient to take his medications correctly prior to discharge. Using teach back can inform the nurse whether the patient understands what the medications are for, how frequently and in what dosage they should be taken, and what might be side effects that should prompt the patient to contact his primary care provider immediately.
Filling the prescriptions in advance before discharge can allow the nurse to observe whether the patient is able to read the labels, open the containers, and take the medications as directed (Visiting Nurse Associations of America, 2012).
Lack of Ability
Inability to conduct education evaluations most often results from lack of knowledge, confidence, interest, or resources needed to carry out this process. Members of the primary audience are accountable for providing the necessary resources- personnel, equipment, time, facilities, and so onto conduct the evaluation they are requesting. Unless these individuals have some expertise in evaluation, they may not know what resources are necessary.
The persons conducting the evaluation, therefore, must accept responsibility for knowing the resources that are necessary and for providing this information to the primary audience. The person asked or expected to conduct the evaluation may be as uncertain about necessary resources as the primary audience is, however.
Lack of knowledge of which resources are necessary or lack of actual resources may create a barrier to conducting an evaluation that can be difficult though not impossible to overcome. Lack of knowledge can be resolved or minimized by enlisting the assistance of individuals with needed expertise through consultation or contract (if funds are available), through collaboration, or indirectly through literature review.
For instance, a survey tool recently was developed to evaluate the perceptions of pediatric nurses in carrying out their role as patient educators, but the self designed instrument required outside experts to determine if the survey content was reliable and valid (Lahl, Modic, & Siedlecki, 2013). In another study, interviews with nurses revealed that they needed much more professional education on how to teach children and their families, including the use of multi method evaluation techniques (Kelo, Martikainen, & Eriksson, 2013).
Lack of knowledge about evaluation techniques can be resolved by enlisting the assistance of individuals who have expertise and are willing to collaborate or provide consultation services. Remember that patient care is a team activity. If a patient, for example, needs to learn how to use crutches correctly before being discharged, the nurse can appropriately request a physical therapist who is expert in evaluating the patient's ability to use them safely.
Fear of Punishment or Loss of Self Esteem
Evaluation might be perceived as a judgment of someone's value or personal worth. Both the learner and the teacher may fear that anything less than a perfect performance will result in criticism, punishment, or evidence that their mistakes will result in their being labeled as incompetent (Lahl et al., 2013). These fears form one of the greatest barriers to conducting an evaluation.
Unfortunately, these fears may not easily be overcome, especially for individuals who have had negative experiences with teaching and learning in the past. Consider, for example, traditional quality assurance monitoring. where results are used to correct deficiencies often through punitive measures, such as losing the opportunity for promotion or losing respect from colleagues.
As another example, many times an educator has interpreted learner dissatisfaction with a teaching style as learner dislike for the educator as a person. In addition, it is unfortunate but not unusual for parents of pediatric patients to say, “If you don't do it right, the doctor won't let you go home...and we will be very disappointed in you.” And everyone probably has experienced test anxiety as a learner at some point in their life.
The first step in overcoming this barrier is to realize that the potential for its existence can be closed to 100%. Individuals whose performance or knowledge is being evaluated are not likely to state outright that evaluation represents a threat to them.
Rather, they are far more likely to demonstrate self-protective behaviors or attitudes that can range from failing to participate in a teaching session or attend a class that has a posttest to providing socially desirable answers on a questionnaire or to responding with anxiety or anger to evaluate questions. It is even possible that an individual may intentionally choose to fail an evaluation as a method for controlling the uncertainty of success.
The second step in overcoming the barrier of fear or threat in being evaluated is to remember that “the person is more important than the performance or the product” (Narrow, 1979, p. 185). If the purpose of an evaluation is to facilitate better learning, as in process evaluation, then the focus is on the process.
For example, in teaching a patient how to perform urinary self catheterization using the same clean technique as she will use at home, the educator carefully and thoroughly explains each step of the process, observing the patient's responses (eg . level of attention or nonverbal reactions) during the explanation.
When the patient tries to demonstrate the steps, how ever, she is unable to begin. Why? One answer may be that the use of an auditory teaching style does not match the patient's visual learning style.
Another possibility might be that too many distractions are present in the immediate environment, making concentration on learning all but impossible. And yet another reason may be psychosocial, such as embarrassment about carrying out this procedure. The educator must attend to these teaching and learning factors for education to be successful.
A third step in overcoming the fear of being evaluated on the outcomes of education is to point out achievements, if they exist, or to continue to encourage effort if success at learning has not been achieved. For example, the nurse educator should praise the efforts of the learner honestly, focusing on the task at hand.
Communication of information about why an evaluation is being done is very important for those who are the subjects of an evaluation as much as for those who will conduct the evaluation. If learners or educators know and understand the focus of an evaluation, they may be less fearful than if such information is left to their imaginations,Last, failure to provide and protect certain information about the learner may be unethical or even illegal.
People must be reassured that their privacy will be protected, otherwise they may be unwilling to be evaluated. For example, any evaluative data about an individual that can be identified with that specific person should be collected only with the individual's informed consent. The ethical and legal importance of informed consent as a protection of human rights must be a central concern and responsibility of anyone involved in collecting data for evaluation purposes.
Give your opinion if have any.