Evaluating Nursing Education Assessment Tools

Nursing Education Assessment Project

Coursework early in a nursing education program covers a broad range of topics and extensive amount of details must be committed to memory. Assessments that are directly tied to coursework are primarily formative assessments, which demonstrate the ongoing learning over the period of the course. Formative assessments generally take the form of quizzes and clinical demonstrations of a particular knowledge set recently covered in during a class or classes. Summative assessments are generally used at the end of a course to assess the overall learning that has taken place during the course; summative assessments include final exams or tests, practicum demonstrations, and capstone projects.

The focus of this assessment project is a formative criterion-referenced test of general, fundamental nursing education knowledge. The items used in the test are included in Appendix A -- Nursing Education -- Fundamental Concepts. Twelve individuals were approached to take the exam as though it was an actual assessment of their knowledge, and of the twelve candidates, ten people agree to complete the exam. The general exam is useful for students to gauge their learning and recall of basic concepts acquired during the early phase of their program. A collection of comprehensive tests at this and more advanced levels can serve as a basis for preparing for state board exams. The results of the examination for the ten individuals are shown in aggregate form below.

Planning and Development

To arrive at a comprehensive test of basic nursing concepts, the planning and development process addressed several common areas in nursing education. The objective was to achieve a balance of question items across several different areas of the fundamental nursing knowledge base. A variety of resources were used to arrive at the item selection, including professional standards and competencies, learnings from previous coursework, and personal experience. These areas are as follows:

Examination Procedures -- 32%

Nursing Actions -- 28%

Nursing Terminology -- 4%

Nursing Theory -- 8%

Management & Supervision -- 4%

Legal Liability -- 8%

Patient Rights -- 4%

Medical Records -- 4%

Care Objectives -- 8%

Levels of Blooms Taxonomy were used to organize the test items and to achieve a balance of test item difficulty and knowledge scope. The levels of Bloom's Taxonomy that were introduced in the original taxonomy in 1959 include the following: Application, analysis, comprehension, and knowledge (Bloom, 1969; Anderson, et al., 2001). However, this assessment uses the revised taxonomy from 2001 (See Appendix B -- Bloom's Taxonomy Revised) and instead uses the following categories of the taxonomy, including further delineation of the knowledge category, as follows:

Remember (Recognizing, Recalling);

Understand (Interpreting, Exemplifying, Classifying, Summarizing, Inferring, Comparing, Explaining);

Apply (Executing, Implementing)

Analyze (Differentiating, Organizing, Attributing);

Evaluate (Checking, Critiquing);

Create (Generating, Planning, Producing) (Armstrong, 2015)..

The six cognitive processes are based on knowledge, which is further divided in the revised taxonomy into types of knowledge utilized in cognitive processing, as follows:

Factual Knowledge (Knowledge of terminology, Knowledge of specific details and elements);

Conceptual Knowledge (Knowledge of classifications and categories, Knowledge of principles and generalizations, Knowledge of theories, models, and structures);

Procedural Knowledge (Knowledge of subject-specific skills and algorithms, Knowledge of subject-specific techniques and methods, Knowledge of criteria for determining when to use appropriate procedures);

Metacognitive Knowledge (Strategic Knowledge, Knowledge about cognitive tasks, including appropriate contextual and conditional knowledge, Self-knowledge) (Armstrong, 2015).

Content Outcomes

Remember

Understand

Apply

Analyze

Evaluate

Create

Knowledge

Examination Procedures

1

14, 17

13

12 -- Procedural; 23 -- Factual

Nursing Actions

16

10, 19, 20, 22

15, 21, 25

3 -- Procedural; 18 - Conceptual

Nursing Terminology

2 - Factual

Nursing Theory

4, 5

24 - Conceptual

Management / Supervision

6

Legal Liability

7, 9

Patient Rights

8 - Conceptual

Medical Records

11 - Procedural

TOTALS

4

6

3

3

1

0

25

Percentage of Each of Bloom's Taxonomy Cognitive Domains in the Nursing Fundamentals Assessment Exam

Remember 16%

Understand 24%

Apply 12%

Analyze 12%

Evaluate 4%

Create 0%

Knowledge

Conceptual 12%

Factual 8%

Procedural 12%

Metacognitive 0%

Evaluation

The reliability of a test and the validity of test items are the basis for determining the dependability and trustworthiness of an assessment, with both of these measures playing a pivotal role in professional examinations (Billings & Halstead, 2009). In order to consider a test reliable, there must be evidence of strong test-retest replication or some other demonstration that the test provides consistent scores within in an evidentiary bandwidth of test takers (Billings & Halstead, 2009). The test was administered to 10 individuals in a nursing education program at a major university, such that levels of experience fell within a range of one to two years of nursing education.

Raw Scores Data

Number of students 10

Number of items 25

Maximum point value 100

Highest score 100

Lowest score 76

Median 88

Mean 86.8

Standard Deviation 8.527

Cronbach's Alpha (test reliability) .998

The internal consistency of the assessment was conducted using Cronbach's Alpha in IBM SPSS. The Cronbach's Alpha score attained was .998, which indicates that the question items do measure the same concept. Moreover, the inter-item correlation matrix displays all positive numbers, with the lowest correlation at .991 and the highest correlation, naturally, at 1.00. The Sum of Squares between people was 66963, with a mean square of 2575 (F=3.144, p = .001). The Intraclass Correlation Coefficient (ICC) shows single measures at .980 (95% Confidence Interval, F = 525.4). This was conducted with a two-way random effect model in which both the effects of the people and the effects of the measures are random. And the intraclass correlation coefficients were measured using the definition of absolute agreement.