Appendix B
Design of Instrument

Participants who successfully logged into the Web site completed three major tasks: a 60-item standardized competency examination, a 30-item achievement examination, and 11 demographic questions similar to the NREMT LEADS study.

Standardized Competence Examination

The 60-item standardized competence examination consisted of 60 questions previously used on NREMT-Paramedic certification examinations and decommissioned from NREMT item banks for the purpose of this study. Item performance from 18,859 candidates for NREMT-Paramedic certification who were previously exposed to the identical 60 items was reviewed prospectively. The 60 items were selected based on the current NREMT Test Plan for the NREMT-Paramedic written examination derived from the NREMT Practice Analysis conducted in 1999 (Brown, Dickison, Misselbeck, and Levine, 2002).

In the NREMT Practice Analysis, 744 randomly selected NREMT-Paramedics rated 199 tasks and patient interventions based upon frequency, difficulty, and potential of harm. The data was analyzed by the NREMT Practice Analysis Committee and a test plan for NREMT examinations was developed. This process helps NREMT provide a valid certification process that assures safe and effective practice at entry-level competency based on responses to items that accurately reflect the current practice of the EMS profession by those who are actually providing patient care in the field. The 60 items chosen for the standardized competency examination were selected to proportionately reflect the NREMT Test Plan (4% variance by section permitted) used for NREMT certification as follows.

In addition, items whose performance closely matched the rating of item difficulty as judged by a panel of experts using the Gross Modification to the Nedelsky technique of criterion-referenced standard setting were selected (Gross, 1985; Gross 1989). This process of standard setting was adopted by the NREMT in 1988 under the guidance of Leon Gross, Ph.D., and has been a successful method of establishing the criterion of entry-level competency for NREMT-Paramedics. Items of varying levels of difficulty were selected in each of the six content areas of the competence examination as identified in Appendix B, Table 1. Due to the small sample size, the content domain, and the use of scenario-type questions, no difficult items were selected in the “Airway and Breathing” and “Ob/Gyn and Pediatrics” sections of the examination merely by coincidence. The minimum required criterion or passing score based on the Gross Modification to the Nedelsky technique for these 60 items was 70 percent, the same minimum overall score which is required for NREMT-Paramedic certification for entry level providers.

Appendix B, Table 1

The most important performance indices to compare with the predicted item difficulty criterion based on the Gross Modification to the Nedelsky technique are the difficulty index and discrimination index. The difficulty index, known as the p-value, represents the percentage of the candidates correctly answering an item. One should expect to see decreasing percentages of candidates answering items correctly as the item becomes more difficult so long as the criterion is established in an appropriate manner. Additionally, the discrimination index, known as the r-value, should increase as the items become more difficult. As the item is judged more difficult, the low achievers should miss the item more often than the higher achievers on the examination. Lower achievers are those who are defined as scoring in the lower one-third of the entire population of exam takers. Items that demonstrate a positive discrimination index of 0.20 or higher are very desirable and contribute to the effective measurement of the examination. The average discrimination index of the 60 items used in this portion of the survey was +0.33 (range +0.03 to +0.74).

Ninety percent of the easy items in the examination are expected to be answered correctly using the criterion based on the Gross Modification to the Nedelsky technique. The actual performance on the easy items by these 18,859 candidates is shown here:

Appendix B, Figure 1

r Value – discrimination index
Lower 1/3 – Scorers in lower 1/3 of all exam takers
All – all exam takers

Sixty percent of the moderately difficult items in the examination are expected to be answered correctly using the criterion based on the Gross Modification to the Nedelsky technique. The actual performance on the easy items by these 18,859 candidates is shown here:

Appendix B, Figure 2

r Value – discrimination index
Lower 1/3 – Scorers in lower 1/3 of all exam takers
All – all exam takers

Forty-five percent of the difficult items in the examination are expected to be answered correctly using the same criterion. The actual performance on the easy items by these 18,859 candidates is shown here:

Appendix B, Figure 3

r Value – discrimination index
Lower 1/3 – Scorers in lower 1/3 of all exam takers
All – all exam takers

There is very high correlation with the predicted criterion and mean performance on easy, moderately difficult, and difficult items of the 60-item standardized competency examination:

Appendix B, Table 2

Achievement Exam

Thirty achievement items were then presented to the respondent. These items were drafted by a select group of experienced EMT-Paramedic educators and providers with the sole purpose of writing very difficult items in order to spread the scores of the group. The achievement items had never been developed for use on NREMT-Paramedic certification examinations. No pilot data or previous performance by candidates for certification were available for the 30 achievement items. Five achievement items in each of the six examination content areas (see Appendix B, Table 2) were developed for use in the survey to balance content and keep the survey instrument to a reasonable length.

Demographic Questions

The final portion of the survey consisted of 11 demographic questions. These questions were designed to determine the respondent’s educational background, length of time performing as an EMT-Paramedic, call volume, and continuing education. Where possible, identical questions from the NREMT’s LEADS (Longitudinal Emergency Medical Technician Attributes and Demographics Study) project were asked to provide linkage to the EMT-Paramedic population at large (NREMT, 2000). The NREMT is currently in the fifth year of a 10-year longitudinal study of attributes and demographics of EMT .