November 2009 – As part of its policy of continuing improvement and exam oversight, the National Commission for the Certification of Crane Operators conducted a Practical Exam Auditors continuing education seminar in September at H&E Equipment in Salt Lake City. The seminar provided initial training as well as updates to NCCCO's Practical Exam Audit Program. Brett Petersen of H&E Equipment offered the site’s training facilities as well as the permanent site cranes for auditor training activities.
NCCCO auditors are responsible for conducting performance reviews of Practical Examiners as well as site inspections to ensure the fairness, integrity, and safety of practical exams leading to CCO certification. In doing so, they also provide the impartial witness critical for fulfilling the accreditation requirements of the National Commission for Certifying Agencies (NCCA) and the American National Standards Institute (ANSI).
At the September training session, NCCCO Manager of Program Development Phillip Kinser presented numerous updates on the various CCO programs. The Auditor Program requires annual training as part of NCCCO’s continuous improvement program. Among important new requirements: auditors must now provide photographs of audit discrepancies. Other fine-tuning includes new restrictions on auditors performing both tests and audits simultaneously, and annual audit minimums for maintaining their auditor status.
NCCCO Practical Test Auditors were briefed recently on CCO program enhancements aimed at ensuring the continued fairness and integrity of the testing process.
Kinser explained that NCCCO maintains an active and ongoing auditor training program for numerous reasons. “All CCO programs require rigorous practical exams, and we require our Practical Examiners to undergo thorough training and auditing to understand the intent of psychometric principles such as validity, reliability, bias avoidance, and Behaviorally Anchored Rating Scales (BARS),” he said. “The auditing process provides feedback to the governing program committees and accrediting bodies (ANSI and NCCA) that the issues of test integrity, fairness, and adherence to exam procedures and policies are being followed.” This process was aimed, Kinser added, at ensuring that candidates seeking certification through NCCCO's programs are provided psychometrically fair and reliable exams.
Substantial time was spent at the seminar reviewing issues typically noted during site audits as well as coaching the auditors on appropriate responses. Examples include covering how to tally scores correctly; ensuring that participants are wearing appropriate personal protective equipment; and setting up the practical exam course precisely in accordance with the provided specifications. "This continuing training,” noted Kinser, “is intended to ensure confidence in the inter-examiner reliability of our examiners as well as the inter-auditor reliability of auditors.”
Practical Test/Audit Program Coordinator Erin Jones organized the event. “By keeping everyone on the same page regarding program improvements,” she noted, “we can be sure that we are fulfilling NCCCO’s mission to protect all stakeholders and we are delivering on that promise. The Audit Program brings our accountability full circle and improves the ability of the CCO certification process to reduce accidents, save lives, and generally make the construction industry and lifting operations a safer environment for all to work in.”
Practical exam sites and/or practical examiners are selected for audits based on criteria such as random selection, unusually high or low pass rates, registration of a new site or in response to issues reported by candidates. Audits may be scheduled with or without prior notification. It is a condition of Practical Examiner accreditation and Test Site status that all site personnel, including the Test Site Coordinator, Practical Examiner, and Proctor cooperate fully with the NCCCO auditor during the audit process.
“In many ways, NCCCO’s Audit Program is modeled on ANSI’s on-site assessments, which are conducted as part of its accreditation process,” commented NCCCO Executive Director, Graham Brent. “As NCCCO enters its third year of accreditation to the international standard ISO 17024, we have had plenty of opportunity to use—and take lessons from—the ANSI process established to ensure that all NCCCO procedures conform to the essential requirements of test fairness and integrity.”