FMCSA may face trouble in assessing carrier safety scores

One recommendation FMCSA has been addressing is that it use a statistical model known as item response theory (IRT) to replace its current safety measurement system (SMS) for prioritizing carrier safety interventions. FMCSA confirmed in the OIG report that it plans to complete testing of the IRT model by September 30 and decide whether to adopt it by September 2020.

But FMCSA did not fully respond to recommendations to improve SMS transparency and carrier safety rankings, according to the OIG. While the Agency acknowledged benefits to making SMS data publicly available on the web, “FMCSA neither explained how these would be user-friendly nor included FAST Act-required estimated costs or an implementation plan,” the report states.

FMCSA’s action plan also lacks details on how it would improve how it assesses motor carrier safety rankings – by using percentile rankings and relative and absolute measures, for example – to help determine which carriers receive safety alerts, the OIG contended. It noted that the agency plans to address this once it decides whether to adopt the IRT model. However, “until the Agency makes this decision, it may face delays in making its [Motor Carrier Management Information System] database, safety measures, and rankings more transparent – whether it continues to rely on SMS or turns to the IRT model.”

In a response letter to the OIG dated September 5, FMCSA Administrator Ray Martinez stated that his agency partially concurred with OIG recommendations to provide certain staffing, enforcement, and data collection costs associated as recommended by NAS.

However, Martinez said his agency “has no immediate plans to collect data on carrier exposure and additional crash data,” after reaching out to the industry and determining that much of the data doesn’t exist.

“Upon completion of the IRT modeling by September 30, 2019, FMCSA will decide how to move forward with its prioritization methodology, including providing estimates and benchmarks to complete the remaining open NAS recommendations,” Martinez stated. “FMCSA plans to make this decision by September 30, 2020, after completing a full review of the IRT model.”