Sequential Monitoring of Item Parameter Drift

Ongoing surveillance of item parameter drift for continuous testing programs and pre-equated item banks. Estimates item difficulty in rolling calibration windows, runs sequential cumulative sum (CUSUM) detection (Page, 1954, ; applied to testing by Veerkamp and Glas, 2000, ) with change-point estimation, separates gradual drift from abrupt jumps, tunes alarm thresholds for a bank-wide false-alarm target by simulation on the program's own design, quantifies score and pass-rate impact, and records recommended actions in an audit log.

driftwatch

Not "did it drift?" but "when, how, and what should we do?"

Most drift checks compare two calibrations at equating time. Continuous testing programs and pre-equated banks need ongoing surveillance. driftwatch borrows sequential change detection from statistical process control.

library(driftwatch)

sim <- dw_simulate(seed = 1)                         # or your responses + bank
est <- dw_estimate(sim$responses, sim$bank)          # item x window estimates
tu  <- dw_tune(est, target = 0.01)                   # threshold for 1% false alarms/item
mon <- dw_monitor(est, h = tu$h)                     # CUSUM + change points + type
mon
dw_actions(mon, anchors = my_anchor_ids)             # recommendations + audit log
dw_impact(form_ids, sim$bank, current_b, flagged)    # score / pass-rate impact

Installation

From CRAN (once released):

install.packages("driftwatch")

Development version from GitHub:

install.packages("pak")
pak::pak("edidatasolutions/driftwatch")

How it works

1. Window estimates. Rasch difficulty per item per window, with examinee ability known from operational scoring. Standardized deviation from the bank: z = (b_t - b_bank) / sqrt(SE_t^2 + SE_bank^2).
2. Two-sided CUSUM on z (reference value k, threshold h).
3. Threshold tuning on your own design. dw_tune() regenerates responses under no drift for the program's actual items, windows, sample sizes and examinees, then reruns estimation and CUSUM. This matters: each item's bank error is shared by every window and accumulates in the CUSUM. The design-based threshold (h ≈ 9.4) is far above the iid-normal one (≈ 6.9), and only the former hits the false-alarm target.
4. Gradual vs abrupt. Step and ramp models are fitted with a profiled change point, and the log likelihood ratio decides between them. With weak evidence the type is undetermined rather than guessed.
5. Impact and action. True-score scoring impact of keeping, removing or recalibrating flagged items. Actions are retire, recalibrate or remove from the anchor set, each with a written rationale, threshold settings, analyst and timestamp.

Validation (known truth, 5 replications, inst/validation/known_truth.R)

300 items, 40 windows, ~80 responses per item per window. 10% of items drift gradually (0.02–0.06 logits/window) and 5% jump (0.4–1.0 logits).

Drift-type classification:

Gradual drift is hard to type soon after an alarm, because a short ramp looks like a step. Reclassify as windows accumulate.

Score impact (60-item form with 12 drifted items, cut at theta = 0.5): keeping banked values mis-states the pass rate by -1.3 points; recalibrating flagged items cuts that to -0.2 points, and removing them to -0.15.

Status and assumptions

Done: dw_simulate, dw_estimate, dw_tune, dw_monitor, dw_impact, dw_actions, dw_twopoint. Rasch difficulty only; examinee ability treated as known. Next: 2PL discrimination drift, ability uncertainty, anchor-set re-linking after removals, and a per-item run-length (ARL) view.