Background
Deriving measures of policy effects from policy documents requires accurate information on what policies were in force, when they took effect, and what provisions they contained. Government publication and preservation practices can make this history difficult to reconstruct. This study presents and evaluates a workflow for constructing longitudinal policy measures in one national policy series.
Methods
A workflow combining document recovery, automated extraction with human validation, and construction of longitudinal measures was applied to NHS England’s Additional Roles Reimbursement Scheme (ARRS) from 2019/20 to 2026/27. Fourteen policy versions were split chronologically into eight development versions and six later versions used to evaluate keyword, deterministic, and large language model (LLM) extraction against an author-verified reference. Three simplified measurement strategies were compared with the validated longitudinal record.
Results
Historical reconstruction recovered 14 substantive policy versions across eight years, while the NHS England landing page exposed documents from only one policy year. Reimbursable workforce categories increased from 5 to 27 between 2019 and 2026. On the evaluation versions, the LLM matched all scored targets for eligibility status, effective dates, and reimbursement amounts, while eligibility conditions were less accurately extracted. Simplified measurement strategies altered the reconstructed exposure history. Using one policy document per year credited 96 category-months before eligibility began, equivalent to 7.3% of the 1,320 eligible category-months, while keyword coding produced 138 category-month errors, equivalent to 10.5%.
Conclusion
In this policy series, the workflow recovered a broader historical record than current web access alone and extracted predefined variables accurately, yet accurate extraction of eligibility conditions and new policy content presented challenges. Valid longitudinal measurement required historical source recovery, explicit coding and dating rules, evaluation on later versions, and human verification. The workflow can offer a structured approach to measuring changes in governmental decisions about eligibility, financing, and conditions over time.