|Action||Three Waivers (ID, DD, DS) Redesign|
|Comment Period||Ended on 3/31/2021|
Section 12 VAC 30 – 122 – 200 requires the use of the SIS and provides some specific requirements for use of the scores generated by that instrument; unfortunately, it does not provide specific regulatory requirements for the implementation of the SIS and overly limits the areas to be considered when applying the results of the instrument.
The amount of skepticism that should be directed towards the Virginia system for implementation of the SIS based on this analysis is significant; as there are clear inconsistencies between the Virginia system and the tested system in the users manual. Interestingly, all have the impact of lowering an individual score:
The Virginia system for implementing the SIS creates significant, distinct and meaningful differences between the scores and level assignments generated by the Virginia system and those that would be generated using the procedures in the tested users manual. These different results as proven above cannot lay claim to the same degree of accuracy, reliability or validity as the SIS which uses the user manual. In fact, when contacted and directly ask about the dominant activity approach an aaidd representative could not identify any study where the dominant activity approach to scoring had been employed much less one where it was proven accurate, reliable and valid and when pressed stated “perhaps some of the international studies”; likewise the representative was unable to identify any information in the public domain or peer-reviewed articles about the dominant activity approach. Regardless of what the international studies show, B. Rammstedt (citation above page 5) indicates these results are not readily transferable to the Virginia population.
Preemptively, because the state provides no opportunity for rejoinder, occasionally state representatives indicated that the instrument is “robust” in response to criticisms; which would be a valid response were it true. However, robustness can be tested comparatively, statistically and empirically but there is absolutely no direct evidence that these verifications for robustness have ever been attempted/completed for the changes implemented by the Virginia system. Additionally, given both the direction and the magnitude of changes from the users manual indicated above, the claim of robustness would fail to even meet the minimal non-statistical requirement for robustness which is provided by T. Plumper and E. Neumayer in their work Robustness Test and Statistical Inference; “most applied scholars even today define robustness through an extreme bounds analysis: a baseline model estimate is robust to plausible alternative model specifications [i.e. Scoring changes] if and only if all estimates have the same direction and are statistically significant.” As the analysis above made painstakingly clear the Virginia system changes the direction in the specifications and any direct statistical comparison between results generated by use of the user manual and the Virginia system would find statistically significant lower scores from the Virginia system; proving that claims of robustness are not applicable to the criticisms in this analysis. Finally, it is important to consider the source aaidd is making millions off of the system by keeping the State customer happy and providing the State the smokescreen of a robustness response without any empirical testing or delineated rationale to support the claim should be greeted with more than a grain of salt.
Recommendations –1st the State could demonstrate felicity to the model they continually use as a justification for this regulation and implement the SIS with strict adherence to the user manual – 2nd the State should recognize the existence of changes from the tested/proven user manual and provide a justification, rationale and empirical evidence for the appropriateness of these changes; making adjustments in level assignments as warranted from the information discovered in this analysis and verification of their use of the instrument – 3rd regulatory protections for individuals subjected to the SIS for determining their level of support needs and hence resources, should be provided directly in the regulation to prohibit any changes in the scoring system that are not verified as appropriate and prevent the implementation of future changes without being subjected to an appropriate system of empirical evaluation and meaningful checks and balances – 4th the prohibition against the individual and respondents having even a blank paper and pencil should be rescinded to promote the preservation of data integrity which is essential as indicated by the Department of Health and Human Services guidelines for Responsible Conduct in Data Management “regardless of the discipline, comprehensive documentation of the collection process before, during and after the activity is essential to preserving data integrity”; without this change an independent check to preserve data integrity will not be possible.
The State has gone to great lengths to sell the SIS and these regulatory changes by dressing them up in the language of scientific certainty and the cloak of reliability and validity---BUT WHAT WE WERE PROMISED IS NOT WHAT WE GOT.