In 12 VAC 30 – 122 – 200, the regulations establish the SIS as the key component for establishing individual support needs and funding levels. The State has placed excessive confidence in the SIS which coupled with a meaningless appeals process places a significant number of individuals receiving waiver services at risk for mis-evaluation of their support needs and hence an unfair reduction in their support resources without any meaningful recourse under the existing system.

1. The SIS score is not settled scientific certainty. For years now state employees with the task of selling the SIS have attempted to give the instrument an air of scientific certainty by reflexively repeating the mantra of “proven reliable and valid” and occasionally throwing in academic reinforcers (“multidimensional”, “comprehensively”, “norm-referenced”, “nationally tested” etc.), whenever a SIS question or concern is raised and used this “certainty” to close any additional discussion. This level of deference to the instrument is sorely misplaced as Notre Dame Professor G. Gutting explains in the 5/17/12 issue of The Stone: “social sciences may be surrounded by the paraphernalia of the natural sciences, such as technical terminology, mathematical equations, empirical data and even carefully designed experiments… [however] most social science research falls far short of the natural sciences standard of controlled experiments… Above all we need to develop a much better sense of the severely limited reliability of social scientific research.” Empirical verification for how skeptically these instruments should be viewed is reported in the 9/24/16 issue of The Economist (pg. 75 – 76), which provides a history and meta-analysis review that finds social science research (especially that which allows the publisher to prosper) has an extremely low rate of statistical power and that replication is “incapable of correcting the situation”. This criticism is directly applicable to the SIS as an instrument as the replication results touted for the SIS are typically apples to oranges comparisons that are insufficient to validate the SIS; Michael Wehmeyer et. Al. Made this clear in The American Journal of Developmental Disability January 2009 (page 12); “First, until there are other widely available indicators of support need, it is difficult to determine the validity of the tool such as the SIS, other than looking at the assessments relationship with and difference from measures of adaptive behavior and tools such as the DDP.” Clearly, as a social science instrument the SIS claim of scientific certainty should be viewed with a great deal of skepticism; however, the state has confidently claimed that the program has been completely successful based on scant evidence that is extremely overly optimistically interpreted. This claim to success can be found in the Virginia Register of Regulations 2/4/19 “an analysis of data and SIS administration procedures highlight that the distribution of support needs levels, while not identical are consistent with the model predictions from 2014… Thus, the waiver redesigned appears to be successful in identifying individual support needs”. This claim should be viewed with serious skepticism; 1^st if the SIS is to be believed then the original HSRI guesstimate was clearly skewed to the downside overestimating the number of individuals in the lowest 2 levels and significantly underestimated the number of individuals in the highest 3 levels making it a flawed basis of comparison; 2^nd the statement “while not identical are consistent” simply strains credulity when compared to the October 1, 2017 report to the Virginia Gen. assembly on the impact of implementing the SIS, which stated that in the area of residential supports the actual distributions matched in only 3 of 7 categories for group homes, 4 of 7 in sponsored residential services and 2 of 7 in supported living services--- how is a match in only 9 of 21 opportunities “consistent”; the results were even worse for day support with only 1 of 7-- making the grand total 10 consistencies in 28 tries – they claim is exposed as farcical. Thus, it should be clear that the use of the SIS should proceed cautiously, with a great deal of skepticism and adequate safeguards to protect the individual during the use of this instrument by the State.
2. Errors in identifying the actual support needs of a specific individual are guaranteed to occur in Virginia. The instrument itself testifies that some percentage of errors will occur and only claims a 95% probability of accuracy at best, openly admitting that some individuals will be mis-evaluated. The way the system is being administered in Virginia serves to increase the number of individuals who will be mis-evaluated in 2 ways: 1^st the SIS is being used for individuals over 72 where the SIS doesn’t even make a claim to reliability or validity and 2^nd the state procedures do not capture the entire 95% accuracy claimed because the 95% accuracy refers to a range of numbers and the State uses the middle number not the highest number in the range thus sacrificing any claim to the full 95% accuracy rate. It is important to note, that these guaranteed errors will occur even when all of the SIS procedures have been strictly adhered to and thus these errors are not subject to the current appeals process. Additionally, the State has adopted procedures/practices that all but assure additional errors will be present in the Commonwealth’s implementation of the SIS; one example, is the prohibition against respondents having any materials with them during the assessment which includes even a blank piece of paper and a pencil to record the agreed-upon rating that is determined during the assessment. This practice assures additional error will creep into the system for 2 reasons: 1^st data entry errors are statistically guaranteed given the large number of entries per person times the number of persons evaluated; however, the restriction prohibits the recording of the numbers to provide a double check once they are reported so the system has 0 controls for the data entry errors which are sure to occur; 2^nd the absence of relevant notes and the pressure for timely completion of the lengthy instrument all but assure lapses in memory and premature moving forward that will result in the respondent not providing all the relevant information in a particular area again resulting in unchecked error. Based on this analysis and the total number of individuals the SIS is being administered to well over 1000 Virginians with developmental disability have been or are at risk of being mis-evaluated and assigned SIS scores in error.
3. The regulations establish a system that is overly reliant on the SIS score for assigning resource levels to individuals. There is a clear consensus in the relevant literature that the SIS score is an insufficient justification for assigning a particular level of resources to an individual. Prof. Michael Wehmeyer et. al. (Citation above) states “one should not expect that the SIS, or any other singular supports assessment instrument, will explain 100% of the variance in funding required to provide these supports.”; an extensive review of the SIS by, Yuwadee Viriyangkura in the Illinois State University theses and dissertations 8/27/13 states that “a funding formula should not be based on a single assessment” and even the authors of the SIS (who are making millions off of it) are very clear in the user manual (page 89) that due to “the complexity of cause-and-effect relationships among support needs, funding levels and outcomes… SIS data will provide only one piece of the puzzle.” While Virginia has included other elements in their funding formula, their limited application and lack of relevance to the individual has created an over/total reliance on the SIS score that all relevant authors agree is extremely poor public policy and will result in mis-identification of needs with unfair denial of resources to individuals. SIS proponents will surely point to the “Virginia supplemental questions” on medical and behavioral needs as a response to this criticism; however, this response should be rejected for several reasons: 1^st – the original SIS which they point to as reliable and valid did include supplemental medical and behavioral sections and even with these all of the authors cited above found it inadequate; again it strains credulity that the simple rewording or addition of items in the Virginia version would be sufficient to address the criticism; 2^nd – the Virginia supplemental questions only figure into the funding formula for an individual if they are found to have “exceptional needs” and are therefore only applicable to individuals in levels 3, 6 and 7 which represents only 18.3% of the total according to the 10/1/2017 report to the Virginia Gen. assembly (cited above) – the result is that 81.7% of all individuals subject to this regulation will have their SIS score be the sole determinant of their assigned level and thus funding tier; there are simply no other inputs at all into the assignment under current practices. It’s possible, some may argue that group home size, type of program and both residential/day reimbursement are also considered in the funding formula; however, these factors are irrelevant to this discussion because it is the level and tier that results from the sole SIS score that dictates the allocation of resources for the individual across all of these other funding elements. Thus, despite warning after warning that sole reliance on the SIS score is poor public policy and will result in the mis-evaluation of the needs and unfair denial of resources to individuals the State has made this a reality for 81.7% of all developmentally disabled Virginians in their programs. One clear example of this concern that has manifest is the one point tier jumpers that move back and forth between a score of 30 and 31 and thus back and forth between tier 2 and tier 3 which has significantly different resource allocation levels, even though there has been no change in the support needs or circumstance of the individual and is likely to change back during the next SIS evaluation; meaning that one of the evaluations is in error, anecdotal evidence would indicate there’s a lot of these but no data could be found one way or the other.
4. The appeals process identified in the regulations is inherently flawed and provides no meaningful recourse to individuals who have been mis-evaluated and/or unfairly denied resources due to their SIS score. The current process only allows appeals based on procedural/process errors and does not permit an appeal of the specific score. As the analysis above clearly indicates even when all of the procedures/processes are strictly adhered to (thus there is no basis for appeal) error, mis-evaluation and wrong scores will be generated and specific individuals will be unfairly denied an equitable allocation of resources with absolutely no recourse whatsoever. Additionally, the practice of prohibiting individuals/respondents from having even a pencil and paper during the assessment precludes them from evaluating whether or not the procedures were followed (without a checklist can you really remember if all of the questions were asked during the two+-hour ordeal) and prohibits them from taking notes to document the procedural error as required by the State to even initiate the appeal; severely restricting the likelihood of successful appeals even when they are warranted. Remember this even precludes the possibility of catching and appealing a direct data entry error which is statistically guaranteed to occur with no recourse for the individual unfairly impacted.
5. Recommendations: 1^st – I concur with other commentators that individuals over 72 years of age (where use of the SIS score is admittedly invalid) should be assigned tier 4; 2^nd – the highest score in the range of scores which have a 95% probability of accuracy should be used in calculating an individual’s SIS score to capture as much of the accuracy of the instrument as possible; 3^rd – a range of additional factors should be used in the calculation of an individual’s level and thus tier, especially for the 81.7% of individuals who do not exhibit extraordinary medical and behavioral needs and thus have their resource level now determined solely by their SIS score; Y. Viriyangkura (citation above) identified a number of States who are using the SIS instrument but have adopted a wide array of additional factors in calculating and assigning resource levels to individuals that address the consensus concern with overreliance on the SIS score, if these states get improved accuracy and error reduction with these procedures why can’t we; 4^th – the State should permit by regulation individuals/respondents to have notes and the ability to document during the actual assessment to provide a check on data entry errors and provide the information necessary to pursue an appeal; 5^th – additional social science instruments should be used in cooperation with the SIS, Julia Harries of the University of Adelaide reported in Supports Needs Assessment of Individuals with Developmental Disability 9/2008 that dual approaches have been shown to improve the capacity of the SIS and indicates they may have utility “when the level of funding derived using a supports needs measure is disputed or results in the individual falling on the cusp of a funding band level”; such an approach may be able to mitigate the indicated concerns without the need for formal appeals; 6^th- provide a meaningful appeals process where an individual is able to directly appeal the SIS score, for logistical purposes this appeal right may need to be limited to individuals where there is only a one point difference that results in tier jumping back and forth but at least this would provide some protection and provide an avenue for redress for individuals in close call situations where error is more likely to have a significant impact.

As is this regulatory set constitutes the State sticking its head in the sand and pretending that there are no SIS errors and based on this willful blindness foreclosing any meaningful avenue of redress for those individuals who are guaranteed to be mis-evaluated and unfairly denied equitable resources based on the States overreliance on the SIS score – this should not be allowed given the significant numbers of individuals who will be impacted this way and the fact that if even just one person is denied justice it harms them and reflects negatively on all of us.