Case study: cognitive errors in court experts’ assessments in autism spectrum cases

PURPOSE

In the Polish legal system almost every person diagnosed with autism spectrum disorder, even though they are most of the time holders of a disability certificate, must appear before a medical council of the County Committee for Disability Assessment within a prescribed time limit to determine the kind of support they are entitled to from the state relative to their disability. Unfortunately, experts who are medical examiners ruling for these committees frequently lack the knowledge or experience necessary to evaluate autism or its consequences for the patient’s every-day functioning. which is often undetectable at the first meeting. In many cases, the evaluated patient is denied the right to receive the services that he or she needs and is entitled to. If, based on a short survey and an interview, the medical examiner issues an expertise stating that a patient does not require the state benefit – even if it is in contradiction to the assessments submitted by the Therapeutic Team, which is a team of specialists involved in the process of the patient’s therapy on a daily basis – many new practical problems arise for the individual.
Such official denial of benefits may be appealed against at the District Court, which appoints expert court witnesses to re-assess each case. Although at this stage, experts are selected on the basis of their specialism, they are often not sufficiently competent to evaluate a disease or disorder with which they are not in daily contact [1]. In such cases, the expert’s assessment is given under conditions of strong uncertainty due to a) limited medical knowledge b) limited time spent on an observational survey, c) highly variable individual ASD symptoms that often deviate from the common or stereotypical ideas, and d) very limited knowledge of the patient being examined. Numerous studies have shown that people usually considered as experts in their field are not fully immunized to simplified rules of inference (i.e. heuristics) or stereotypes [2-10]. As Shanteau points out in his research, because of this high degree of uncertainty, psychologists and psychiatrists belong to a group of experts who are susceptible to making relatively frequent errors.
Such mistakes made by experts ruling in ASD-related cases can have very serious, negative consequences for the lives of people on the spectrum, such as reducing their access to the much-needed state benefits. For this reason alone, identifying such mistakes and uprooting their causes, as well as making experts in psychiatry and psychology aware that they are susceptible to making such mistakes, is the first step towards overcoming these difficulties. In order to identify the contradictions and cognitive errors in court experts’ assessments, a Qualitative Comparative Analysis was conducted on a patients’ medical and court documentation consisting of two sets: the assessments by the group of specialist therapist working with a patient on a daily basis were compared to those made by the court experts on the basis of a short observational survey.

CASE DESCRIPTION

Patient’s profile

The patient whose case is discussed in this article is a 5-year-old boy of Polish nationality with normal development until the age of 4 months (eye contact, social smile) and typical motor development (crawling stage omitted). Later, his development as well as non-verbal and verbal communication stopped; the boy’s social smiling and responding to play were withdrawn. At the age of 12 months, he was reported for the diagnosis of autism spectrum disorders. The formal diagnosis was made when the child was 18 months old. The boy did not communicate or make contact; he did not play with others being concentrated on objects rather than people. He showed many types of repetitive behavior and responded to frustration by hitting his head against the floor. After the diagnosis, he was given psychological therapy using elements of the Early Start Denver Model. By the time he was 2 years old, a gesture of pointing, imitating sounds and the first words appeared. Following this development, therapy was carried out for one year using the 3I method. Further social-emotional development was delayed; however, the boy reached his next developmental stages. At the age of 4, he started pre-school education in the integration group of a non-public Montessori nursery under the supervision of psychological counsellor, speech therapist and sensory integration therapist. Currently he attends an integrative class in a public school, and his cognitive development is above average.

Context

During the meeting of the County Commission for the Assessment of Disability, the patient was denied a proper disability certificate, which would take into account his actual level of functioning. Thus, his family was denied the possibility to receive financial support from the state. The patient’s family appealed against the decision three times, to increasingly higher court levels, and had court expert witnesses appointed (a psychiatrist and a psychologist) to re-assess the case. The documentation created in this process served as the material for analysis presented in this paper. In order to identify contradictions and cognitive errors in court experts’ assessments, a Qualitative Comparative Analysis was made of the patient’s medical and court documentation, consisting of two sets: the assessments of the group of specialists working with the patient on a daily basis were compared to those of the court experts which were based on a short observation survey. The study involved a comparative analysis of the medical and court documentation of a patient with ASD which was used in the procedures determining the level of his disability. The goal of the research was to identify the potential cognitive errors made by the evaluating experts.
In order to identify contradictions and cognitive errors in the expert assessments, the Qualitative Comparative Analysis (QCA) [11-12] was applied to the two sets of documentation. The QCA method allows for the in-depth analysis of the investigated phenomena, taking into account the context in which they occur [13]. These were the assessments of court experts and physicians issued as part of the legal case described above. These were subjected to a qualitative comparative analysis with the assessments of the Therapeutic Team which involved a number of specialists, i.e. psychologists, speech therapists, psychiatrists, sensory integration specialists, etc. who worked with the patient on a daily basis, knowing perfectly well his individual specificity.

The documentation analysed

The documentation was divided into two sets consisting of: a) Therapeutic Team1 documentation, and b) external experts documentation (District Court files, court expert witnesses, medical examiners, disability committees)2. The Therapeutic Team’s documentation was supplemented with an interview with the patient’s family and with video recordings. These two groups were compared to determine the reasons why the external court and committee experts had denied the patient his right to state aid and whether they had made any errors in their judgments. The benchmark and point of reference were the assessments of the Therapeutic Team, as these which were developed by a team of autism-related specialists who worked closely with the patient. The documentation analysed in this study consisted of:
1) 14 Therapeutic Team assessment (total of 26 pages);
2) 9 court, committee and expert witness assessments (total of 33 pages);
3) 3 disability certificates (total of 3 pages);
4) 2 functional diagnoses (total of 5 pages).
The subject of the analysis in the above-mentioned documentation were the statements contained therein, both by the Therapeutic Team and by external experts, on the patient’s health condition, the severity of his disorder, the way he functioned, the methods of observational tests by the experts, and statements determining whether the patient required assistance in meeting his basic needs in life in a manner exceeding the scope of care for a healthy child.

COMMENT

Because of the short time available to examine each patient, the assessments and decisions made by experts during the legal proceedings or at meetings of disability committees are often based on intuitive decisions [14-17]. As shown by many studies conducted since the publication of the famous article by Kahneman and Tversky [18], simplified methods of inference are used by all people when making intuitive judgments.
An analysis of the aforementioned court and medical documentation has shown that problems related to patient evaluations by medical examiners and expert court witnesses include:
1) insufficient length of the observation survey and evaluation of the person with ASD;
2) lack of specialist knowledge;
3) too much confidence in personal expertise: overconfidence bias [19];
4) relying too much on information collected during a short observation (the error of the “what you see is all there is” type – WYSIATI) [20];
5) selective use of information obtained during the case and from the Therapeutic Team’s assessment which fits better with a negative outcome.

All of these factors often result in mistakes being made by experts, such as:
1) The issue of an assessment that understates the severity of the patient’s disorder when symptoms visible during the observation (usually lasting several minutes instead of 5-8 hours, as is needed for a full autism diagnosis process) do not fit into the stereotypical image of a person with autism (representativeness heuristic bias). As many researchers indicate, the symptoms of ASD can greatly vary, from a complete withdrawal from interpersonal relationships to their unusual intensity, which is, however, not accompanied by appropriate cognitive effects.
The patient’s documentation indicated the presence of aggressive behaviours (hitting, kicking, biting, throwing objects), tantrums that were difficult to pacify and lasted up to an hour, hypersensitivity to hearing and touch (the child sometimes cried in response to an unexpected, even delicate touch), problems with concentration and creating shared attention. According to the therapists and the patient’s caregivers, these problems were aggravated when the child had spent more than 4 to 5 hours four days a week in the nursery, due to sensory overstimulation in intense environment. As a result, the patient’s sensory integration therapist included recommendations for longer absences from preschool to allow the child’s sensorily overloaded nervous system to recuperate.
Referring to these indications, the medical appraiser appointed by the court of primary jurisdiction concluded that “such behaviours are also typical for healthy children”, and therefore they do not call for the need for additional childcare. However, research conducted on the behaviour of children with autism shows that they occur much more frequently, last much longer, and are less often subject to spontaneous withdrawal than in neurotypical children. The committee’s examination did not take into account the fact that the behaviours described in the patient’s file may be much more severe than indicated by its observation.
2) During their brief examination, the court experts focused on the areas of functioning that were stereotypically assessed as affected by ASD, rather than on the actual areas in which the patient had significant deficits. This means that the experts focused on determining whether the patient corresponded to the stereotypical perception of an ASD person, for example whether they spoke or engaged in interpersonal relations. An expert guided by a stereotype of ASD person as not using speech, and being withdrawn from any form of contact, might misjudge the actual level of severity of ASD disorder or question the diagnosis without giving it a proper consideration. Such evaluation method also misses those areas in which the patient actually has difficulties if they do not fit in the typical image of an ASD person.
In the case described here, the patient’s documentation indicated difficulties in functioning such as problems with using the toilet, with dressing and undressing independently, eating, relationships with peers, physical and verbal aggression, auditory and touch hypersensitivity, and other issues listed in the “patient’s profile” section.
The observational examination conducted by the appraiser of the original court consisted of asking the child two questions: “What is your name?” and “Do you go to the nursery and do you have friends there that you like?”. The questions did not refer at all to the spheres in which the child’s problematic behavior was reported. Instead, they addressed some areas stereotypically identified with ASD, such as lack of speech skills and absence of interest in interpersonal contacts.
Furthermore, the preschool psychologist’s comment in the patient’s file – “He is unable to establish relationships with children, although he shows increasing interest in them. His relationships are superficial, always expecting others to follow his rules” – indicates that the fact that the child had contact with other children in the nursery did not mean that he had no problems with social interaction. Therefore, the medical examiner’s single question on this topic could not have been sufficient for the appraisal of  the extent of the boy’s disorder.
3) The issue of internally contradictory assessment. During a short examination by a medical examiner or court expert, a child diagnosed with autism might not exhibit the typical symptoms for this type of disorder, even in the most sensitive areas, the so-called diagnostic triad3. This does not even give the expert any grounds for concluding the presence of spectrum disorder, which may be in an apparent contradiction to the assessments by the specialists working in the Therapeutic Team. Failure to notice or ignore such fundamental internal contradictions in the assessment may result from the so-called belief bias [21], among other things. The assessment by the experts in the court of second instance was contradictory because certain stereotypical behaviours and communication difficulties, which are two of the three most important diagnostic criteria for autism, were not observed during the examination. Nevertheless, the experts’ assessment confirmed that the child had ASD even though it was not their task. This shows how selective the experts were in treating the evidence provided by the Therapeutic Team. On the one hand, they did not question the diagnosis made by the Team, but on the other, they did not consider the child’s difficulties in satisfying the basic needs of his life as reported by all members of the Therapeutic Team.
In response to the objections of the patient’s parents to the assessment issued, the experts stated that “During a single ambulatory examination the child may not have presented behaviours typical for autism. Knowing this, the experts relied on the available documentation and did not question the previously made diagnosis of childhood autism.” If the experts were aware of the diagnosis, it is all the more surprising that they denied the presence of the child’s challenging behaviours as described in detail in the Therapeutic Team’s assessment.
4) The tendency to challenge the findings of the Thera­peutic Team that cares for people with ASD on a daily basis because of the WYSIATI effect. Some expert assessments completely ignore the assessments by external specialists working with children on the spectrum. They only consider the information collected during a short observation study, while denying the possibility of the occurrence of behaviours utterly different from those observed at the time of the examination. The fact that the patient’s behaviour may differ on a daily basis is not taken into account at all, even though the expert’s own examination is a one-off assessment lasting only several minutes. During that time, the person examined may, for various reasons, show behaviours that are unusual for him or her. However, the expert’s susceptibility to the heuristic and confirmation or belief bias, might mean that eventually the assessment by better-informed specialists is disregarded.
According to the interview with the child’s mother, the appraiser’s assessment in the court of first instance stated that she not only disbelieved that the child experienced the problems described by the Therapeutic Team but even that the child had autism spectrum disorder at all. Consequently, the medical examiner denied the child’s mother the right to receive care benefits so that she could continue to provide intensive daily care for the child. This decision was issued contrary to the recommendations of all the members of the Therapeutic Team, who unanimously stressed in their assessments that the mother’s participation in the child’s therapy process was essential and that it was necessary for her to provide constant care for the child. The experts in the court of second instance stated that the child was “communicative, bold and cheerful”. However, neither the Therapeutic Team nor the interview conducted with the child’s mother revealed any adaptation difficulties – in fact, both said that in situations that were new to the child he was usually cheerful, and the difficulties surfaced only when the child’s biological or emotional needs were not met, for example when something did not go his way. During the examination, the child was never confronted with such a situation and therefore did not display any difficult behaviours.
As one assessment from the Therapeutic Team points out: “Examples of situations causing the child a significant psychological discomfort and emotional tensions were, among others, the appearance and smell of food, changes in activity related to the rhythm of the day, enforcement of rules agreed individually with the boy, situations in which his place at the table was taken by another child, games with competitive elements”. The child’s response to such situations included “getting very angry, hitting, kicking, biting, overturning chairs, being verbally aggressive, screaming about killing others and blowing up something. Such situations occurred several times a week and required interventions by teachers [...]. There were situations when the steps taken by the teachers were not enough and it was necessary to call the parents in order to collect the child from the nursery early. In such moments, only the child’s mother was able to calm him down, which involved the need for him to stay at home for a few days” (the educator’s assessment, 13.07.2018). Here we can see a clear contradiction between the child’s composed and cheerful behavior observed during the examination and the assessment of the Therapeutic Team which was nevertheless ignored by the court experts.
While this study does not perhaps provide for any irrefutable conclusions, notably similar problems have been reported with other court expert assessments in the past, which should provide the basis for more detailed research in a future that would involve a larger number of similar cases. Taking into account a short time devoted to this type of evaluation, possible individual differences, and complexity of the autism spectrum phenomenon, as well as the researcher’s susceptibility to expert errors in psychiatry and psychology, it can be stated that the results constitute a major argument for conducting such comparative analyses on a larger scale. The problem of unreliable assessments issued by medical appraisers or court experts in Poland is of a systemic nature. This is shown by the number of cases in which ASD people and their families appeal to the decisions and assessments issued in the court proceedings to have the evidence of their disability confirmed or approved. Self-advocacy and neuro-diversity movements have been campaigning to change the assessment and evaluation system for years but have hardly been unsuccessful. Exposing the cognitive errors that can be found in expert evaluations in a full-scale study might constitute an important step towards the improvement of the current state of affairs.

Conflict of interest

Absent.

Financial support

This paper was written as a part of the research project nr DI2018 001348, financed from the budget funds for science in the years 2019-2023, within the “Diamond Grant” programme of the Polish Ministry of Science and Higher Education.

Ethnic statement

The patient’s family agreed to the use of patient-related data presented in this article. The project under which the study was conducted was approved by the Committee for Ethics of Scientific Research at the Maria Curie- Skłodowska University in Lublin.

References