The AASPIRE Outcomes Project
Why Does AASPIRE Care About Measuring Outcomes?
Many autistic people experience a “services cliff” as they transition to adulthood, and struggle to find high quality healthcare and social services. Thankfully, people are starting to focus more attention to services for autistic adults. That means there are a lot of new ideas out there for programs or services that are supposed to help improve the lives of autistic adults. But just because something sounds good, that doesn’t mean it actually helps anyone.
It’s important to know what actually helps autistic adults. The way researchers can tell if something helps (or hurts) is by seeing if it changes what we call an “outcome.”
However, we can’t tell if these services work unless we know how to accurately measure the outcomes that matter!
There are a few big questions:
- What outcomes matter? (And who gets to decide?)
- How do we measure outcomes accurately? (And how do we make sure these measures work well for a wide range of autistic adults in the real world?)
- How do these outcomes change over time?
- What can help autistic adults thrive?
We are concerned that – without a concerted effort – services may target outcomes that are less important to autistic adults; or researchers may evaluate these services using survey instruments that don’t actually work well for many people on the spectrum (or that aren’t practical to use in the real world).
Creating and rigorously testing a set of accessible, practical outcome measures now can have a large impact on the field for years to come.
What Types of Outcome Measures are We Talking About in this Project?
Some outcomes are best measured directly. For example, if the outcome is blood pressure or weight, you would want to check a participants’ blood pressure or weigh them, not ask them what they think those things are.
But often, we need to use surveys. For example, if the outcome is depression, quality of life, or job satisfaction, you will need to ask participants questions about these things. You can’t weigh depression. And often, it’s better to ask a bunch of questions about those things. For example, if we just asked someone if they were depressed, they may not recognize they are depressed. But if we ask them a bunch of questions about the symptoms of depression, we may get a better sense of how depressed they are. We would call this set of questions an “instrument” for measuring depression.
In the health sciences, we call survey instruments that ask about outcomes “Patient-Reported Outcome Measures (PROMs).” This project is focusing on PROMs for use with autistic adults, or if needed, supporters who can act as “informant reporters.”
What are Some Challenges with Survey Instruments in Adult Autism Research?
First of all, a lot of studies don’t even collect data directly from autistic adults. Other studies try to collect information directly from autistic adults, but they use instruments that were made for general populations or for parents. These may not work well for autistic people. Ultimately, inaccessible, impractical surveys lead to bad data, which leads to the wrong conclusions, which leads to bad policies!
Since our founding in 2006, AASPIRE has used a CBPR approach to assess and adapt survey instruments for use in our own studies. We have found that many survey instruments that have been created for general populations may not be accessible to autistic adults. You can learn more about the problems our community partners have identified in existing instruments – and the process we use to make surveys more accessible to autistic adults and people with intellectual disabilities – by reading our paper on Instrument Adaptations.
What are the Overall Objectives of the AASPIRE Outcomes Project?
The main objectives of this project are to:
- Collaboratively create the AASPIRE Measurement Toolkit – a set of accessible survey instruments that can measure the outcomes most important to the autistic community.
- Rigorously test the AASPIRE Measurement Toolkit surveys in a large group of autistic adults with diverse characteristics.
- Use the survey instruments to see how outcomes change over time, and what can help autistic people thrive.
- Share the AASPIRE Measurement Toolkit with others so that they can test the effectiveness of services for autistic adults.
What are the Main Parts of the Outcomes Project? (And How’s it Going?)
Phase 1: Decide what to include in the Toolkit
We used a “nested Delphi process” with our CBPR team and a Delphi panel of autistic community leaders, service providers, and academic researchers to:
- come up with a list of the most important outcomes to measure and
- identify and review existing survey instruments that can measure these outcomes.
We focused on the outcomes that would be most important to measure when evaluating the effectiveness of services for autistic adults. And we looked for instruments that would work well in the real world.
Progress: We finished this step. You can read a paper about how we came to consensus (agreement) on the 15 most important outcomes to measure when evaluating services and interventions for autistic adults.
Phase 2: Create accessible survey instruments to measure these outcomes
Our CBPR team then worked together to create survey instruments to measure each of these outcomes. In some cases, we were able to adapt existing instruments to make them accessible to autistic adults. In other cases, we had to create our own. We made two versions of each instrument: one for autistic adults and one for caregivers.
After we were done creating drafts of each instrument, we conducted cognitive interviews with 37 autistic adults and caregivers to make sure that our survey instruments ask important questions and are easy to use. We used what we learned in this part of the study to make further changes to the survey instruments, as needed, until we felt they were as accessible and relevant as possible.
Progress: We have finished this process. You can see the full list of outcomes and instruments we included in the AAPIRE Measurement Toolbox. We are working on a paper with more details about how and why we adapted the instruments.
Phase 3: Test the AASPIRE Measurement Toolkit with a large cohort (group) of autistic adults
We recruited a diverse cohort (group) of 850 autistic adults from
- the healthcare system,
- the disability services system, and
- the community
These participants took a big survey at baseline (when they first join the study) and again 3 to 4 times over the course of about a year.
We used this data to make sure that the instruments in the AASPIRE Measurement Toolkit work well (in other words, that they have good “psychometric properties”). Overall, we found that all the instruments were:
- Accessible (people could understand the questions and answer them
- Reliable (they measured things in a consistent way)
- Valid (they measured what they were trying to measure), and
- Responsive to change (scores changed in the right direction when the outcome they were measuring changed).
We made all these survey instruments available for free in our AASPIRE Measurement Toolkit
These survey instruments will enable researchers to collect higher-quality survey data from autistic adults. They will also help clinicians and service providers understand if their interventions or services are effective in improving the health and wellbeing of autistic adults.
Current Stage: This step is finished, but we are still working on lots of papers about our findings. You can see a preprint describing our methods and our study sample and another preprint describing how well the instruments work overall. We have finished more detailed analyses on many of the instruments and will be publishing papers about them soon.
Phase 4: Follow outcomes over time
We are using our survey data to:
- See how outcomes change over time.
- Better understand what supports and services may help autistic adults thrive.
Current Stage: We already collected data on these outcomes at 3 time points over 12-18 months. We are analyzing this data now and plan to publish a few papers on our findings. We hope to get more funding to keep collecting data over a longer period of time. If we are successful, we will open the study back up to recruitment and data collection.
This project was funded by a grant from the National Institute of Mental Health (award R01MH121407).
Principal Investigator: Christina Nicolaidis, MD, MPH, Portland State University
Community Project Lead: Ian Moura