Systematic Review of Gaps in Single Case Design Research: Evaluation of Study Quality and Rigor Using the Single Case Analysis and Review Framework
DOI:
https://doi.org/10.15763/issn.1936-9298.2024.1.1.61-83Keywords:
replication, internal validity, single case analysis and review frameworkAbstract
As the use of single case research design (SCRD) to answer socially significant research questions increases, so too should the rigor and quality of those designs. Higher rigor and quality decreases the chance of threats to internal validity and increases the chance of replication, both of which are critical to determining the effectiveness of an intervention. We conducted a systematic review of systematic literature reviews (k =18) that scored the quality and rigor of SCRD papers using the Single Case Analysis and Review Framework (SCARF; Ledford et al., 2023). On a continuous rigor/quality scale of 0-4, the 1,251 articles in all included reviews obtained an average of 2.14 with a standard deviation of 0.62 suggesting large gaps in the rigor/quality of SCRD. We discuss the implications of these gaps and offer recommendations for researchers to increase the rigor and quality of their SCRD studies.
References
**Met inclusion criteria and included in the review.
*Met inclusion criteria but unable to access data.
Barnett, D., Hawkins, R., McCoy, D., Wahl, E., Shier, A., Denune, H., & Kimener, L. (2014). Methods used to document procedural fidelity in school-based intervention research. Journal of Behavioral Education, 23(1), 89–107. https://doi.org/10.1007/s10864-013-9188-y
Barton, E.E., Meadan-Kaplansky, H., & Ledford, J.R. (2018). Independent Variables, Fidelity, and Social Validity. In J.R. Ledford & D.L. Gast (Eds.), Single case research methodology (3rd ed., pp. 133-156). Routledge. http://dx.doi.org/10.4324/9781315150666-6
**Barton, E. E., Murray, R., O’Flaherty, C., Sweeney, E. M., & Gossett, S. (2020). Teaching object play to young children with disabilities: A systematic review of methods and rigor. American Journal on Intellectual and Developmental Disabilities, 125(1), 14–36. https://doi.org/10.1352/1944-7558-125.1.14
Barton, E.E., Meadan-Kaplansky, H., & Ledford, J.R. (2018). Research Approaches in Applied Settings. In J.R. Ledford & D.L. Gast (Eds.), Single case research methodology (3rd ed., pp. 133-156). Routledge. http://dx.doi.org/10.4324/9781315150666-6
Bergmann, S., Niland, H., Gavidia, V. L., Strum, M. D., & Harman, M. J. (2023). Comparing multiple methods to measure procedural fidelity of discrete-trial instruction. Education & Treatment of Children, 46, 201-220. https://doi.org/10.1007/s43494-023-00094-w
Chazin, K. T., Ledford, J. R., Barton, E. E., & Osborne, K. C. (2018). The effects of antecedent exercise on engagement during large group activities for young children. Remedial and Special Education, 39(3), 158–170. https://doi.org/10.1177/0741932517716899
**Chazin, K. T., Ledford, J. R., & Pak, N. S. (2021). A systematic review of augmented input interventions and exploratory analysis of moderators. American Journal of Speech-Language Pathology, 30(3), 1210–1223. https://doi.org/10.1044/2020_AJSLP-20-00102
**Chazin, K. T., Velez, M. S., & Ledford, J. R. (2022). Reducing escape without escape extinction: A systematic review and meta-analysis of escape-based interventions. Journal of Behavioral Education, 31(1), 186–215. https://doi.org/10.1007/s10864-021-09453-2
Creswell, J. W., & Creswell, J. D. (2023). Chapter 1 the selection of a research approach. In Research design: Qualitative, quantitative, and mixed methods approaches (6th ed., pp. 40–61). SAGE Publications, Inc. http://dx.doi.org/10.5539/elt.v12n5p40
Elliott, T.C., Morgan, G. & Ayres, K.M. (2024). Techincal Adequacy of the Resaerch Informed Classroom Evlauation - Play (RICE-P) Tool for Evaluating Special Education Teachers. [Manuscript in preparation].
Ennis, R. P., Lane, K. L., & Flemming, S. C. (2021). Empowering teachers with low-intensity strategies: Supporting students at-risk for EBD with instructional choice during reading. Exceptionality, 29(1), 61–79. https://doi.org/10.1080/09362835.2020.1729766
Ennis, R. P., Jolivette, K., Fredrick, L. D., & Alberto, P. A. (2013). Using comparison peers as an objective measure of social validity: Recommendations for researchers. Focus on Autism and Other Developmental Disabilities, 28(4), 195–201. https://doi.org/10.1177/1088357612475078
**Eyler, P. B., & Ledford, J. R. (2023). Systematic review of time delay instruction for teaching young children. Journal of Early Intervention. Advanced online publication. https://doi.org/10.1177/10538151231179121
Falakfarsa, G., Brand, D., Jones, L., Godinez, E. S., Richardson, D. C., Hanson, R. J., Velazquez, S. D., & Wills, C. (2022). Treatment integrity reporting in behavior analysis in practice 2008–2019. Behavior Analysis in Practice, 15(2), 443–453. https://doi.org/10.1007/s40617-021-00573-9
Gast, D.L. & Ledford, J.R. (2018). Research Approaches in Applied Settings. In J.R. Ledford & D.L. Gast (Eds.), Single case research methodology (3rd ed., pp. 1-26). Routledge. http://dx.doi.org/10.4324/9781315150666-1
Ganz, J. B., & Ayres, K. M. (2018). Methodological standards in single-case experimental design: Raising the bar. Research in Developmental Disabilities, 79, 3–9. https://doi.org/10.1016/j.ridd.2018.03.003
**Gibbs, A. R., & Tullis, C. A. (2021). The emergence of untrained relations in individuals with autism and other intellectual and developmental disabilities: A systematic review of the recent literature. Review Journal of Autism and Developmental Disorders, 8(2), 213–238. https://doi.org/10.1007/s40489-020-00211-0
**Gibbs, A. R., Tullis, C. A., Conine, D. E., & Fulton, A. A. (2024). A systematic review of derived relational responding beyond coordination in individuals with autism and intellectual and developmental disabilities. Journal of Developmental and Physical Disabilities, 36, 1–36. https://doi.org/10.1007/s10882-023-09901-z
Gusenbauer, M. (2019). Google Scholar to overshadow them all? Comparing the sizes of 12 academic search engines and bibliographic databases. Scientometrics, 118(1), 177–214. https://doi.org/10.1007/s11192-018-2958-5
Han, J. B., Bergmann, S., Brand, D., Wallace, M. D., St. Peter, C. C., Feng, J., & Long, B. P. (2023). Trends in reporting procedural integrity: A comparison. Behavior Analysis in Practice, 16(2), 388–398. https://doi.org/10.1007/s40617-022-00741-5
Hardy, J. K., McLeod, R. H., Sweigart, C. A., & Landrum, T. (2022). Comparing and contrasting quality frameworks using research on high-probability requests with young children. Infants and Young Children, 35(4), 267–284. https://doi.org/10.1097/iyc.0000000000000223
Haddaway, N. R., Page, M. J., Pritchard, C. C., & McGuinness, L. A. (2022). PRISMA2020: An R package and shiny app for producing PRISMA 2020-compliant flow diagrams, with interactivity for optimised digital transparency and open synthesis. Campbell Systematic Reviews, 18(2), e1230. https://doi.org/10.1002/cl2.1230
Hariton, E., & Locascio, J. J. (2018). Randomised controlled trials – the gold standard for effectiveness research: Study design: randomised controlled trials. BJOG: An International Journal of Obstetrics and Gynaecology, 125(13), 1716–1716. https://doi.org/10.1111/1471-0528.15199
**Herrod, J. L., Snyder, S. K., Hart, J. B., Frantz, S. J., & Ayres, K. M. (2023). Applications of the Premack principle: A review of the literature. Behavior Modification, 47(1), 219–246. https://doi.org/10.1177/01454455221085249
Holcombe, A., Wolery, M., & Snyder, E. (1994). Effects of two levels of procedural fidelity with constant time delay on children’s learning. Journal of Behavioral Education, 4(1), 49–73. https://doi.org/10.1007/bf01560509
Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., & Wolery, M. (2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71(2), 165–179. https://doi.org/10.1177/001440290507100203
Howland, J. L., Wright, T. C., Boughan, R. A., & Roberts, B. C. (2009). How scholarly is Google Scholar? A comparison to library databases. College and Research Libraries, 70(3), 227–234. https://doi.org/10.5860/0700227
Kratochwill, T. R., Hitchcock, J., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M. & Shadish, W. R. (2010). Single-case designs technical documentation. http://ies.ed.gov/ncee/wwc/pdf/wwc_SCRD.pdf http://dx.doi.org/10.1177/0741932512452794
Lane, J. D., Ledford, J. R., & Gast, D. L. (2017). Single-case experimental design: Current standards and applications in occupational therapy. The American Journal of Occupational Therapy, 71(2), 7102300010p1-7102300010p9. https://doi.org/10.5014/ajot.2017.022210
Lane, K., Wolery, M., Reichow, B., & Rogers, L. (2007). Describing baseline conditions: Suggestions for study reports. Journal of Behavioral Education, 16(3), 224–234. https://doi.org/10.1007/s10864-006-9036-4
Ledford, J. R., & Wolery, M. (2013). Procedural fidelity: An analysis of measurement and reporting practices. Journal of Early Intervention, 35(2), 173–193. https://doi.org/10.1177/1053815113515908
Ledford, J. R., Lane, J. D., Zimmerman, K. N., Chazin, K. T., & Ayres, K. A. (2016/2023). Single case analysis and review framework (SCARF). http://ebip.vkcsites.org/scarf/
Ledford, J. R., Lambert, J. M., Pustejovsky, J. E., Zimmerman, K. N., Hollins, N., & Barton, E. E. (2023). Single-case-design research in special education: Next-generation guidelines and considerations. Exceptional Children, 89(4), 379–396. https://doi.org/10.1177/00144029221137656
**Ledford, J. R., & Windsor, S. A. (2022). Systematic review of interventions designed to teach imitation to young children with disabilities. Topics in Early Childhood Special Education, 42(2), 202–214. https://doi.org/10.1177/02711214211007190
Ledford, J.R., Lane, J.D., Tate, R. (2018). Evaluating Quality and Rigor in Single Case Research. In J.R. Ledford & D.L. Gast (Eds.), Single case research methodology (3rd ed., pp. 365-392). Routledge. http://dx.doi.org/10.4324/9781315150666-13
**Ledford, J. R., & Pustejovsky, J. E. (2023). Systematic review and meta-analysis of stay-play-talk interventions for improving social behaviors of young children. Journal of Positive Behavior Interventions, 25(1), 65–77. https://doi.org/10.1177/1098300720983521
**Ledford, J. R., Trump, C., Chazin, K. T., Windsor, S. A., Eyler, P. B., & Wunderlich, K. (2023). Systematic review of interruption and redirection procedures for autistic individuals. Behavioral Interventions, 38(1), 198-218. https://doi.org/10.1002/bin.1905
Ma, Z., & Travers, J. C. (2022). Adjusting intervention intensity to support students with autism spectrum disorder. Intervention in School and Clinic, 57(5), 291–297. https://doi.org/10.1177/10534512211032596
Maggin, D. M., O’Keeffe, B. V., & Johnson, A. H. (2011). A quantitative synthesis of methodology in the meta-analysis of single-subject research for students with disabilities: 1985–2009. Exceptionality, 19(2), 109–135. https://doi.org/10.1080/09362835.2011.565725
McDonnell, J., & O’Neill, R. (2003). A perspective on single/within subject research methods and “scientifically based research.” Research and Practice for Persons with Severe Disabilities, 28(3), 138–142. https://doi.org/10.2511/rpsd.28.3.138
Morgan, D. L., & Morgan, R. K. (2012). Single-case research methods for the behavioral and health sciences. SAGE Publications.
*Nanda, S. (2021). Peer-mediated instruction and interventions to increase social skills of children with ASD without intellectual impairments in inclusive preschool and elementary school settings: A meta-analysis [Doctoral dissertation, University of Washington]. ResearchWorks Archive. http://hdl.handle.net/1773/48012
Perepletchikova, F., Hilt, L. M., Chereji, E., & Kazdin, A. E. (2009). Barriers to implementing treatment integrity procedures: survey of treatment outcome researchers. Journal of Consulting and Clinical Psychology, 77(2), 212–218. https://doi.org/10.1037/a0015232
Petursdottir, A. I., & Carr, J. E. (2018). Applying the taxonomy of validity threats from mainstream research design to single-case experiments in applied behavior analysis. Behavior Analysis in Practice, 11(3), 228–240. https://doi.org/10.1007/s40617-018-00294-6
Preas, E. J., Halbur, M. E., & Carroll, R. A. (2024). Procedural fidelity reporting in the analysis of verbal behavior from 2007–2021. The Analysis of Verbal Behavior, 40, 1-12. https://doi.org/10.1007/s40616-023-00197-w
Privitera, G. J. (2020). Research Methods for the Behavioral Sciences (3rd Edition). Sage Publications.
Reichow, B., Barton, E. E., & Maggin, D. M. (2018). Development and applications of the single-case design risk of bias tool for evaluating single-case design research study reports. Research in Developmental Disabilities, 79, 53–64. https://doi.org/10.1016/j.ridd.2018.05.008
*Rubio, E. K., McMahon, M. X. H., & Volkert, V. M. (2021). A systematic review of physical guidance procedures as an open-mouth prompt to increase acceptance for children with pediatric feeding disorders. Journal of Applied Behavior Analysis, 54(1), 144–167. https://doi.org/10.1002/jaba.782
*Rubio, E.K. (2021). Evaluation of intensive behavioral interventions for severe avoidant/restrictive food intake disorders in young children [Doctoral dissertation, Georgia State University]. ScholarWorks. https://doi.org/10.57709/24097625
**Scheibel, G., Chen, P.-Y., Zaeske, L. M., Wills, H. P., & Zimmerman, K. N. (2023). Improving implementation fidelity with teacher-directed self-monitoring interventions: A systematic review. Journal of Positive Behavior Interventions, 25(4), 253–269. https://doi.org/10.1177/10983007221137368
**Scott, V. (2022). On the efficacy of treating escape-maintained inappropriate mealtime behaviour with and without escape extinction: A meta‐analysis of escape‐based interventions [Doctoral dissertation, Brock University]. DSpace Software. http://hdl.handle.net/10464/17040
Shepley, C., Zimmerman, K. N., & Ayres, K. M. (2021). Estimating the impact of design standards on the rigor of a subset of single-case research. Journal of Disability Policy Studies, 32(2), 108–118. https://doi.org/10.1177/1044207320934048
**Snyder, S. K. (2023). A comparison of outcomes of indirect, direct and functional analysis assessment methods and implications for classroom based treatment for challenging behavior [Doctoral dissertation, University of Georgia]. Exlibris.
St. Peter, C. C., Brand, D., Jones, S. H., Wolgemuth, J. R., & Lipien, L. (2023). On a persisting curious double standard in behavior analysis: Behavioral scholars’ perspectives on procedural fidelity. Journal of Applied Behavior Analysis, 56(2), 336–351. https://doi.org/10.1002/jaba.974
Sanetti, L. M. H., Dobey, L. M., & Gritter, K. L. (2012). Treatment integrity of interventions with children in the Journal of Positive Behavior Interventions from 1999 to 2009. Journal of Positive Behavior Interventions, 14(1), 29–46. https://doi.org/10.1177/1098300711405853
Snodgrass, M. R., Chung, M. Y., Kretzer, J. M., & Biggs, E. E. (2022). Rigorous assessment of social validity: A scoping review of a 40-year conversation. Remedial and Special Education, 43(2), 114–130. https://doi.org/10.1177/07419325211017295
Swanson, E., Wanzek, J., Haring, C., Ciullo, S., & McCulley, L. (2013). Intervention fidelity in special and general education research journals. The Journal of Special Education, 47(1), 3–13. https://doi.org/10.1177/0022466911419516
Tanious, R., & Onghena, P. (2019). Randomized single-case experimental designs in healthcare research: What, why, and how? Healthcare, 7(4), 143. https://doi.org/10.3390/healthcare7040143
Tate, R. L., Perdices, M., McDonald, S., Togher, L., & Rosenkoetter, U. (2014). The design, conduct and report of single-case research: Resources to improve the quality of the neurorehabilitation literature. Neuropsychological Rehabilitation, 24(3–4), 315–331. https://doi.org/10.1080/09602011.2013.875043
Tate, R. L., Perdices, M., Rosenkoetter, U., Shadish, W., Vohra, S., Barlow, D. H., Horner, R., Kazdin, A., Kratochwill, T., McDonald, S., Sampson, M., Shamseer, L., Togher, L., Albin, R., Backman, C., Douglas, J., Evans, J. J., Gast, D., Manolov, R., … Wilson, B. (2016). The single-Case Reporting guideline In BEhavioural interventions (SCRIBE) 2016 Statement. Physical Therapy, 96(7), e1–e10. https://doi.org/10.2522/ptj.2016.96.7.e1
Wellons, Q. D., Roach, A. T., & Sanchez-Alvarez, S. (2023). Is social validity an afterthought in single-case design studies in school psychology research? Contemporary School Psychology. Advanced online publication. https://doi.org/10.1007/s40688-023-00460-w
What Works Clearinghouse. (2020). What Works Clearinghouse Standards Handbook, Version 4.1. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee/wwc/handbooks http://dx.doi.org/10.1037/e301462004-001
Wolery, M. (1994). Procedural fidelity: A reminder of its functions. Journal of Behavioral Education, 4(4), 381–386. https://doi.org/10.1007/bf01539539
Wolery, M., Dunlap, G., & Ledford, J. R. (2011). Single-case experimental methods: Suggestions for reporting. Journal of Early Intervention, 33(2), 103–109. https://doi.org/10.1177/1053815111418235
Zimmerman, K. N., Ledford, J. R., Gagnon, K. L., & Martin, J. L. (2020). Social stories and visual supports interventions for students at risk for emotional and behavioral disorders. Behavioral Disorders, 45(4), 207–223. https://doi.org/10.1177/0198742919874050
**Zimmerman, K. N., & Ledford, J. R. (2017). Beyond ASD: Evidence for the effectiveness of social narratives. Journal of Early Intervention, 39(3), 199–217. https://doi.org/10.1177/1053815117709000
*Zimmerman, K. N., Ledford, J. R., Severini, K. E., Pustejovsky, J. E., Barton, E. E., & Lloyd, B. P. (2018). Single-case synthesis tools I: Comparing tools to evaluate SCD quality and rigor. Research in Developmental Disabilities, 79, 19–32. https://doi.org/10.1016/j.ridd.2018.02.003
Downloads
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Tyler-Curtis Elliott, Kevin Ayres, Joseph Hart, Jennifer Ledford
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
