Welcome to the PQHS Research Methods Meeting Site
Research Methods Meetings on the 1st and 3rd Wednesdays @ 10 AM
To join the meetings Click here
10/16/24: Goal-Oriented Data-Centric Learning Presenter: Hongfu Liu, PhD, Assistant Professor of Computer Science at Brandeis University Bio: Dr. Hongfu Liu is an Assistant Professor of Computer Science at Brandeis University. His research interests lie in core machine learning and AI-assisted applications. He has published over 100 papers (e.g., NeurIPS, ICLR, ICML, IJCAI, AAAI, KDD, ICDM, SDM, CIKM, CVPR, ICCV, TPAMI, and TKDE). These publications have received over 4,000 citations with an h-index of 37. He has also won several awards including the First Place Award in MS-Celel-1M Grand Challenge in ICCV 2017, the NVIDIA CCS Best Student Paper Award in FG 2021, the 2021 INNS Aharon Katzir Young Investigator Award, the top reviewer in UAI 2022, the highlighted/notable Area Chair in ICLR/NeurIPS 2022/2023, and the 2022 Global Top-25 Chinese Young Scholars in AI (Data Mining Area) by Baidu Scholar. He has served as an Associate Editor of IEEE CIM and as a (Senior) Area Chair of ICLR, ICML, and NeurIPS. Brief Description:Data-centric learning focuses on enhancing algorithmic performance from the perspective of the training data. In contrast to model-centric learning, which designs novel algorithms or optimization techniques for performance improvement with fixed training data, data-centric learning operates with a fixed learning algorithm while modifying the training data through trimming, augmenting, or other methods aligned with improving utility. Data-centric learning holds significant potential in many areas such as model interpretation, subset training set selection, data generation, noisy label detection, active learning, and others. In this talk, I will introduce our recent advances in data-centric learning based on influence functions. |
10/2/24: SMART Study Designs in Medical Research with an Example in Diabetes Management |
7/17/24: Trimming in propensity score: improve the robustness and validity of causal inference in observational studies
Presenter: Shiwei (Echo) Liang, MS, MA Ph.D. Candidate in Population Health Sciences, UMass Chan Medical School Bio: Echo is mentored by Matthew Alcusky, PhD, PharmD, MS and her research interest is in pharmacoepidemiology and health system including insurance claim data, prescription drug effectiveness, health intervention, as well as the development of healthcare system evaluation to promote health surveillance. She is working on her dissertation about antidepressants for depressive symptoms and agitation in ADRD among nursing home residents. Brief Description: This presentation will provide an overview of trimming in propensity score in observational research, with a focus on fixed threshold trimming. In addition to discussing the technic of trimming methods in healthcare research, the application of trimming in propensity score to study the effectiveness of treatments will be discussed.
7/10/24: Informed Presence Bias, The Bias Inherent to Electronic Health Record Review
Presenter: Zachary Ballinger, MD, General Surgery Resident, UMass Chan Medical School Pediatric Surgery Research Fellow, UMass Memorial Medical Center Children's Hospital, MSCI Candidate Bio: Dr. Ballinger is a resident physician in the general surgery program here at UMass. He is currently on two years of dedicated research time with the Pediatric Surgery department, where his work focuses on clinical outcomes research using both single institution and large national databases. Recently, his work has focused on disparities in metabolic and bariatric surgery in the pediatric population, and management of blunt kidney injury in pediatric patients. Brief Description: This presentation will provide an overview of selection bias in Electronic Medical Record data and research, with a focus on the concept of informed presence bias. In addition to discussing the principles of this bias, strategies for bias mitigation and control will be discussed.
6/26/24: Navigating the Threshold: Applying Regression Discontinuity Design in Healthcare Research.
Presenter: Justin Rucci, MD, MSCI, Advanced Fellow in Health Systems Research, VA Boston Healthcare System, Center for Healthcare Organization and Implementation Research, Assistant Professor, Boston University Avedisian & Chobanian School of Medicine, Division of Pulmonary, Allergy, Sleep & Critical Care Medicine. Bio: Dr. Rucci is a physician-scientist and recent graduate of the UMass Chan MSCI Program. His research evaluates practice pattern variation across intensive care units and how this variation impacts patient outcomes. Most recently, Dr. Rucci seeks to understand how the use of a learning health system model within intensive care units could support robust mixed methods research and enhance critical care delivery. Brief Description: This presentation will provide an overview of the quasi-experimental method regression discontinuity design (RDD). In addition to discussing the premise of RDD and describing a stepwise approach to conducting an RDD study, the session will provide practical examples of how RDD can be applied to healthcare research.
6/12/24: Multimorbidity patterns in Medicare beneficiaries with multiple myeloma.
Presenter: Atinuke G. Oyinbo, MPH, PhD Candidate, Department of Quantitative Health Sciences, UMass Chan Medical School Bio: Atinuke is PhD Candidate in the Population Health Sciences program at UMass Chan specializing in cancer epidemiology. Her research focuses on investigating the impact of chronic conditions and healthcare policies on cancer care. She is also involved in studies focused on understanding and addressing social disparities in cancer screening and survivorship care. Brief Description: The presentation will describe the first aim of my thesis research, which examines the patterns of multimorbidity in older adults newly diagnosed with multiple myeloma using clustering analysis, and their associations with sociodemographic factors.
6/5/24: Reducing algorithmic bias in AI
Presenter: Kumba Sennaar, MS, MA, Brandeis University Bio: Kumba Sennaar is a PhD candidate in Social Policy at Brandeis University and founder of William Kelly Consulting. Her research focuses on reducing health inequities by centering the end‐user voice and perspective as a conduit to disrupting algorithmic bias. Sennaar earned an M.A. in Social Policy from Brandeis, M.S. in Biotechnology from Johns Hopkins University and a B.S. with honors from Rensselaer Polytechnic Institute, in Science, Technology & Society (STS). Her articles on applications of AI have been cited by consulting firms and by journals including the Boston University Law Review, Fordham Urban Law Journal and the Harvard Data Science Review.. Brief Description: I will present my recent TEDx talk which discusses practical ways nontechnologists can participate in AI research and engage in Q&A on the discussion topic
5/29/24: Precision Digital Medicine Study of Alzheimer’s Disease
Presenter: Honghuang Lin, PhD, Professor of Medicine, UMass Chan Medical School Bio: Dr. Lin is a Professor of Medicine and Co-director of the Program in Digital Medicine. His research focuses on the development of computational methods to study molecular mechanisms underlying cardiovascular disease and Alzheimer’s disease. He is also leading multiple studies to explore digital technologies and wearable devices for continuous health monitoring. Brief Description: Dr. Lin will discuss the integration of genetic risk and digital biomarkers to study Alzheimer’s disease
5/22/24: Risk Adjustment 101, Part II
Presenter: Arlene S. Ash, Professor and Chief, Division of Biostatistics and Health Services Research in PQHS, UMass Chan Medical School Bio: Since 2014, Dr. Ash is a pioneer in risk adjustment methods, and has led teams incorporating both medical and social risk into global payments and quality measures for MassHealth (Massachusetts Medicaid and Children’s Health Insurance Programs). She is keenly interested in understanding and addressing inequities associated with gender, age, race, and socioeconomics; also, quality, equity and efficiency in health care financing and delivery. Brief Description: I will continue the discussion from two weeks’ ago about risk adjustment for managing health care delivery systems, this week focusing on some new directions in modeling. You may find slides for the 2 talks combined in the following link (Link).
5/15/24: Hybrid Designs, what they look like, and how we use them in RCTs
Presenter:: Bruce Barton, PhD, Professor, Population and Quantitative Health Sciences, UMass Chan Medical School Bio: Dr. Barton was named Research Professor in UMass Medical School’s Department of Quantitative Health Sciences (Division of Biostatistics and Health Services Research) in 2010 and, subsequently, the Director of the Quantitative Methods Core. He has also served as the Team Leader of the Research Methods Team (Biostatistics) in the Center for Health Policy Research at UMMS and as an Adjunct Professor at Tufts School of Veterinary Medicine. Dr. Barton is a biostatistician with over 40 years of experience in medical research studies, especially randomized clinical trials, starting with the National Surgical Adjuvant Breast Project breast cancer clinical trials at the University of Pittsburgh in 1975. He has been the PI/Director of 31 Data Coordinating Centers during that time-frame, funded both by NIH and by industry. His experience in clinical trials includes multi-center/multi-national trials with patient-level and higher-level (cluster) randomizations. Brief Description: I will be discussing hybrid study designs in RCT and show several bad and good examples. The current discussion will be informal and touch on the study design aspects.
5/8/24: Risk Adjustment 101
Presenter: Arlene S. Ash, Professor and Chief, Division of Biostatistics and Health Services Research in PQHS, UMass Chan Medical School Bio: Since 2014, Dr. Ash is a pioneer in risk adjustment methods, and has led teams incorporating both medical and social risk into global payments and quality measures for MassHealth (Massachusetts Medicaid and Children’s Health Insurance Programs). She is keenly interested in understanding and addressing inequities associated with gender, age, race, and socioeconomics; also, quality, equity and efficiency in health care financing and delivery. Brief Description: I will discuss why risk adjustment is crucial for managing health care delivery systems, how (and how well) the commonly used models work, and new directions in using computer-assisted technology for generating clinically credible and transparent risk models. Questions from the audience are encouraged. Depending on how much we cover, and audience interest, we may continue with this topic at our meeting on May 23.
5/1/24: Collaboration Opportunities with the Division of Health Systems Science
Presenter: Allan J. Walkey, MD, MSc, Professor of Medicine, Chief, Division of Health Systems Science, Physician, Division of Pulmonary, Allergy and Critical Care, Department of Medicine, UMass Chan Medical School Bio: Dr. Walkey is a Professor of Medicine, inaugural Chief of the Division of Health Systems Science, and a Pulmonary Critical Care Physician at UMass Chan Medical School and UMass Memorial Hospital. His research evaluates care delivery to critically ill patients, leverages practice variation to compare effectiveness of existing standards of care and implements evidence-based interventions. Brief Description: I will be discussing why we need to study Health Systems Science, describe the history of our division, describe who we are and what we do, and how we hope to engage in collaborations.
4/24/24: The Impact of the Opioid Epidemic on Crime and Incarceration in the US
Presenter: David Swedler, PhD. is a Senior Healthcare Research Analyst with ForHealth Consulting at UMass Medical School Bio: Dr. David Swedler is a Senior Healthcare Research Analyst with ForHealth Consulting at UMass Medical School, where he is embedded with the MassHealth state Medicaid agency. His work at MassHealth has focused on statistical methods and health disparities analysis. Prior to coming to ForHealth in 2023, he spent 6 years as a methodologist at the non-profit Pacific Institute for Research and Evaluation, and 2-and-a-half years as reach faculty at the University of Illinois at Chicago School of Public Health. He has an MPH in Epidemiology from Michigan, a PhD in Health Policy from Johns Hopkins, and did a postdoc in Occupational Epidemiology at Harvard. Brief Description: This analysis is from work that we did at PIRE for the plaintiffs on the national opioid litigation. My coauthors, Ted Miller and Mark Cohen, have decades of research on the costs of crime in the US. We sought to apply data on opioid drug distribution and overdose deaths to established methods of analyzing crime and incarceration at the state-level over time. While the results are interesting in and of themselves – we find a significant amount of property crime, violent crime, and prison and jail inmates are associated with the opioid epidemic – this project involved a few advanced methods worth discussing in a research forum. In particular, I am going to talk about how we identified “missed” opioid overdoses, how we allocated fentanyl deaths by licit and illicit origin, and the creation of a counterfactual version of the states to assess how big the impact of opioids on crime and incarceration was. This project was very broad, so I will leave time to discuss any implications you’d want to raise.
4/17/24: Applying Physics Reasoning in Epidemiology
Presenter: Dustin Burns, PhD, GStat, Senior Managing, Scientist, Exponent
4/3/24: How David [Elis for Rachael] forced Goliath [Yale] to modernize its mental health policies
Presenter: Paul Johansen, MA (Yale ’88, also a UMass Med alum), Miriam Kopyto, BA (Yale ’23), Lucy Kim (Yale ’24), Zack Dugue (CalTech ’25), Lily Colby, Esq. (Yale '10).
3/27/24: Probing Research Methodologies and Technology Stacks in Artificial Intelligence for Healthcare Analytics
Presenter: Vineet Khullar, PhD, Sr. Analyst Cigna Healthcare, Boston, MA
3/20/24: How to model mediation with different types of outcomes
Presenter: Richard Goldstein, Ph.D., Consultant
3/13/24: Searching for a useful quality measure in TKA surgery
Presenters: Hua Zheng, PhD, Assistant Professor, Orthopedics and Arlene Ash, PhD, Professor, PQHS
3/6/24: Using Health Services Research to Advance Equity
Presenter: Debbie Peikes, PhD, MPA - Vice President, Measurement & Evaluation
Mark Friedberg, MD, MPP - SVP, Performance Measurement & Improvement
Deanna Fulp, Sr. Director, Health Equity Programs
2/28/24: Quantifying Regression to the Mean in the PRISM Study
Presenter: Julie Flahive, MS, Biostatistician, Quantitative Methods Core, UMass Chan Medical School
2/21/24: How Employer-Sponsored Health Insurance Costs are Impacting Earnings Inequality Among US Families
Presenter: Kurt Hager, PhD, Instructor, PQHS, UMass Chan Medical School
2/14/24: Integrating Wearable Devices into Lifestyle Interventions and Healthcare System Workflows: Closing the Feedback Loop
Presenter: Garrett Ash, PhD, Assistant Professor of Medicine (General Internal Medicine) Assistant Professor of Biomedical Informatics and Data Science, Yale School of Medicine
2/7/24: A journal review, “Sensitivity Analysis in Observational Research: Introducing the E-Value” by Vanderweele and Ding (2017)
Presenter: Arlene Ash, PhD, Professor and Jonggyu Baek, PhD, Associate Professor, PQHS, UMass Chan
1/24/24: Yale’s Biostatistics Webinar: "Measures of Selection Bias for Proportions Estimated from Non-Probability Samples
Presenter: Rebecca Andridge of The Ohio State University
1/17/24: Using Synthethic Control Method (SCM) to strengthen causal inference with observational data
Presenters: Arlene Ash (Division Chief for Biostatistics and Health Services Research) and Jeroan Alison (Chair), in the Department of Population and Quantitative Health Sciences, UMass Chan
1/10/24: Disparities in Palliative Care Use for Patients with Blood Cancer Who Died in the Hospital
Presenter: Tien-Chan Hsieh, MD, Fellow, Hematology Oncology Division, UMass Chan
12/20/23: Characterization and Outcomes of Multiple Myeloma in Puerto Rico
Presenter: Maira Castaneda-Avila, PhD, Assistant Professor, PQHS, UMass Chan
12/13/23: Can Risk Modeling Research Help Sustain Care Delivery Innovations for People Living Homeless with Chronic Behavioral Health Conditions and Other Barriers to Stability?
Presenter: John Gilvar, MA, Gilvar Consulting
12/6/23: What is ARPA-H and how might we participate?
Presenter: Arlene Ash, PhD, Professor, PQHS, UMass Chan Medical School
11/29/23: The maternal health crisis: Can we move the needle?
Presenter: Sara Teppema, FSA, MAAA, FCA, Chief actuary, Wildflower Health
11/15/23: Evaluation of the Pediatric Palliative Care Network (PPCN) Program
Presenter: Ying (Elaine) Wang, PhD, Associate Professor, UMass Chan Medical School
11/8/23: Development and Evaluation of the Algorithm CErtaInty Tool (ACE-IT) to Assess Electronic Medical Record and Claims-based Algorithms' Fit for Purpose for Safety Outcomes.
Presenter: Sonal Singh, MD, MPH, Associate Professor, Family Medicine and Community Health, UMass Chan Medical School
11/1/23: Biomarker Qualification
Presenter: Jeffrey B Driban, PhD, Professor, PQHS, UMass Chan Medical School
10/25/23: Introduction to the HCSRN Virtual Data Warehouse
Presenter: Hassan Fouayzi, PhD, Assistant Professor, PQHS, UMass Chan Medical School
10/18/23: Medicare for All - Implications for UMass Memorial and Acute Care Hospitals across Massachusetts
Presenter: Eric Mick, PhD, Associate Professor, PQHS, UMass Chan Medical School
10/4/23: The rationale and logic of propensity scores
Presenter: Arlene Ash, PhD, Professor, PQHS, UMass Chan Medical School
9/27/23: The Art of Sample Size Estimation
Presenter: Bruce A Barton, PhD, Professor, PQHS, UMass Chan Medical School
9/20/23: Missing Data on Sexual Orientation and Gender Identity in the Behavioral Risk Factor Surveillance System
Presenter: Willian Jesdale, Assistant Professor, PQHS, UMass Chan Medical School
9/13/23: Pain and its management in nursing home residents
Presenter: Kate Lapane, PhD, Professor, PQHS, UMass Chan Medical School
9/6/23: Making Capitated Payments to Primary Care Practices Conform to Expected Differences in Resource Needs
Presenter: Arlene Ash, PhD and Meagan Sabatino, PhD candidate, PQHS, UMass Chan Medical School
7/25/23: A journal club discussion
Presenter: Meagan J Sabatino, PhD candidate, PQHS, UMass Chan Medical School
7/18/23: Flexible Services Medically Tailored Meal Evaluation Consortium
Presenter: Kurt Hager, PhD, Instructor, PQHS, UMass Chan Medical School
7/11/23: Advancing Regularized Regression
Presenter: JungAe Lee, PhD, Assistant Professor, PQHS, UMass Chan Medical School
6/27/23: Introduction to Multiphase Optimization Strategy (MOST)
Presenter: Bo Wang, PhD, Professor, PQHS, UMass Chan Medical School
6/20/23: The Fuss about P-Values
Presenter: Henry Feldman, PhD, Professor, Biostatistics and Research Design, Boston Children's Hospital
6/13/23: Regularized Regression: Ridge, Lasso, Elastic Net, and Beyond
Presenter: JungAe Lee, PhD, Assistant Professor, PQHS, UMass Chan Medical School
6/6/23: Cervical cancer screening behavior among women in Malawi: with a specific focus on the HPV screening test. Presenter: Deogwoon Kim, PhD, Postdoctoral researcher, PQHS, UMass Chan Medical School
5/30/23: BioMarkerAI: Combining Statistical and Machine Learning Methods for improving identification of Coding and Noncoding Genes as Putative Biomarkers
Presenter: Chan Zhou, PhD, Assistant Professor, Biostatistics & Health Services Research, PQHS, UMass Chan Medical School
5/16/23: WebENAR discussion part II: Statistical Issues in Responsible Conduct of Research
WebENAR presenter: Sarah J. Ratcliffe, PhD, University of Virginia School of Medicine; and discussant:
Presenter: Bruce A. Barton, PhD, Professor, PQHS, Director of QMC, UMass Chan Medical School
5/9/23: WebENAR discussion part I: Statistical Issues in Responsible Conduct of Research
WebENAR presenter: Sarah J. Ratcliffe, PhD, University of Virginia School of Medicine; and discussant:
Presenter: Bruce A. Barton, PhD, Professor, PQHS, Director of QMC, UMass Chan Medical School
5/2/23: Mean or Median?
Presenter: Austin Lee, PhD, Professor, Division of Biostatistics and Health Services Research, PQHS, UMass Chan Medical School
4/25/23: Interpretable, Generalizable, and Fair Predictive Modeling for Long COVID
Presenter: Feifan Liu, PhD, Assistant Professor, Division of Health Informatics and Implementation Science, PQHS, UMass Chan Medical School
4/11/23: Review of Lloren A et al. (2019)’s disparity model
Presenter: Jonggyu Baek, PhD, Assistant Professor, Division of Biostatistics and Health Services Research, PQHS, UMass Chan Medical School
4/4/23: Less is More: Why more than half items in the most popular oral health-related quality of life (OHQoL) measure needed to be deleted for more meaningful use?
Presenter: Chengwu Yang, MD, MS, PhD, Associate Professor, Division of Biostatistics and Health Services Research, PQHS, UMass Chan Medical School
3/28/23: Measuring Age Acceleration in a Pooled Cohort
Presenter: Sarah Forrester, PhD, MPH, Assistant Professor, Division of Epidemiology, PQHS, UMass Chan Medical School
3/14/23: PCORI funding in total knee replacement (TKR) surgery
Presenters: Shao-Hsien Liu, PhD, MPH, Assistant Professor, Division of Epidemiology, PQHS, UMass Chan Medical School; Hua Zheng, PhD, Assistant Professor, Orthopedics and Physical Rehabilitation, UMass Chan Medical School
3/7/23: The Commonwealth’s Roadmap for Behavioral Health Reform: Along the Road from Development to Implementation
Presenter: Christie Hager, JD, MPH, Associate Professor, Division of Biostatistics and Health Services Research, PQHS, UMass Chan Medical School
2/14/23: AGO’s 2022 Cost Trends Report: How Current Approaches to Risk Adjustment May Entrench Barriers to Care and Divert Funds Away from Low Income Communities
Presenter: Sandra Wolitzky, Deputy Division Chief of the Health Care Division; and Chloe Cable, Assistant Attorney General, Health Care Division, Office of the Attorney General, One Ashburton Place, Boston, MA
2/7/23: Rapid Turn-Around Qualitative Methods
Presenter: Laël Nethania Ngangmeni, MBS, MD/PhD student (GS2), UMass Chan Medical School
1/24/23: Detection of cognitive impairment in Parkinson’s disease using speech markers: Impact of sex and age on machine learning models
Presenter: Kara Smith, MD, Associate Professor of Neurology and Neurosurgery, Co-director, Movement Disorders Division, UMass Chan Medical School
1/17/23: Informal discussion of CHIA methods for addressing SDOH factors in policy work, as described in two manuscripts
Presenters: Research team in the Center for Health Information and Analysis (CHIA)
1/3/23: A Model for Estimating the Financial Impact of “Medicare for All” on Healthcare Systems
Presenter: Eric Mick, ScD, Associate Professor, PQHS, UMass Chan Medical School
12/20/22: Analyses of Select Sexually Transmitted Infections (STI) in Massachusetts Using the All Payer Claims Database (APCD)
Presenters: Jeffrey Williams, BS, Biostatician, Quantitative Methods Core, PQHS, UMass Chan Medical School
Karen Clements, MPH, ScD, Assistant Professor, PQHS, UMass Chan Medical School
12/13/22: Adaptive Designs in Clinical/Intervention Trial: A Brief Tour of the Recent Developments
Presenter: Samiran Ghosh, PhD, Professor and Vice Chair, Department of Biostatistics and Data Science, University of Texas School of Public Health
12/6/22: Association of DNA methylation with genetic variation, cardiovascular disease and mortality
Presenter: Tianxiao Huan, PhD, Staff Scientist, The Population Sciences Branch, National Heart, Lung, and Blood Institute, National Institutes of Health and with The Framingham Heart Study
11/15/22: Accumulating Evidence for the Value of Food as Medicine
Presenter: Kurt Hager, PhD
11/1/22: Lessons from an RCT trial of Fitbit at UMass
Presenter: Bruce A. Barton, PhD, Professor, Division of Biostatistics & Health Services Research, Department of Population and Quantitative Health Sciences; Director, Quantitative Methods Core, UMass Chan Medical School
10/25/22: Extending Our Analysis Comparing Chronic Diseases in the Long-Life Family Study with the Medicare Population
Presenter: David C. Hoaglin, PhD, Adjunct Professor, Biostatistics & Health Services Research
10/18/22: The CREID: complementary randomized controlled trial and real-world study for efficacy, effectiveness, or implementation design, with illustrations for sequential, parallel, reversed sequential, and iterative designs
Presenter: Chengwu Yang, MD, MS, PhD, Associate Professor, Biostatistics & Health Services Research
10/11/22: FLnc: Machine Learning Improves the Identification of Novel Long Noncoding RNAs from Stand-alone RNA Sequencing Data
Presenter: Chan Zhou, PhD, Assistant Professor, Biostatistics & Health Services Research
10/4/22: Huge improvements from simple changes in measuring and predicting success in total knee replacement
Presenter: Arlene Ash, PhD and UMass' FORCE-TJR team
9/27/22: Statistical Tales from the QMC
Presenter: Bruce A. Barton, PhD
9/13/22: Predicting Race And Ethnicity To Ensure Equitable Algorithms For Health Care Decision Making
Presenter: Dr. Arlene Ash
7/26/22: P4E design
Presenter: Dr. Mark Friedberg
7/19/22: Interim findings of the 1115 Waiver Demonstration evaluation of MassHealth’s expansion of SUD services
Presenter: Karen Clements, MPH, ScD, Assistant Professor, PQHS
7/12/22: Reproducible Research with R
Presenter: Ben Gerber, MD, MPH, Professor and Division Chief, in the Division of Health Informatics and Implementation Science, PQHS
6/28/22: The Effect of Medicaid Accountable Care Organization on Quality of Care, Outcomes and Disparities for Children With Asthma in Massachusetts
Presenter: Sarah L. Goff MD, PhD, Associate Professor, Health Promotion and Policy, School of Public Health and Health Sciences, UMass Amherst
6/21/22: Estimation of conditional treatment effect and prediction of binary outcome using the joint use of propensity and prognostic scores
Presenter: Jonggyu Baek, PhD, Assistant Professor of Biostatistics, PQHS
6/14/22: Focus on Youth in the Caribbean (FOYC): a nationwide implementation of an HIV prevention program in The Bahamas
Presenter: Elizabeth Schieber, PhD, Postdoctoral research fellow, PQHS
6/7/22: A choice between Gosset and Satterthwaite
Presenter: Dr. Austin Lee
5/31/22: Equitable Engagement of Family Caregivers in Serious Illness Care
Presenter: Jennifer Tjia, MD, MSCE, FAAHPM of the Division of Epidemiology
5/24/22: A Novel Machine Learning Algorithm for Creating Risk-adjusted Payment Formulas
Presenter: Randall P. Ellis, PhD, Professor, Department of Economics, Boston University, Boston MA
5/17/22: Separating racial/ethnic disparities into a within-provider effect and a between-provider effects: an example using the Kitagawa decomposition
Presenter: Dr. Michael Shwartz
4/26/22: Statistical Methods for Quantitative Intersectionality Research (multilevel analysis of individual heterogeneity and discriminatory accuracy (MAIHDA) model and sequential conditional mean model (SCMM) used in Ariel’s intersectionality dissertation)
Presenters: Ariel Beccia and Jonggyu Baek
4/19/22: Addressing Disparities in COVID-19 Vaccine Access in Arkansas: Health Equity Strike Teams
Presenters: Drs. Michelle Smith and Austin Porter
3/22/22: Introducing a powerful new research platform, TriNetX, for HER data at UMass
Presenter: Dr. Adrian Zai, Chief Research Informatics Officer, and Ugur Celik, Data Scientist at UMass Chan
3/15/22: Health Equity-Based Quality Scoring
Presenter: Agniel et al. (2019). “Incentivizing Excellent Care to At-Risk Groups with a Health Equity Summary Score.” Meagan Sabatino, PhD student, CPHR, UMass Chan Medical School
3/8/22: Measuring the neighborhood-level effects of structural racism
Presenter: Zachary Dyer, MD/PhD candidate
2/22/22: Subgroup Analysis the FDA Way
Presenter: Dr. Bruce Barton, PQHS
2/15/22: Network analysis applied to evaluate transdisciplinary research teams in public health
Presenter: Dr. JungAe Lee, Assistant Professor, PQHS
2/8/22: The talk is about to review the three main core claims databases from MarketScan: Commercial, Medicare Supplemental, and Multi-State Medicaid. The presentation slide is attached with this forwarded email
Presenters: Folks from the IBM
1/18/22: Communication strategies for promoting COVID-19 vaccinations: Insights from the scientific literature and rapid response research
Presenters: Dr. Stephenie Lemon, Dr. Grace Ryan (post doc), Melissa Goulding (graduate student), Amy Borg (program director) and Princilla Minkah (research coordinator)
1/11/22: Q&A opportunity for the upcoming ‘PQHS call for proposals to support P-level Grant Development’
Dr. Catarina Kiefe, Chief Scientific Officer and Professor, PQHS
1/4/22: Intelligent Health Monitoring for Machine and Human: From Artificial Intelligence to Realization of Wearable Electronics
Presenter: Dr. Xian Du, Assistant Professor, UMass Amherst.
12/21/21: Incorporating intersectionality theory into epidemiology and population health research: Methodological challenges and advances
Presenter: Ariel Beccia, PhD candidate, PQHS
12/7/21: A paper discussion on recent maternal mortality trends in the US, that I looked for after reading a NY Times opinion piece “What We See in the Shameful Trends on U.S. Maternal Health,” by Sema Sgaier and Jordan Downey
Presenter: Dr. Arlene Ash
11/30/21: Talk about controlling the Family-Wise Error Rate (a.k.a., alpha error) in any study design involving inference – including RCTs, but other designs as well. He will cover the background of the “usual suspects” – group-sequential designs and fully adaptive designs – and then show how to plan (including sample size) and how to analyze those designs. He will demonstrate an example from his recent work with Sanifit. He will also cover the work of Frank Bretz at Novartis, used by the FDA in their guidance on multiplicity.
Presenter: Dr. Bruce Barton
11/16/21: The All of Us Research Program: Opportunities for Researchers”. Dr. Epstein will present a brief background of the All of Us Research Program, a nationwide, NIH-funded initiative to enroll one million or more US residents in a prospective research cohort, with a focus on precision medicine. She will discuss how patients are recruited and enrolled, the types of data that are collected, provide an overview of how researchers can gain access to the data, and what types of analyses that might be possible. She will also talk about the next steps for All of Us (e.g. release of genomic data).
Presenter: Dr. Mara Epstein, Meyers Health Care Institute, Department of Medicine
11/9/21: Investigation of false discovery rate in microbiome data for paired samples”. Brief description: Microbiome data that characterize human (or animal) health and disease have been gaining enormous popularity among scientists. While novel statistical research is active in this field, easy access is a method from such as differential expression in genomics. The fundamental difference, however, is that the differential abundance in microbiome means the difference in relative abundances based on taxon counts, needing special attention to the “compositionality.” With such data, controlling false discovery rate (FDR) has proven difficult, we aim to examine the well-known Benjamini and Hochberg (1995) procedure under unreplicated block design. Nonparametric tests are useful to deal with compositional data. For paired samples, the Wilcoxon-signed rank and Quade test are found to be more suitable for the FDR-based inference in microbiome data rather than paired t-test, signed test or Friedman test. To reach this conclusion, the key assumptions of FDR procedure with microbiome data were demonstrated.
Presenter: JungAe Lee
11/2/21: Standardizing the coding of SDoH data for research and clinical care at UMass Chan: Can we do it, and if so, how?
Presenters: Drs. Adrian Zai and Arlene Ash
10/19/21: A JAMA paper that he was involved in from a pharma company project in sickle cell disease, “The effect of Poloxamer 188 vs. Placebo on Painful Vaso-Occlusive Episodes in Children and Adults With Sickle Cell Disease”.
Presenter: Bruce Barton
10/5/21: “How can we fix ‘risk adjustment’? Abstract: Risk adjustment models that rely on administrative records from health care systems (demographic and enrollment information plus information on diagnoses gleaned from encounter records – and sometimes from prescribed drugs) are widely used to summarize the medical – and sometimes social – risks of individuals cared for by various provider groups and integrated health care delivery systems. The various goals of risk adjustment models include 1) allocating a fixed total budget in a way that pays more to systems that care for more complex patients, 2) setting targets for quality outcomes that recognize existing differences in what can be expected based on patient factors (e.g., hospitalization is more likely for patients with multiple serious diseases), and 3) trying to understand – and control – the apparently inexorable rise in health care spending in Massachusetts and the nation. The important function of “leveling the playing field” is threatened when equally sick patients are made to look sicker within some parts of the system than in others through aggressive “upcoding” of diagnoses. The speaker has spent years pondering how to identify and correct for differential upcoding and hopes to use this research meeting to gather additional ideas and involve others in this work.
Presenter: Arlene Ash
9/28/21: A discussion about the federal initiative to develop a “National Small-Area Social Determinants of Health Data Platform” in which we will learn about AHRQ’s SDOH (Agency for Healthcare Research for Quality’s social determinants of health) Beta-version, for which they are seeking input as to how useful it is and what they should do to enhance it.
Our discussion will follow slides from a recent AHRQ webinar as follows;
AHRQ: OS-PCORTF Project “Enhancing Patient-Centered Outcomes Research (PCOR): Creating a National Small-Area Social Determinants of Health Data Platform” Webinar Recording Available
Dr. Patricia Keenan and Dr. Jim Kirby recently presented on the project, “Enhancing Patient-Centered Outcomes Research (PCOR): Creating a National Small-Area Social Determinants of Health Data Platform.” This goal of this project is to leverage existing federal data sets and other publicly available data sources to develop a national standardized database of readily linkable SDOH variables. The database will include key information at multiple geographic levels such as income, employment, food, housing, education, health status, and healthcare access and utilization. The project has developed publicly available data files and supporting documentation (e.g., variable codebook).
The beta version of the AHRQ SDOH database includes county-level data from 2009 through 2018 and zip code-level data from 2011 through 2018 across five SDOH domains: social context; economic context; education; physical infrastructure; and healthcare context. The data can be used by external researchers to link to other datasets or look across counties and zip codes assessing SDOH-related characteristics.
• Download a recording of the webinar here
• Access the webinar slides here
• Access previous OS-PCORTF webinars here Jay Himmelstein and Arlene Ash
9/13/21: Simulated impact of transition to Medicare for All at UMMHC; The main aim is to understand the financial impact of transitioning to Medicare for All on the UMass health care system. To model this for UMMHC, we need to set out our assumptions about how “Medicare for All” would be implemented, and what that might mean for the environment in which we would operate. This presentation will present several scenarios under consideration with the express purpose of soliciting feedback and discussion of critical components of the simulation from attendees.
Presenters: Suzanne Stinson, Arlene Ash, Taylor Larusso, Eric Mick
7/27/21: A discussion about “MassHealth behavioral health in the Year of the Pandemic” – discussing the pivot from massive system reform to pandemic response and then both at once, with an eye towards opportunities for research.
Presenter: Christie Hager, JD, MPH, Managing Director, Public & Private Health Solutions in ForHealth Consulting, and the MassHealth Office of Behavioral Health
7/20/21: A meeting to discuss about "Evaluating Adoption, Impact, and Factors Driving Adoption for TREWS, a Machine Learning-Based Sepsis Alerting System" (link).
Presenter: Carl Hollins III, Research Associate
7/6/21: A meeting to discuss about "Algorithmic Bias Playbook" (link) the JAMA Internal Medicine sepsis article (about the poor performance of EPIC's algorithm; link).
Presenters: Drs. Arlene Ash and Adrian Zai
6/21/21: Excess Deaths from COVID-19 and Other Causes During the first ten months of the COVID-19 pandemic, deaths in the U.S. increased by 23% more than expected. However, COVID-19 deaths accounted for only 72% of these excess deaths. Derek Chapman, PhD, and Roy Sabo, PhD will present their team’s research on excess deaths during the COVID-19 pandemic. At the conclusion of this seminar, attendees will be able to (1) explain how social determinants of health may impact COVID-19 mortality; (2) understand methods for estimating excess deaths; (3) describe how excess death rates vary by race/ethnicity and state/region; and (4) discuss potential causes of excess deaths.
Presenters: Kerek A. Chapman, PhD is the Interim Director of the Center on Society and Health and an Associate Professor in the Division of Epidemiology, Dept. of Family Medicine and Population Health at Virginia Commonwealth University. Roy T. Sabo, PhD, is an Associate Professor in the Department of Biostatistics at Virginia Commonwealth University.
6/1/21: Exploring Secure Messaging Content and Associations with Diabetes Physiological Outcomes. Used this presentation to elicit feedback (analytic methods, ideas for other analyses, etc) at an upcoming AcademyHealth symposium.
Presenter: Dr. Stephanie Robinson, Research Health Scientist in VA Bedford Healthcare System, Research Assistant Professor, Boston University School of Medicine
5/18/21: A casual (informal) discussion about a research design question regarding how to evaluate a program to deliver medically tailored meals to Medicaid members with chronic diseases (such as diabetes).
Presenter: Matt Alcusky
4/27/21: Discussion about the Virginia Health Opportunity Index.
Presenter: Rexford Anson-Dwamena, Senior Epidemiologist/Spatial Analyst (GIS), in Virginia Department of Health
4/20/21: PolicyMap training.
Presenter: Betsy Minnich
4/13/21: The Medical Care Blog from one of its founding co-editors Jessica Williams, PhD, Assistant Professor of Health Policy and Management at the University of Kansas Medical Center. PQHS Research Coordinator Franny Eanet, MS, will present “A Novice’s Introduction to Blogging” based on her recent experience writing a blog post “Rewarding ACOs that Manage Complex Patients,” and PQHS MD/PhD student Zach Dyer will share his experience writing his award-winning blog post “Want to Be an Antiracist? Expand Medicaid (Or End It).”
Presenters: Franny Eanet, Jess Williams, Zachary Dyer
3/23/21: An efficient segmentation algorithm to estimate sleep duration from actigraphy data”. A brief description: sleep duration is a recognized determinant of mental health, obesity, and CVD across the lifespan. Actigraphy is a viable alternative as an objective measure of sleep duration compared to self-reported sleep diary. In the paper, a new efficient algorithm to obtain robust and consistent sleep duration is proposed 1) by identifying temporal segments via pruned dynamic programming and by applying a calling algorithm for sleep evaluation of each identified temporal segment. A new proposed algorithm is also applied in the Multi-Ethnic Study of Atherosclerosis (MESA) and the Early Life Exposure in Mexico to Environmental Toxicants (ELEMENT) study.
Presenter: Jonggyu Baek
3/9/21: Early effectiveness of COVID-19 vaccination with BNT162b2 mRNA vaccine and ChAdOx1 adenovirus vector vaccine on symptomatic disease, hospitalisations and mortality in older adults in England
Presenter: Arlene Ash
2/23/21: The knowledge base behind your advocacy legislation guaranteeing primary care for all residents of MA using a global payment
Presenter: Wayne Altman
2/9/21: Approaches to sample size estimation for complex study designs with multiple primary outcomes. These study designs typically require simulation to determine the sample size and this example from a recent FDA submission provides insight into the process. Future sessions will dig deeper into the details and other aspects of the testing of a new drug.
Presenter: Bruce Barton
2/2/21: Connecting Randomized Controlled Trials and Real World Studies: Seamless Design and Implementation, with Illustration of a Large Child Abuse Prevention Study
Study description: For decades, randomized controlled trials (RCT) have been deemed as the “gold standard” for evaluating the efficacy of interventions. However, due to its inclusion and exclusion criteria, RCTs have well-known limitations of their generalizability to the general population in the real-world setting. As a result, real-world studies (RWS) are gaining increasing interests in recent years, given their abilities of generating more realistic and generalizable results than RCT, or even offering stronger evidence for efficiency of an intervention in a real-world setting. And RWS has been increasingly valued by regulators and payers. Even with conceptual distinctions, RCT and RWS can happily co-exist and complement each other. Here we report our efforts towards this direction, through seamless design and implementation of RCT and RWS in two phases of a large child abuse prevention study, with published articles.
Presenter: Chengwu Yang
1/12/21: The issue of race/ethnicity imputation given that MassHealth data have 40-50% missingness in one of the consultants on the project from Clark University’s Geography Department. She will share not only the methodology and validation, but also she will share some of the outputs.
Presenter: Judith Savageau
12/22/20: Predicting adolescent HIV risk behaviors and intervention responsiveness for precision HIV prevention: a deep learning approach
Presenters: Feiban Liu and Bo Wang
12/15/20: The Area Deprivation Index (ADI) isn’t what it seems to be.”
Brief description: the ADI, a summary score based on 17 census variables measured at the census block group (CBG) or census tract level, is used widely in research, public health planning and resource allocation as an index of neighborhood socio-economic distress. We will explore the surprisingly important role of “technical” choices, such as how/whether included variables are standardized and how CBGs are weighted, on what the ADI actually measures, and contrast it with our (UMass) neighborhood stress score (NSS) developed for use by MassHealth starting in 2016. The ADI and NSS rate some CBGs in Massachusetts at opposite “poles” of distress. We will explore what drives such anomalies and solicit opinions from attendees as to what choices to use as we update the NSS.
Presenter: Zachary Dyer
12/8/20: Risk adjustment and primary care
Learning objectives:
1. Understand what risk adjustment is and the underlying rationale for its range of applications
2. Understand the broad mechanics of how “risk adjustment” is done (using our work with MassHealth as an exemplar)
3. Understand why risk adjustment is important for primary care both as and “enterprise” and for individual PCPs
4. Learn how your coding practices and affect how you or your organization is either paid or judged on quality outcomes
Presenter: Arlene Ash
11/17/20: Hospital Admission Quality Measure for Individuals with Diabetes: Importance of Accounting for Morbidity and Social Determinants of Health” The following is a brief description. MassHealth includes a hospital admission quality measure for individuals with diabetes (DM). For fairer comparisons across health plans, it is important to account for the challenges inherent in caring for medically and/or socially high-risk members with diabetes. We estimated increasingly rich models predicting hospital admission for individuals with diabetes using demographics, medical diagnoses and SDH factors as predictors. We compared these models’ predicted rates with actual admission rates for subgroups of interest among 48,172 MassHealth managed care eligible members with DM, ages 18-64, and for the health plans that enroll these members.
Presenter: Hassan Fouayzi
10/27/20: Benchmarking expectations for interventions to reduce emergency department use among today’s high-users
Presenter: Edwin Boudreaux
10/20/20: Illustrating variability in ED visit use with the boxplot – a valuable, underused graphical format: "Complex Patients Have More Emergency Visits: Don’t Punish the Systems That Serve Them” that asks the basic question: When holding health plans accountable for patient outcomes, how can we distinguish populations that do better because they were “low risk” to begin with from those receiving better-than-average care? This paper is currently under an invited resubmission review at Medical Care. We enclose the Abstract for context in addition to the figure (link). We will discuss the advantages and limitations of the general boxplot format, as well as the particular design choices made for this figure.
Presenter: Arlene Ash
10/6/20: Dr. Bryce Bludevich, a General Surgery resident at UMass, will present a on-going research, “Association between the modified Frailty Index and outcomes following Lobectomy”. Description: Using the National Surgical Quality Improvement Program (NSQIP) database we identified patients from 2010-2018 who underwent an elective VATS or open lobectomy for any indication. Using the Charlson Comorbidity Index and the modified Frailty Index (11-point and 5-point) we aim to determine which index best predicts early (within 30 days) post-operative complications. Dr. Bryce Bludevich presents the current on-going research with Dr. Feiran Lou and and Dr. Isabel Emmerick in hopes that they find anyone interested in collaborating the project.
Presenter: Bryce Bludevich
9/29/20: Public Policy Considerations in Research Design
Presenter: Robert W. Seifert
9/15/20: Sample integrity in high dimentional data: outliers and missing data analysis
Presenter: JungAe Lee
8/25/20: Obermeyer et al. (2019), “Dissecting racial bias in an algorithm used to manage the health of populations” to examine and critique the technical modeling they use and present them in our statistical methods meeting. This would be a great opportunity for anyone to learn and discuss about analytic techniques for examining equity and fairness.
Presenter: Sarah Forrester
8/18/20: Loss of Community Tenure as a Risk Adjusted Quality Measure”. The following is a short study description. When patients receive services at a hospital, rehabilitation center, mental health facility, nursing home or emergency department, they incur a loss of community tenure (CTloss). We will examine MassHealth members with a diagnosis of bipolar, schizophrenia, and/or psychosis (BSP) disorder in 2017, calculate and model days of CTloss in 2017 (a number between 0 and 365), and describe issues that arise in modeling CTloss for use as a risk adjusted quality measure in the BSP population.
Presenter: Nienchen Li
8/11/20: An introductory talk about our modeling work for MassHealth (both to predict payment and to produce risk adjusted quality measures) – the kinds of things that we do and don’t worry about, and how we test models to see whether they are “good enough
Presenter: Arlene Ash
8/4/20: A statistical issue about building a multilevel model of resident and facility factors related to developing suicidal ideation among nursing home residents. One of the main challenges that her team has been encountering is the issue of ideation being a relatively rare outcome.
Christine Ulbricht
7/21/20: Adolescent HIV-related behavioral prediction using machine learning: a potential foundation for precision HIV prevention. Dr. Bo is the first author, and Dr. Liu and Dr Ash are co-authors.
The following is a brief summary.
Precision prevention is increasingly important in HIV prevention research to move beyond universal interventions to those tailored for high-risk individuals. The current study aims to develop machine learning approaches for predicting adolescent HIV risk behaviors (multiple sex partner and ever had sex) and understanding behavioral, social and environmental determinants of those risk behaviors. We used a comprehensive longitudinal dataset from a HIV intervention study on 2564 grade-10 students in the Bahamas. We explored four classic machine learning algorithms; random forests and support vector machines achieved best performance on predicting ever had sex and multiple sex partner respectively, yielding AUC scores of 0.87-0.89 on unseen testing data. Cost-sensitive learning helped to combat the data imbalance issue for multi sex partner. Important predictors were also identified for both target risk behaviors. This exploratory research holds great potential for implementing targeted intervention strategies towards precision HIV prevention.
Presenter: Feifan Liu
6/2/20: High-frequency hospital users: the tail that wags the dog?
Description: To describe the characteristics of high-frequency hospital users (with 4 or more hospitalizations in a year) and the consequences of including or excluding their data from 30-day readmission measure
Presenter: Hassan Fouayzi
5/19/20: “Factors Associated with 90-day Mortality in Patient Undergoing Thrombectomy for Acute Ischemic Stroke”, to investigate baseline clinical and procedural factors associated with 90-day mortality in patients undergoing mechanical thrombectomy for emergent treatment of acute ischemic stroke. Findings may distinguish patients who may not benefit from endovascular intervention, help clinicians evaluate post-thrombectomy prognosis, and identify potential areas to improve upon mortality and complication rates in the modern era of mechanical thrombectomy. For more details, see the attached.
4/28/20: The COPE- Clinical Manifestations Study (COPE-ECM) and solicit practical and conceptual advice regarding how to identify the study sample.
The COPE- ECM study will recruit 600 individuals who have undergone SARS-COV2 (COVID-19) testing at UMMHC facilities: 300 COVID+ and 300 COVID-. These individuals will be interviewed by phone following a Computer-Assisted Telephone Interview (CATI) protocol and be compared for clinical presentation as well as clinical outcomes between the two groups.
The goals are to:
1) Identify possibly novel symptoms that suggest a differential diagnosis of COVID vs. other disease processes with overlapping symptomatology.
2) Develop a predictive tool to distinguish between people most and least likely to test positive based on just symptoms and other patient characteristics.
3) Compare the subsequent clinical trajectories of COVID-tested patients:
a. Between COVID+ and COVID- patients.
b. Among COVID+ patients, explore differences in the clinical course based on early symptoms and other characteristics.
The EPIC COVID registry will be used to identify and COVID+ and COVID- patients in this diverse cohort will be surveyed, so as to be able to explore these data for evidence of racial/ethnic and socioeconomic differences. Interviewers (both English-speaking and Spanish-speaking) will use pre-populated CATI scripts with patient-specific data from the EMR.
Presenter: Arlene Ash
3/31/20: “Machine Learning-based Outcome Prediction & Hypotheses Generation for Substance Use Disorder Treatment” to get valuable feedback from UMass Chan colleagues to improve the paper before it goes to the targeted journal and 2) to find new UMass Chan colleagues who might be interested in doing research with Dr. Oztekin in healthcare analytics, medical informatics, clinical decision support using machine learning, data mining, etc.
Presenter: Asil Oztekin's team
3/17/20: A discussion about the attached paper (Austin 2011), “An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies”. The paper describes four different types of propensity score methods at the introductory level: 1) matching, 2) stratification, 3) weighting, and 4) covariate adjustment.
Presenter: Arlene Ash
2/25/20: A discussion about an attached paper (Kolak et al., 2020), “Quantification of Neighborhood-Level Social Determinants of Health in the Continental United States”. Briefly, the authors in this article suggest a new set of social determinants of health measures. It goes beyond single index measures to create 4 separate indices, measuring different types of social determinants, all at the census tract level. This approach gives more information about factors that might be driving health outcomes in a particular area. It would be a useful approach for policy studies and may have other applications in observational research.
Presenter: Robin Clark
2/11/20: “Stratified randomization: what, why, and how”.
Presenter: Bruce Barton
1/28/20: Introductory propensity Score methods
Presenter: Arlene Ash
1/21/20: Sample size and power calculation by simulation
Presenter: Bruce Barton
11/26/19: A literature review and the paper by Obermeyer et al. (2019) “Dissecting racial bias in an algorithm used to manage the health of populations”
Presenter: Matthew Alcusky
11/12/19: A literature review Downs et al (2010) ,“Vulnerability-Based Spatial Sampling Stratification for the National Children’s Study, Worcester County, Massachusetts: Capturing Health-Relevant Environmental and Sociodemographic Variability.
Presenter: Hassan Fouayzi
10/22/19: Geocoding in SAS and R
Jonggyu Baek
10/8/19: Neighborhood-level risk factors: Childhood opportunity index (COI) Reading: The Child Opportunity Index: Improving Collaboration Between Community Development And Public Health
Presenter: Eric Mick
10/1/19: Q: to caculate patent costs in a step wedge design implemented in Ed Boudreaux’s System of Safety (SOS) study
Presenter: Karen Clements