Evaluation and Quality Improvement
This page provides tools and resources from ASTDD and CDC, followed by materials from many other sources:
- ASTDD Evaluation and Technical Assistance Annual Report
- General Evaluation Methods and Tools, including logic models, program evaluation and quality improvement
- Evaluation Hot Topics, such as partnerships and coalitions, communication strategies and plans, collaboration efforts, oral health surveillance systems, oral health state plans, sealant programs, mobile/portable programs, quality improvement, and policy and advocacy efforts
- Using evaluation information to communicate and demonstrate impact and outcomes.
ASTDD Program Review and Technical Assistance On Evaluation Issues
ASTDD offers State Oral Health Program Reviews and many types of technical assistance to state oral health programs. Technical Assistance Request form. ASTDD uses four consultants to provide evaluation technical assistance to state oral health programs (SOHP.) Their short bios follow. If SOHP would like TA from any of these consultants, contact Bev Isman at email@example.com
Carissa Beatty, MPH, CHES serves as an Evaluation Consultant for the Association of State and Territorial Dental Directors (ASTDD), and provides evaluation planning and implementation support to state oral health programs and to the Association. In her role as Evaluation and Networks Manager at the Emory Centers for Public Health Training and Technical Assistance, she develops, implements, and evaluates training programs for national, state, and local public health organizations. Professional interests include program planning; logic model development; strategic use of data to support partnerships, collaboration, and social innovation; and effective reporting / communication (e.g., data visualization). Carissa is a Certified Health Education Specialist (CHES), and holds an MPH from Emory University.
Dr. Mary Davis is principal and owner of Project Y Evaluation Services, LLC specializing in public health evaluation and applied research consulting, training, and technical assistance services. Dr. Davis has extensive experience with most evaluation designs and data collection approaches, including mixed-method designs. She has trained public health students and practitioners in evaluation methods through workshops, formal courses, technical assistance webinars, and employment opportunities. Further, she has provided evaluation capacity building assistance to organizations, such as coalitions, public health departments, and foundations. In her more than 20 year career, she has worked in academia at the University of North Carolina, Gillings School of Global Public Health as well a national member association and a for-profit consulting firm. Her areas of expertise include preparedness, health department accreditation, quality improvement, and partnerships.
JoAnna Hillman, MPH, is the Director of Research and Evaluation for Emory Centers for Training and Technical Assistance. Within Emory Centers, she leads the Evaluation and Quality Improvement Program, where she oversees the planning, coordination, and implementation of evaluation and quality improvement activities for Emory Centers' projects and partnerships. Prior to joining Emory, JoAnna worked for CDC as a Health Scientist in Informatics. At CDC, she provided technical assistance for the design, development, implementation, and evaluation of health information systems and co-developed trainings for health professionals to support HIV/AIDS health services in resource-constrained countries. JoAnna also has state and local public health experience working on complex interagency initiatives for health indicator improvement, outbreak and emergency response planning, and capacity building for public health workers. Her professional interests include evaluation design, survey development and methods, qualitative methods, quality and process improvement, and evaluation capacity building.
Maggie Pustinger, MPH, CHES serves as an Evaluation Consultant for the Association of State and Territorial Dental Directors (ASTDD) and provides evaluation planning and implementation support to state oral health programs and to the Association. She serves as an Evaluator at the Emory Centers for Public Health Training and Technical Assistance. Her professional interests include logic model development, survey design, evaluation capacity building, and process improvement. Maggie is a Certified Health Education Specialist (CHES) and holds an MPH from Emory University.
ASTDD Evaluation and Quality Improvement Materials
Evaluation of Training Programs. In this webinar, ASTDD evaluation consultants Mary Davis, Rebecca Fils-Aime, and JoAnna Hillman provide an overview of how to create a simple evaluation strategy for a training program. This session will help you understand how to apply assessment and evaluation during various stages of training development and implementation to increase the effectiveness of learning; outcome targets and indicators for training program evaluations; and models for evaluating training programs. (December 2021)
Qualitative Methods Webinar Series (1 of 2): "Getting to the How and the Why." In this first webinar of a two-part series, ASTDD evaluation consultants Mary Davis and JoAnna Hillman provide a complete overview of planning and designing qualitative evaluations. This session will help you understand when would be most appropriate to use a qualitative method for a program or activity's evaluation and offers helpful tips for setting your qualitative evaluation up for success. (February 2021)
Qualitative Methods Webinar Series (2 of 2): "Analysis, Reporting, and Wrap-up". In this second webinar of a two-part series, ASTDD evaluation consultants Mary Davis and JoAnna Hillman provide an overview of analysis and reporting for qualitative evaluations. This session will help you understand the approaches to qualitative analysis, the tools available to support qualitative analysis and how to approach reporting. (March 2021)
Survey Methods: From Design to Implementation - In this webinar, ASTDD evaluation consultants JoAnna Hillman and Mary Davis, and Emory Centers evaluator, Christiana Reene, provide a complete overview of surveys from planning to wrap up. It addresses good survey practices including proper question construction, developing questions that provide data you can use, and how to use logic (branching, skip, display). The session will help you identify common issues and help you to avoid pitfalls. (December 2020)
Evaluating Oral Health Surveillance Systems - In this webinar, ASTDD consultants Kathy Phipps, Mary Davis, and JoAnna Hillman provide guidance on how to evaluate oral health surveillance programs using the CDC evaluation framework. The webinar includes resources and examples from state oral health programs. (September 2020)
ASTDD Performance Management Toolkit for State Oral Health Programs - The purpose of this toolkit is to provide state oral health programs with an orientation to performance management (PM) as well as resources to further learning and competency related to PM. The toolkit grew out of an identified need among Association of State and Territorial Dental Director (ASTDD) members and requests for information and technical assistance. (August 2017)
Adopting Performance Management Strategies to Improve Oral Health in Your State - Performance Management and Quality Improvement are more than just buzz words. Performance Management and Quality Improvement approaches and tools can help state Offices of Oral Health improve their efficiency and effectiveness. This webinar provides an introduction to Performance Management and Quality Improvement approaches and tools as well as examples of how these are used in state Oral Health Offices. (April 2017)
Partnership Evaluation Webinar - This webcast enhances knowledge of key concepts related to partnership evaluation, evaluation tools, and things to consider when undertaking partnership evaluations. B.J. Tatro, PhD addressed topics including when to do partnership evaluation, how to match evaluation questions and evaluation tools to the partnership's stage of development, methods to use, stakeholders to involve, and how to utilize results to strengthen partnerships. Examples from states that are planning or conducting internal and external partnership evaluations were used and resources were shared. (February 2015)
Partnership Evaluation Webinar PowerPoint
Toolkit for Planning, Evaluating, and Improving Interagency Collaboration for Oral Health Programs - The purpose of the Toolkit is to provide a framework and specific strategies for planning, improving, and evaluating inter-agency collaboration to more effectively and efficiently address our most complicated and complex national oral health issues through strategic alliances and organizational partnerships. The toolkit contains a Handbook, a Workbook, and worksheets.
ASTDD Communications Plan Templates and Resources
Tools for planning and evaluating a single project or document and a year-at-a-glance template for multiple communication strategies.
CDC Evaluation Materials
CDC Program Performance and Evaluation Office-webpage includes CDC Framework for Program Evaluation and multiple resources and background materials on the framework. https://www.cdc.gov/eval/framework/index.htm
CDC Sealant Efficiency Assessment for Locals and States (SEALS)
SEALS software helps states and communities evaluate the effectiveness and efficiency of their school dental sealant programs.
Developing an Effective Evaluation Plan - The purpose of this workbook is to help public health program managers, administrators, and evaluators develop an effective evaluation plan in the context of the planning process. It is intended to assist in developing an evaluation plan but is not intended to serve as a complete resource on how to implement program evaluation. Rather, it is intended to be used along with other evaluation resources (2011)
Impact and Value: Telling your Program's Story
This CDC workbook describes how to develop a system for collecting and creating stories that will allow you to best communicate the accomplishments of your programs. (2007)
Practical Strategies for Culturally Competent Evaluation
This guide from the CDC's Division for Heart Disease and Stroke Prevention and the National Asthma Control Program highlights the prominent role of culture in evaluation and provides important strategies for approaching an evaluation with a critical cultural lens to ensure that evaluation efforts have cultural relevance and generate meaningful findings that stakeholders ultimately will value and use.
Success Stories & Writing Resources
Writing resources to create impactful success stories, plus a searchableÂ repository of stories from National Center for Chronic Disease Prevention and Health Promotion (NCCDPHP) grantees.
ASTDD State based Oral Health Surveillance System Conceptual Framework and Operational Definition https://www.astdd.org/docs/state-based-oral-health-surveillance-systems-cste-whitepaper-oct-2013.pdf
CDC Updated Guidelines for Evaluating Public Health Surveillance Systems https://www.cdc.gov/mmwr/preview/mmwrhtml/rr5013a1.htm
Tip Sheet: Types of Evaluation
This tip sheet from CDC's Division of STD Prevention briefly summarizes the most common types of evaluation, differences between process and outcome evaluation, as well as situations in which one would choose the different types of evaluation.
Updated Guidelines for Evaluating Public Health Surveillance Systems
(Article from MMWR, July 27, 2001;50(RR13):1-35.
General Evaluation Methods and Tools
Reversible Decay: Oral Health Is a Public Health Problem We Can Solve: This brief report shows how assessment results can be presented in a user-friendly format to non technical audiences. More information on the assessment can be found atÂ http://dentaquest.com/oral-health-resources/reversible-decay/.
WK Kellogg Foundation Evaluation Handbook
An online handbook that covers all aspects of planning for evaluation, implementing your plan and analyzing the data, as well as communicating findings and using the results.
This guide from the WK Kellogg Foundation provides a step-by-step process to develop and use different types of logic models; it includes many templates, checklists and exercises to develop a model that fits your program.
The Community Tool Box: Chapter 36. Introduction to Evaluation
An introduction to key terms and frameworks in evaluation from the University of Kansas. Includes sections on how to engage community stakeholders in evaluation and working with a professional evaluator.
American Evaluation Association
Find evaluation focused conferences and other professional development opportunities, or identify AEA members available for evaluation consulting or to serve on evaluation teams due to specific expertise in particular methodologies.
Western Michigan University, The Evaluation Center, Evaluation Checklists
Checklists for multiple types of evaluation.
Making Data Talk Workbook
The National Cancer Institute's workbook explains how to present scientific and health data to decision-makers and the public in engaging and effective ways. It is based on the book "Making Data Talk: Communicating Public Health Data to the Public, Policy Makers, and the Press."
Evaluation Hot Topics
Equity and Evaluation
Better Evaluation an international resource guide to evaluation processes, management, communications and other resources https://www.betterevaluation.org/
Equitable Evaluation Initiative-Resources and framework to conceptualize, implement, and utilize evaluation to promote equity. https://www.equitableeval.org/
Expanding the Bench-evaluation network that promotes diverse perspectives and experiences that inform evaluation methods to benefit communities. https://expandingthebench.org/
Partnerships and Coalitions
Evaluating Community Partnerships Resource List
Resource list compiled by Public Health Ontario, Partners for Health.
Butterfoss FD. Evaluating partnerships to prevent and manage chronic disease. Prev Chronic Dis 2009;6(2):A64
Performance Measurement for Public Health Policy
Joint publication of the American Public Health Association (APHA) and the Public Health Foundation (PHF) from 2012.
Chriqui, J., O'Connor, J., and Chaloupka, F. (Spring 2011). What Gets Measured, Gets Changed: Evaluating Law and Policy for Maximum Impact. In Using Law, Policy, and Research to Improve the Public's Health
Seal America. The Prevention Invention, Step 10, Program Evaluation
Good guide for determining school-based dental sealant quality, determining program cost, program participation and other measures.
Evaluation, Performance Management, and Quality Improvement: Understanding the Role They Play to Improve Public Health; Craig Thomas, PhD Liza Corso, MPA Harald Pietz
Presentation that provides a nice overview of how Program Evaluation aligns with Quality Improvement and Performance Management in public health.
Embracing Quality in Public Health:Â A Practitioner's Quality Improvement Guidebook
This second edition of Embracing Quality in Public Health reflects five years of study, action, and learning among the authors and many other generous, thoughtful individuals who contributed their time and talent to informing this resource. It is a comprehensive resource on planning and implementing QI activities in a health department.
Population Health Improvement Partners. Step by Step QI Guide
NACCHO QI Road Map
The Quality Improvement Roadmap provides health departments with guidance on progressing through six phases of QI maturity. This resource also describes six foundational elements of a QI culture that health departments should cultivate over time.
The Institute for Healthcare Improvement, in collaboration with the Public Health Foundation, hosts a Public Health Resources page addressing quality improvement, including changes, measures, and tools for improvement that can be used in a public health setting.
Public Health Foundation
The Public Health Foundation provides QI tools, training, resources, and technical assistance to state and local health departments.
Applying Quality Improvement Measurement to Population Health Initiatives
Article in AMCHP PULSE July/August 2016.
Population Health Improvement Partners. Step by Step QI Guide
Walks you through each phase of a QI project and provides examples, tools and templates.
Moving from Goal to Impact: A Quality Improvement Approach to Advancing Children's Oral Health in Medicaid
This CMS Technical Assistance Brief (2015) describes a step-by-step strategic planning approach to help states design effective oral health action plans. It highlights approaches that represent the interests of a diverse set of key stakeholders, are feasible to implement, are measurable, and are likely to achieve the desired impact on children's oral health.
QI Resources for Epidemiologists. AMCHP 2014
Material for this publication originated from a roundtable discussion with epidemiologists about their needs to successfully take part in QI projects.
EQIPP Oral Health Course
This course from the American Academy of Pediatrics delivers all the tools and resources you need to identify and close practice gaps in key clinical oral health activities! This quality improvement course helps you address: performing oral health risk assessment, providing oral health anticipatory guidance and education, ensuring fluoride varnish application for high-risk patients, and establishing a dental home. The course may be taken by pediatricians, nurse practitioners, and allied health care professionals. The cost is $199 for non AAP members.
Examples of Performance Management approaches that can be adapted for oral health.
https://www.phii.org/sites/default/files/resource/pdfs/1 FINAL Biosurveillance Report_092013.pdf
https://www.phii.org/sites/default/files/resource/files/Understanding Surveillance Data Exchange Processes Guide and Worksheet.pdf
https://www.phii.org/sites/default/files/resource/pdfs/Healthy Weight Overview and Implementation Guidance.pdf