Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.” – Albert Einstein
Why Evaluate Mentoring?
Effective data-based evaluation of mentoring is important for at least three reasons:
1. Helps to overcome biases and make good decisions about needed programmatic or mentoring relationship change.
2. Enables appropriate allocation of departmental resources and efforts towards supporting the most effective elements of mentoring.
3. Allows sharing of success with stakeholders and participants. Evaluation is an important tool in your mentoring program toolbox.
When to Evaluate
- Align with your departmental calendar and events (with the natural ebb and flow of your department). Select times when your faculty are not already overburdened or absent.
- Ideally, pre-and-post measures would be collected to assess change.
- Interim measurement if possible (depending on how long your faculty members are involved in the program). A short “check-in” survey or email to all participants within the first few weeks of involvement in the program provides an additional data collection point.
Who to Evaluate
It is important to collect information not just from the mentees, but from the mentors in the program as well (and other stakeholders). Collecting information from mentors provides an important perspective on both mentees’ and mentors’ experiences. Information collected from mentors, especially about benefits of mentoring, can inform future mentor recruitment efforts.
Anonymous versus Confidential Evaluations
An important decision to make is if the information you collect from individuals will be anonymous or confidential. This needs to be made clear to all individuals providing data. Anonymous means that not even the department chair or mentor program director knows who provided the response. Anonymous surveys increase response rates but limit your ability to intervene or seek more information should a respondent report a negative outcome. Confidential surveys clearly articulate who will have access to the responses and assure the respondents that their answers will not be shared beyond those individuals.
- Response rate ˂ 30% reflects a burden placed on the respondent
- Response rate of 50-80% is ideal
How to Evaluate
Use more than one method to collect data on your program. This provides stronger evidence that the information converges on similar themes.
| Pros and Cons of Measurement Methods | ||
|---|---|---|
| Methods | Pros | Cons |
| Archival analysis | Records are already available, increased accuracy of data | Some records may be incomplete or missing |
| Focus Groups | Provides in-depth information, opportunity to ask for clarification | Fewer responses, time-intensive to record responses and analyze them |
| Interviews | Can obtain sensitive information | Time-intensive, potential response bias |
| Observation | Easy to collect estimates of participation and engagement in real time | Can be time-intensive if looking for more nuanced behaviors |
| Photographic evidence | Visual images are powerful ways to convey success | Expense to hire photographer, poor-quality pictures are not useful |
| Surveys | Easy to administer and tabulate responses | Response bias, may not ask correct questions |
How to Ask Questions: Top Ten Practices
1. Use memory prompts and time frames in surveys, interviews, and focus groups (e.g. in the last 6 months….)
2. Write short, succinct questions.
3. Go beyond “yes” and “no” questions (categories) and collect continuous data when possible.
4. Place demographic information at the end of surveys.
5. Be selective in choosing what questions to ask, or even what data to collect.
6. Use online surveys when possible.
7. Test your questions!
8. Word questions so that you can compare the data you collect with information collected in prior years, or data from other programs.
9. Ask simple questions that do not require complex calculations.
10. Make sure your surveys are visually easy to follow and think about how your survey would be used on a mobile device.
Tips for Increasing Response
1. Keep survey short – limit to 10 questions if possible.
2. Advance announcement of survey by department chair, dean or other administrative leader.
3. Send reminder survey announcements.
4. Report results back to the respondent pool to increase likelihood of future responses to survey requests.
What to Evaluate
What you evaluate should be guided by your program’s theory of change or logic model.
What you measure is related to the activities, expectations, and desired outcome of your mentoring plan.
While it should be individualized to your department, the logic model below is illustrative of a mentoring program logic model that might contain elements common to many departments.
This resource was developed as a collaborative effort between the UNC Center for Faculty Excellence and the UNC School of Medicine Mentoring Task Force.