Question Bank and Co-Curricular Resource Evaluation

Process

Scope and Goals

This project came about from multiple angles. The University of Illinois College of Medicine (UICOM) provides students who are about six months away from taking their first board exams a question bank resource with which to study for the board exam. UICOM also provides a co-curricular resource that includes a question bank, thousands of relevant videos and other content to help students study. Faculty use the resource as well by reviewing content within the system and utilizing many of the videos as pre-work for students to complete so they can contribute to interactive class discussions.

UICOM has course evaluation data with open-ended responses to questions that ask about 3rd party resource use for studying and student leaders sharing information about 3rd party resource use with administration. These data reflect that while students do use resources provided by the college, they also use and pay for additional study resources that are not provided by the college like Boards & Beyond and Amboss.

Objective: The main objective of this project was to investigate resources that are not provided by the college and are known to be used by students. Criteria for resources to evaluate were based on student feedback.

Organizing the Project

Timeline: This project started in late 2022 and should be completed by late April 2023.

Recruiting Faculty: I reached out to learning specialists and faculty members who have expertise in the subject matter and are willing to participate in the evaluation process. The faculty and learning specialists who agreed to participate also represent the three campuses for the college. I explained the goals of the project, the criteria for selecting resources, and the timeline. I provide them with the necessary training and support, such as access to the resources, evaluation rubrics, and other communication tools.

The question bank resources were evaluated by learning specialists and faculty evaluated co-curricular resources. Two evaluations were created in Qualtrics: one for learning specialists to complete for question bank evaluation and the other for the faculty to complete. Each group was asked to provide feedback via an evaluation tool for a minimum of 3 (out of 5) products which required them to complete the evaluation for each project. A virtual hard copy of each evaluation tool was provided to each group to use as a guide during review of the tools.

Contacting Vendors: Once the list of 5 question bank and 5 co-curricular resources were finalized, I contacted the vendors for each product to request trial access for faculty and learning specialists. I was clear about the evaluation process and the timeline and requested additional materials or support that would facilitate the evaluation process.

Conducting the evaluation: I have provided the faculty and learning specialists evaluators with access to the resources, the evaluation rubrics, and the communication tools. I set a deadline for completing the evaluation, and have scheduled regular check-ins to discuss progress and provide feedback.

You can see the evaluations used by both groups here.

Co-curricular Resource Review Evaluation

Question Bank Resource Review Evaluation

Analyzing the Results and Making Decisions

Analyze the results: I will collect and analyze the data from the evaluation process, using the evaluation rubrics and other criteria. The evaluations completed by the faculty and learning specialists will help us identify the strengths and weaknesses of each resource, and compare them to determine the best fit for the project goals.

Making a Decision: Based on the analysis of the evaluation results, we will make a decision on which resources to select, if any. We will then determine if the tools that are most highly recommended fall within the budget allocated for these types of resources. If so, we will then communicate the decision to the stakeholders, and provide feedback to the vendors and the faculty evaluators.

Implement and Monitor: If budget allows, we will implement the selected resources in the project, and monitor their effectiveness and impact. We will collect feedback from the students, faculty, and learning specialists, and make adjustments as necessary. We will use the entire process to inform future decisions and improve the process.

Results and Takeaways

This project is in-process.