Question Bank and Co-Curricular Resource Evaluation

Process

Scope and Goals

This project came about from multiple angles. The University of Illinois College of Medicine (UICOM) provides students who are about six months away from taking their first board exams a question bank resource with which to study for the board exam. UICOM also provides a co-curricular resource that includes a question bank, thousands of relevant videos and other content to help students study. Faculty use the resource as well by reviewing content within the system and utilizing many of the videos as pre-work for students to complete so they can contribute to interactive class discussions.

UICOM has course evaluation data with open-ended responses to questions that ask about 3rd party resource use for studying and student leaders sharing information about 3rd party resource use with administration. These data reflect that while students do use resources provided by the college, they also use and pay for additional study resources that are not provided by the college like Boards & Beyond and Amboss.

Objective: The main objective of this project was to investigate resources that are not provided by the college and are known to be used by students. Criteria for resources to evaluate were based on student feedback.

Organizing the Project

Timeline: This project started in late 2022 and was completed by early 2024.

Recruiting Faculty: I reached out to learning specialists and faculty members who have expertise in the subject matter and are willing to participate in the evaluation process. The faculty and learning specialists who agreed to participate also represent the three campuses for the college. I explained the goals of the project, the criteria for selecting resources, and the timeline. I provide them with the necessary training and support, such as access to the resources, evaluation rubrics, and other communication tools.

The question bank resources were evaluated by learning specialists and faculty evaluated co-curricular resources. Two evaluations were created in Qualtrics: one for learning specialists to complete for question bank evaluation and the other for the faculty to complete. Each group was asked to provide feedback via an evaluation tool for a minimum of 3 (out of 5) products which required them to complete the evaluation for each project. A virtual hard copy of each evaluation tool was provided to each group to use as a guide during review of the tools.

Contacting Vendors: Once the list of 5 question bank and 5 co-curricular resources were finalized, I contacted the vendors for each product to request trial access for faculty and learning specialists. I was clear about the evaluation process and the timeline and requested additional materials or support that would facilitate the evaluation process.

Conducting the evaluation: I provided the faculty and learning specialists evaluators with access to the resources, the evaluation rubrics, and the communication tools. I set a deadline for completing the evaluation, and scheduled regular check-ins to discuss progress and provide feedback.

You can see the evaluations used by both groups here.

Co-curricular Resource Review Evaluation

Question Bank Resource Review Evaluation

Analyzing the Results and Making Decisions

Analyze the results: I collected and analyzed the data from the evaluation process, using the evaluation rubrics and other criteria. The evaluations completed by the faculty and learning specialists helped us identify the strengths and weaknesses of each resource, and we compared them to determine the best fit for the project goals.

Making a Decision: Based on the analysis of the evaluation results, the college made a decision to select ScholarRx. The decision was relayed to the stakeholders, and feedback to the vendors and the faculty evaluators.

Results and Takeaways

I left the University of Illinois College of Medicine in January 2024, however, I am aware that the product that was selected was ScholarRx.