Learning Management System (LMS) Refresh

Overview

Client
University of Miami Miller School of Medicine

AI-generated image of a person with red hair in a bun and wearing glasses sitting in front of a computer giving a thumbs-up.

Responsibilities
Project management, time management, vendor relations

Target Audience
Medical school faculty, staff, and students

Tools Used
Microsoft O365, Smartsheet

Budget
N/A (internal project)

Year(s) / Project Duration
2024-ongoing

Process

Overview

There has been significant dissatisfaction with the existing learning management system (LMS) used for the clinical portion of the curriculum at the University of Miami Miller School of Medicine. The system used for clinical education is Elentra. This project developed functional and technical requirements for LMSs supporting the NextGenMD curriculum, which then enabled us to better formally evaluate available LMS solutions. With the resulting data, we can determine the best path forward to ensure the success of the curriculum. The focus for this review will be for Phases 2 (clerkship year) and 3 (final courses and electives) as Phase 1 utilizes a LMS that is stable and supported by UM (Blackboard). The project will result in recommendations to Miller School leadership to aid and identify a simple and sustainable learning management solution for the NextGenMD curriculum. 

Scope and Goals

The project scope is aligned with the curriculum and includes the following areas: 

Phase 1 Scope 
Phase 1 is the first 14 months of the curriculum. It is a student-led, faculty-facilitated case-based collaborative approach to learning foundational sciences. Phase 1 has early clinical experiences including serving as patient navigators and learning emergency medical skills. 

Phase 2 Scope 
This 12-month phase consists of four 12-week blocks of integrated clerkships. Basic science, longitudinal topics, simulation, standardized patient experiences, and competency-based assessments are integrated into every clerkship. It is possible some of the integrated clerkships will be separated out within the next two years. 

Phase 3 Scope 
This final 17-month phase offers students a more personalized, varied curriculum experience. Students can explore career goals and interests, and they complete dual degrees and other scholarly projects. By the end of Phase 3, students have achieved competency as sub-interns and are ready for an internship (residency). 

Organizing the Project

Project Dates and Milestones 

Target Start Date: Project kickoff on September 30, 2024 

  • Stage 1 – Clarify Requirements: 6 Weeks (September 30 – November 8, 2024)  
  • Stage 2 – Market Research: 8 Weeks (November 11 – January 8, 2025) 
  • Stage 3 – Product Evaluation: 12 Weeks (November 11 – January 31, 2025) 
  • Stage 4 – Report Development: 4 Weeks (February 3 – February 28, 2025)

Target Completion Date: Deliver final report to leadership with recommendations by May 31, 2025 

High Level Risks 

  1. Elicitation and Documentation of Requirements: There is a risk associated with the ability to accurately elicit and document requirements from subject matter experts (SMEs). This could lead to misunderstandings or incomplete requirements, which may affect the project’s success. 
  1. Scope Creep: Without well-defined project objectives, there’s a risk of scope creep, where additional requirements are added without corresponding increases in resources, potentially leading to project delays or budget overruns. 
  1. Curricular Standards: Without standardization and adherence to best practices within the curriculum, there is a risk that no LMS can be adapted or identified to meet our needs. 
  1. Resource Availability: The project may face risks related to the availability of necessary resources, including personnel, technology, and materials. 
  1. Technological Changes: Rapid changes in technology could render project solutions outdated or require additional training and resources before they are implemented. 
  1. Communication Risks: Ineffective communication can lead to misunderstandings, delays, and errors in project execution. 
  1. Stakeholder Engagement: Insufficient involvement or buy-in from stakeholders may lead to resistance or lack of support for the project initiatives. 
  1. Integration Risks: Challenges in integrating new systems or processes with existing infrastructure could lead to technical issues or delays. 
  1. Legal and Compliance Risks: The project may encounter risks related to legal requirements, licensing, or compliance with industry standards. 
  1. Security Risks: Risks associated with data security, privacy, and protection against cyber threats. 

Assumptions and Constraints 

Assumptions: 

  • It is assumed that all necessary stakeholders and subject matter experts will be available for requirement gathering sessions and will provide timely and accurate information. 
  • The project team assumes that the current technology infrastructure will support the new system without significant upgrades. 
  • It is assumed that the project will have access to all necessary resources, including personnel, technology, and materials, as planned. 
  • The project assumes that no significant changes in organizational processes or policies will occur during the project timeline that could impact project delivery. 
  • It is assumed that all vendors involved will adhere to the agreed-upon timelines and deliverables. 

Constraints: 

  • The project timeline is constrained by the target completion date, which may limit the scope of work that can be accomplished. 
  • Resource constraints, including the availability of skilled personnel and subject matter experts (SMEs), may impact project progress and outcomes. 
  • Contractual constraints with vendors may limit negotiation flexibility and the ability to make changes to the project scope or deliverables. 
  • Regulatory and compliance constraints may dictate certain project requirements and limit the choice of solutions. 

These assumptions and constraints should be reviewed regularly and updated as necessary throughout the project to reflect any changes in the project environment or external factors. It’s important to document these elements in the project charter to provide clarity and set realistic expectations for project stakeholders. 

Functional Roles 

These are roles that will directly interact with the system. Which of the following best describes your role in the curriculum? How do you (or will you) interact with the system? We may evaluate functional areas of a system differently depending on the role.  

  • Leadership (Deans, Directors) 
    • Data analysis and interpretation
    • Indirect interaction
  • Administration / Operations (Program Managers, Coordinators, Instructional Design, etc.)
    • Data entry
    • Updating
    • Report generation
  • Faculty (Instructors, Preceptors, LCEs, etc.) 
  • Students
  • Technical Support (Systems administrators, End User Support, Development, Data Management) 
    • Report development 
    • Data extractions 
    • User account management and auditing 

Functional and Technical Requirements 

Detailed technical requirements will be documented within the LMS Product Evaluation Matrix Excel workbook. It details the requirements that must be present to meet the stakeholders’ needs and expectations.  

  1. Integration Capabilities: Ability to incorporate single sign on (SSO), APIs, LTIs. 
  1. Client Software Requirements: Compatibility with institutional default browsers 
  1. Communication Tools: Features that facilitate communication between students and instructors, and among students. 
  1. Data Management: Ability to import / export data and data retention features. 
  1. Data Analysis Tools: Ability to generate reports and dashboards for data analysis. 
  1. Implementation and Upgrades: Ease of implementation and upgrades / enhancements. 
  1. Security and Privacy: Regulatory compliance and custom roles, permissions, etc. 
  1. User Management/Administration: Features that allow for specific role-based access and control 
  1. Student Experience and Personalization: Features that allow for mobile friendly UX, progress tracking, etc. 
  1. Faculty Experience and Personalization: Features that make the system more user friendly and personalized. 
  • Mobile friendly UX: Responsive design or user mobile application with every feature faculty might need. 
  • Ability to use speech-to-text to ‘write’ narratives, provide feedback to students, etc. 
  • Ability for faculty to save drafts of narrative assessments. 
  • Identify narratives in need of improvement based on pre-determined criteria. 
  • Clinical preceptor portal. 
  1. Accessibility: Features that enable full accessibility for all users. 
  1. Availability and Usability: Ability to always provide access for all users, excluding downtime for upgrades. 

Other Items for Evaluation 

Detailed technical requirements will be documented within the LMS Product Evaluation Matrix.xlsx Excel workbook. It details the requirements that must be present to meet the stakeholders’ needs and expectations. More specific details around requirements can be found within the LMS Review Project – Functional Technical Other Requirements.docx. 

  1. Support and Training: The provider should offer adequate support and training to help users make the most of the system. 
  1. Sandbox / Test Environment for Evaluation: Ability to create dummy courses to test features. 
  1. Strength of Financial Proposal 
  1. Market Share 
  1. Health and Stability of the company 
  1. Chance of being approved or denied through university governance 

Project Stages (phases) 

This section describes how the project will be carried out. 

  1. Clarify and document all functional and technical requirements of the NextGenMD curriculum – By November 8, 2024.  
  • Stakeholder survey to identify primary role and ensure we have all user needs 
  • Stakeholder meetings (by phase/area) to review survey results and clarify priority of user needs. 
  1. Conduct comprehensive research to identify viable LMS products. This includes both market research and information obtained from other medical schools. 
  • Use the AAMC GIR Survey and SCOPE Surveys to identify what other schools are using 
  1. Evaluate the LMS products we identify against the documented requirements. 
  • Rank products on a preliminary score sheet to filter it down to ~5 
  • Schedule demonstrations with top vendors based on initial rankings 
  1. Prepare and deliver a detailed report outlining the evaluation process, findings, and recommendations, to the Executive Dean for Medical Education. The report will include resources required to implement and maintain the desired future state solution.  

Project Strategies 

Strategies for communication and utilizing our time and resources effectively.  

  • Provide project status updates to leadership during Curriculum Council, Executive Curriculum Committee, and OME Team meetings.  
  • Leverage existing curriculum operations team meetings to discuss project-related topics. 
  • Hold interviews with each functional area (Phase 1, Phase 2, etc.) to clarify requirements. Interviews could utilize existing implementation team meetings for each area.  
  • A single large meeting will be organized to review our findings and answer questions.  
  • All product demos will be recorded and available to project stakeholders.  

Operational Framework 

  • The project team will meet weekly.  
  • Where will project information be stored?  
  • How will the team(s) communicate outside of meetings?  
  • What tools will be used to track project performance (tasks, schedule, etc.)?  

Project Closeout 

  • Who is responsible for maintaining the project outcomes (product, service, etc.)? Identify the service owner and support model used.  
  • Incorporate essential information into operational plans and systems/service documentation. 
  • Incorporate recurring licensing and infrastructure costs into the budget.  

Products for Evaluation 

Current and potential products for evaluation during the project include: 

Terms and Definitions 

Define any unique terms or acronyms which are used during the project.  

Learning Management System (LMS): The Learning Management System for the NextGenMD Undergraduate Medical Education Curriculum extends beyond a traditional LMS. It includes educational delivery, assessment of student performance, clinical rotation scheduling, program evaluation (mapping), etc.