Improvements to Accuracy of ASR in Video Recordings

Video content is a part of the delivery system for the medical curriculum at the University of Illinois College of Medicine. Live lectures and other activities are typically recorded. Faculty also create short videos on specific topics. The latter is often used as prep work for an upcoming session. There are two lecture capture systems used at UIC, Panopto and Echo360. UICOM uses the latter one. Students can access these recordings at any time for study / review.

Until the late 2010s, the university and therefore the college of medicine did not electronically transcribe any video content provided to students. Even though Echo360 had a third-party tool (Google AWS, at the time) that was used to create transcription for all video content, it was officially not turned on at our institution. In late 2018 and early 2019, we turned on this feature and let students know video content now included transcription.

The college has one of the most diverse student bodies in the country and there are typically many international students in each cohort for whom English is not their first language.

Problems

Video content is a part of the delivery system for the medical curriculum at the University of Illinois College of Medicine. Live lectures and other activities are typically recorded. Faculty also create short videos on specific topics. The latter is often used as prep work for an upcoming session. There are two lecture capture systems used at UIC, Panopto and Echo360. UICOM uses the latter one. Students can access these recordings at any time for study / review.

Until the late 2010s, the university and therefore the college of medicine did not electronically transcribe any video content provided to students. Even though Echo360 had a third-party tool (Google AWS, at the time) that was used to create transcription for all video content, it was officially not turned on at our institution. In late 2018 and early 2019, we turned on this feature and let students know video content now included transcription.

The college has one of the most diverse student bodies in the country and there are typically many international students in each cohort for whom English is not their first language.

So, I developed a website to serve the faculty and staff and contain important content like:

• Terms that are used in the curriculum and their definitions and situations of use
• Examples of educational technology that are used in the design and delivery of curricular content and how to use / who to contact to use them
• Specific forms that are used with descriptions of their purpose
• Diversity, equity, inclusion, and accessibility issues to consider when designing and delivering curricular content
• Best practices in the design and delivery of curricular content, including team-based learning activities
• How to incorporate educational games in the delivery of content
• How to create a video tutorial, from the technology to consider to best practices in recording based on multimedia design
• Best practices and guidelines for coordinators for each phase of the curriculum for all three campuses

Research

After discussions with Echo360 representatives, we found out that there was not a built-in dictionary for medical, pharmacologic, and certain higher level scientific terms in the ASR tool they used (which was Google AWS). For example, if a faculty member said ‘hypoallergenic’, the system might not have comprehended it accurately and the transcript would show ‘hyper-allergenic’ which is incorrect or sometimes the word it comprehended was completely indecipherable.

Here is an example of how the system incorrectly heard the term ECG from a faculty member who was speaking. The correct term is ECG. After listening to this part of the lecture. it was very clear he said ECG, not eggs. Select the image to see it larger.

My Role

My role at the college of medicine is to lead instructional design efforts. With my background in instructional technology, I manage a lot of related projects.

Solutions

Initially, some of our medical students tried to correct the incorrect transcripts themselves and soon discovered how long it can take to correct them. We determined that a better option, until Echo360 had the appropriate dictionaries in their system, was to hire non-pre-med students to edit the transcripts. We were approved to hire up to 5 part-time students to help with the project. I worked with a colleague to interview and hire students to do this work.

We developed a job title (Medical Lecture Transcriber) and description based on existing similar jobs in the university system. Then, we published the job via Handshake, a tool for students to find jobs on campus, as well as other sources like LinkedIn. We were able to hire four non-pre-med students who had science backgrounds and one 4th year medical student to edit transcripts.

I reviewed some of our Echo360 lecture recordings and selected recordings in which the transcripts might need the most improvement. We used Google Sheets to allow the student workers to select a specific recording to edit. Many of these student workers were studying various aspects of science, so a value added to editing the transcripts was perhaps an opportunity for them to learn something that was relevant to their own studies. During training, we made sure the students understood it could easily take 3-4+ hours to edit and correct a 1 hour recording transcript.

Results and Takeaways

After running this process during spring term 2020, it became clear that the students were not keeping up with the demand for editing like we planned. At the end of the semester, we decided to discontinue utilizing students to correct transcripts.

While that was an unsuccessful attempt at correcting a problem, we learned a lot about our own systems and how to improve them.

A postscript is the lecture capture vendor, Echo360 partnered with a different company to provide automatic transcription, Speechmatics. There is a huge difference in the quality of the transcripts as Speechmatics does have an advanced medical, scientific, and pharmacological dictionary. The students are grateful for the improvement.