How University of Central Florida Clarified Criteria for Evaluating Adaptive Courseware
With a growing number of adaptive courseware products on the market, the task of choosing the right tool or combination of tools has become more difficult for institutions. What are the criteria to use during the evaluation process to ensure that courseware meets the technical needs of an organization and its standards for accessibility and equity?
The Center for Distributed Learning (CDL) at the University of Central Florida (UCF) recently asked questions like these to clarify its selection criteria for adaptive learning courseware while re-evaluating its primary platform and considering new options.
Among the several adaptive courseware solutions UCF uses, the most common is Realizeit. Instructional designers and programmers in the CDL partner with faculty to develop courses on the platform, either from scratch or by adapting existing courses. For example, CDL professionals work with faculty to design new course content, incorporate open education resources, and create original assessments.
To help onboard new instructors, especially in large enrollment and gateway courses, the CDL also created two training courses within Realizeit to guide instructors through designing adaptive learning activities while giving them a student’s-eye view of how the platform works.
Dr. Baiyun Chen, program director of the Personalized Adaptive Learning (PAL) initiative in the CDL, says Realizeit was originally chosen in 2014 because of the strong adaptive component to the platform. It has been a valuable tool, but she acknowledges that building a course in Realizeit is time intensive and not all instructors want to take on such a big project, even with the CDL resources and support available to them.
For that reason, Chen’s team of instructional designers in PAL began evaluating new adaptive courseware products in 2019 to serve a larger segment of UCF’s instructors.
Criteria to guide the courseware search
Chen and her instructional design colleagues, who have built award-winning adaptive learning programs at UCF, started their search for new adaptive learning courseware options by reaching out to vendors for demonstrations. Once they had an idea of the current adaptive learning landscape, they began evaluating the tools they had seen.
“At the time, we didn’t find good criteria we could use,” says Chen. The PAL instructional designers ended up creating their own criteria based on a simplified version of the Courseware in Context framework, as well as on input from UCF faculty. These criteria took into account price, course content, assessment types, objectivity, alternative content pathways, and how the new tool compared to their current tool.
The PAL team ultimately chose to implement Acrobatiq alongside Realizeit as a result of the evaluation. However, when CourseGateway launched in 2022, the UCF team decided to use its evaluation rubric to re-evaluate the tools they were using, including Realizeit, Acrobatiq, Canvas, and their internal tool, Obojobo.
“It was a good practice for us to reflect on what we really need for students and instructors in terms of adaptivity,” says Chen, who has since joined the CourseGateway Product Advisory Board and contributed her expertise to product reviews.
The CourseGateway rubric provides guidance for evaluation in terms of ease of use, student participation, availability, flexibility, scalability, and suitability for active learning, as well as how equitable a courseware solution is.
As a result of the most recent re-evaluation process at UCF, the PAL instructional designers felt more confident that the tools they were using were right for the institution. In particular, they found that Realizeit was still the preferred courseware for many of their needs, especially in terms of how the system is able to personalize learning by algorithmically adapting content and assessment questions.
Continuous improvement to serve students and faculty
At over 68,000 students, UCF is one of the largest institutions in the United States, which means the CDL must always be looking for new ways to help students and faculty benefit from digital learning technologies.
“Our general education courses routinely have two hundred to eight hundred students,” says Chen. “It’s impossible for an instructor to teach large-enrollment classes like that and pay attention to different students, so adaptive learning is a perfect solution.”
Along with a willingness to re-evaluate their tools, the PAL also regularly consults with platform vendors to educate themselves about new technologies. They also meet with faculty internally to learn from their experience.
For example, the PAL instructional designers are currently exploring options to work on an adaptive textbook in collaboration with Portland State University and University of North Carolina Charlotte. They’re also helping another set of faculty develop two new courses based on openly licensed Realizeit courses designed by faculty at other institutions.
Likewise, Chen suggests that institutions evaluating adaptive tools not be afraid to reach out to peer institutions to learn from their experience. Lastly, when evaluating courseware, Chen says CourseGateway uses “a great, comprehensive rubric that any institution can use to evaluate any type of tool or courseware.”