Quizzes.Next
A Case Study for the next generation assessment tool in Canvas LMS
Overview
The Canvas Learning Management Platform serves the learning needs of schools, teachers, and students around the world with over 30 million global users. Quizzes is the primary assessment tool built into Canvas. The new tool is referred to as Quizzes.Next which was the projects name during development.
Roles
My roles were Product Designer and Team Lead for Quizzes.Next. I began as an individual contributor, then as the project expanded I became the Team Lead of Product Design over Assessment within the organization. As a Team Lead I was responsible for 2-3 other Product Designers working on other assessment related products (Quizzes, Outcomes, Gauge, MasteryConnect).
Problems to Solve
Outdated, sometimes cumbersome UX
Limited features compared to more modern competitors
Aging, brittle codebase preventing upgrades, not scaling to needs
Scope
Design a Quizzes replacement with the following:
Parity with existing Quizzes functionality
Updated intuitive user experience
Advanced assessment interactions like drag-and-drop and hot spot selection
Constraints
Quizzes.Next would be built as an LTI (Learning Tools Interoperability) application as a way to break free from the existing codebase and speed up engineering. The tradeoffs of LTI were known but probably underestimated (see Lessons Learned).
Build
By design, an instructor’s “Build” view of a quiz closely resembles the student’s “Take” view. The Preview feature quickly gives a true preview of the student view including question interactivity and submission results. This is a big improvement over the old Quizzes which requires the instructor to exit the build view before being presented with a preview option.
Quiz builders can quickly add blank questions, text passages (stimuli), or pre-authored banked content at any access point between content blocks.
Watch this demo video to see the quiz building experience in action.
*footage is from an early version of Quizzes.Next
Settings
Settings could easily be toggled on and off. Some settings had sub-options that would appear once that setting was toggled to the “on” position. User input for sub-options would remain intact but would disable if the parent setting was toggled to the “off” position.
Reports
The Statistics & Item Analysis report helped gauge the performance of individual questions and the Quiz as a whole. As an example, if a large percentage of respondents answered incorrectly with the same answer this shows that the concept of this question could have been taught incorrectly by the instructor or perceived incorrectly by the students.
The Outcomes Performance & Mastery report was designed, built, and maintained by a separate product team. While not my work I have included a screen for reference.
Moderate
The Moderate view provides the instructor a comprehensive view of all attempts completed or in progress. From here the instructor could view the attempt details for each student, view individual attempt results in a Grade view, and provide accommodations like giving extra time or extra attempts on an individual basis.
The Attempt Log captures system and interaction details from a Quiz taker during attempts. Details captured by this log can be a powerful deterrent to cheating.
Looking at the following screen, notice the timestamp 00:29 shows that taker left their Quiz tab, returned at 00:50, and used the paste function at 00:53.
Instructors can quickly review results and give marks for manually graded questions (Essay and File upload questions) in the Attempt Results view. The Regrade feature allowed instructors to retroactively correct any mistakes in their Quiz and regrade any submissions according to the edits made.
Student Experience
Before beginning the Quiz taking experience students are presented with a summary of assessment parameters. Honor Code acknowledgment was an instructor-set option that could also be presented in this view.
Within the assessment taking framework students had collapsible a side navigation that could be used to jump through content (in longer assessments) as well as provide indicators for pinned and completed questions. When an assessment has a time limit a timer appears in the main header. One feature specifically requested during the validation phase is the ability for students to collapse the timer as a means to reduce test taking anxiety.
Quizzes.Next Results
Milestones
Discovery for Quizzes.Next began late 2015, building started in early 2016
Minimum Viable Product (MVP) released as a limited beta for a select group of clients (2017)
Beta made generally available to all Canvas users on an opt-in basis (2018)
Migration path provided to move content from old Quizzes to Quizzes.Next, new courses given the option to use either tool (2019)
Quizzes.Next to become the official assessment tool of Canvas, sunset on old Quizzes with migration available for deprecated content (dates TBD)
Drag-and-drop implemented throughout Quizzes.Next. This early prototype shows how we solved for accessibility in questions with drag-and-drop interactions.
Lessons Learned
During crunch time some poor implementation slipped through our process. At the time we agreed to “fix things later” so as not to backslide on delivery goals. Fixing things later rarely (if ever) happens.
Building Quizzes.Next as an LTI provided freedoms at the cost. Tying Quizzes in with Canvas required more time and effort which has resulted in a watered down user experience at certain integration points.
Continuous Improvement
Software is never done. Once it has been “delivered” it’s time to start anew in the cycle of continuous improvement.
As the Moderate view has expanded the initial layout got stretched to its limits. I have been taking a fresh look at the design and information hierarchy, seeing where improvement could be made. As Canvas was pushing for better responsiveness on smaller screens I wanted to carry this into Quizzes.Next and eliminate the use of a table in the Moderate view if possible. We were just discussing how to gain user feedback on this approach at the time of my departure.
Some other pieces being revisited: the student “take” view and header and footer that wraps every question type in “edit” view.
This case study covers the frameworks around quiz building and quiz taking. The other major pieces of Quizzes.Next are covered in the next sections, or use these links to jump ahead.
Quiz Question Types
The Content of Quizzes.
Item Banks
Storage banks allowing instructors to access and share question content across courses.
Gauge
A tool wrapped around the Quizzes product allowing for assessment deployment across school districts.