The first time I taught a large class (about 600 students in Calc II), I was totally unprepared for how much effort it took to simply run an exam, from room scheduling and proctoring to dealing with conflict exams (and conflict-conflicts and even conflict-conflict-conflicts). Then came grading, and re-grading, and student complaints about unfair TA grading. And all this doesn’t even include actually writing the exam!
Now we are using fully-automated exams in the CBTF (Computer-Based Testing Facility), which is providing more authentic tests with dramatically improved logistics, as described below.
Automated testing: the CBTF concept
Traditional exams in large classes are so painful to run that we only do them a few times a semester, despite evidence grounded in the psychological theory of the testing effect that shows that frequent testing helps students to learn (Roediger et al., 2011; Roediger and Karpicke, 2009; Pennebaker et al., 2013). In addition, traditional exams for large classes have to be paper-based, because there is no other way to get 600 or more students taking an exam simultaneously in computer labs. This severely limits the types of testing that can be done, especially for anything involving computing.
To fix the problems with traditional exams, Craig and Tim came up with the idea of a Computer-Based Testing Facility (CBTF). This is a room where students can go to take tests at any time of the day. Each test runs for about four days, and within this time period students can sign up for any available time slot. Different courses run tests at the same time, and at any given hour there will be students taking a variety of tests next to each other. Tests are randomly generated so every student gets different questions, and tests are auto-graded so students get their score and feedback immediately.
The CBTF started in Fall 2014 and has seen rapid growth each semester since, with over 20,000 tests being run in Fall 2015. In the current semester (Spring 2016) there are courses from four different departments using the CBTF:
- Computer Science: CS 125 (Bill), CS 225 (Anna and Cinda), CS 241 (Lawrence), CS 357 (Mike and Eric), CS 421 (Mattox and Elsa), and CS 450 (Andreas)
- Mechanical Science and Engineering: TAM 210/211 (Mariana K. and Gabe), TAM 212 (Elif and me), and TAM 251 (Randy, Yuhang, and Mariana S.)
- Electrical and Computer Engineering: ECE 110 (Chris, Patricia, Hyungsoo, and Serge)
- Materials Science and Engineering: MSE 206 (Dallas) and MSE 304 (André)
The CBTF was originally started by Craig and is funded by the College of Engineering under the SIIP grant program run by AE3. It is now developed and managed as a partnership between faculty (Craig and me) and Engineering IT (Dave, Kim, and many others).
Full automation for professors
The goal of the CBTF is to function as a black box for professors. You can feed in test questions (typically using PrairieLearn) and without any further action you will get back test scores for all your students and analytics on the test and individual questions.
Everything except the creation of the test questions is automatically handled by the CBTF. This includes student scheduling, proctoring, conflict exams, students who sleep in and miss their test, students who get sick, DRES accommodations, student identity checking, rescheduling if problems arise for any reason, and preparation of FAIR reports for academic integrity violations.
The flip side to having everything automatically handled is that tests in the CBTF run according to an organized system, and arbitrary requests can’t necessarily be accommodated. For example, you can’t ask to have students take exams in teams or only at specific times of the day, because that isn’t supported. You also can’t necessarily schedule every exam on the exact days of your choice, because some negotiation is needed to squeeze all the courses into the available slots. Generally speaking, however, virtually all courses get exams on the weeks they want, just not necessarily on the precise days of the week that they prefer.
When is it worth using the CBTF?
While using the CBTF can save a huge amount of hassle and effort, there is a non-trivial startup cost to using it. This mainly consists of preparing computer-based questions (normally in PrairieLearn) and checking them to make sure they are correct.
I like to think of the different exam technologies as trading off the upfront preparation cost against the incremental cost to repeat the exam for more students. At one end of this tradeoff are oral exams, which may require no preparation at all but scale horribly. Traditional written exams and paper-based multiple-choice (MC) exams take progressively more effort to write, but scale up more efficiently to large numbers of students.
The CBTF is at the opposite end of the tradeoff space from oral exams. It takes more effort to implement auto-grading questions for use in the CBTF, but once done it is almost literally zero effort to scale up the exam to arbitrary numbers of students.
One big benefit of using the CBTF to reduce the effort of testing is that this frees up your time as an instructor to spend more effort designing high-quality tests, using test analytics to improve the course, and exploring novel strategies for teaching and testing. This is just one example of the general idea of automating wherever possible to free up time for innovation.
Enabling authentic tests
Although the CBTF is farthest removed from oral exams in the initial-cost/repetition-cost tradeoff, this does not mean that exams in the CBTF are even more contrived and restricted than multiple-choice exams.
On the contrary, computer-based tests in the CBTF open up all kinds of exciting opportunities for more authentic tests. For example, in TAM 212 Elif and I are asking more realistic and more complicated questions because students are able to use Matlab to solve them. In CS 233, Craig has students drawing auto-graded finite state machine (FSM) diagrams, while Cinda, Lawrence, and Mattox have CS 225/241/421 students writing code with full access to compilers and debuggers. The CBTF can support anything that can be machine graded.
Asynchronous exams: flexibility for students
The extreme flexibility of the CBTF comes from the fact that the students don’t all need to take an exam at the same time. That is, an exam takes place asynchronously over several days. Students choose their own time to take the exam and they can reschedule online anytime before their exam slot. Reminders are sent automatically by email in advance of exams, so the course instructor does not need to be involved in the scheduling process at all. If a student misses their time-slot then the CBTF proctors can automatically handle rescheduling.
The main concern with asynchronous exams is whether students are able to cheat by passing information about the content of the exam. Our main strategy for handling this problem is to randomize the exams (PrairieLearn handles this by default), both by having questions with random parameters and also by having different questions variants for exams that are randomly selected for each student. Data from multiple classes shows that the highest exam scores are typically on the first day, with declining scores each subsequent day. While not proof that students aren’t cheating, it does give some confidence that it at least is more difficult that one might initially suspect.
The best natural experiment we have so far is from TAM 212 in Spring 2015 when Exam 1 (the first midterm) was a traditional pen-and-paper synchronous exam while Exam 2 (the second midterm) was spread over five days in the CBTF. The average “overachievement” student scores on Exam 2 controlling for Exam 1 performance are shown below, with a negative slope over the five days. This seems to show that any cheating or extra study time is a smaller effect than the well-preparedness that students are indicating by choosing to take the exam earlier.
As well as allowing painless scheduling for instructors, the free choice of exam time-slots is also very popular with students, as shown below in a survey with 406 students from TAM 212 in Spring 2016.
Frequent exams and second-chance exams
One of the most exciting aspects of the CBTF is that freedom from exam logistics means we can break away from traditional midterms/finals and start thinking about which testing format actually best supports student learning.
For example, Bill uses weekly quizzes in CS 125, while Gabe and Mariana K. switched to bi-weekly quizzes for TAM 210/211. This allows students to receive accurate feedback much more frequently and picks up problems faster than waiting for a high-stakes midterm exam. The ease of running exams also lets us offer possibilities such as “second chance” or “retake” exams, where a student can come back the following week to try another version of an exam (currently used in several classes, including by Mariana S. in TAM 251 and Craig in CS 233).
Frequent testing and retry testing means that students are forced to confront their actual state of knowledge and can see whether their study practices are really producing learning. This helps to counter the “familiarity versus knowledge” issue that many students have, where they mistake familiarity for content expertise (Roediger et al., 2011). This arises because students frequently study by re-reading material (Karpicke et al., 2009) and do not test their abilities (Karpicke, 2009; Kornell and Bjork, 2007). This leads to substantial mis-estimations of performance (Rebello, 2011), with good students under-estimating and poor students over-estimating their abilities (the Dunning-Kruger effect). When students only have a few high-stakes exams then they can always explain to themselves that they only did poorly because “the exam was unfair”, but when it happens week after week they are forced to confront the usefulness of their own study habits.
Getting started in the CBTF
If you want to use the CBTF for your course, or just learn more, leave a comment below or contact me, Dave, or Craig. There are also papers with more information:
- M. West and C. Zilles, “Modeling student scheduling preferences in a computer-based testing facility”, in Proceedings of the 3rd Association for Computing Machinery Learning at Scale Conference (Learning@Scale 2016), DOI: 10.1145/2876034.2893441, 2016.
- C. Zilles, R. T. Deloatch, J. Bailey, B. B. Khattar, W. Fagen, C. Heeren, D. Mussulman, and M. West, “Computerized testing: A vision and initial experiences”, in Proceedings of the 122nd American Society for Engineering Education Annual Conference and Exposition (ASEE 2015), DOI: 10.18260/p.23726, 2015.
- M. West, G. L. Herman, and C. Zilles, “PrairieLearn: Mastery-based online problem solving with adaptive scoring and recommendations driven by machine learning”, in Proceedings of the 122nd American Society for Engineering Education Annual Conference and Exposition (ASEE 2015), DOI: 10.18260/p.24575, 2015.
Education at Scale 