Last week I gave a demonstration of Numbas at the CAA Conference in Zeist. As part of the submission process I had to submit a formal paper, which it turns out isn’t being included in the published proceedings because it was on the practice track. Instead, I thought I’d reproduce the paper here, since it offers a good, brief overview of Numbas. If you’d like the PDF version of the paper, click here.
[Could not find the bibliography file(s)Numbas is a free and open-source e-assessment system developed at Newcastle University. This demonstration highlights the key features of Numbas and the design considerations for an e-assessment tool focused on mathematics.
About Numbas
A typical Numbas question.
Questions are randomised according to a set of variables, defined by arbitrary mathematical expressions. A question consists of a statement, one or more parts, and an advice section. The advice section typically contains a fully worked solution to the question and is revealed once the student has finished the question.
A part can provide steps, revealed if the student clicks a button, which break the part into more parts to guide the student through the question, and allow the student to receive partial marks for an incomplete answer.
Numbas supports several question types, including most of the types you’d expect – number entry, multiple choice/multiple response, and text entry. Marking is performed on the client, so feedback is immediate.
A Numbas test is a standalone package containing an HTML page, with all the exam logic implemented in JavaScript and running on the client side. Because a Numbas test is just a web page, questions can include any of the rich content found on the web, including videos, interactive graphics, and other elements.
Questions are created using a web-based graphical editor.
Formative vs summative assessment
We are strongly of the opinion that computer-based assessment of degree-level mathematics is most suitable in a formative mode rather than summative. The kinds of questions which can be marked automatically by a computer are mainly restricted to mechanical, turn-the-handle techniques.
More creative, problem-solving tasks are much harder to automate and repeat.
Numbas tests are ideally suited for use as an inexhaustible bank of practice questions to practise basic techniques. Students can receive immediate, high-quality feedback on their answers, and each question has a “Try another question like this one” button which generates a new randomised question of the same type.
Suitable topics for assessment with Numbas
Numbas is used at Newcastle to assess undergraduate-level mathematics, both in the mathematics degree and in other service courses. We believe computer-assisted assessment of mathematics is most useful around the transition to university, when we need to ensure that students are proficient in the basic skills of algebra and calculus. Statistical modelling tasks of the sort undertaken in first-year courses in STEM and other numerate disciplines are also particularly adaptable to randomisation and automated marking.
While a few final-year modules use Numbas to provide an opportunity for practice at new but mainly algorithmic tasks, such as cryptography and coding theory, there are very few topics at that stage of the degree that are suitable for purely computer-based assessment.
Conversely, Numbas as it exists today has limited value in primary education, which requires a simpler interface and presentation is much more important than sophistication in generation and marking.
Current use
Every module in Stage 1 and Stage 2 of the mathematics degree at Newcastle University includes at least two Numbas tests. Students can access each test in “practice mode” for one week, during which they can make as many attempts as they like and get full feedback, followed by “assessed mode” the next week, for which they can make only one attempt[?].
A publicly-accessible instance of the Numbas editor is available at mathcentre. Its question bank contains over 4,000 questions written by lecturers and teachers all over the world, including over 700 questions created at Newcastle and released under the Creative Commons Attribution licence. Numbas has been translated into Norwegian Bokml, Spanish and Dutch.
Client-side design
Numbas runs entirely on the client. This means it’s very fast – no calls to external computer algebra systems to generate questions or mark answers – and can be distributed to students as a standalone webpage, or integrated into a VLE without any setup.
While many other mathematics assessment systems have a dependency on server-side computer algebra systems, Numbas contains a computer algebra system written entirely in JavaScript, so expressions can be interpreted and evaluated entirely on the client, with no delay while a network request is made. This system is quite fast – a question can generate random data sets of several thousand elements and compute associated statistics in a fraction of a second.
By their very nature client-side designs are not as secure as server-side implementations and an enterprising student could find ways to cheat without suitable safeguards. This is only an issue for summative high-stakes examinations and can be easily circumvented – we recommend that all testing takes place in an invigilated environment. For less-important summative assessment, such as the fortnightly in-course assessment required in all maths modules at Newcastle, we trade some convenience for security and provide the students with a standalone app which is the only means of accessing the site storing their Numbas tests. This way, students are prevented from breaking the software to cheat, but can still copy off each other, look up answers on the internet, or do any of the other things badly-behaved students also do for paper-based homework.
Input of mathematical expressions
The syntax used by the student to enter a mathematical expression as their answer must be simple, memorable and unambiguous (discussed by Sangwin and Ramsden in [?]). Numbas uses a syntax similar to that which a student would type on a calculator or computer algebra system such as Maxima. Importantly, the student’s expression is rendered in mathematical notation beside the input field and updated immediately as the student types, so they can be sure that the system has interpreted their expression as they intended.
Students by and large have no problem with this syntax[?]. One error which occurs quite often is the omission of the multiplication symbol between variable names, or a variable name and an opening bracket. To avoid this, the question author can provide a list of expected variable names in the student’s answer; the student is shown a warning if they use any variable name not in the list.
The student’s answer is rendered immediately next to the input field.
Usability
Usability is an important aspect of any software, and often neglected in mathematical assessment systems. Good user experience is an important goal of Numbas, both for the student and question author.
Student experience
The interface is clean, and accessible by e.g. screen readers. Because everything runs on the client, feedback is immediate.
Over several years of use and based on student feedback, the interface has been made more intuitive. Because Numbas runs through the web, it can be accessed from any device, including mobile phones and tablets.
Author experience
The barrier to entry for question authors is low – using the graphical editor, one can create a simple question with no knowledge of programming or LaTeX.
As the author gains experience, they can gradually add mathematical notation written in LaTeX (we believe this is the most straightforward way of entering mathematical notation on a computer) and more complex variable definitions.
The rich-text content editor provides instant renderings of LaTeX notation, cutting out the write-compile-preview loop.
Questions can be run immediately inside the editor, so the author can see the question as a student would, and find errors in rendering or marking.
Numbas allows the creation of template questions which can easily be adapted to new subjects without changing the underlying maths. For example, a question on linear regression can provide several variables representing parts of the statement text and parameters for the underlying statistical distribution from which random values are generated. An author can edit these variables as simple text strings or numbers, respectively, without modifying the more complex parts of the question.
Complexity of questions
Standard questions encountered in a high school or early undergraduate maths course are easily implemented in Numbas. However, questions covering more advanced topics or with more sophisticated designs are of course more complex, particularly when the question is randomised: first of all, defining a set of variables which generate a question with sensible setup and solution (or even any solution at all) is not always trivial, and often effectively boils down to a programming exercise; secondly, marking a mathematical expression might not involve just deciding if the student’s answer matches a given model answer exactly, instead asking if the student’s answer has certain mathematical properties, or is given in a particular form.
Numbas has been designed with the aim of minimising this complexity and providing abstractions of common patterns to avoid repetition.
Variable generation
Question variables in Numbas are generated declaratively. Generating random numbers, text, or other data is very simple. Other variables can then do further computations with those values.
The variable definition interface allows you to work interactively, with variable values generated immediately with each change to the definitions. The variable testing tool allows you to generate large numbers of sets of variables and compute statistics about their values. This helps the author get an idea of how often particular cases occur. You can also specify a condition on the variables which must be satisfied, to avoid pathological cases.
The variable definition interface.
Marking
The algorithm used by Numbas to mark mathematical expressions is conceptually simple, but good enough to cover a wide range of question types – the student’s answer and the model answer are evaluated over a randomly selected set of variable values, and compared. If the student’s answer matches the model at every point, it’s marked as correct.
Often, questions assess a mathematical expression not just for the value it represents, but for its form – for example, when asking a student to factorise a quadratic, it’s important that the answer is of the form (x + a)(x + b) and not x2 + ax + b. Numbas has a sophisticated pattern-matching system which can be used to make assertions about the form of the student’s answer and tailor feedback accordingly.
An abstraction for marking schemes that doesn’t involve programming would be very useful. STACK[?] uses an acyclic directed graph with tests at each node, which we find both cumbersome and limited. For now, Numbas allows authors to write custom JavaScript marking algorithms which can manipulate the student’s answer to decide how it should be marked.
Rendering mathematical notation
Numbas uses the MathJax library to render mathematical notation. Question authors can write mathematics using LaTeX syntax, with question variables substituted in.
The substitution of random variables into mathematical expression raises a significant problem – how to present the resulting expressions in a natural way. Numbas has a sophisticated and customisable system of simplification rules based on a unique pattern-matching system, which cleans up mathematical expressions before they are rendered.
Preview of LaTeX in the question editor, and the result as shown to the student.
Customisation
A large driver towards the creation of Numbas was the lack of customisability in previous systems. Numbas has separated the display of exams from the underlying logic. This means authors can create new themes to change the look of their tests. This can be cosmetic changes, such as a different logo or changes to the stylesheet, up to total reimaginings of how the test is delivered, as seen in Peter Rowlett’s printed worksheet theme [?].
Numbas also has an extensions interface. Extensions can provide new data types, functions and resources to questions. Examples include a library of statistical functions, data types representing permutations and linear codes, and functions to draw interactive diagrams with JSXGraph [?].
Dynamic rendering of the graph of a student’s answer using JSXGraph.
Integration with a VLE
Numbas uses the SCORM 2004 standard to integrate with compliant VLEs, such as Moodle and Blackboard. This way, the VLE handles session management and data storage, leaving Numbas as a standalone assessment tool. Numbas tests can be accessed just like any other web page outside a VLE, but without score tracking.