Bill Foster was director of the e-learning unit at Newcastle University until his retirement in 2016. In this post he tells the story of how Numbas came to be developed under his guidance.
Numbas is the latest in a long line of assessments systems: it is the spiritual successor at Newcastle to the i-assess assessment system which itself was based on the CALM system developed in the late 1980s.
In the following I will cover the implementation of CALM at Brunel University (1991/2), and at Newcastle University the implementation of i-assess (2006) followed by the development and implementation of Numbas (2012).
Brunel: CALM
CALM was developed by Cliff Beevers and his team at Heriot-Watt University. The main attributes of CALM were that it included randomisation of variables within questions, allowed the input of mathematical expressions based upon a robust methodology, allowed detailed reporting of student attempts and gave immediate and useful feedback. As a result it was suitable for use in first year mathematics courses at HE.
Bill Foster and Cliff Beevers at the EAMS conference 2017 in Newcastle upon Tyne
The first implementation I was involved with was at Brunel University where in 1991/2 with a small team and the help of Jim Renshaw from the University of Southampton we installed CALM under UNIX, aimed at supporting the mathematics course for all first year engineering students. This was taught by the Mathematics Department as a service course. This decision was made as a result of good evidence from Heriot-Watt and Southampton in terms of performance and engagement in first year mathematics courses together with assessment work on the Mathwise project through which I had experience of CALM and the excellent results it was achieving.
The Drivers:
- First year entrants had a very mixed mathematical and numeracy background; more practice was needed for the weaker students.
- The tutoring resources for this service course to 500 students were not adequate to provide the necessary feedback to students on their attempts on set problems.
- The CALM based practice system was seen as a partial remedy to the problem of giving such practice and feedback.
- The necessary computing infrastructure was now available, given the recent introduction of large computer clusters.
- Service teaching from the Mathematics Department was, as ever, under threat given financial constraints – so the introduction of new teaching and learning techniques were seen as countering any moves to take such teaching in-house.
The Implementation:
- Primarily aimed at calculus.
- Designed to be of a formative nature, but a small amount of marks were given for completing periodic assessments. This was enough of an incentive to most students to complete assessments in their computer lab sessions.
- Regular large classes (100) were held in a computer cluster with 3 tutors.
- Found to be very popular with students, especially those with weak backgrounds in maths.
- Good evidence of improved performance and engagement.
Newcastle: i-assess
I started work at Newcastle in 2001. In 2005 I was tasked to find an e-assessment system suitable for use in our first year undergraduate mathematics and statistics courses as well as service courses.
Once again, as at Brunel, the main driver was to supply feedback on in-course assessment. Student cohorts were significantly increasing in size and it was felt that the existing system of postgraduate marking and supplying feedback was inadequate and too expensive. National Student Survey results on feedback indicated that students felt that they did not receive the level of feedback expected.
Following from this, the brief was to supply students with practice material relevant to their courses but at a level of difficulty which would give confidence in basic techniques, increase engagement and to supply immediate feedback to their attempts.
Why was i-assess chosen?
I reviewed the available e-assessment engines which were suitable for mathematics courses in the first year at HE. I became aware of the commercial system i-assess and tested it along with other candidates. The level of support from the i-assess team was good, the system allowed the writing of questions at the right level of sophistication and validity. Also and very importantly, i-assess was based on CALM and so I knew that its design, both in terms of functionality and pedagogy, was very suitable for the use we intended. The School of Mathematics and Statistics had formed a committee to oversee the implementation and agreed to its use.
Implementation
There was very little time to implement i-assess and the main problem I had was in writing the questions, together with Anthony Youd, for all the stage 1 courses. Together with the excellent support from the i-assess team and the School’s IT team this was achieved for the academic year 2006/2007.
In general the introduction of i-assess was a success and was viewed very positively by students; engagement increased markedly in terms of the take-up of in-course assessment. There were problems with memory leakage causing problems with access to i-assess but this was overcome by maintaining (rather exhausting!) email contact with all affected students.
The main problem encountered was in the writing of questions for stages 2 and 3 as in subsequent years the School decide to expand i-assess use given the success for stage 1. It became apparent that the system did not have all the required functionality and the writing of questions became difficult for more sophisticated applications. Also the initial brief of using the system in order to give confidence allied to practise of basic skills was forgotten, with attention shifting to the replacement of paper-based in-course assessment. More demanding questions were asked for and the e-assessment system became the de-facto in-course assessment system.
Increasing frustration
i-assess ran successfully from 2006 until 2012 but frustrations with the editor together with other issues were mounting and it was felt that we needed a change to system which met our future demands. During this time we looked at various commercial packages such as Maple T.A. but were disappointed with the support provided and some of the functionality. However, in 2010 two events enabled the development of our own system.
Newcastle: The Development of Numbas
Firstly, I obtained a grant for the development of outreach material in mathematical sciences. An early decision was made to make it freely available, so the creation of our own in-house assessment and practice system meeting our requirements, was seen as a suitable vehicle for the supplying of outreach material.
Secondly, I was able to recruit Christian Lawson-Perfect, an exceptional talent, as the chief developer of the new system, which we decided to call Numbas.
Between 2010 and 2012 Christian with help from Anthony Youd wrote the new system. The School decided in 2012 to implement Numbas across its maths degree and abandon i-assess.
Because of the above history, Numbas still retains the basic design and methodology of CALM but has extended its functionality greatly.