End-of-year survey of Newcastle’s students’ attitudes to CBAs

As it’s the end of the academic year, I decided to survey our students to get an idea of their attitudes about the Computer-Based Assessments (CBAs) that form part of their courses.

First of all, I’ll give some background on the CBA system:

  • A CBA is a Numbas test, delivered through a Moodle server set up specifically for the purpose.
  • CBAs are worth a percentage of the total credit for each module; in Stage 1 maths modules this is 10%, but it’s less for other modules.
  • Each CBA runs in “practice mode” for one week, where the students can make as many attempts as they like with full feedback, followed by one week of “assessed mode” when they must make one attempt which counts towards their credit for the module, with feedback revealed only when the work is finished.
  • Because of some anxiety about security, this year students in the school of mathematics and statistics have accessed their CBAs through a custom program accessible through our remote desktop service, RAS. This program prevents students from using their browser’s built-in development tools to manipulate the test. It’s definitely inconvenient, but the school decided the benefit of ensuring students couldn’t cheat in this way outweighed the inconvenience.
  • It’s each module leader’s responsibility to inform students about CBA deadlines. There have been a few instances of lecturers not knowing or forgetting this, and students relying on checking the Moodle site instead.

Questionnaire

We emailed every student taking a module which uses CBAs (891 students). The students filled entered their responses in an online form, with this prompt:

We’d like to gather some information about your experiences with the CBA system. Your answers to this survey will be anonymous. For each of the fields below, please indicate how strongly you agree with the statement.

I picked a few statements from Nick Parker’s 2011-2012 survey, and came up with a few new ones to cover things like RAS. The statements were:

  1. The CBA system is easy to use.
  2. The practice mode is useful.
  3. The questions are well written. (“Well written” means the questions tell you what to do clearly, and don’t contain typos or other errors.)
  4. The question feedback is useful.
  5. The RAS program is easy to use.
  6. The RAS program is reliable.
  7. I work with others on CBAs. (changed from “I work on the CBAs independently from others” to see if it would encourage more students to admit to collaboration)
  8. I enjoy the CBAs.
  9. I use the CBAs in practice mode after assessed mode has finished.
  10. Module credit is my main motivation to attempt CBAs.

They were also given two text areas, labelled “Suggestions for improving the CBAs” and “Any other comments”.

An email with a link to the form was sent to every student enrolled on one or more modules with CBAs (893 students).

Results

I received 79 responses in one week – fewer than 9% of the students contacted! This number is less than, but comparable to, the 116 respondents to the 2011-2012 survey. Those students were all taking the same module, however, so I’d expect more variation in my data to account for experiences with individual lecturers.

question_results

Some of the questions can be directly compared with the 2011-12 survey.

q1-cbas-easy-to-useindividual_questions-0

This year’s students rated the system as less easy to use. This could be because of the added inconvenience of the RAS program – the negative response to statement 5, “The RAS program is easy to use”, is evidence for this.

q2-practice-mode-is-usefulindividual_questions-1

The ratings for “the practice mode is useful” have stayed almost exactly the same.

q4-question-feedback-is-usefulindividual_questions-3

Students rated the question feedback much more highly this year.

q7-i-work-independentlyindividual_questions-6

Changing the statement “I work independently” to “I work with others” did seem to encourage more students to say they collaborate on CBAs. However, over 70% still say they work on their own.

q8-i-enjoy-the-cbasindividual_questions-7

While the proportion of students who say they enjoy CBAs is still less than half, it has increased, and the balance is now towards “agree” instead of “disagree”.

Student comments

  • 5 students complained that the system does not give partial credit for partially-correct answers, either for expressions which contain a small error, or for errors carried forward from previous parts.
  • 7 comments complained that the RAS program is slow or unreliable, making CBAs impossible to complete.
  • 4 comments complained that the process of logging into the CBA system through RAS is too long – students must log into a university PC, then onto RAS, and finally into Numbas.
  • One student asked for hard copies of CBAs to be handed out in lectures.
  • 3 students complained that the system doesn’t always recognise a correct answer in the wrong form. While it’s impossible to know if they really did have an answer equivalent to the model, one student asked that questions on modular arithmetic should work in the appropriate field.
  • 2 students asked to be notified by email of which CBAs are running at any time.
  • 2 students commented that the amount of feedback given is not consistent across all questions – some questions have no worked solution.
  • 2 students said they liked that the CBAs are made available for practice after the assessed mode finishes. One student asked for more CBAs to be made available for practice before exams.
  • One student said that they can’t complete CBAs on their tablet now that it runs through RAS.
  • There was some confusion about how much credit CBAs are worth – one student thought the number of marks awarded in a CBA corresponded directly to course credit (so a question worth 6 marks in one CBA is worth more than a question worth 2 marks in another).

Discussion

Based on their answers to the questions, the students just about tolerate the RAS program. However, the comments reveal that it has reliability problems which make running the CBAs very hard for some students, and the inconvenience is putting students off the CBAs as a whole. The RAS program limits what we can do with the CBAs – we can’t include videos, and it’s a shame that they can’t be more accessible on mobile devices, when Numbas on its own is very good on mobile.

There is a widespread practice of copying model answers from practice mode and using them as a guide for the assessed mode. The students think this is unfair. If the CBAs were more explicitly for formative use, they might realise this isn’t as much of a waste of time as they think!

When students said their alternate correct answers weren’t accepted, the vast majority turned out to be in fact incorrect. However, there are a few questions where there are several valid solutions, but we restrict the solution set to a unique answer by asking for the least value, or the single answer in a given range. Maybe we should just accept the other answers, which would involve writing more complicated marking scripts.

The practice mode is highly valued, and students appreciate having the CBAs available in practice mode for revision.

We could do more to tell students what CBAs are available or due at any given time.

Overall, the students seem to like the CBAs, and the increase in positive responses to the question “I enjoy the CBAs” is very encouraging. If we moved to purely formative use of CBAs, I think most of the students’ complaints would be nullified – we could do away with the RAS program, and questions to do with the exact amount of credit awarded would become less important. However, several members of staff are of the firm opinion that students wouldn’t do the CBAs if they weren’t worth any module credit, and the strong positive response to statement 10, “Module credit is my main motivation to attempt CBAs”, backs this up.