Numbas training session - 2020-07-15

OK Hello everybody, it's 10:00 o'clock here in the UK now, so I think we'll get started.

Welcome to this Numbas training session. I'm just going to introduce Numbas and go through a few things.

This session is for beginners. So, a lot of people are just beginning to use Numbas and haven't used it before, so I'm going to talk through the basics of what it does and how to use it.

OK, so uh, Chris Graham should be here.

He's my colleague at Newcastle.

He said he went off to get a cup of coffee and hasn't reappeared yet, so he will be here.

[Chris] I'm here, caffeined up. [Christian] You're here? Good. Hi.

So Chris is going to be watching the chat, and telling me when people want to interrupt me.

OK, so we have a quite a few people in already, around the world.

I'm aware we have quite a lot of colleagues from Indonesia joining us.

So hello to all of you.

So let's just go through the plan for the day - and make sure I pick the right mouse up.

So.

First of all, some rules for using Zoom.

This is an informal session so don't worry about things going terribly wrong, interruptions, and all that.

I'll try and roll with it.

But please mute yourself when you're not speaking, just to make it clear for everybody else.

There's a raise hand button.

If you click on the participants button, you will see everybody and then at the bottom there's a list of things you can do to get our attention.

One of them is raise your hand. And please use the text chat to ask questions or to comment on what's going on.

Chris will be watching that and will stop me if there's something I need to deal with.

If there's something that you didn't hear or don't understand, or you want to ask a question about before I move on, please do.

I don't mind stopping what I'm doing and going into a bit more detail in things.

It's very hard to get an idea of what people are following with, so please tell me if you want a bit more detail on something.

Right, so. The plan for today is I'm going to introduce Numbas, tell you what it is, and how we use it in Newcastle and how it's used around the world and then give a quick demo of the system itself.

After that, I'll pause and find out what all your backgrounds are, what you want to use Numbas for, to guide the rest of the session, and then I'll have a look at using the Numbas editor and go through the process of writing a first question.

And then we'll have Q&A, and I think I removed that line from my slides at one point - no, I didn't.

Right. OK, so... I need to make sure I've got the chat window up, which has disappeared again.

Hello, chat. So we will begin.

Numbas is an open source web based e-assessment tool, developed here in Newcastle University in the school of math, stats and physics.

It initially was for our maths course.

It's aimed at numerate disciplines. The sciences, and maths in particular, have particular needs to do with assessment, a lot of calculation and a lot of automatic marking can be done.

Where subjects in the humanities will typically be working on essays and have human markers.

So there's a lot of automatic assessment you can do in maths, and Numbas is really set up to do very well at that.

So there are a few things that are really the key features of Numbas, that are the important design points.

It's scalable.

Everything runs - all the exams run on the client's device, on the student's computer.

So you can have thousands of students using Numbas and it doesn't really matter.

Reliability is obviously very important, because we use Numbas for course credit.

And accessibility is one of the other really big design points.

Ease of use is important.

I know from other e-assessment systems that it's typically an afterthought because you want to get the actual assessment done and then people start thinking about, "is it easy to use?", but that's really something from the beginning in Numbas.

And the aim in that question authors who aren't experts should be able to write questions and give them to their students.

So making things easier for question authors is an important design goal.

Feedback is important.

It's important that students get told how their answers were marked.

Really, that's how you learn. It's all well and good typing an answer into a box, feedback saying, "This is correct, this is incorrect" will get you some of the way, but it's important to do a bit more than that.

Customisability is important.

I think maths is such a huge subject and there's so many different things you can do, you can quite easily run up against wanting to do something that's not in the built-in feature set.

So Numbas supports extensions, custom marking algorithms, even custom part types, which I'll get onto later on.

The way you get it to students.

A Numbas test is just a web page, so you can just download that from the editor, put it somewhere, students will do the test, but if you want to keep track of scores you need somewhere for the data to go.

So there are a variety of options for delivering it through a VLE - a Virtual Learning Environment - such as Blackboard, Moodle, Canvas, that kind of thing.

And finally, as I said, it's aimed initially at maths and the sciences, so there's a lot of features to support assessing mathematical subjects.

Right.

So now I'll move on to my demo.

So here we go. I've been busy rewriting this demo, so this is my in progress thing, it's the first time I've used it.

Chris, could you just mute all, please?

Right, so, this is a Numbas exam.

It's in the new menu mode, so we get given a menu of questions to choose from.

Another way of delivering exams is you give the student a list of questions and they have to work through them, they're numbered like an exam.

So I'll go straight into this one and show this is what a question looks like.

A question has a statement at the top that could give some information that you need to know in order to answer the rest of the question.

Not in this demo one, this just gives some explanation.

And then one or more parts. A part is where you ask a student to enter an answer.

They submit it, Numbas does some marking and gives some feedback and some points.

So in this question, we just have to enter a number.

The prompt is, "I eat five apples a day.

How many do I eat in a week?" And I think the answer is 35.

So that's correct.

So you get immediate feedback. You get this nice green tick and you get an explanation of how the score was awarded.

Another real test of automatic assessment is, does it also mark the wrong answer wrong?

So this answer is incorrect. So that's sort of the absolute minimum of e-assessment.

Write a number, check it's the right number, and give feedback.

You can ask the student to enter a mathematical expression.

So here is a polynomial that I have to differentiate.

I think I can just about do that in my head.

So here we go.

You can see as I'm typing I get a rendering of the answer to the right here, so I can see that it's been interpreted by Numbas correctly.

So if I'd written something like that, I might have meant, "seven over x minus two, like all of that, under the brackets, and I see that that's not grouped together.

So I need to put some brackets on. There we go.

I'll just submit that and make sure I got that right.

There we go.

So, moving on, you can ask the student to enter a matrix, which certainly comes up, linear algebra, that kind of thing.

So here I have to enter an identity matrix, so I think that's something like that.

And optionally I could resize that. You can turn these options on or off, what you allow the student to do.

Short text string.

I think we basically never use this part type, but it exists just in case somebody wants to use it.

So you would have to type some text.

And then there's a lot of multiple choice part types.

So here I have to choose one item from this list.

I think a watermelon is the biggest fruit. There we go.

Choose several from a last, I've got to tick every prime number here.

That's not prime, that is, that is. There we go.

And this grid thing here.

So I've got a list of countries and I have to match them up with their capital cities.

I've got a few distractors here as well.

So capital of Argentina is Buenos Aires, capital of France is Paris.

Let's get that one wrong. Capital of China is not Brussels.

And there we go.

And finally, here's a custom part type.

It's not one of the built-in ones, it's something I've written, a customized marking algorithm, it's a really simple part type, it just offers options yes or no.

The abacus was indeed invented before the mobile phone.

So I can get on to what custom part types mean later on.

So that's, I think, as much as I want to show you of that.

We'll maybe come back to this demo later on.

What else do I want to show you?

Yeah, I'll come back to that later on.

Over the years that we've been doing e-assessments...

E-assessment at Newcastle started before I worked there.

It's been well over a decade now, but Numbas is coming up for 10 years in use.

Initially, we started using computer aided assessment to replace homework, so that students got some practice during their course and it was worth course credit.

That was very successful.

Over the years, we've realized that computer aided assessment is particularly good for formative assessment, where the assessment is for the student's learning and growth rather than so that we can award a final mark for the course.

Numbas is used summatively for some final exams.

But we think formative use is really where it's best employed.

So, one of the really big things about e-assessment - and I didn't demonstrate that - is that the questions can be randomized.

So you can set up a question on a certain topic and randomize it and the student can try it over and over again, until they're happy that they can do the thing that you're assessing.

For summative assessment you have a couple of problems.

How to prevent cheating is a big one, and I guess everyone has experience with this now, with moving to online teaching.

There were always problems with doing stuff at home.

Even written homework, it seems to become more of a problem as soon as you introduce the computer, in people's minds.

But then apart from that, how do you ask sufficiently challenging questions in maths?

So you might want to assess students' ability to write out a proof or explain a motivation for something, and you can't do that with automatic assessment.

So there are limits to what you can do with automatic assessment.

So at Newcastle we use Numbas for a variety of things.

This sort of goes chronologically through the academic year here.

So before students get to Newcastle, once they've accepted their place, we send them out an induction pack with information about the university and what they're going to do on their course.

We also send them some practice material to make sure that they are up to speed with the maths that we expect them to already have before they start the course, covering the English A Level.

And then once they get here in week one, they're given a diagnostic test which is in fact the same test as the one they were sent in the pre entry pack, which we then used to identify weak points for students and tell them to go and get help from our maths support people.

Throughout the semester we have banks of practice material available through our VLE, which was Blackboard and is going to be Canvas, which students can just access whenever they like, so if they want to get some practice on something they just had a lecture about.

Typically, students access these just before tests and before the final exam when they're doing revision.

We use Numbas for some in course assessment, largely just to keep students engaged.

And we're going to have to do a lot more of that, I think, with remote teaching next academic year.

We're still working out exactly how much material we're going to have to write.

So typically they're given a test that's available for a week, they'll get on attempt at it, and shown their feedback once they finish the test.

And there are a few courses around Newcastle that use Numbas for a final exam, particularly stats service courses where the students are just expected to do some pretty rote techniques, and it can all be marked automatically, and a few engineering modules as well.

Actually, before I move on there, on the subject of final exams.

in the semester that's just finished where all the exams - all the assessment - had to happen remotely, all of our first year exams were through Numbas and we think that was pretty successful.

We might talk about that later on, I think. Yeah, Chris Graham gave a talk at the recent EAMS conference, which you can pop a link to, about our experiences with that.

So Numbas is used - this is at Newcastle again - in the maths and stats degree, in physics, and then across all these other subjects.

Engineering, chemistry, they have a lot of their own stuff.

Business studies and psychology, these are statistics service courses that we teach a first year statistics for psychology course.

The Biomedical Science course is new, and Sports Science Course.

Those have a lot of Numbas material.

And there are other uses that I'm just not aware of.

Right, so outside Newcastle Numbas is used very widely. I just checked the stats again and we've now passed the 1000 institutions mark, for people who have registered on the Numbas editor.

Whether they've used it continuously or seriously, I have no way of knowing.

And we have just over 4000 users registered on the public editor and a bank of over 8000 questions available under Open Access licenses to use.

Because of the way Numbas works, everything is entirely standalone, we don't really have great ways of checking how thoroughly Numbas is used around the world, so we just sort of have to guess based on what people are doing on the public editor.

So the numbers are underestimates. But who knows how much they're underestimates.

Right.

So.

This public editor is hosted through mathcentre, which is a UK-wide, I think, repository of Open Access maths resources for higher education.

So the Numbas editor is accessed through that. It's web based, open to everybody, free to use.

You can either have a look at the stuff that's been published and collect it together into your own test, or write your own questions.

And I think I'll get on to show you that shortly, um, yeah, it's a load of screenshots here.

So I think I'll stop here and get an idea of what you want out of this workshop.

I definitely will - the next thing after this will be - I'll go and show you how to write a simple question.

But maybe let's get an idea of - you could write in the chat - where you are, what subjects you are going to use Numbas for and what experience you have with e-assessment already, while I get ready with the editor.

Stony silence here, so maybe I'll just move on.

Here we have some answers.

To use as a teaching tool, both formative and summative.

Alex Corner says, "looking at using it for regular homework and concept checks." Linear algebra, trig and complex numbers, graph sketching.

We can talk about graph sketching.

The answer is largely "no" at the moment.

Elementary school math.

Now that's interesting.

First year engineering. We have a lot of engineering material.

Chemistry lectures.

Yeah, OK. And somebody in finance, so I hope I'm not going to have to talk about actuarial notation.

OK, thanks for that.

I'd be really interested to know what our Indonesian colleagues are using Numbas for.

I noticed a huge number of you all signed up overnight for this workshop, so...

Formative assessment at the start of a first year calculus and ODES course, and summative assessment if the formative trial goes well.

OK, so all largely stuff I'm familiar with, so I can definitely help you all with that.

OK, so keep sending those in, and Chris will keep me up to date with what's going on there.

Right, so.

Going to the Numbas editor. So what I'm going to do is - actually first of all, I'll show you the documentation.

This is where you should start. And in fact, show you how I got to the documentation?

So this is the Numbas editor, I've already got onto the create a new question screen because I'm an administrator on here and so you would see everything if I showed you the home page.

Actually, no, let's log out.

This is what you get. The address again is numbas.

mathcentre.ac.uk. I'm not expecting you to follow along with this, but please feel free to if you'd like to.

Chris might put the link in the chat as well.

So you create an account here, log in again, and hopefully not reveal my password to anybody.

Here we go. And this is what you get shown. So I'm going to go to the documentation, which is this help link at the top right here.

I'll talk about what's in here.

So there's a set of tutorials under the getting started heading, taking you through creating an account, have a look round the editor, writing a question, and then how you actually deliver Numbas exams to your students.

It's a good idea - we're going to go through this today - once you've done that, please make sure you have a skim through the rest of the documentation.

There's a good document on how to plan the design of a question, which I might talk about in a while.

This "How do I?" page has a lot of tips for little things that might not be completely obvious.

In fact I'll show you that.

Stuff like how to get videos and diagrams into a question, stuff to do with formatting bits of text, tips and tricks for randomizing questions.

Right, and I'll go back. And then underneath there's a reference for the entire editor.

At the moment it's just sort of a list of for every screen and every field in the editor and a description of what it does.

I'm working on writing up a more - what's the word - a separate book which will motivate a lot of the design features of Numbas.

And then there's a load of stuff about more advanced things which you might get onto after using Numbas for a while.

Right.

So what I'm going to do is, first of all go through - have a look around the editor.

What I'll do, rather than looking at that page, is I will go back to the editor itself and drag the zoom bits out of the way.

There we go. So what you can do is click the explore button at the top and have a look around the stuff that's been published.

Everything you create on the editor is by default visible only to you.

You have to actively publish it so other people can see it.

So don't worry about your students finding your questions unless you pressed publish.

So we've published quite a lot of material, and collected it in a couple of projects.

There's this "Content created by Newcastle", which is largely stuff coming out of our mouths and stats degree course.

So we've got quite a few things here.

As ever, it's tricky keeping stuff up to date. We're going to have a big shift of stuff over from our private editor to the public one at some point this summer, so we'll be updating that fairly soon.

But there's a whole lot of stuff there that you can use.

And we also have a project on transition to university.

So supporting students who are moving maybe not into the maths degree, but into a degree that uses maths.

We've found that a lot of students get into courses like psychology and business studies and sometimes even economics, and don't realize just how much maths they need.

This is a load of material that is made available to our students through our maths support website.

And there's plenty more.

It's not just Newcastle. There's some stuff on engineering.

Ben Brawn's been doing some stuff in high school in the USA, so linked to the Common Core standard.

There's a lot.

OK, actually, let's carry on looking through things.

So here's the transition to university project.

I click on browse, here, I can have a look through a lot of things.

Click on percentages, why not?

So here's a list of questions and I can click on play for one of them.

And see that question.

So I'm not going to do that.

In my head.

While I'm here actually I'll just show you, I think these prices are randomized.

Yeah.

So the price of this laptop is randomized and the discount is randomized too, so I can click try another question to have another go at something else, and there's also reveal answers.

Where it will show you the correct answer and then the person that wrote this question has also written a fully worked solution.

So here we've got all the randomized bits in it.

That's really good for a student who's not understood how to do this and give them some help.

Maybe they can have another go at another instance of this question.

So you can do that.

You can go around the editor.

Now there's this little trolley Icon. There's no money anywhere, this is just collecting questions into a basket.

If I collect a lot of questions, they go up into my basket and I can create an exam from those.

We'll call that demo exam.

Everything in Numbas goes into a project. When you create an account, you get a personal workspace that's just for you.

Or you might create a project for a particular module you're teaching or a particular purpose.

So here I've created this exam.

I just need to make sure I've got the chat window open, it keeps disappearing.

Right, OK?

So yeah, here's my exam. This is the exam editor.

I've got my list of questions.

I can reorder them. I can set a pass mark, which will decide at the end of the exam, the student will be told, you've passed or you haven't.

There are a few options.

I can pick a different theme. I've got a couple more here than anybody else can access, because I'm doing development work.

And pick an interface language.

Numbas has been translated into quite a few languages, as you can see.

Some more completely than others. As we do updates, there are new bits of text that need to be translated, so some have fallen behind a bit.

So if Numbas isn't currently available in your language, you can translate it for us.

I'll maybe come back to that later, how you can do that.

And then finally, there's a whole load of options about what you allow the students do, what kind of feedback they get, for different use cases.

So you can ask for a password.

We use that in invigilated tests.

For a summative assessment, you probably don't want to let the student regenerate questions, and keep trying them until they get the right thing.

There are some options around letting students move backwards and forwards.

I think my opinion is that it's always good to let students move around, but some colleagues differ.

And some more options here.

And then there's the question navigation mode.

Actually, if I run this exam... So the standard mode for a numbers exam is you get a list of questions.

You see down here we've got question one, two and three and the student has to work through them all and then press end exam.

We realize that's not the best way of presenting material for formative use, where students are just accessing it at will, and just want to practice a particular thing.

So there's also the menu mode, which is what my demo example's set up with.

So you just go straight into a list and you get to choose things.

There's no end exam because it's not intended to be for summative purposes.

You can still keep track of your scores to see what you've been through, what you've achieved.

I think that's really good for practice use.

Right, there are some timing options. You can set a time limit and you can decide how much feedback the student can get while they're doing the exam.

So, for something that is worth course credit, you probably don't want to give the students feedback while they're entering their answers, because they might use that feedback to change the answer and brute force their way towards getting full marks without really understanding why their answer is correct.

So I think that's it for an exam.

Yeah, OK.

I think what I'll do next is I'll move on to the next bit of the documentation, if I can find it.

Which is writing a question.

So again, rather than looking back and forwards between the documentation and the editor, I'm going to try and do this from memory, so I'll go back to the editor and I need that group chat again.

And click new question.

I'm going to call it my first question, it's not, as you can see I've typed that before, but I'll pretend.

Right, OK, so here's the question editor.

And we've got a load of tabs for the different parts of the question.

As I said earlier, a question begins with a statement which is just a block of fixed text where you give the student any information they'll need to answer the rest of the question.

You then have a list of parts where you're asking the student to enter something.

The advice is where you write that worked solution that appears either when the student gives up and says reveal answers or once they've finished the exam and they're reviewing their feedback.

And there's a load of other stuff.

I skipped over variables.

This is how you set up the randomization. I will do some of that in a bit.

So the really simple question I'm going to write here is the question from that demo earlier about eating apples.

Chris came up with this one because he realized he eats a lot of apples, but it's a nice easy bit of arithmetic just to show off basically how Numbas works.

So the first thing I need to do is add a part.

Now the answer I'm expecting to get is a number, so I'll pick number entry.

And I have to give a prompt and then set up how the marking is going to work.

So, the prompt here is going to be: "Chris eats three apples per day.

How many apples does he eat per week?" Right, so there's my prompt, and actually what I can do now is click test run.

And see how this question would look for a student, and I'm going to get an error because I haven't set up the marking.

So.

Click on marking settings and I get a whole load of things here.

The most important bits are the minimum and maximum accepted value.

The number entry part takes a range for cases where you have to do some calculation involving decimals, and there might be some rounding.

You might want to give a bit of tolerance of error to the student.

In this case, it's a whole number, so it's going to be the same thing in both boxes.

It was three a day, which is - right keyboard - 21 a week.

Right, now I'll click test run again.

Here we go, right.

So there's my prompt,my input box, I'll write 21.

It gets marked correct.

Anything else is marked incorrect. So really simple.

That's straightforward.

So if you're just transferring questions that have been paper homeworks in the past and you just want to get a computer based version of those.

It might be as simple as that.

Yeah, you write out the text and then say what the answer is, and it'll get marked for you.

Things get more complicated when you want to randomize things.

So what could we randomize in this question? The obvious candidate is the number of apples that Chris eats per day.

So the way we'll do that: define a variable.

I went to the variables tab, clicked add a variable, and I get this interface.

I need to give it a name and then a definition. Numbas uses a language called JME, which we inherited from the previous system that we used.

The syntax is very similar to most other programming languages, the kind of stuff you would type into a graphical calculator.

There's documentation on all of that linked from under here.

The documentation has a page about JME.

Right, so I need a variable to say how many apples Chris eats per day.

What I'm not going to do is call it N. It doesn't explain very much.

I'm going to call it apples per day.

And then I get to pick a data type.

JME Code is I get to write some code to define this variable.

So I could write something like random two to five.

So it's a random number between two and five. And over here I get shown up a generated value and I can click regenerate and see the sorts of values it's going to create.

None of this is surprising to me at the moment because it's just picking at random from those numbers, but when you've got more complicated randomization, this is really handy.

But this is quite a common thing, so this data type drop-down has a few templates for setting up this kind of thing more easily.

I'm going to pick random number from a range and I get a couple boxes here, so random number between two and five.

The step size is 1, so whole numbers there.

And then finally, there's a box for description where I say what this variable represents, what it's used for.

When you write a question, it's very easy to set things up, while you're sort of hacking away at the code and then if you come back in a couple of months and need to fix something, it's not at all obvious what each variable is for or why it's generated in the way that it is.

So this box gives you an opportunity to document how things work.

So I'm going to repeat some information here because I gave such a good descriptive name for this variable.

There we go.

Right, so now I need to use this.

I'll go back to my part prompt, and replace that three with some curly braces and the name of my variable, so that substitutes the value of the variable into the text at that point.

These curly braces just substitute it in as plain text, which is fine here, because I've got just a number.

When you've got mathematical notation, things get more complicated and we'll get on to that in a bit.

Right.

So let's see how that works. Here we go.

Now, what are the chances that I got three again.

I'll try again.

Chris eats five apples per day. So that's been nicely randomized, but I've forgotten to do something, I forgot to change the marking.

So what's five times seven?

35. Not marked correct, because the answer is still set up to be always 21, so I need to change that.

Right, so here it says 21.

I'm going to say seven times apples per day and copy that into the max accepted value field as well.

OK, there we go.

So we'll do test run. Chris eats four apples per day.

Four times seven is 28.

There we go. So this question's now randomized. That's good.

Chris Jobling's just asked, is there a number to word function that can be used in questions?

It would need localization.

I can't remember if there is one, actually. I've definitely done that in the past and I can't remember if that's a built-in function now.

Let's go and have a look.

Click on the documentation.

Click on the right thing.

There we go.

What would it be called? Word.

I think the answer is no, but there should be.

There could be.

It would be a nice thing to have. I guess you're thinking about cases where convention would say that you have to spell out a number rather than use digits.

The big wrinkle with having that built in is that it would have to be localizable, and the rules for writing numbers in different languages are not in agreement with each other, so it would get complicated.

Maybe an extension to do that would be good.

Right.

So. What have I done?

I've randomize this question. Under my variable now I can see where it's used, so I can see it's used in the prompt and in the Min and Max accepted value for this.

This is a really handy thing for being able to have a look at variable names, and if you've got a lot of variables work out which ones correspond to the answers for things.

Right. So now I've written this question.

What else might I want to randomize?

Actually, I did that calculation there, number of apples per week, inside the field for this number entry part.

What would be nice would be if I had a variable called apples per week.

So I only have to do the calculation once and I can reuse that elsewhere.

So let's make a new variable, call it apples per week.

And it's seven times apples per day.

There we go. So over here on the preview, two apples a day is 14 apples a week.

And I can regenerate these.

And underneath I can see that this variable depends on the variable apples per day.

Go back to apples per day and it's used by apples per week.

Right, so I think that's a decent minimal question.

Let's have a peek at the documentation to see what the next thing to do is.

Right, so I've written a question, I've written a statement, set up some variables.

Oh yeah, worked advice.

Where are we?

Over in the advice field.

So I have to give a description of how to get the right answer to this question.

So maybe you might be missing the information there are seven days in a week.

And then maybe I'll do some maths notation here, so I should multiply apples per day by 7 to get that many apples per week.

So there's my text, I've used curly braces to put those variables in.

But I can click this preview button to see just this bit of text without going into running the whole question itself.

So there we go. If I run the question again.

There we go, I've revealed answers, under advice, there's my worked solution.

Lovely. Right, I could see a couple questions came up there.

Um, Augustine asked in the case of decimal answers, is it possible to restrict the number of decimal places.

It sure is! So let's make another part, and make up a decimal answer.

I mean.

There we go.

Something like this, so some calculation you want the student to round off.

So I can set the correct answer here.

Then further down, there's an option for precision restriction.

So I can say I want a certain number decimal places.

Two of them. Then there's an option.

In some contexts, it's really important you give exactly the right number of decimal digits.

I think in engineering, you use the number of digits after the decimal place to convey the precision to which you know the number.

So 1.2 isn't the same as 1.

20 because the margin of error is different in those. So you can require that or not.

Let's say we do require that at the moment.

And you can give partial credit if the student gives their answer to the wrong position.

So what Numbas is going to do, it's going to first of all see how many digits the student has given their answer to, round off the correct answer to that many digits, and then compare.

So.

There we go.

1.23, I think that is. I get a little hint here about writing my answer, just in case the prompt hadn't said that.

You could turn that hint off if you want. There we go, and if I'd given that whole number there - let's say I got the right number, but I didn't give it to the right precision.

OK.

Lynn says, is there an option to record a video solution for the advice.

Yes, certainly, I'll go back to my demo exam.

I said I'd come back to it. So I've got this question here.

You can put images in a question.

That there is a Numbat, which I'm trying to make the official mascot of Numbas.

You can use SVG, which is a vector image format for making up randomized pictures or things that scale nicely.

And then I've got a video down at the bottom. So if you've got a video on YouTube or Vimeo then it's quite straightforward to stick it in.

For a lot of our practice material we have these videos, which a colleague who's now retired recorded.

Couple of hundred of them. We'll stick these videos either in the advice or in steps, which I haven't talked about.

So I'll go back to this question here.

If you want to give the student a bit of a helping hand, a hint or walk them through the steps of a calculation, you can use steps.

So what that is...

I'm going to do information only.

These are extra parts that the student can choose to reveal or not.

So.

That's just some text there, and the way it works in the interface is you can click show steps and you get shown it.

So I could have several.

If this was a really involved calculation, I can have a few more number inputs for each of the steps along the way there.

So the student would get guided through it.

They get some scaffolding, but also they have the opportunity to get partial credit for as far as they get correct answers.

Right.

So what we often do is we stick a video in the steps there.

I could optionally - where's my option here?

Penalty for revealing steps - I could take away some points if the student shows the steps, so they won't get full marks for having seen the hint, but they can still get something.

Right.

Over on my demo. For each part type, I've got a question so I can show a load of options, and it might be worth quickly running through these, if not how to author them.

So for the maths expression part there's a straightforward one.

You can have randomized variables in the answer.

Implicit multiplication is something students always do.

When you're writing a mathematical expression on paper, it's conventional to not put a multiplication sign between a couple of variables or a number and a variable.

So Numbas allows for that, but there are also cases where you want the variable to have more than one letter, so that's an option.

You can ask for the student's answer to be in a certain form. The way Numbas marks a mathematical expression is it spots the variables in the expression.

So here 'a' is a variable.

It picks random values for those and then evaluates, both the student's answer and expected answer at those points.

If the values they produce are the same, then the expression is considered equivalent, so the student gets the marks.

That falls down when you're really interested in the form of the students answer.

So this one.

This is sort of a collect together powers thing. These two forms would both give the same results numerically, but you're interested in, "Has the student collected together all these powers?" So you can use a pattern match restriction to make sure they've written something like "a to the power of a number." And then this stuff about how you can mark things like there's an unknown function involved, or mark formulas.

This will be the demo exam linked on the Numbas website as soon as this training is finished, I nearly got it all wrapped up before we started.

Number entry. We've talked about precision restrictions, different notation styles.

So in the UK we use a dot for the decimal separator and commas for powers of a thousand.

In Europe typically they have it the other way around, and there's different notation in India.

Switzerland has its own unusual notation.

Or scientific notation.

So you can select those.

I sort of alluded earlier to the fact you can accept a number in a range.

You can use scientific notation.

Numbas now uses very high precision number representations, so you can talk about extremely big numbers and extremely small numbers without running into floating point rounding errors.

I could take a fraction.

Rather than a decimal number, which is important here because 22 over 7 doesn't terminate.

And you can optionally ask for the fraction to be reduced.

Chris Jobling's asked, "Is everything is done in JavaScript?" That is right.

How much would be revealed if the student turned on developer tools?

Well, they could see pretty much everything if they knew what to do.

So that's a really good point. In formative use, that doesn't matter because it's for the students own benefit, really, they can cheat whatever way they like.

Cases where it's a high stakes exam, you need to if the student's not physically present, and being invigilated, you need to use a lock down browser.

There are a few options for that.

Newcastle has a license for Respondus.

There's a really good open source tool called safe exam browser which you can use.

So those effectively just show the student the page, don't let them open developer tools or anything like that, and that's the best you can do.

Numbas is made this way because having all the stuff done on the client makes it really easy to use in a really wide range of places.

It lets us offer Numbas to the whole world without having to run some huge server that would get overloaded under use, so doing stuff on the client really makes it a lot more versatile.

And having a locked down browser is a good solution.

Even systems that do all the marking on the server tend to recommend using a lock down browser anyway.

Unless anybody has more questions, I think I'll skip some of these other part types.

If there's something you're particularly interested in, I can show you.

Choosing from a list, I mean there's a lot of options.

You can put things in columns. Each choice has its own number of marks available.

So for example in this one there might be an answer that's nearly correct, you want to give a bit of credit for, so I think that one gets one mark, but there's one which gets all the marks.

You can get distractor text to explain why an answer is wrong.

It's quite a self-referential thing, but in a real question you would say something like, explain why the answer is not correct.

And you can randomize the order of choices, and you can have variable numbers of choices or make the marking depend on random variables.

OK.

Right, say something about shared projects.

I'd sort of alluded to that, looked at that a bit earlier, didn't we?

So what can I do without showing too much here?

So we have an absolute tonne of projects.

Typically at Newcastle we'll make one project for each module and collect together all the materials for that module there.

So here on the public editor we've got a few things, our maths support projects supporting particular subjects.

And then the transition to university stuff.

The way a project works, every question and exam in Numbas belongs to a project.

A project has a list of members who can edit stuff in the project, and you can give people just viewing access if you want.

So everything inside the project is visible to the members of the project.

So you get this timeline of material that's being worked on, activity inside the project, you get a breakdown of status of content to help you collaborate.

I'll go into some of these things. Let's pick some stuff a while ago.

I've done a load of work here. This is a project we worked on with some student interns so they had a lot of back and forth.

Gosh, how many years ago was it?

Was it 3 or 4? [Chris] It's four.

Can't believe it. [Christian] Four years ago? Wow. Doesn't time fly?

Just keep going back until we find, I don't know.

There we go to that one.

So yeah, I sort of glossed over the collaboration features.

And that happens on the editing history tab.

Oh, here we go. So, uh, Hannah wrote this question.

When she was finished, she labeled it as needs to be tested.

I'll go way up to the top here.

There's a button here, you can give some labeling on questions, just sort of where it is in the editing process.

We have a rule that somebody who writes the question cannot say that it's ready to use.

Somebody else has to look at it. So, Hannah wrote it, so it needs to be tested.

It looks like Chris went in and ran it, then had some feedback for her so he left a comment.

Some suggestions of changes.

It went back and forth as she was fixing stuff.

Vicki came and gave some more feedback, and so it's been quite a long back and forth about how this question should work.

I think we tried to get as many people as possible looking at these questions because they were going to be used by a lot of people.

And I've written a lot of stuff. When was it finally ready to use?

There we go. So finally I was happy with it and said ready to use.

I wasn't very polite there.

I didn't leave a comment saying this is good.

I normally did. And then later on, somebody has come along and said there was a problem.

So because this was published, everybody else has been able to look at it.

Yeah, so that person noticed the typo and I spotted it.

So that's largely the workflow.

We used this for writing our end of year exams this summer.

We had a team people spread around the region, writing content and then back and forth with feedback and editing.

Proofreading. Catching up again.

Mochamed Apri says can we draw a randomized graph of a function?

Yes, we can. There's my demo. Here we go. There's a few options for this.

There's no really good option for me.

There's no one absolute sure fire answer for drawing diagrams.

There are a few, based on what you want to do and how much coding you want to do.

So.

First thing that we ever used was a package called JSXGraph.

There's an extension in Numbas, which lets you insert JSXGraph diagrams?

And that gives you geometry tools. You can draw geometrical diagrams, plots of graphs, charts, that kind of thing.

In order to set it up you have to do some coding in JavaScript though, so I think that's quite a big barrier for most people.

But this question is quite nice because here I've got to write a formula for the height of this object.

A projectile that's shot upwards.

As I write and expression it gets plotted for me.

We have plenty of questions where there's just like a static graph is shown to the student, but this one's a nice dynamic one.

What's really good about this question is that it cements the link between the algebraic form of a function and its plot.

Students when they're learning this - and I know I did when I was young - take a while to get the hang of what the different bits of a formula mean and how they affect the plot.

So I'm going to try and do this one. A hundred T minus T times T squared over 2.

Something like that. Good old equation of motion.

I think that's the right answer.

Yeah, it is. So the diagram here is used for illustration.

You can also have interactive elements that - we have some questions where you move points around and that fills in an input box, so it can go both ways.

Another option is GeoGebra.

GeoGebra is really easy to use and very powerful.

It's again an open source system, they have a really nice graphical editor.

Here I've made a diagram of a couple of points and sort of bisected them, a circle through them and then a perpendicular bisector.

So it gives you a load of again geometry tools.

But then there's a lot about stuff drawing graphs and interactivity.

So that's a really nice, easy to use method of getting diagrams in.

And I can use question variables to set up the GeoGebra diagram, but getting stuff back from GeoGebra out into Numbas is a bit more complicated.

They used to have a good feature for assessing constructions.

But they've taken that away and so it's sort of a one-way process now.

But for illustrating. providing some interactivity so the student can get a feel for how things move about, that's really good.

And then finally sort of in the middle, there's Eukleides, which is a language I found years and years ago, and I've re-implemented in JavaScript and made the Numbas extension for, which makes it easy to create diagrams.

It's sort of halfway between JSXGraph and GeoGebra.

You don't have to write JavaScript.

The language is a lot more simple. You can say "line between these two points, and circle at this point with this radius", without a load of extra code and syntax.

But it lets you create diagrams based on random variables.

If these questions were randomized, we'd just draw diagrams and they would be fixed.

It's the fact that they need to update with the random values that's the hard part.

OK, so that's about diagrams.

Got a question from Rinovia, "How do I embed exams in LMS other than Moodle, MS Teams and Google classroom?" So I'll go to the documentation to talk about that.

Going to the page on delivering examples to students. So, an exam can run standalone in the browser.

Standalone means that the data doesn't go anywhere.

For both, for scalability and for data protection reasons, we don't offer a database where you can save student data to.

So you need to have something through your learning environment.

So you can do that.

When you - let's see, where was it? - run something through the editor you can click the share button and that address is something you can embed or link to.

And your page and test will run completely stand alone. To save data, you need somewhere for it to be saved.

So a couple of virtual learning environments offer a SCORM Player.

Blackboard and Moodle do in particular.

Blackboard's SCORM player is very buggy so we don't recommend it, but just for formative use you can do that.

Moodle's is very good and can be relied on.

If you don't have access to SCORM then you need to use LTI.

There's a Numbas LTI tool. The way LTI works is you start in your learning environment.

The student clicks the link to launch an activity and The LTI standard says how information from the learning environment gets sent off to some other service, in this case the Numbas service.

Launch the activity, the student does the thing, and then a score gets reported back to the learning environment.

So this needs a server to be handling all the Numbas attempt data.

That's open source, we've released, but you do need to set it up locally, so you need your IT people to set up a server, a virtual machine or physical one, running this software, and then it connects to your VLE.

There's documentation on what's involved there, what you need.

It really doesn't need very much in terms of resources And it's quite straightforward to set up.

And we're happy to offer some support setting things up, but we don't offer a public LTI tool, again for scalability and data protection reasons.

We are thinking very hard about what we can offer in a way that would make it feasible for us to offer something easier to use.

But anyway, that's there and a lot of places are using it.

I know a fair few universities in the UK and around the world are already using that.

So that's what we use in Newcastle.

We have an LTI connection to our learning environment.

I'm not sure. I don't think MS Teams - Microsoft Teams - supports LTI, so you would be linking to standalone tests.

You can get the students to send you their scores. You can - actually, I'll show you - where are we?

Just run a test.

If I end the exam, there's a button to print a results summary and that will give this results page plus all of their questions.

So they could send you a PDF of their transcript, which is a bit slow and manual, but if you don't have a way of saving data, that's what you could do.

Yeah OK, and Google classroom, I feel like I've looked into it before.

I'm not sure if it supports LTI. It might or might not, but you would have to set up this server.

Right?

No more questions there. Where was I? I was going through my demo.

I want to talk about some more things that Numbas can do.

A few of these things are quite new this year actually.

The first one is alternative answers.

It really often comes up that you've either got certain errors that you're expecting the student to make and you'd like to give some feedback based on that, or partial marks, or if you have a question that has more than one acceptable answer, to accept those.

So the way you can do that is with alternative answers.

If I go back to my first question. There we go.

Right.

So here if I want to think of another error here.

What could the student do? I suppose instead of multiplying by 7, they might add 7.

Click add an alternative answer.

So this is a copy of that part, and the marking settings were copied over and I can change whatever I like.

So instead, I'm going to say the correct - the accepted - value for this version is plus 7, plus 7.

Right?

I don't want to give any marks for this answer because it's completely incorrect, but I do want to give the student some feedback.

So for something that was almost right, you could give partial credit.

So I can give a feedback message and say something like, "Did you add 7 Instead of multiplying 7?" So in this version, the correct answer is this thing.

I don't want to make a copy of that, I want to run it.

And here we go. So if I make this error and enter the answer 3 plus 7, I get this specific feedback message, rather than just saying it's incorrect.

It takes a lot of time to think about errors that students would make, but sometimes there are obvious errors you're expecting.

Part of the design process for a question should be thinking about how a student could get a question wrong.

And so you should be able to program in those answers and give students more tailored feedback.

That's really handy.

That's a nice easy way of setting those things up.

So here in my demo question the example was, I'm going to give partial marks for getting close.

So I've got two numbers to add here.

Without thinking too hard, I could just add up the first digits.

I get some feedback here.

I'm within 100 of the correct answer. So I've been given 3 marks.

I might see, can I get a bit closer?

So I get closer and closer, I get within 10, and I get more marks.

I think the correct answer is 2087. There we go. I got full marks for that.

So there's a lot of alternatives there that accept progressively narrower ranges around the correct answer.

And they gave different feedback, different number of marks each.

OK, so that's alternative answers. I'll come to explore mode after adaptive marking.

I think explore mode takes a bit more work.

So adaptive marking. A motivation for this was error carried forward.

If you have a question with several parts and the student gets the first part wrong, then they're probably going to get the rest of them wrong as well.

So students think this is sort of unfair. It's not a fair assessment of whether they can do the techniques that are being assessed in the later parts.

So you can accommodate this by saying, "If the answer to this second part here depends on the answer to the first part", and that's a question variable where you replace the value of the question variable with whatever the student wrote there and re- calculate what the correct answer for this second part would be.

So then if the student gives that answer, then you know that they've used the wrong value that they had earlier, but they've done the right thing with it.

So you can give them some marks. So here if I enter - I'm fairly sure one is a wrong answer to this.

Yeah, there we go.

Then I'm told that one bag of apples cost £1.

10 so I will spend £1.

10 on one bag. That gets marked correct and I get some feedback here saying this was marked using your answer to previous parts.

So even though that's not the right answer, this can be 9 times that, I get the marks for this part's calculation.

And if I do reveal answers, the correct answer is still like that.

So that's fairly easy to set up. Let's have a look at that question.

Adaptive marking.

So I had my second part here.

Then the number entry gap.

There we go, adaptive marking. So I replace the variable number of apples to buy with the answer to the part whose title was "how many apples to buy." So when the student submits an answer to this part, it first of all tries to mark against the correct answer.

So try without replacements first.

If that doesn't work, if you don't get full credit, it then tries doing this replacement.

So it recalculates whatever the correct answer was going to be.

Total cost, which is itself a variable, so cost per apple times apples to buy, we've replaced that value.

Which gives you a new range of acceptable answers.

And then that gives you some marks.

Whichever version gives the most marks is used.

I can optionally apply a penalty if that error carried forward marking is used, which the engineers asked for this year.

You can use adaptive marking a bit more creatively. The chemists at Newcastle have set up digital labs, so the students will go and do an experiment in the lab, take a lot of measurements and then they have to do some calculations with that - sort of work out yield or something.

I don't know what chemists do. Anyway, the way they used adaptive marking was, they had the first part where they say, "enter your data." The rest of the question is based on that.

They use adaptive marking to set up the question variables following what the students did.

So that's worked very well for them. I think they used that quite a lot.

Which leads onto my next thing I want to show you.

Yeah, Chris Jobling's just asked in the chat, are the transition questions available for other unis to use.

Yeah, those are published, so if you click that explore icon, you get that those things are all released under a Creative Commons Attribution license.

Right, so the next thing is explore mode, which is this year's big new feature.

And the motivation for this, well, there are a few things that we wanted to do.

I had colleagues saying, we want to offer the student a choice of hints at different levels of usefulness, or on different things, so it might be do some really complicated calculation, and you might want to offer students a reference to their notes for a small penalty, or you might want to offer them a really scaffolded, guided work through the whole calculation, for which they would get quite a big penalty.

And that wasn't possible with steps.

It was, either show of them or none.

So that was one use case, another one is, sort of, an interactive exploration of a topic.

So you might...

I'll show you this example here.

This one's an experiment. The idea is you're going to go through a series of steps and set up an experiment, then do the experiment and then evaluate it.

So it's about coin flipping. I've got to decide whether a coin that I've got is fair or not.

How do I do that? In the standard view, this might tell you what to do, say do it ten times, work out how often it was heads, is it fair or not?

Which doesn't give the student much choice.

Here I think I've only got one path through this, but I could have more, sort of "choose which technique you want to do", "what you want to measure".

So explore mode makes it possible to choose your path through things.

How many times will you flip your coin? So I'm going to say I'm going to flip my coin four times.

So that is then set as a question variable. If I move on, that gets stored.

The next question is, "how many times do I have to come up heads for me to think the coins biased?" I think anything other than exactly fair is.

probably biased, so I'll say 3.

And actually before I enter that answer, I had an option for a hint here.

So to help me decide what number would make sense, I could say, "Here's a hint: with an unbiased coin, how many flips would you expect to be heads up?" And the answer there should be 2 and that's based on the number that I put in earlier.

So going on that, two would be unbiased, so I might say "well, three's probably biased." If I'd chosen to do more flips I might pick something a bit further away.

So next enter the data. So flip the coin. There's my number 4.

And, let's say it landed 3 heads and a tails.

There's a custom marking algorithm which interprets this here.

That gets saved to a question variable as well. So next I have to say what percentage of flips landed heads.

If that's beyond me, I've got a hint here. Count up how many heads, how many tails?

Just in case I needed to check.

Because if I'd done a lot, I might lose count.

So that's a good way of getting me past that.

So I think 75%. Yeah, there we go.

I can move on. So you see this this is sort of leading me through down one pass.

You can design questions where you get a choice of paths.

You can move around. It's really versatile.

Finally, based on all of that, I have to decide if it's biased.

I think it is, yes. Now it's not, no.

Because I got that wrong, I get this hint, "Calculate critical likelihood." So again, work out what my threshold percentage should be.

So yeah, that's really powerful.

I'll show you how it works in the editor again.

I'll just find that question, explore mode.

This feature is brand new and I'm really keen to use it in a lot of ways.

I could probably talk for ages about it. I think I'll talk more about it in the advanced session next week.

I'll have to make sure I stop myself here.

So the way it works: for each part, I have a list of options for the next parts to move to.

Where's my...

move in a little bit more.

So for each option you can say when the option becomes available and you can do some variable replacements like with the adaptive marking.

So that's really powerful.

Right.

Finally, I'll just talk about a couple of extensions.

I've referred to extensions a couple of times.

These are ways of adding extra features.

Either functions to use in variable generation, or embedding things like GeoGebra worksheets.

So here's a couple of them.

The random person extension solves a problem that we had, particularly with our maths support material.

We had a lot of word problems with named people in them, and I noticed that they were very largely male characters and they were very largely names of whoever was sitting next to the person writing the question.

Which, led to a bit of an inclusion problem, frankly.

So there's plenty of evidence showing that students need to feel that they are the subjects of teaching material, textbooks and assessment material.

So if the student never sees, or commonly sees names of - if you're a female student, you always see male names and things, if you're a student from a different cultural background to your lecturer, you might not feel like you belong on the course.

Anyway, what you want to do is make people feel like they belong on the course.

So what this extension does is it randomly picks names for characters.

I got some data from the UK on relative frequencies of names.

It picks from those. So basically this job's taken out of my hands.

This only applies in a UK context.

But if you can get hold of data for where you're from, then the same thing should apply.

There you go.

So that's what that does. It generates person and then randomises pronouns and things like that.

Now. Something that came out of the work with the chemists was that they always ask for units with their numbers, which we've never done in the pure maths course.

So I went away and found a package that deals with quantities with units and wrote an extension to add that into Numbas.

So now here we have a part type where the student has to give their answer along with some units.

And it's not just about marking, it's about random variable generation as well.

You can generate values with certain units attached to do arithmetic on those, so 9 centimetres cubed times 0.

7 moles per decimeter cube is going to end up in terms of moles, I think.

So all arithmetic to do with combining units together is handled very nicely.

So that makes that quite straightforward to work with.

Oh, here's a good example of steps.

I've no idea how to do this calculation, so, uh...

Hey, I'm taking through here.

There's a little layout bug there.

But I could work through these things. So I think, ... Is that correct?

No, no no, sodium hydroxide, right?

dm cubed, oh no.

Right, I don't know how to answer this.

Yeah. Let's reveal answers and see what the answers were.

Oh, there we go.

I didn't read it properly. So anyway, you see, the answers to these things have units attached and the student's answer is only correct if they've got the right number and the right units and there are a lot of options around what the right units means, would CM cubed be acceptable here?

No, because it's converting from centimeters cubed.

In another question where you're just interested in the dimensions like centimetres instead of kilograms, you might accept any of the orders of magnitude, for the right kind of measurement.

Right. So, we've got half an hour to go. I think I've demonstrated all the things.

I see Linda's asked, will we be able to see these questions afterwards?

Yeah, my intention was to finish this and get it published before the session started, on the Numbas website.

There's a big link called "See a demo" which at the moment goes to the old demo.

It will go to this new demo shortly and it will have a link to see the same thing in the editor.

So you can see how these questions are implemented.

By the way, the Numbas website has some information.

I said earlier on I would talk about how you can help contribute.

First thing - the easiest thing to do I think - if you speak another language, is to translate Numbas into that language.

There are some links here to go to the translation management site.

You can contribute translations.

As I said earlier, the translations fall out of step as the English version gets more text put in it, so even ones that were complete like Swedish I could do with help with.

So you sign up to this site and you get shown a list of text and you just have to write in there the translated version.

So if you can do that, that would be really helpful.

The code's all on GitHub. Help with code would be extremely welcome.

Doesn't happen too often, but if you spot a bug or even if you're just not sure how to do something in Numbas, you can file an issue on GitHub, or email us, or the user group.

There's a link there to the user group, which gets a fair bit of traffic.

We watch that and other people might be able to help you if you've got a question that's been answered before, so that's how you get help with things.

Yeah, so, I think I'm done with my demos.

Matti Harjula, hello Matti, said, "a question related to mathematical expression parts, can you build a question asking for an example of something?" Yeah, if you come to the advanced session next week, I'll talk about the custom marking algorithms feature, which lets you change how answers are marked.

The graphical editor gives you a load of options, but there's always going to be stuff you want to do that wasn't foreseen by me.

OK, right, so I think I'll stop my slide - actually, I'll skip to the end there, just remind you of some addresses and things.

So you can go to the Numbas website.

We're always happy to receive emails through the Numbas email address.

Don't count on a quick response, but don't worry about wasting our time because so many of the features in Numbas have come about from people asking questions outside Newcastle.

I'm always happy to help support use of Numbas elsewhere.

OK, so I've turned my screen share off.

I think I'll take questions now if there's anything else that you want me to take a look at.

One of the problems with Zoom is I can't see who's typing, so I could either sit here in silence indefinitely or someone could say something.

Alright, OK.

So Vivien asks, "What are the challenges using Numbas for summative assessment?" So the main challenge is what you can assess.

Everything has to be marked automatically, there's no feature to go back in and manually mark parts of a student's exam.

So for pure maths that limits what you can ask, you can't ask the student to write a proof.

For other subjects, you can't ask the student to write a long text answer like an essay or something justifying their answer.

So that's something to bear in mind.

For cases where everything can be marked automatically and fairly, so these kind of service courses, lots of math but the way they've traditionally been assessed is that there is a right answer.

For those you need to think about robustness of your questions.

If something is done in a high stakes environment where students don't get immediate feedback, you really need there to not be any errors in your way of setting things up.

The conditions under which students can cheat, you need to think about.

With all our students being at home, I think that made people more willing to be pragmatic in the way that we traditionally have at Newcastle.

You have to accept that students are going to have a friend that they can talk to, all those things that aren't relevant, aren't particular to electronic assessment.

Even if students were doing their exams on paper they would be able to do those things, but a lock down browser is an absolute must for Numbas, because otherwise a student who knew what they were doing could access everything.

So yeah, I think that's it.

You have to test your questions very well.

You need to think very hard about alternative forms students could enter their answers because you don't have that sort of feedback loop of the students saying "I got this wrong, it should be marked." Otherwise you'll have to go back and manually fix the marks for a lot of things.

I don't know, Vivien, if you joined halfway through, I talked about some options for a lock down browser earlier on.

At Newcastle we have a license for Respondus, which is set up to work through our Blackboard and we can't really use.

We wrote - I wrote - George wrote a separate app just for Numbas, but there's an open source tool called safe exam browser which is perfectly acceptable and is used very widely.

Completely on the level with the commercial offerings, I think.

And what that does is it launches the test, it provides some kind of secret key to the server so that the server doesn't let in anybody with a normal browser, launches the test and doesn't give a student access to developer tools where they could inspect the code of the Numbas software.

Suzanne Anderson, asking about students losing connection.

So it depends on how the test is being delivered.

Stand alone tests, where the students are going to email you their transcript at the end, it runs completely on their client, so nothing can go missing unless the computer breaks entirely.

For SCORM, which I'd only recommend using through Moodle.

It does save the data as student goes.

So if the connection breaks, or the computer crashes, they can go back in and resume from whatever was last saved to the server.

But if your internet connection goes down, then you might carry on filling in answers before you realize that and that stuff hasn't been saved to the server.

What the Numbas LTI tool does is it also saves to the device's local storage, so as long as you open the attempt again on the same device, it will resume from wherever you last were.

Either what's on the server or what's saved on the device, which ever was the latest version of the data, so really data won't go missing, using the LTI tool.

And that was very heavily tested over the past few years we've been using it in Newcastle, and particularly in the recent exam period.

OK.

I'll continue waiting for more questions.

I'll try and think about what else I can show you, if anybody else has anymore.

We've got a little bit longer in the timetabled activity, let's skip back and see what did I miss from the documentation.

I didn't talk about LaTeX, I can do that, and I didn't talk about the question planning, which I think is good general advice, so I'll I'll go back into screen share and do that unless a question has popped up.

So.

You probably spotted some mathematical notation throughout some of these questions.

Numbas uses LaTeX to represent mathematical notation.

It's very widely used in maths and the sciences.

If you don't know it already, it's not too hard to pick up.

And there's some links in the documentation to good places to learn LaTeX.

The way it works is you write dollar signs and inside that you have some LaTeX code.

Here in the other - I'll just zoom that in a bit.

I've got a preview of what I've written. There we go.

Now there's a problem with substituting random variables into LaTeX.

Actually, let's set up a couple of variables first.

Call A, random number between minus five and five.

B, again a random number between minus five and five.

Oops, just clicked the stop share button.

Just wanted to make sure I had the chat window open still.

Right. Right, so I'll go back to this part, right?

So there's a command called slash var, which substitutes a variable in in place without looking at context.

So here we go.

The value of a is five. I might want to say, a times x plus b times y, something like that.

The problem here is, what if these values are negative?

I've got lucky straight away.

So that minus two was just subbed in place, which is no good because it has a plus sign right next to it.

If you'd been writing this, you would have written five X minus two Y.

So you really need to know some context about where things are being substituted in.

Which is why there's a separate command called simplify.

Which. Simplify is not a word you should ever use with a student, I don't think.

It's really ambiguous.

In different contexts, simplify means different things.

Here, indeed it can mean different things and I'll show you.

So you write simplify, have some curly braces containing things, and then you write a JME Expression because the system needs to interpret it as a mathematical expression, as a tree of operations, and then do some stuff to it to work out how to draw it as LaTeX which then gets rendered in the page.

So here we go.

It's noticed that it's got a plus and a minus and instead just written a minus sign.

So that's how you substitute variables into LaTeX.

Now there's another wrinkle here about what simplify means.

Suppose I've just got two numbers that I'm adding up.

By default, simplify applies a rule that says if you see two constants collect them together, sort of gather things up.

It tries to apply simplification rules to make an expression look more natural.

In this context, I don't want to do that because I want to show those two numbers separately.

So it's configurable, which I could do with some square brackets - and all these rules are listed in the documentation.

So when I do this I have to go off and look at the documentation to work out which rules I want to turn on and off.

So here with just the basic rules that make sure you get something that's valid notation, so like never have a plus and minus next to each other, but otherwise don't mess with stuff.

Here I get five minus two.

I can regenerate and see that it makes nice choices.

OK, I've got a question from - sorry, I don't know how to pronounce Polish names - Maciej.

"How do you get explore mode?" You have to make that choice when you create the question.

Because explore mode works so differently to the list of parts mode, it's not possible to switch between them on the fly.

So I go to create a question and then parts mode I'll pick explore.

Right, what else did I say?

I haven't looked at documentation on how to plan a question.

Question writing is a skill. Writing questions that can be marked automatically, there are lots of things you have to bear in mind, and then writing questions that could be randomized is even harder.

You certainly develop that skill as you do it more and more.

We've collected some guidelines in this document.

We wrote this for our student interns because the students were writing questions that, the prompts that they get the student didn't give them enough information or relied on some conventions that weren't obvious, or the marking was set up in a way that it was impossible to submit a wrong answer or there were certain correct forms of answer that weren't accepted so we sort of wrote this down to remind them they have to think about all of these things.

So.

The general idea is: First of all, before you even come into the Numbas editor, think about what you want to assess.

Don't think, "I want to question on this chapter of the course." Think about exactly what particular thing you need to assess whether the student can do.

So the example I give to start with is, "complex numbers" is not a thing you're assessing.

"Recall the definition of complex conjugate and find the conjugate of a given number" is a thing that you can assess.

The student can either do that or not.

And it says "make this the title of the question".

The way that we organize stuff in the editor is the title of the question says what the student needs to do.

So if we need a question on that we can find it again.

My predecessor at Newcastle gave question titles like MAS106 question 3 which is no use to anybody.

I mean, it tells you which module it's in and where it appears in the exam, but other than that, no use at all.

So think about what you want to assess, and then think what will you ask the student to do in order to assess that?

So will they have to perform a calculation, and give you the results?

Will they have to give you an example of something?

Will they have to make a decision?

So quite often a statistics questions end with, is there a significant difference?

That kind of thing. Or they might have to identify something.

You might give them a load of data or a lot of text and they have to pick out, and have to interpret it, and demonstrate their reading comprehension.

So you might ask a student to do several of these things at once.

So once you've thought about what you want student to do, then think, what could they do to get it wrong?

This came up surprisingly often writing these recent exams, that a lecturer would set a question, they hadn't thought about what a student would do wrong.

So that the marking scheme that they suggested would make it possible for a student who's understood nothing to get half the marks or make it so that you couldn't submit something that we thought was a reasonable attempt at an answer, but the way they'd set it up Numbas would say this is completely invalid.

You have to think about that.

So for example here I've got Differentiating cos of two X.

There's a whole lot of different things a student could do wrong.

Some of them you might want to get partial marks for, like forgetting a minus sign, forgetting to apply the chain rule.

You might want to give some specific feedback there.

They might do things in the wrong direction, so you might integrate instead of differentiate.

Or you might make a typo and that last one you probably want to prevent the student from submitting their answer, because that's not what they meant.

There's a separation between whether the student's answer is a valid answer, meaning it's the kind of thing you're expecting, and whether it's correct.

And you should prevent the student from submitting invalid answers, but allow them to submit reasonable incorrect answers.

But don't get too far into this because trying to think about every possible error and programming that in individually is a huge time outlay on your part and often not much more benefit to the student than just getting a tick or a cross and going, "Yeah, I know what I did wrong." So once you decide all these things, think about the structure of the question.

Implement it in Numbas. Don't randomize it at first.

It's easy to get really bogged down in working how to randomize a question before you've even got anything working.

And in fact, something might come up once you've made a static version of a question that you see which bits are easier to randomize.

Something I haven't mentioned here, which I used to give in these demos, is a really good technique for randomizing questions.

For example, if we've got a quadratic, you might give the student a quadratic of the form A X squared plus B X plus C, ask them to factorize it.

You might think, "I'll randomize the values A, B and C and then workout what the roots are." The problem you have then is, you very rarely get nice roots.

You maybe want the roots to be whole numbers.

Certainly not complex numbers.

Doing it backwards, starting from the answer, so let's randomly pick the answers and then work out what quadratic to show to the student, is a much easier way of doing it.

So working backwards from the answers is a good technique for randomization.

And then finally, make sure you've given the student all the information they need.

I mean, you've thought hard about how to design the question, so you might forget to mention that variable X is positive, or you might forget to tell them exactly what form they need to enter their answer in.

It's possible to do very badly on this, and the question that you can work through and answer and do the correct answer and the incorrect answer and you think it's fine, but someone else looks at it has no idea what to do.

This is something that's really worth having someone else looking at a question for.

And then finally randomization.

And then - I can't remember if that's a deliberate mistake, saying proofread and I've missed off the P.

Let's call that a deliberate mistake. Yeah, so I haven't talked about the admin bits here or how to publish stuff.

So in the settings tab there's a description field which appears when you search for questions.

So you can describe what the question's for, what the student has to do, maybe which bits of the course it links to.

I've got some tips here on what you should put in.

You can tag things to make them easier to find.

Select a licence if you're going to publish it. You need to say under what terms other people can reuse it.

There's a load of Creative Commons options.

All rights reserved, you probably don't want to be publishing stuff if you think it's all rights reserved.

And then there's some taxonomies which you can use, although they're not very widely used, but if you're committed to tagging things properly you could really make use of that.

If I need to publish things, actually that's already been published.

Let's go over here.

Access. This is my thing. So I've got a check list of stuff that I should probably do before I publish.

This question does not have a statement at the moment, I've done some parts and variables.

These are sort of optional. Name, fill out the description I haven't done yet, and I haven't picked a license yet.

Once I've done that it'll suggest to me that I can publish it.

At the moment I can override if I'm really sure.

And you can grant individual access to things without publishing them by a couple of links here.

Or you can type in somebody's name to give them access through the editor if they've already got an account.

Chris Graham - there's two Chris Grahams there.

Chris was granted access. Let's remove that just in case it's the wrong one.

Yeah, so.

Right, I think that's probably all I want to say now and now we're coming up to the two hour mark, so I think we'll stop there.

Thank you all for coming along.

We are running an advanced session next week for people who've already got quite a bit of experience using Numbas.

I'll be going through some of the more complicated parts, taking questions from people there, so you might want to come along and watch that.

Otherwise, I hope that was helpful.

Have fun using Numbas. We're always available through the Numbas email address or the users group.