Numbas advanced training 2020-07-22

Welcome to this advanced training session for Numbas.

I don't really have a plan for this session. We've got two hours in which basically you can ask me how to do stuff.

The way things are going to run, uh, I just had my my zoom rules up there on the screen.

I'll share them again very quickly and talk through them.

It's an informal session as I've said, so don't worry about people intruding on your zoom call.

I've already seen a couple of kids joining in.

But please mute yourself when you're not speaking so that we don't get a lot of interference.

You can use the raise hand button before you ask a question, but since normally Chris joins in with these things and he can tell me when stuff's happening but he isn't here.

Now if you click on participants, now I get all the host controls here, but you'll get less...

Anyway, press one of these buttons and I might see it.

I think what we'll do today maybe is just unmute yourself and shout at me.

And please use the text chat.

I see a couple of people have already started doing that.

So good morning to everybody.

So that's what we're going to do.

I've got a few things that I could talk about, otherwise I'm happy to take suggestions.

In the chat, maybe say where you're from, what you're using Numbas for, while I get to...

where's my next tab? I've prepared some ideas here for things that we could talk about.

What I'd like you to do is use zoom's annotations, so you should get a button looking something like annotate.

It might be in a different position on your screen.

And then pick a stamp.

I'll go with a nifty little heart here and you can just click on the things that you would like me to talk about, and I think that's a way that we can vote for what I'll talk about first.

So I'll leave that up for awhile and clear my drawings.

Here we go.

What I'm going to do, in fact...

can I turn off my zoom interface so that the screen share works properly?

Just hold on a second, sorry.

Share screen shows in windows.

Now there we go.

Share screen again. Right there we are.

Go for it. And I'll keep the chat open.

So we have people from Swansea. I've had a lot of chat from people in Swansea lately.

Manchester Met, hello to you Jonathan.

University of Newcastle in Australia.

So we're spanning the globe with Newcastles.

Sydney using it for first year, cool.

Western Sydney. Glasgow, flipped learning, varied level maths class of geomatics students Geomatic.

I think I know what that is. But you might have to put in the chat exactly what this.

Kaplan pathways, is that the Nottingham people?

Ruth, are you from Nottingham?

And got a couple of Pretoria people in. We had a good project making up some material for engineers in Pretoria a few years ago.

And that was the longest and shortest flight -- shortest trip outside the UK I've ever had.

Quantitative economics.

Annotations aren't sticking? Well, it looks like there is some going down.

It looks like there's a strong need to look at explore mode.

Sam from Leeds using Numbas for stats modules.

Interested in generating randomized expressions, simplifying all that, diagnostic testing.

Maybe we can't see the annotations says Julio, so maybe you can only see your own?

I can see lots. So I will make it...

Oh really, you can't even see your own.

Right? To annotate this, maybe somebody who's not a host could unmute and explain where the button is.

For me it's at the top of the screen.

[Unknown person] Yes, it's at the top, when you click on view options and then it gives you an annotate option.

[Christian] Right. And then once you got on to annotate, you can click on a stamp, but I mean we've got a fair few votes and I think I'm going to start with explore mode.

And then there's a few more votes for custom marking.

I'll give you a couple more minutes - a while - to vote for things.

Somebody said that their queries might not be advanced enough.

I mean, advanced is just anything beyond you've never used Numbas before.

And there will be people asking for stuff that can be very complicated.

Let's see, Adele is asking something about JavaScript, so I'll try and remember all of these things, so I think maybe what we'll do is get straight into talking about explore mode first.

So that's the thing I'm most keen on at the moment.

This is very new in Numbas. So what will I do?

I can't keep these annotations, can I?

That's sad.

Clear everything.

Right, So what am I going to do? I'm gonna move, going to stop annotating.

By the way, this session is being recorded, it said on the sign up form that it would be recorded, so I hope that's OK with everybody.

If it's not, if anything happens during the session that you want cut out, email me afterwards, and I will do that.

OK, so what I've done is I've made a project on the Numbas editor called advanced training.

I may publish this once we've finished, but I was going to do stuff through screen share, because doing stuff on your own is hard to keep track of over zoom, so let's make an explore mode question.

I think what I'll do is I'll just describe how it works.

And the kind of things that I thought it might be used for.

Just keep moving zoom bits around.

There we go.

Zoom is awkward to use.

Right! Just catching up with chat, Claire says is this available via an iPad.

You mean zoom?

There is a zoom app. Maybe you could add some more information if it hasn't answered that question.

Right.

Oh, the annotate function.

I believe so. I don't know how. We've moved on. OK.

Right, so explore mode. The idea with this is to give the student more choice about how they move through a question or to make a question adapt to what a student does.

So classically, in a Numbas question you would have a statement and then a list of parts that are all generated when the question is.

And those would add up together to give you a total score for the question.

We had steps where you could make up some extra parts that the student can choose to reveal and give some marks for that and maybe add a penalty onto the marks for the original part.

But you couldn't offer a choice of steps, you couldn't reveal just one step at a time.

So the start of explore mode was can we do something better with steps?

But then also for a long time, for years and years, I've wanted to do something more like a choose your own adventure book.

If you've ever done any kind of interactive fiction on the computer, where you maybe get description of a place where you can do certain things and you can move around between places.

Or if you played a choose your own adventure, fighting fantasy book when you were younger.

You do a thing that says if you want to go North, turn to page 36.

That kind of thing.

I think that would be a really - it's a really good model for interactive exploration of a topic.

To sort of let students choose what they want to do within a set of scenes that you've made up.

That was sort of in my head for a while, and then in 2016 at the EAMS Conference, Matti Harjula, who I think is present, presented his Stateful mode for STACK, which does that sort of thing.

The way Matti came at it was, you want to add in some state information to a question that gets updated as a student enters answers.

So over the last year I've been implementing that and came up with a few use cases, and I'll talk about the basic model.

So the idea is you only get shown one part at a time.

And you can move between parts.

So the first thing you see is the first part.

Let's make an information part and say... Now, what was the intro text for Zork?

I'll just try it.

[inaudible] I didn't quite hear that.

I think whoever that is, maybe you should have your mic muted.

Right. So first part. And then create another part.

I'll just use information only for these.

Really make it like a choose your own adventure. Right.

And so I've got two parts here. I need to give them names for this to really make sense, so I'll go "another part" and call the first one "first part".

Now, when I run this question, I'll only get shown this first part because I need to make up the structure.

It's not the case that you just go from one to the other.

So what I can do is add a next part option. So I'm at first part and I might want to go to another part.

This option is always available.

So now when I run the question I get the first part and I get this option, what do I want to do?

I can go to another part.

So, that's the basic idea.

Let's make this a bit more complicated.

What's a good thing to do? A choice of hints? So let's make a number entry part and try and think up something to do.

Actually, let's not try and make up a question 'cause it always ends up being complicated.

Let's make something completely trivial just to show structure.

This part is going to be called "write number one".

The accepted answer is one.

The prompt is "write the number one".

OK so I'm gonna make that the first part. And these other two are going to be hints now.

So I'll have a vague hint and a better hint.

OK. I'll skip to the next part option there.

There we go.

So what I can do is I can have options for both of those.

Vague hint and better hint.

Right. Let's have a look now.

OK.

So here's my question. Is it possible for Christian to remove the zoom group chat?

Now I thought I'd hidden the window, is it still showing it for you?

The answer is no.

In that case, intermittent.

The wonders of Zoom. It never used to show up and now it does.

On last week's recording it did. So it came up every now and then.

I turned off the option. I don't know what I can do.

Need to keep an eye on it in case something comes up. Right so.

Back to my question, here I've got this number.

I've got a choice of hints.

So I could pick a vague hint and it says there's nothing more to do because I haven't ticked the option to send you back, or I could go back and pick another hint.

Now the question is, how does marking work here? This question says it's not marked, which is a surprise because I thought there was a mark available for this.

And the reason for that is marking in this mode is really complicated.

So because you might have several ways to go through the question, or you might be able to come back to the same part more than once, you can't just add up the scores from one of the parts that you visited.

It might, if you can go back to a part again, you might be able to get infinite points, or if there are several paths through, it won't be easy to see, just from the total, what the student's actually done.

So scores go into objectives, which is where parts give you marks and those add up.

Then penalties which get applied when you choose a particular option.

So I think objective here, we should have an objective "write number one".

Limit for that.

Let's make it 10 marks. I will have a couple of penalties.

Actually no, we'll have one penalty.

Hints. Maximum two marks.

Right.

So Dinh Tranh says if the student uses the back button in the browser, does it cause issues?

I think you'll get...

Now it's not available here because it's in a new tab. I think if you had come to a test and the back button was enabled, it would just go back to whatever that previous page was in an exam.

It will warn you that there's an exam in progress and it will say, do you really want to leave?

I don't think overriding the back button would be the right thing to do as it gets very complicated when you do that.

Right.

But that's a good point. A good user interface point.

OK, so these points.

I've got objectives, so I'm going to go back to my write the number one and set the marking settings.

This counts towards the objective write the number one and you get 10 marks for doing it.

And then these next part options.

For the vague hint, I'll take one point away.

And for the better hint, I will take two marks away.

Chris says, has anyone written an RPG in Numbas?

Not yet. It's only about a month old, this feature.

I would love it if someone did though.

I'm going to get on to do more complicated stuff.

Right. I probably want to spend about half an hour on explore mode, so we'll see how far we get.

Right, so. Here's my thing now, so I have a vague hint and I get that penalty.

I still haven't fixed this suggesting going back.

So that had minus one, but if I write number one here I've got my 10 marks, but I've had that penalty taken away so that covers the offering a choice of hints use case.

The next thing we want to offer a choice of paths through a question.

I think what I'll do. Let's go to my demo question.

Something already exists. I'll find that exam.

There we are. Right, so I've got a whole load of explore mode questions here.

Factorizing a number, is that?

I want a choice of paths. Um...

Maybe find the roots? Actually, I'm not sure if I ended up making a good example of that.

They all tend to be fairly linear.

What I realized writing these questions is it gets really complicated if you start allowing a lot of different paths.

I think what we'll end up using this for is things like stats questions where.

Sorry, someone is unmuted.

Oh yeah, Julia says she made one. Yeah, we will go and have a look at Julie's question if that's alright.

Put a link in the chat there, I'll click on that in a bit.

Yeah, stats questions where you might want to offer the student a choice of tests to perform.

The first one I tried to make up was a calculus question where you choose like do you apply the chain rule or product rule and it tries to assess you on that.

We realized that it wasn't really worth the effort.

Right, so I'll just work through this question.

So here I have to find the roots of this quadratic and at each stage I'm offered a hint.

Where I get to - first of all it's offering a hint, find the discriminant and then use whether it's zero or not to decide how many roots there are.

So then I can see a hint, and there's a formula for the discriminant.

So I could go back and calculate it and all that What do you think of those?

I think that has one distinct root.

There we go. Get some points there for doing that in my head.

So now that I've answered this question, this other option has become available.

If we have a look in the editor.

No I won't go back again.

So anyway there was an option - you can give a condition for when an option is available.

And this one said when it's answered or correct.

So then I get taken to this part where I have to write the root.

I forgot to write a prompt for it.

Anyway. The root was minus four and I get that point.

OK, so that sort of offered a couple of choices and I can see at this navigation thing up here, I can go back to the bits I've seen.

I'm not sure about this interface.

I don't know if that's the right thing to do.

If you have a really long question, it ends up getting very complicated.

But in this one you can navigate around by clicking the "Go back to the previous part" and just sort of avoid this thing.

Couple of questions from Dmitri.

Is it possible to provide a penalty for wrong answers on the main part?

No. And maybe it should be.

A couple of people have asked for negative marking for certain things, and we'll see.

If you can come up with a good use case for it, I'll have a look at whether that should go in.

And then number 2. Is it possible to penalize the usage of advice?

So sort of show the advice but get a penalty.

I think you would do that with an option to go to an information part that would apply a penalty.

The advice is shown once the questions over: either you've given up or you've finished the exam, you're reviewing.

Alright, I'll try and work through.

Matti asks does a student need to always choose to change the scene or can the act of answering trigger that change?

The student has to choose.

So you can make an option available but I think If you are answering and it immediately moved, you wouldn't see the feedback that you got for that part.

So I think it's a... Let the student choose when they want to move.

So I guess you're imagining sort of making something unavailable after the student's submitted an answer.

I might have an example of locking a part after leaving it.

So you might ask the student to enter an answer and then the next bit will contain a big giveaway for what the answer for that should have been.

Sometimes you might have to set a problem where you do a calculation and then the next stage says some more information that would have helped you get the right answer in the previous bit.

So what you can do is once I've locked this part -- once I've answered this part, I can go to here and then that answer's locked in so I can't go back and change it again.

So.

Tell me what you were thinking of. If you think that it should be possible to move the student automatically, Ione is asking could you have alternative answers with partial marks in the main part?

Yep, so alternative answers is another new thing, and it's sort of separate to explore mode so you can have alternative answers and you can even it an option available based on which alternative was used.

Uh, there we go. So here's an example of that.

I should have given myself a prompt for whatever to do here.

I'm going to guess it's follow the order of operations wrong, so do this left to right instead of grouping by the brackets.

So I'll try 11 times 5, 55, minus 5 gives me 50.

So because I've done that, it's showing me this, get a hint.

Did you evaluate the expression from left to right? And then do that.

Right.

Wow, you are doing really well with questions. Chris Jobling says navigation, maybe forward and back buttons.

It's the forward button that's the hard part. When you've got more than one choice, what does forward mean?

And there is a back button. Would it be fair to say the main benefit of an explore mode question is that it allows the student to choose whether or not to view a hint before returning to the question and applies a penalty?

That's one use of it.

Another really good use of it is exploration of a topic.

This question's to do with designing an experiment, I think.

It's not just about showing hints, it's to do with adapting to what the student has said and what you do in this experiment.

Yeah, so Julie says a huge benefit is putting student answers into variables to use later on, which is what this question does.

It tries to set up an experiment about deciding if a coin is fair.

So you'd probably all come up with the same methodology for this: flip it a load of times and see if it comes up more heads or more tails significantly.

So this sort of asks you for a few things to set up the experiment.

It says how many times will you flip your coin?

Enter a small number so that I can actually answer this thing.

The next stage is to define what biased means.

So that's not something that has an objective correct definition.

So I might say... It gives me a prompt, actually. How many heads would make you think the coin is biased?

If I'd done this experiment in real life and allowed students to make up their own definitions, I might have a few more prompts, a few more ideas for what biased means.

Maybe let them pick that and they can choose the methodology of the experiment is based on that.

But here I'd say maybe 4 heads would make me think this thing is suspicious, and if I hadn't answered that I can have a hint.

Sort of a prompt to come up with what this number should be, to think about an unbiased coin.

What would that do? So then maybe more than that would be biased.

I'll just carry on going through. So then I have to enter some data, and each of these steps, those answers are getting saved in question variables.

When you move on to a next part, you can replace the value of a question variable with something from the student's answer or with a static value, or increment a counter.

So that's giving you this state thing.

So then let's say I got 5 heads.

I'm going to whizz through this question now, so then - actually it doesn't really matter.

That's the point of that.

Daniel said, can we see the details of the part where you double the number?

Was that the lock a part after leaving it, you want to see that in the editor.

Right, so we have this thing.

The first part was - I'll just hide the chat for a second.

It says write a number and the next part options are all the parts here.

So it says it's available when an answer is submitted, there's no right answer to this thing.

It just says write any number.

So as soon as the student has entered a valid number, this part - this option - will show up and it says when you choose this part, set the value of x to -- Could you make sure you're muted please, Tatiana?

There we go.

Right.

So that's that part and then this other one, if you - No.

When unanswered or incorrect. So yeah, so if you haven't entered an answer it offers this option to get some inspiration.

I don't know what this part says. Oh yes, it just suggests a random number.

So then, once you're in the double the number part, it will show you what your number was and says what is 2 X.

I think the point for this question was that once you've picked a number, it gets locked, so you're fixed in.

So if it turned out that you've chosen the number that was particularly poor for whatever purpose, it wouldn't let you go back and change your mind.

I think for when a student is just accessing something at will, you should let them do whatever they like, but there might be reasons you want to do this.

Right, uh? Yeah, so I think I dealt with a few of these things.

Can branches only be made visible if we detect a certain sort of mistake?

Yeah, I showed that.

OK.

Can it b used adaptively for release of subsequent question parts of summative tests, so that students don't see too much?

Absolutely, yeah.

So a thing you can really do is, we came across this a lot writing this year's exams that a lecturer would write a question where part a asked them to do something and then part b really gives away the game in terms of what's -- I don't know, It was some number theory stuff where the number has to be prime, effectively, that the student was supposed to write in the first part.

So if they can't see that text until they've answered that question, then that's a benefit.

Right.

Yeah, so then I can mute people. Yeah, I think what I'll do is I'll turn off allowing people to unmute themselves.

Mute all and then don't allow people to unmute themselves, there we go.

So if you really want to say something, I will unmute you in a bit.

Could you show us how to write an answer to a variable?

Sure. I'll get back to my question I was writing earlier.

Here we go.

I think.

I'll do this and then, unless you have more questions to do with explore mode, we'll move onto another thing in a bit, yeah?

So I'm going to add a variable.

And you have to give a stock value, but it might never be used.

Maybe there should be an option in the editor too.

Say this variable's gonna get student input so it doesn't even ask you for a value.

Interface thing.

So here we go.

Let's just write a number.

And I'll accept anything, so I'll go from minus Infinity to Infinity.

OK.

And then maybe I'll add a part option to show that number.

There we go.

We'll have... Did I call it a?

I think I called it a. Right.

So we'll add a next part option. Tell me if I'm going too fast with these things.

It's easy to whizz around the interface when you know what you're doing.

So an option to go to this "Show the number" part and add a variable replacement.

Replace the variable a with the student's answer.

I could replace it with the credit awarded, so a number between zero and one, to do with how correct the answer was, or a JME expression, and there I get access to all of the marking notes for that part.

If you've had a look at custom marking algorithms, you'll see that when a park gets marked, it produces loads of these notes that have values to do with the student's answer, so you could pick out one of those, and that's how the one that detects -- so there's one called something like "alternative answer used" which you could use to do some logic about what the student's done.

I have an aim to create a question which replicates our old DIAGNOSYS system which gives a big bank of questions to establish what incoming engineering students can do, sort of GCSE and A level maths.

And it had a graph of skills.

So if you answer a particular hard question that relies on a load of other skills that you don't need to ask if the student can do those more simple skills, you could sort of infer that they already can.

So I want to do that in explore mode at some point.

Anyway, so I can just replace this variable a with the student's answer.

And that's it.

It works the same way as adaptive marking, if you've done that: if there was another variable that depended on a that would get re-calculated.

So let's write a number. And click on show the number and it shows me 53.

There we go.

We could have something called b that's two a, I don't know.

And then if I click on show the number, it should be -- it would recalculate that.

So it's really easy to implement this.

OK.

Right, let's check the chat again.

Nothing else. Right. So let's go back to my board, what else did we pick?

We wanted to do something on custom marking and I just mentioned that, so let's do that.

Oh, sorry question from Elizabeth.

What would be a good way of making a question change if you need several related numbers to show up for a particular variant?

Can I just would you mind if I unmuted you Elizabeth or you could say in the text.

I'll click unmute, whichever way you want to do it.

[Elizabeth] Can you hear?

Yep, OK so. The really simple example I'm thinking about, although it probably comes up in other things as well is if you're trying to get people to round to a set number of significant figures or decimal places, you can say alright then I have this particular number.

And to make sure they're actually rounding properly, I need to chop it off at this many significant figures.

But, I need the question to be able to appear various times, so I need to be able to have you know, five or six possibilities.

So they can repeat it and get a different different question, but I need to be able to control the number of significant figures related to the thing.

I don't always want it to be 3, for example, [Christian] Oh right.

[Elizabeth] So I kind of would like a way to be able to say well, for this iteration of the question then I need this number, this number and this number.

And for the next iteration of the question, I'm going to have this main number and then this many significant figures attached.

There's like more complicated examples, but that sort of Illustrates the sort of link sets of numbers between different variants of the question.

OK.

[Christian] I mean, that was such an abstract word that I found it difficult to follow.

I think what you want - you want to maybe set the number of significant figures to ask for as a variable.

So you might want to set the number of significant figures to a random number and then if I set up a -- [Elizabeth] I don't want it to be a random number necessarily, because - [Christian] Well, whatever, it might depend on something else.

Yeah.

[Elizabeth] It'll depend on what the main number is.

[Christian] OK, so... [Elizabeth] I want to make sure they've got the learning thing of of have they actually rounded it up?

It will come back to a bigger number.

It'll either stay the same or it will come out to a bigger number depending on - [Christian] Oh right, so you want to make sure it's a number where it will make a difference.

OK, and they're not just chopping it off.

You're talking about this example where the student gets to pick the number they enter.

[Elizabeth] Yeah, I want to control whether they - make sure they get questions where they both, if they're rounding the number will increase, because it should make sure they don't fully grasp the idea of rounding.

[Christian] There's a really simple way of doing that. So pick a random number is the first part, but you don't know what the last digit is going to be, so if it's less than five then you round down.

So a way of ensuring that it's going to round up is you multiply that by 10 and then add on a number between 5 and 9.

So this will always produce a number that has to round up in the last digit.

And you can - I don't know - If you wanted that to be somewhere else in the number, you could...

There you go, so I've shifted that along a couple of digits and now it's this hundreds digit that is always going to round up.

So does that - [Elizabeth] It kind of helps in this particular use case potentially.

I've copied the screen briefly and I'm going to have a think about it, but it doesn't actually answer the, I mean, equally I do things like spherical trigonometry.

And you can potentially sort of go "alright, I'm only going to do questions where it's all above the equator" or "I'm only going to do questions where it's all below the equator".

But if we want to sort of have a question which can change but control which question comes out, you need to be able to say "for this question, I need these three numbers, but for another version of the question I need these three numbers which I've checked will also be a sensible..

." [Christian] So it's a case where setting up the definition so that you get the right thing might be really tricky.

So instead let's just change the definition of this to be 1 to 1000.

There's a variable testing thing so you can write a condition that you want to be true, so you would for your spherical trig thing or whatever you would say all these three numbers need to have whatever property, so I'm going to say something like mod 10 of n is greater than or equal to 5.

So if we have a look over here.

It's now only - It keeps generating sets of variables until that condition is met.

So here it's chucked out any times that the last digit is less than five.

And this condition can be in terms of any combination of the question variables.

That's always going to be true, because I think a is fixed to one at the moment.

Actually, while I'm here, this test condition button generates as many sets of variables as it can for the time you give it, and will tell you how often that condition is true.

So if you write something that's unsatisfiable it will tell you, or something that takes 10,000 goes, you'll have to increase the number of runs it will try to do.

When a student runs a question, if it doesn't make a set of variables that satisfies the condition within the limit, it just fails and says "Numbas died." OK, so does that get you some of the way?

[Elizabeth] Yeah, that's really interesting thanks.

There's no way of sort of like saying well I've got a matrix of variables and I want row one or something?

[Christian] Yeah.

[Elizabeth] I don't know. [Christian] Yeah, you can write anything in terms of the variables here.

So, if you have like a matrix M, then the first row would be - you might want to be 123 or something, you can write whatever you like.

[Elizabeth] I'm gonna go off mute and I'm gonna let you go.

[Christian] OK!

Right, going back. Yeah I did say I have look at Julia's question because I thought it was really good.

Apart from anything else, it's really pleasing to see someone else use the tool without having to email me a million times about how it works or just find it doesn't work at all, so that's good.

So this question. It's about solving linear systems of equations.

And I get to make up my own one.

I really like this because something that I've often wanted to do with students is that writing a question means that you have to really understand the mechanics of the maths much better than just answering a question.

So for years I wanted to run a class where we get students to write Numbas questions because they have to understand what they're doing a lot better, so this kind of thing gets a lot of the way there.

So what I have to do is come up with a set of simultaneous equations.

I'm gonna try my best to not make one that's trivial.

This is going to be like this many X plus this many Y equals that.

I think that's right, from last time I tried this. And Julia's set a limit of 5 rows because I think more than 5 rows would take forever.

Let's reverse engineer this and say X is going to be one, and Y is going to be 2.

So that's going to be 8 and to make my life easy, I'll say one and one add up to three.

I've just noticed that I've set my interface language to Spanish.

Is that right?

Yeah. And that maybe Julia has.

I don't know.

I know what it is - the last person who clicked test run was running Spanish, that's it.

That's a bug.

Right. So then I've put this system of equations in.

It said that's fine. Then I've got some options about what I want to do.

I can multiply a row by something, I can add a multiple of one row to another, so that sort of skips a couple of steps and I can swap some rows around.

Or I can be at the end. I guess that's going to say, what are the roots?

So I think what I'm going to do is take away two of this from that.

So I can say multiply some rows. It's got a load of gaps?

Actually, that's why there's a limit of five rows, isn't it Julia?

Because you need to fix the number of gaps you could have.

I'm going to keep one lot of row one, two lots of row 2.

And then the next thing is calculate the multiplications and this is going to take me back to a bit where I can choose what step to do.

So this is my matrix now.

And I've chosen to take a multiple of this, so I'm gonna say take away one time row 2 from row 1.

On a user interface thing, these could be dropdowns, because I don't know whether these are zero indexed or one indexed or whatever.

And we can maybe annotate these with one and two first.

There we go.

So that's really nice.

I like that question a lot, and Julia's put the link in the chat.

Catching up, Jonathan's asked about \var.

Yeah, this is something I always cover in the intro session, but that's fine.

The reason I use \var is just to get it drawn in LaTeX. For a plain number on its own, curly braces is fine, but when you've got maths notation going on, you probably want consistency in the fonts.

If I just do curly braces here and the preview button...

Why don't those variables exist?

Because there's an error in the testing condition: unsatisfiable.

There we go.

Right, so you see, the plain text font and the LaTeX font look different.

So there's a consistency thing. But you also might want \var if you have it in the middle of some notation.

Something like that.

Don't use \var too much.

Normally the best thing is to use simplify so that you get - you don't get plus minus next to each other and all that.

There's a page in the documentation about this, rather than going through all this, I will - Substituting variables into display maths already talks about all this.

So I already have something written about this.

Dmitri says is it possible to accept an answer when certain condition holds?

Yes, I'm going to do custom marking algorithms next, so I'll I'll do that.

Apparently I made a "how to accept any symmetric matrix" question.

Yeah.

Search for that. Get a server error, that's not good.

What's going on?

I'm gonna guess it's in "How To" As usual, whenever I give a demo, the list of bugs accumulates.

Marking, I'm going to guess it's in...

"accept any symmetric matrix", there you go, it's the first one.

I see actually, you've all been really good catching up in the chat, telling where to go look in the documentation for that.

Julia, you said you wanted to say some things about your question.

Absolutely, so I'll just ask you to unmute yourself.

[Julia] Well, I just wanted to say that so I've also put in that, depending on the size of the matrix, that's how many prompts you'll see for multiplying the rows and so on.

So if you have two, like you had two rows, you'll only see two multiply rows, if you have three rows, you'll see three, and so on.

[Christian] OK, yeah, that wasn't clear.

[Julia] And then if you put the matrix of - like with three rows, say - and the augmentation with only two rows so it doesn't match, it will give you a warning and won't let you submit.

[Christian] Right, yeah, so it checks for consistency between those two things.

That's really good. Thank you.

Right.

Shift that out of the way again. So, custom marking, Go here...

So this is a matrix entry part and the prompt is "Write a symmetric matrix".

In marking settings I've had to give a correct answer.

Now there is no one correct answer to this so I could - the term in the interface is "expected answer", which still isn't quite right.

So the identity matrix is symmetric, so that's what I've chosen.

Now, the way you check that the matrix is symmetric is with a custom marking algorithm.

Actually, we'll go to the documentation about that.

So the idea with this is, it's a real design goal in Numbas that you should be able to easily set up the really common stuff, so like ask the student to enter a mathematical expression and it's correct if it's equivalent to one that you've written.

That's really common. That's a built in thing in Numbas.

So all that's available through the graphical interface, you get this nice form with a box to type that expression in.

That's all you need to do to set up that part type.

But there is always stuff that you want to do that's not in the built in feature set.

In other systems you have to write code.

10 years ago, 20 years ago, e-assessment systems were largely stuff like "you write a Perl script".

That's the way WeBWorK works.

If you've had a look at STACK, that works differently with those potential response tree things.

I don't think there really is that immediate, "I just want to do this really common task, I just fill in one box" thing.

So the question is how you make that nonstandard marking possible in as simple a way as possible.

And what Numbas has at the moment is these custom marking algorithms, where you write some JME code, sort of like the way you define variables.

So I'll explain what a marking algorithm does. The student writes an answer, and when they press submit, that's when the marking algorithm takes over.

There's a load of stuff to do with alternative answers and adaptive marking and all that, constructing the context in which the marking happens, then you get down to a point where you have access to the student's answer, the settings for the part, and all the question variables.

So what the marking algorithm has to do is, it has to decide if the student's answer is valid.

So if there's a mismatched bracket in an expression, you can't mark it and clearly the student didn't intend to submit that, so you just prevent submission and say it's invalid.

You can give some kind of feedback message about what's gone wrong, if you want to help them.

Once it is valid, you work out how much credit to give to the answer and any feedback messages.

So these things in Numbas used to be built in, there used to be a load of JavaScript that did this for each part type.

But in order to make them customizable so you can write your own, now they're written using JME.

So what's the best way to explain it?

Let's have a look at one.

Here we go. So I can see here, this is the built-in marking algorithm for the matrix part.

What it is, it's a load of - it's a series of notes.

So you get a name, then in brackets you can have a comment about what it means, just for reference for the question author really, and then an expression.

So that can produce a value like this, so the number of rows in the student's answer, or it can produce some feedback.

So we'll find one of those.

Oh here we go.

I said "or", that was inclusive or: it can produce both a value and some feedback.

So here I have a note: are all of the cells in the matrix given to the same precision?

So I've got this big long condition here.

I've worked out the precision for each cell up here.

So it evaluates to true if they are, and if they're not, it gives this feedback note: "Not all cells are given to the same precision." Because this is a built-in thing that's a translated string.

If you are writing a question, you probably just write an actual sentence there, and it also has the value false.

If I keep going along, there's a lot of stuff about invalid cells here, so it gives you a warning and also fail.

When you get to that, the marking just stops entirely.

Right, where are we?

Going to get to the marking script itself.

Every marking algorithm has two notes, one called mark which is used to give the final feedback to the student and one called interpreted answer which is what the part reports as the answer the student gave.

So for adaptive marking and for explore mode where you're replacing variables, it uses this value.

So not just - for this matrix part, it's not just the grid of strings the student wrote, but it's a matrix value.

In the mark note we apply a load of other notes, so first of all there's this "any empty", which I think says it's invalid if there's an empty cell anywhere.

Where are we? There we go.

Assert that there are no empty tiles, otherwise end.

Any invalid, and then apply the precision restriction, all same precision, that's right.

Is it the wrong size? That would I think give it 0 marks.

And then it does some stuff on, if all the cells are correct then it's just correct.

There's just this function that just produces the stock message, "Your answer is correct".

Otherwise if we're doing per cell marking, set the credit to whatever proportion it should be, otherwise mark it incorrect.

And finally apply the precision penalty.

I'll find that.

Wrong precision, where is it?

There we are. If there are any cells given to the wrong precision, then multiply the credit by whatever that partial credit factor is set to.

So the way this works, throughout each of these statements joined together with semi colons, they produce feedback notes.

Those all get made up into a list.

And then to work out the credit for the parts and the feedback messages, it goes down the list adding credit, subtracting credit, multiplying.

The correct function sets the credit to one and then it gets to the very end and that's the final feedback.

So and then this custom one, I'll get onto that in a bit, but let's try entering an answer here.

Here we go. So I've entered the one by one matrix with one in it.

Which is, I think it's symmetric, and I can see my mark note, I just get "your answer is correct".

If it's invalid it will tell me that it's invalid.

And none of the other marking happened.

It might have been better to show this with a stock part first.

So that's not symmetric and I haven't given any feedback here.

But I can see for each of these marking notes, I can have a look at what its value is.

Any empty, no, any invalid, no.

Where's my "is symmetric"? False.

So that's roughly how a custom marking algorithm works.

So to make this mark any symmetric matrix correct, I had to change the mark note, so I still want the any empty and any invalid notes because I don't want to try and work with an invalid answer, but once I know that it is valid the only condition is that it's symmetric, and that's defined to be if the matrix is equal to its transpose.

So there's this function "correct if" which says if this thing is true, then give full credit, otherwise give zero credit.

I could have - if I'd wanted to give some more feedback for when it's not symmetric, I could have done something like, If it's equal, that's correct.

"Your matrix should be equal to its transpose and it isn't." There we go.

There we are, and now I sort of have to change my answer a bit to get this to re run, and there we go.

So that's come out of the symmetric note and that's the feedback student gets.

So there we go.

All right, I haven't looked at the chat in a while.

OK.

What happens in Numbas if student gives a mathematically valued answer in a format Numbas doesn't recognize, for example, "write a number", the student answers pi plus one?" They'll get something saying it's invalid.

So let's show you what that would look like.

Let's go to my training project. I'm going to create a question.

Let's call it marking.

Inside that window.

Right, so what was it? Enter a number between one and 10.

A student might write π. If I write π here, it will get "Your answer is invalid.

You must enter an integer or decimal." You could write a custom marking algorithm to accept that.

You might want to choose a different part type, you might want a match text pattern or you might want a mathematical expression part type.

Actually, that's a good idea for a nice custom marking algorithm, let's check.

Oh no, pi.

So the choice of part type is really about what kind of input method you want.

I haven't talked about custom part types yet, I've got plenty of time, I will.

So I want to change the marking algorithm.

And the built in one has got something to parse the student's expression.

Let's see how much of the existing marking that I want.

Let's copy that.

So the student expression note tries to do some parsing and if it can't be parsed then it fails, so we'll keep that.

Then the rest is all to do with looking at the answer as a mathematical expression, but what we're going to do is check that it's a number.

Something like that.

Now what's it called?

There we go. I'm just going to try that.

Oh, there's a weird error message.

So because I've changed a lot of stuff in Numbas version five, there are now some more cases where errors come up and that should be a sensible error message telling me what I've done wrong.

I don't really know at the moment.

I'm going to guess...

is that right? No, that's bad.

OK, that works.

Oh, the reason that doesn't work is because that note doesn't exist.

Let's just... And that should definitely have given me a better error message than that.

OK, so.

Let's just check this student value note, there's a tonne of these here.

Right, and if I try X, what does it do?

The name x.

OK, so I can get an error.

So maybe I want to catch some errors there.

I'm going to skip that when I'm doing this.

Write a note to get the type of the student value.

There we go. 5.

Integer, yeah, and in fact Numbas has a load of different data types for numbers, to make sure you don't lose any accuracy in different contexts.

So it has integers, it has rationals, it has the built-in floating point numbers that it always used to use, high precision decimals and both the number and the decimal type can be complex, so they can have imaginary parts.

What I want is a function that says, is it equivalent to a number type?

So I'm going to go off to the help.

I think I have a function that can do that for me.

I'll be sad if I don't.

Uh, right.

Here we go. JME function reference.

Sub expressions. Because there's a built in function in JavaScript that does this but apparently not in the JME, no.

OK, I'll have to do it differently.

What I'll do is I'll try and force it to make it to a number type.

There we go.

So that works and that does not.

Can't automatically convert from name to number.

OK. So I think the way I'll get this to work is I'll say try to convert to a number, otherwise fail.

Something like that.

And I've got that problem again.

There we go. What does mark look like at the moment?

Let's just check that.

There we go.

Actually, let's make that incorrect if it's not the right thing and correct if it is.

And get rid of that line.

I realize at the moment I've just been talking to myself for a while.

I will check the chat in a second.

It still says incorrect. That's not good. Have I misunderstood how the try works?

Try.

Oh yeah, yeah, yeah. The second argument is an error message, which I don't really want, which is why I forgot about it.

There we go. That's not good still.

I should've maybe been less adventurous, I assumed this would work.

Yeah, so it...

Man..

right? There we go, try that.

Yeah, so because it's a number, it didn't trigger that thing and we'll just say it was correct if we get to that point.

So what I was doing was I was saying try to convert it to a number data type.

If that fails, then say that the student's answer is incorrect and end the marking there.

So anything that gets applied afterwards just gets thrown away.

If it did work, then will get into this next line where we mark it correct.

So there that's marked correct, one's right, x is not right.

There we go, it worked. So I can write something like pi.

So really that asking about can you enter pi, I can answer anything here, couldn't I?

So 1 + 2 something like that and that evaluates to number.

So that's the kind of thing you need a custom marking algorithm for, where you want to change what is accepted.

OK, uh, right, I'll try...

Hopefully that's helped you, who knows?

Is it possible to pass variables between the algorithms of parts?

Ruth, could you just explain what you mean by that?

I can unmute you. [Ruth] Yeah, I'm not sure I explained that very well.

So if there was, if you got something, an answer to part a that might be right or wrong, just even the variable for it, could you pass that through to the custom marking for Part B?

So they were interacting with each other?

[Christian] Yeah, that's what adaptive marking is about.

If you're not in explore mode, in explore mode then I showed how you do those variable replacements.

Not in explore mode...

Let's hide zoom again.

So, here's my number entry something or other.

Let's add another part, it doesn't really matter.

I'm just going to pick at random. So I go in adaptive marking, I need a variable for this to go in.

What will come out there?

Right, so I'm going to replace with the student's answer to part a, and I thought I could pick a note from the market algorithm, but that's fine because I can go over here and just change what the interpreted answer is.

So I can say something like Is it an integer?

So I think probably that works best in explore mode because you might want to pick out a few different things from the part.

Maybe this is a feature request that adaptive marking should - the bit that you use should be a drop down here for which bit you use.

So thanks for letting me know.

[Ruth] I think it's that my coding isn't very good, so when it's not a drop down I get a bit lost, but that does make sense.

[Christian] Yeah, that's a good point - I try to, whenever there's a point where you could put some code in, I try to think what are some good dropdown common options you could have.

So over in the explore mode, there were options for the student's answer, whether they submitted it or not, things like that.

[Ruth] Is it possible for - so this is within a question, so different parts are talking within a question - is it possible to have different questions talking within an exam?

[Christian] No, the model is that questions are completely standalone, so that we can use a question in more than one exam.

And the advice we give to people when they're writing questions is if you've got several parts in a question they should all be linked in some way.

If you can answer Part B without having ever seen part A, those are really different questions.

So the question is a set of stuff that all needs to be together.

[Ruth] Thank you. [Christian] Which might not much how you like to present things, but that's the only way of keeping things sane in Numbas, I think.

So. Yeah, hope I've answered that for you and I'll move on.

Mauricio asks, is it possible to pass GeoGebra values to Numbas to use it as a calculation engine?

So, GeoGebra does give you access to the values of variables in it, which - I've had demos in the past that have said stuff like move this point to a particular place and then the part will get marked based on whether you did that.

GeoGebra used to have a really good tool for checking the student had done a certain construction.

That seems to have disappeared from the web editor now.

So you can do that.

I won't do it now because it involves a fair bit of scripting and I can't remember how to do it.

Certainly the GeoGebra extension that exists for Numbas can do the other direction.

You can give it a list of question variables, and stick that into GeoGebra.

So it's...

Yeah, maybe I should add in the ability to get stuff out.

The problem is with when this happens.

If you wanted to use those to generate question variables, you'd have to wait until GeoGebra has loaded before you can get the values out, which Numbas doesn't support at the moment, it doesn't have the ability to wait for a certain thing to happen, it just sort of expects that it can generate all the variables in one go.

And I'm very, very wary of adding dependencies for other things, particularly for stuff like doing calculations or computer algebra stuff.

I'd rather have that implemented in Numbas.

Julia worked out what I was doing wrong.

Can we still use JavaScript, says Daniel.

Yeah you can. What you need JavaScript marking algorithms to return, what to do, has changed and it's not very well documented.

I really, really encourage you to use the custom marking algorithms in JME because there's so much stuff that you'll end up having just replicated.

You can write a custom JME function over in the extensions tab.

If there's something you want to do that you need to go off to JavaScript to do, write a function that does it and then call it from the marking algorithm.

I think that's the right thing to do.

Yeah, every time I write a custom marking algorithm I have to look up documentation.

Will this test be available for us to look at?

Yeah, I'll publish these things.

Can marking notes have comments in? Yep, let's go...

where was it? The way it works, it's whenever you got a blank line that splits off a note, but I can still write comments in, that sort of thing.

Yeah, so that's a good idea to explain how these things work and I really should go back and stick lots more comments in the built in marking algorithm so you can see what they do.

I'm not very happy with this interface, by the way.

I'd rather have something like the variable definition one, where you get to see - where there's a lot more opportunities to document and see how things work.

In the custom part type editor, it's a lot more like that.

Jonathan, you said, can you ask a question? I'll unmute you just in case that question is still something you want to ask?

I haven't already...

[Jonathan] Hi. Thanks a lot for doing this Christian, and apologies if this is a somewhat rudimentary question.

I was - I know a lot of you teach mathematics. I'm doing the sort of mathematical side of an economics program for undergraduates, so the maths involved is not that complicated to test, compared to some of the stuff you do, which gives me hope that you might be able to help me with this.

With our second years we're doing partial differentiation and optimization and that sort of thing and I found that every time I write a question I need to go into the sort of solution section and obviously every time it's going to generate new variables, new coefficients which might be called a,b,c and new variables also for the indices.

Is this sort of custom marking thing, is this somewhere where I might find almost a predefined function that essentially tells instead of me having to type it out and work every question out hand by hand, as in subbing the variable in front and then dropping the power by one, is there a sort of predefined function that essentially says whatever the hell that the expression that's generated is, differentiate it with respect to whichever variable.

Do you know what I mean, would that be this, or... ? [Christian] Yeah, down here where I was writing my answer, the line above is the expected answer.

So if that had had variables subbed into it, it would show those.

So I'll just hide all this stuff again.

Make a nice example.

[Jonathan] I hope I've explained the question clearly.

[Christian] Am I on the right track? Before I -- [Jonathan] Well, I I think so.

I just mean normally the only sort of answer I've been doing when it when it comes to inserting the parts, I've been doing the expression part rather than a number entry because here's a function, let's differentiate with respect to X [Christian] And you know how to do that, because you've randomised -- [Jonathan] Yeah, created variables in the variable section that generate different coefficients in front of the x and the y.

And then yeah, that's exactly what I do, so I do that when I'm typing in the solution.

Which is is fine, but if the structure of the original expression is somewhat different, there might not be a b coefficient or a c and it kind of then it depends, do you know what I mean?

If this is the answer, then I'll keep doing it this way, that's fine, but I just thought I'd ask.

[Christian] I'll show you this, so I'm going to make up a couple of variables.

[Jonathan] And I also notice that it's sometimes a bit funny with regards to requiring an asterisk symbol.

[Christian] Ooh, that's a good point.

So let's just make sure we've got these variables, a and b.

So a is going to be - I'll pick a random number from a range one to five.

Actually no, that's a power so I need that not to be 0.

And b is going to be zero or one, just to show you that I want the zero case to come up.

So here's my expression. I'll write out what it was.

It was something like x to the a plus b x squared.

I think that's what I had in mind. So differentiate that.

So the prompt the student will get looks like that.

And actually, you can see.

I'll put some text in to show what a and b are so we can see what's going on.

Right.

So a is 3, so we get x cubed, and b is one, so it's one x squared and the simplification has omitted that one because you wouldn't write that by convention.

Let's make - I need be to sometimes be two.

Let's go and change it. There we go.

Right, let's try again.

So we'll try when it's two, I'd love to come up soon, There we go.

So you get plus 2 X squared.

Now I've written the correct answer - yeah, the correct answer for this is a times X to the minus 1 plus 2 B times X.

OK. And by default it applies a load of simplification rules.

So if B is 0, this thing just gets omitted.

And in the marking algorithm tab, here we go, I've got - I can switch to see the variable values.

A is 3 and B is 2, so -- [Jonathan] It just pulls that from what you've written in the marking settings, isn't it?

[Christian] Yeah.

So does that answer your question?

[Jonathan] Well, I guess I've done that much.

I just wondered if I guess I wondered - that works for that structure of questions, but what if you have more than X's and Y's and you perhaps have Z's or Q's or L's or whatevers.

You then need to sort of write down the new general format solution in marking settings every time, I guess it's fine if that's the case, I just wasn't sure if there was almost pre-designed functions built in that involve programming that's beyond my capabilities, where I could just [Christian] So you're asking specifically about differentiating?

Right, I hadn't -- [Jonathan] Yes, I'm just - it's the only case I've come across to be honest, we know as humans it's the same procedure every single time, which makes me think there might be a function that already exists.

[Christian] Yeah, differentiation is easy, integration is harder.

Other systems have access to a proper computer algebra system.

These systems have millions of rules for spotting how to do integrals.

So Numbas never used to have... [inaudible] Numbas never used to have these features, but I did write a differentiation function recently.

So I think I can do, is it X, with respect to X?

I've already got a variable called x, let's get rid of that.

There we go. So differentiate that expression with respect to X and it will do it.

[Jonathan] That was exactly what I meant.

[Christian] OK, good, glad we got there.

[Jonathan] And can you then pull that through -- [Christian] Don't know how to differentiate - this hasn't really been tested, which is why I haven't really shown it, and there's a pretty glaring edge case.

Sorry. What were you saying? [Jonathan] I'm just wondering now, so when it goes to you putting into the parts section what the solution should be.

Can you literally right down 'expr'?

[Christian] Yeah.

[Jonathan] OK, but that needs to be defined in the variables section for every question?

[Christian] Yeah, well, I can type something like that.

There's a pretty gnomic error message.

Not happy about that. Yeah, I hope that works.

It should work. I think that works.

Right. OK, so does that answer your question?

[Jonathan] Yeah, sorry, it's good to know that that's there.

[Christian] It's there, but it looks like it needs some more work [Jonathan] I'll triple check it before I use it in a summative piece of work.

Thanks for your help, thank you. Right...

Dmitri says can I assign a student's answer to a variable in non explore mode?

Yeah, using adaptive marking.

So for Part B you would add an adaptive marking thing to replace a variable.

So just for that part the value of the variable would be replaced and you can - the idea with that is for error carried forward marking.

So you might want to not do the replacement, but you can pick always replace it.

The chemists at Newcastle do a lot of digital labs where students enter their data in the first part and then that's used in subsequent parts and they have to set up this adaptive marking thing for each one.

So explore mode will make that a lot easier for them. JSXGraph, we haven't really got onto JSXGraph.

I also want to show you custom part types shortly.

Yeah, the point about being able to try stuff out in the variables interface.

I think just typing something and seeing what happens is often tonnes quicker than going off to...

...

its value, absolutely. That's half the point of it. So I think we can show this diff function here.

I've got something saying my internet connection is unstable.

Has my sound just broken up.

Can somebody say yes or no? Uh, I'll keep talking.

I'm back now, OK, I don't know what you missed.

I was just talking about David saying is it possible to use a function?

Yeah, so I'm inside of one. I don't know what kind of function you mean.

If you define a custom function over in the extensions, you can use that.

David says can I show you an example?

Sure. Do you want to do screen share?

Let's let's turn mine off and I'll find David. Can I unmute him from here?

Ask to unmute.

David, I think you're unmuted now, are you trying to find the thing you want to show?

I can't hear you, if you...

Maybe we'll come back.

You say in the chat what it was you wanted to show.

David says, how do I share screen?

We can't hear you, David. He's given a link, alright.

So I'll do that, I'll share screen. Right now I need to turn on the chat again, there we go.

[inaudible] Right, uhm, is this going to be about JSXGraph?

So what's the question?

Alright, looking at variables.

Bye Danielle, yes, the Australian contingent are going to have to go.

Yeah, it would be much easier if we could hear you David, maybe we'll come back to you if there's time.

Or you could email afterwards if I don't get back to you.

Right.

What was I going to do? Custom part types. Oh, bye Jonathan, as well.

Right. So we've got custom marking algorithms and you very quickly start reusing them, which is good.

But if you then need to make a change to it, you have to go back through every part that uses it and make that change.

So custom part types are a way of making something reusable and providing a better interface for editing.

So the way you get to those is you go to your profile and then there's these options down here.

I haven't talked about extensions today, I don't think we've got time to.

You go on part types.

I'll create a new one.

I don't know what it's going to do yet.

So the idea is both to make something reusable and also document it.

So there's a lot of fields for, you can give a description, you can give some documentation, which I use - for part types that correspond to extensions like the quantities one, there's a link to the documentation for that.

It can require an extension.

Again, I've got a load of unreleased extensions available here, but I can for example - the quantities part type needs the quantities extension.

And the way it works, the setup is you have a lot of settings which appear in the editor like the normal part types, to configure how it works.

And then you choose which input method the student's going to use.

So this is all being abstracted out. Like I used the maths expression part type earlier to take a number, but it would be nice if I had a - as a question author, if the editing interface was adapted for this particular use.

So I can choose which input method I want to use, and those have their own options.

So if it's radio buttons, I can have a lot of choices.

And each of these you can untick static to say it's going to be in terms of some settings.

OK, and then once you've set up the input box, you write the marking algorithm, and this is like the variable generation thing: you have these two marks that are needed and you can add others.

And in each you can write a description to explain how they're generated.

So what I'll do is, I'll go back to an existing one.

This quantities part type. So this has a few settings, quite a few in fact.

So you give a correct answer.

There's an option for what kind of hint you want to give, so that's next to the input box.

You might have noticed for when you've got a precision restriction it automatically says give your answer to 2 decimal places.

So a part type can say what that text is going to be.

And then some ticky box options, like do you allow a unitless answer?

Some more drop downs. I'll keep going.

And for each option you can give a hint for question authors that appears in the editing interface to make it more obvious what to do.

Penalties is, this is a percentage value, so the question author gets given a slider.

And this is going to be an expression that evaluates to the margin of error.

So then configuring the input, you have to give a value for the expected answer, when expected answers are revealed.

And an expression which gives the hint I was just talking about.

Then we've got a load of notes.

So this is the mark note. And there's a load of stuff.

I'll go back to start. So the first one is valid number.

Where is that? Down here. Which depends on student number, which depends on other stuff, so you can sort of work backwards through these links to work out where the parsing starts.

So I think that's a bit better interface for writing these things.

OK, so Dmitri you want to unmute.

And David's sorted his sounds out so I might come back to you, David.

Right.

I think Dmitri, let me just remind myself what your question was a while ago.

It wasn't the seed one. Oh right, use the student's answer to a variable in non explore mode.

OK, yeah, sure I'll unmute you.

Where are you? [Dmitri] OK, thank you, so my question was the following.

Suppose the answer on the problem is a square root of b and I don't want to ask the student to write the LaTeX expression of something like this.

So far I did like I put a gap fill and they put two gaps, one for A, one for B.

But of course it may be like 2 square root of 8 or 4 square root of 2 also answered as correct.

Is it possible in the marking algorithm to assign variables to both gaps and check that the product of squares of two numbers is 32 or so?

[Christian] Oh right, yeah, this is really combining both bits of a gap fill, which we can do.

Yeah.

Let's have a go. I won't make a - I might make that example, let's give this a try.

So I'll make a gap fill and I think I won't go with mathematical expression even though I could.

No, fine, we will, so you can enter the square root.

So there are going to be two mathematical expression parts and what we want is something whose square is 12, maybe.

I think that the square root of 12 is 2 root 3, yeah?

Right.

I'll go to gaps. I'm not going to do anything with the prompts, maybe I'll say times those.

Right.

So the way a gap fill marking algorithm works is that it tries marking each of the gaps, just to get the student's answers to those gaps.

And then the standard algorithm will go through each gap, submit it, like run the submit method, so apply all the alternative marking, alternative answers stuff and then collect all the feedback and just sort of make a big list of that.

And that's the feedback for the parent part.

We're not going to do that because what we want to do is combine those two answers together.

So...

Actually, I'm going to make this note called student numbers.

And what it's going to do is evaluate both of the student's answers.

We'd have to add in a load of error checking to check that they've written something that does produce a number, which we did in the other example.

That's why I hesitated about making these a maths expression thing.

Let's assume we do that. I'm not going to type anything that's not a number.

And this answers -- it gives me a list of the student's answers to those gaps.

Why don't I have the option to answer these things here?

I'm going to reload this page. This may be another bug.

Yeah, there we go.

OK, I'm going to have to watch this video, just to make note of all the bugs that I spotted.

So once I've answered these things, this student numbers note should...

Oh hey, what are the answers?

Hmm.

Alright, let's try type of answers.

Just see what that is. And it's gone again.

Right, not a good time for these bugs to be appearing.

Clearly there's something going on with it remaking stuff.

OK.

For some reason those are coming out as integers.

So maybe -- oh fine, whatever.

Let's see if I can do that.

And that's again broken and... There.

Something like that. Who knows why that's come up here?

I think that should work.

I thought that should work. That's correct, but the interface is wrong.

I'm going to run this question.

Right.

And I've got all my other problems here, haven't I?

I'll just fix that so that it comes up with the right things.

I should have made a new question.

Right, here we go. So. So I should be able to type two times root 3.

And actually, I wasn't checking that the product was -- that the square was 12, I was checking that the product was twelve, wasn't I?

So I'll go back. I will get there.

Right.

So check the product squared is 12. Two times square root of 3.

That's correct.

And because these individual bits can't be correct on their own, I need to turn off the ticks and the feedback state for those gaps, which I will do.

Go.

Do not show score feedback icon.

Do not show score feedback icon. So now that means if I write something else.

Square root of 12 times 1, which would it would have shown up incorrect next to each of those boxes because they didn't match what I was expecting, I haven't done anything with the marking algorithms for the individual gaps.

But now, this part is correct.

So does that answer your question, Dmitri?

I'll unmute you again. [Dmitri] Yes, yes.

Thank you very much. And sorry I had also one question.

It was missed because you scrolled down then up. Do you have a seed function for randomness?

If I want to fix the randomness? [Christian] No, I think you would, I guess just fix your random values.

The interface, it does - when you're editing, you can lock a certain value, by clicking that lock button.

So if I click regenerate here, the Five is going to stay the same.

So when you're editing, you can do that and it shows you which variables are random.

If I lock this then it locks everything that it depends on as well.

So I can fix the value of things while editing, but I think...

I guess maybe if you've got a question, it's randomized and you want to make it static as it would be delivered to students, it might be easier to say, fix the random seed.

But that's not possible at the moment.

And I don't think I would consider it a priority.

Right.

We've got ten minutes left.

I don't want to overrun.

Claire asks, JME code, is it a derivative of some other language?

It's made up. It's specific to Numbas but the syntax is very similar to other things.

At Newcastle we used to use a commercial system called i-assess, but that was developed from the lineage of systems beginning with CALM, which was developed at Heriot- Watt in the 80s, which used this kind of syntax.

I think it was never formally specified.

So yeah, the JME system is completely specific to Numbas, and all of the functions and operators and the syntax is documented in the Numbas documentation.

So if it's not there, it doesn't exist.

I hope that helps.

Yeah.

OK, so we've got 10 minutes, and, uh, I guess David we'll try and deal with your question so I can unmute you.

[David] OK, how is that?

[Christian] Yes, I can hear you! [David] You might be able to see me now as well.

Yeah.

Go back to my question. It's probably easiest for me to run through it.

If you just do a test run and I'll show you what it does.

Basically, when you've got electronic components, you've got three components up at the top of that screen, and what I do is I generate random component values for a resistor, an inductor and capacitor, and there are -- there's a fixed set of values that these components have, and what I find I'm doing is, I'm generating values for them every single time.

What I'd like to be able to do - if you could go to the variables - is basically saying can I have a value for a component?

That component value one you're looking at the front there.

That is the actual base digits that are used to generate all electronic component values.

I then multiply it by 1, 10 or 100.

And then I give it a magnitude, which is like an SI prefix, so anything between Pico, Nano, Micro, Milli, Kilo, Mega, Giga.

And I'm generating this every single time for all of my components.

So if you go to, say, R value for example, I pick a random component value.

And then I pick a multiply for it underneath. And then I pick a magnitude.

[Christian] Oh yeah, I see you've got R and you've got C and -- [David] So what I'd like to be able to do is say give me a value and give me an SI prefix Uh, I think some other stuff in there because ideally I'd like to be able to say, give me a magnitude between Kilo and Mega.

Because it makes sense for a resistor to be either a base unit or a kilo or Mega, it doesn't make sense for a resistor to be nanos or micros.

I'd like to be able to give it a range, but even if I could just say it, give me a number and a prefix and then I could use a test to sort out whether I wanted it for that component.

It's that sort of thing, if that makes sense. [Christian] OK, I will...

I'll show you the rough idea. Where's my thing?

I'm going to define a function.

Actually no, let's define a variable first, which is a list of things.

Change that to 2, I've just typed the first few prime numbers.

I'm going to define a function which picks at random from that.

So you can add parameters, if I'd wanted to, which I'm not going to do.

Yeah, let's call that.

So it'll take a list, and it's going to return something.

I don't think it actually matters what you say that the output type is.

So then over in the variables, I can define a variable and use that function random option, and it took a parameter of the options.

So this, it could, -- It could use several things, so I could make something called scale, which you had like 100.

And if I go and change my function definition to use that scale, it works.

It's defined.

It's not coming in the right -- Man! I thought it had access to the question variables but maybe I need to pass it in, so I'll add a parameter, scale has to be a number.

There we go. Right, so that works.

So that could have been a list of possible scales to choose from.

I mean there's no reason why that couldn't be?

So does that solve that problem for you?

[David] Is it possible to have the options list actually in the function, not in the variables?

[Christian] Yeah, absolutely. So I'll get rid of that variable, that scale.

Like that.

And it still tries to take a couple of parameters, doesn't it?

So let's get rid of those. So this now doesn't take any parameters.

There we go. So you can do that.

What you can do, because you might want to use this value more than once, you can use 'let' to define a variable just within that sort of scope.

So the let function, you have a variable name and then a value, and the very last parameter is an expression that evaluates.

So this is going to say define an option, or let options be this list and then pick from it and multiply it by a scale.

So I could add another one. So these are sort of evaluated procedurally.

So these are my new variables. And then there's my expression, option times.

Those things. Hopefully, if I've gone over here, it works.

There we go. And you don't just have to return a number from a function, you could return a list or dictionary, if there's a few elements.

[David] I'd like to return a number and a letter.

[Christian] Let's make that return a list now, and instead of returning the product of those two things, it'll return a tuple.

So there we go.

[David] Yeah, that helps, that gets me started.

I end up with a load of messy handling, if you look on the variables on my question below, on generating a number and then multiplying it by the magnitude to get a scale and then working out what the prefix is going to be, checking if it's u for mu, and...

[Christian] You could look at using the quantities extension because it handles all of that.

So you could -- it does the parsing of units and you can do arithmetic on quantities with units attached.

Which I think would make this a lot simpler. [David] Alright, so I'll have to look at that now, because what I do is I pass some of this to JSXGraph as well.

I don't think you can pass the mu symbol, but you can pass a u and then get that to convert it and it gets messy, and I do the same thing every time because I teach electronics.

[Christian] You should be able to pass the mu.

Maybe look at that later. Right, OK, so that's answered that. I'll do one last check of the chat.

Where did we get to?

So Sam asked if there's been any development on SCORM in Blackboard.

No.

Blackboard, I think, have just stopped developing their SCORM player.

Question, should all the JME functions be documented there?

Yes. If there's one that isn't, tell me.

I have a script which checks that there's documentation for every defined function, but I might not have run it recently.

Dmitri asks, can I use TikZ? No.

Somebody has written a JavaScript port of TikZ. I think it involved some enormous trans compiled library, so it's not included in Numbas, but that would be a good extension idea.

I didn't get onto talking about diagrams and it's now twelve o'clock.

In the Numbas demo, which I recently updated and I went through in the beginners' tutorial.

It's under main demo...

There are a few options for doing diagrams.

I mean, TikZ is very good. It's very expressive.

But it doesn't have any idea of interactivity. So quite often you're going to want to make an interactive thing, or animated diagram.

TikZ is designed for LaTeX, producing PDFs.

So it's not necessarily the best option. So you can use JSXGraph which David referred to.

You can use GeoGebra, which is a whole separate thing and it gets loaded off from elsewhere, and there's Eukleides, which I think will be the closest experience to using TikZ.

I think it's very powerful.

I've used it a fair bit.

Um, right, so twelve o'clock. I need some lunch.

I think we'll stop there. Thank you all for coming along and being so engaged.

I'll just repeat that this is being recorded and unless I receive any requests for things to be clipped out in the next couple of days, this will just go online on the Numbas blog.

Yeah.

So I think we'll run one of these again.

I enjoyed it, I certainly picked up a few bugs that I need to fix.

So we'll put something on the Numbas blog and the users list about that.

In the mean time, you can always email the Numbas email address.

You can use the Numbas users group.

So yeah, I hope to hear from you all at some point in the future.

So thank you very much coming and we'll end it there.