<section class="section1"><h1>Assessing programming in Numbas</h1> <p style="text-align: center; font-size: 1.1em;">Chris Graham</p> <p style="text-align: center; font-size: 1.1em;">George Stagg, Aamir Khan, Christian Lawson-Perfect</p> <div id="branding"> <div> <a href="https://www.ncl.ac.uk" target="_blank"> <img alt="Newcaste University" class="nulogo" src="../../../static/images/logos/newcastle-university-logo.png"/> </a> <p>School of Mathematics, Statistics & Physics E-Learning Unit</p> </div> <div class="container"> <div class="col"> <a href="https://numbas.org.uk" target="_blank"> <img alt="Numbas" class="numbas" src="../../../static/images/logos/numbas.png"/> </a> </div> <div class="col"> <a href="https://www.advance-he.ac.uk/cate-team/e-learning-unit-school-mathematics-statistics-physics-newcastle-university" target="_blank"> <img alt="Collaborative Awards for Teaching Excellence (CATE) winners 2020" class="cate" src="../../../static/images/logos/cate.png"/> </a> </div> <div class="col"> <a href="https://chirun.readthedocs.io/en/latest/" target="_blank"> <img alt="Chirun" class="chirun" src="../../../static/images/logos/chirun.png"/> </a> </div> </div> </div> <hr/> <section class="section2"><h2>In this talk</h2> <p>Motivation for the Numbas programming extension.</p> <p>How the extension works.</p> <p>Application of the extension to our programming modules.</p> <hr/> </section><section class="section2"><h2>Background</h2> <hr/> <section class="section3"><h3>Programming in MSP</h3> <p>Programming is embedded in our mathematics and physics programmes, with compulsory modules at stages 1 and 2.</p> <ul> <li>Stage 1: <ul> <li>Problem solving with Python </li> <li>Introduction to probability and R</li> </ul> </li> <li>Stage 2:<ul> <li>Numerical methods (Python) </li> <li>Computational statistics (R)</li> </ul> </li> </ul> <hr/> </section><section class="section3"><h3>Module structure</h3> <p>Teaching weeks are typically structured as:</p> <ul> <li> <p>1 hour lecture: covering theory and examples</p> </li> <li> <p>2 hour practical: hands-on practical with embedded exercises</p> </li> </ul> <blockquote class="left green"> I loved the format; one theory lecture and a two hour practical really worked as I learnt the concepts and then was given contact time to make sure that I understood them </blockquote> <hr/> </section></section><section class="section2"><h2>Challenges</h2> <hr/> <section class="section3"><h3>Student experience</h3> <p>Students have different backgrounds and experiences with programming and computers.</p> <p>...from A-level computing experience to computer-phobic.</p> <figure class="figure1" data-title="A student enjoying computing, and one who is upset by it"><img alt="A student enjoying computing, and one who is upset by it" src="images/computer-users.png" width="55%"/></figure><hr/> </section><section class="section3"><h3>Answering queries</h3> <p>Our experience was that demonstrators often received "low-level" queries about exercises, often requiring no more than signposting.</p> <blockquote class="left yellow"> How do I start this exercise? </blockquote> <blockquote class="right blue"> I don't understand - what does this mean?! </blockquote> <hr/> <p>Or... students gathered around one competent student working on problems in a very non-interactive way</p> <figure class="figure2" data-title="Students gathered around a student writing computer code"><img alt="Students gathered around a student writing computer code" src="images/computer-bunched.png" width="55%"/></figure><hr/> </section><section class="section3"><h3>Exercise feedback</h3> <p>Handouts contained written exercises to complete.</p> <p>Students work through the material at very different speeds, so timing feedback is difficult:</p> <ul> <li>Solutions released during session, end of session, the next week...?</li> <li>Work through on big screens, but when?</li> </ul> <hr/> </section><section class="section3"><h3>Summative Assessment</h3> <p>Programming is difficult to assess.</p> <p>Little variation between student answers - difficult to detect collusion.</p> <p>Marking is very time-consuming and repetitive.</p> <hr/> </section></section><section class="section2"><h2>Numbas for programming</h2> <hr/> <section class="section3"><h3>Why Numbas?</h3> <p>Many of the lucrative features of maths e-assessment also apply to basic programming: automatic marking, instant feedback, randomisation...</p> <p>Numbas was ideal:</p> <ul> <li>familiar to our students</li> <li>tap into well-developed features: randomisation, steps, adaptive marking, explore mode, etc</li> <li>mix coding questions with other part types</li> </ul> <hr/> </section></section><section class="section2"><h2>Programming in Numbas: first iteration</h2> <p>First attempts used a "follow along" approach:</p> <ul> <li> <p>Student is presented with a problem in Numbas and completes problem in R/Python.</p> </li> <li> <p>Numbas follows along with the analysis in JME, to be able to mark the student's response.</p> </li> </ul> <p>Useful and we still use it for some questions, but limited as it does not directly mark student code.</p> <hr/> </section><section class="section2"><h2>Programming in Numbas: second iteration</h2> <p>The version used in 2020/21:</p> <ul> <li> <p>Student enters a code answer</p> </li> <li> <p>Student's answer plus unit tests are sent to a server running R / Python</p> </li> <li> <p>Server sends back responses which Numbas can use to mark</p> </li> </ul> <p>Very useable, but relies on a server to run code and is fixed to Newcastle.</p> <hr/> </section><section class="section2"><h2>The Numbas programming extension</h2> <p>Released earlier this year.</p> <p>Uses WebAssembly/Emscripten to compile Python (Pyodide) and R (WebR) to run code entirely self-contained in a web browser.</p> <p>No reliance on a server to run code; nothing tied to Newcastle. Therefore shareable!</p> <hr/> <iframe src="https://numbas.mathcentre.ac.uk/question/117159/get-the-first-item-in-a-list-with-a-hint/embed/?token=8b7564b3-1567-41cb-8fed-199c66e0d798" style="width: 100%; height: 620px; "></iframe> <hr/> <section class="section3"><h3>Behind the scenes</h3> <p><a href="https://numbas.mathcentre.ac.uk/question/117159/get-the-first-item-in-a-list-with-a-hint/" target="_blank">Custom part type</a></p> <figure class="figure3" data-title="The code part type"><img alt="The code part type" src="images/codepart.png"/></figure><hr/> </section><section class="section3"><h3>Handout exercises</h3> <p>Exercises embedded in the handouts as Numbas questions:</p> <ul> <li>immediate marking and feedback</li> <li>alternative answers to catch common errors</li> <li>randomisation to try similar questions over and over</li> <li>full worked solutions</li> </ul> <p>And can be scaffolded with steps, or multiple parts to build up solutions.</p> <hr/> <section class="section4"><h4>Embedded exercises</h4> <p>Exercises are embedded directly in handouts:</p> <figure class="figure4" data-title="Embedded exercises"><img alt="Embedded exercises" src="images/exercise.gif" style="width: 100%; border: 1px solid #CCC;"/></figure><hr/> </section></section><section class="section3"><h3>New experience of handout exercises</h3> <p>Students are able tackle exercises and learn at their own pace.</p> <p>During Covid, modules ran successfully with asynchronous handout material.</p> <p>Since returning to campus, the handout exercises have transformed the demonstrator-student dynamic.</p> <hr/> </section><section class="section3"><h3>Assessment</h3> <p>Formative "Test Yourself" question sets accompanying each week of content.</p> <p>In-course assessments and final exams were hybrid, ~60% auto-marked: focus manual marking where it is most effective.</p> <hr/> </section><section class="section3"><h3>Student feedback</h3> <blockquote class="left green"> I like that I can work through the handout so that I'm learning in the best way for myself, at my own pace </blockquote> <blockquote class="right yellow"> Really efficient way off working, using the handouts and tests!! </blockquote> <blockquote class="left blue"> The feedback from our assignments was detailed and personal to us and gave us information on what we did well and where we can improve. </blockquote> <hr/> </section></section><section class="section2"><h2>More examples</h2> <hr/> <section class="section3"><h3>Scaffolding with steps</h3> <iframe src="https://numbas.mathcentre.ac.uk/question/114116/sum-of-triangular-numbers-with-optional-steps/embed/?token=6776abe8-d7cb-43f9-8573-09a96a18d181" style="width: 100%; height: 620px; border: 1px solid #CCC;"></iframe> <hr/> </section><section class="section3"><h3>Explore mode</h3> <iframe src="https://numbas.mathcentre.ac.uk/question/114122/exploration-activity-define-a-class-add-methods-to-it-then-use-it/embed/?token=a8dde3bd-fd0c-40e1-ac34-7bac09f6a09e" style="width: 100%; height: 620px; border: 1px solid #CCC;"></iframe> <hr/> </section><section class="section3"><h3>Mark a plot (experimental)</h3> <iframe src="https://numbas.mathcentre.ac.uk/question/114879/make-a-plot-with-matplotlib-and-mark-its-properties/embed/?token=40be0d3e-cc68-4b20-b29a-edb2c68ded40" style="width: 100%; height: 620px; border: 1px solid #CCC;"></iframe> <hr/> </section><section class="section3"><h3>R example</h3> <iframe src="https://numbas.mathcentre.ac.uk/question/114974/query-the-built-in-mtcars-dataset/embed/" style="width: 100%; height: 620px; border: 1px solid #CCC;"></iframe> <hr/> </section></section><section class="section2"><h2>Find out more</h2> <ul> <li><a href="https://www.numbas.org.uk/blog/2022/02/assess-programming-in-python-and-r-with-numbas/" target="_blank">Numbas blog post on the programming extension</a></li> <li><a href="https://numbas.mathcentre.ac.uk/exam/26300/programming-extension-demo/preview/" target="_blank">Demo exam</a></li> <li><a href="https://github.com/numbas/numbas-extension-programming/" target="_blank">Extension documentation</a></li> </ul> <p>Thank you to my colleagues (George, Christian, Aamir) in our E-learning Unit.</p> </section></section>