A practical guide to creating quizzes: part 5

Practical guides

In part 1, we looked at the characteristics of online quizzes and explored how they could be used to assist or assess learning. In part 2, we explored the various question formats and the types of learning for which they are best suited. In part 3, we moved on to the writing of the questions, in particular the traps to avoid. In part 4, we saw how quizzes can be presented as games. In this final part, we look at the steps you can take to make your quizzes robust and reliable.

Being thorough

Assuming your quiz is being used to test knowledge, then you need to take some care to ensure that it performs this function effectively. Prepare at least one quiz question for each of your knowledge objectives. You cannot be sure that a learner has achieved mastery if you test only a sub-set of your objectives. To be absolutely sure the learner has not simply got lucky by guessing answers, you may well prepare more than question for each objective. Don’t write questions to test skills, unless you are absolutely sure quiz questions are capable of assessing these effectively, which is likely to be rarely.

As we discussed in part 2, you need to select a question format that’s appropriate to the type of knowledge you are testing, For example, if you need to test recall of a technical term, use a text input question and not a multiple choice, which only tests recognition of the term. Don’t be tempted to select different formats simply to increase variety – that’s not your purpose here.

If your objective is that a learner is able to come up with a response quickly, then add time limits to your questions.

Discouraging guessing

Some people reckon they can pass any multiple choice quiz by guessing the right answers. Your job is to prove them wrong. In part 3 we looked at techniques you can use to make life difficult for the chancer – no give-away distractors, no obvious right answers. A simple improvement would be to prepare at least four options for each multi-choice question, and even better five. That does make it even harder to write the questions, but then there really is no pain, no gain when it comes to question writing.

Guessing

Hopefully all that deep thinking is to come up with the right answer, not just to make a good guess

Another technique you could try is to include a ‘don’t know’ or ‘not sure’ option for each question. This would score no points. Then penalise wrong answers with negative scores. This ups the stakes for the learner who wants to guess the right answers.

Discouraging cheating

The greater the reward for passing an assessment, the more tempting it becomes to cheat. Really high-stakes assessments are beyond the scope of this guide, but you should be aware of the difficulties in authenticating whether the person answering the questions really is who they say they are. All sorts of complex and expensive technologies are available to authenticate remote users, including finger-printing and retina scanning, but the only way you can be really sure that the learner is who they say they are and is getting no help from a third-party or some reference source is to have them attend a testing centre which has an invigilator present. Most quizzes are not that serious, so there’s no point getting carried away with the security!

Retinal scan

Sophisticated technologies such as retinal scanning can be used to aid the authentication of learners, but are only really necessary for very high stakes assessments

A more routine way to avoid cheating is to randomise the order in which the questions are presented and the order in which options are displayed within the questions themselves. That way, no-one can simply write down the question and option numbers and pass them on to others. A step further is to create a bank of questions from which the system selects the questions to display, which means that every learner will receive a different set of questions. Yes, this is a lot more work, but the chances of successful cheating will be much reduced.

Quiz options in Moodle

Every quiz tool will provide you with different configuration options. This shows some of the options available in the Moodle quiz module, including the ability to set time limits and shuffle questions and options.

Providing feedback

Assuming you are using a quiz as a form of assessment, then if you tell the learner whether they have got each question right or wrong, you are making it easy for them to pass the quiz on a second attempt, without necessarily curing any misunderstandings they may have had. To avoid this problem, you could create a completely different quiz for second attempts, or have the system draw questions from a bank, as described above.

At the end of the quiz, inform learners whether or not they have passed. If your software allows it, let them know how they performed against each of the topics addressed by the quiz. Pass or fail, provide advice to learners on what they should do next.

If the quiz is being used in a formative manner (to help the learner progress towards the learning objectives), rather than summative (to assess mastery), then it is vitally important that you provide helpful feedback for every question. Ideally this should be provided for each option of each question, rather than just for all correct answers and all incorrect answers. The purpose of this feedback is to correct errors and misunderstandings and to reinforce key learning points.

Scoring fairly

Another consideration is how you score correct answers. Most authoring tools will allow you to specify the number of points you will award to each correct answer. In a simple multiple-choice question, this is straightforward enough – you either allocate the same number of points to each question or award more points for particularly difficult questions.

The difficulty comes with questions that ask for multiple responses. The first consideration is whether these questions should score higher than MCQs because they are actually asking the learner for a series of decisions, not just one. Another issue is how you apportion the points across the various options. Let’s say there are five alternative options, three of which are correct. Ideally, each correct option will score 20% of the available points. But the learner should also be rewarded for not choosing incorrect options, so each option not chosen should also score 20% of the total. Whether you can achieve this with your authoring software remains to be seen!

That concludes this practical guide. A PDF version will be available shortly.

Next up: how to create reference material.

A practical guide to creating quizzes: part 4

Practical guidesIn part 1, we looked at the characteristics of online quizzes and explored how they could be used to assist or assess learning. In part 2, we explored the various question formats and the types of learning for which they are best suited. In part 3, we moved on to the writing of the questions, in particular the traps to avoid. In this fourth part, we have a little light relief, as we see how quizzes can be presented as games.

Quiz games are still quizzes, in that they can be used assist and assess learning, but they employ gaming techniques to increase learner engagement. We all know how compulsive games can be, so it takes little in the way of imagination to appreciate how much they can add to what would otherwise be a very dry process of drill and practice.

To demonstrate a wide variety of quiz game techniques, I’ve taken examples from a quiz making  package called Quizit, unfortunately no longer available. Similar results could be achieved by those with coding skills using Adobe Flash Professional or HTML 5, or by using a number of off-the-shelf quiz game tools.

Ident

Players try to identify as many as possible of the six pictured people, objects or places.

This first example, a classic ‘picture board,’ requires players to type in the name of the pictured object.

Quotient

Players try to get as close as possible to the right answers for each question. Each option is graded as to how right or wrong it is and scored accordingly.

In this variant of a multiple choice quiz, player get rewarded for how close they can get to the right answer. The rather irreverent feedback is delivered randomly from a pool, depending on the accuracy of the answer.

Summit

Players attempt questions of increasing difficulty, with the aim of getting to the highest level that they can (10 being the highest). Players have three ‘lives’, which allow them to have another go when they make a mistake.

Levels are a classic gaming feature. As the player moves up the levels, the questions get correspondingly more difficult.

Guess

Players attempt to guess the identity of a person, object, place or event from the clues provided. The more time they take, guesses they make or clues they ask for, the lower their score.

This game is unusual in that it works entirely as a ‘conversation’ between questioner and player. All input is by natural text. Time pressures add to the level of engagement.

Target

Players demonstrate how well they know the subject of the quiz by estimating a series of percentages relating to the subject’s behaviour. They nearer they get to the actual percentages, the higher their score.

This game works with a slider, which the player uses to make estimates.

Sprint

Players answer a series of questions as fast as they can. They can have as many attempts as they like at each question, but this reduces their score accordingly.

This time players can have multiple attempts at every question, but in the process waste time and points.

Teamplay

Three teams or three individual players each answer a series of questions, to see who can answer the most correctly.

This competitive game can be used with teams of players in a classroom.

faceoff

Two teams or two individual players each answer a series of questions, to see who can answer the most correctly.

This variant on the competitive game pits two players sitting round the same computer against each other.

Coming in part 5: Making your quizzes robust

A practical guide to creating quizzes: part 3

Practical guidesIn part 1, we looked at the characteristics of online quizzes and explored how they could be used to assist or assess learning. In part 2, we explored the various question formats and the types of learning for which they are suited. In this part, we move on the writing of the questions, in particular the traps to avoid.

Question writers are faced with two tricky problems:

  1. They do not want to make the correct answers too obvious but, on the other hand, they don’t want to leave these answers open to challenge. So, they put an awful lot of care into how they phrase the right answers, to make them absolutely clear. In the process, they often give the answers away.
  2. They need to tempt the user with distractors (wrong answers) that are genuinely distracting. To be distracting, they must be plausible. The trouble is that plausible distractors – tempting but still unambiguously wrong - can be hard to come by, particularly when you’re testing knowledge of rules and principles. Still sure you want to write quiz questions?

We’ll explore both of these traps, with the help of some examples.

Obvious answers

In this first case, most learners will pick the fourth option. Why? Because it is the longest. That’s because the question writer will spend a disproportionate time making sure the correct answer is absolutely precise. As a result, it stands out. Also, the question writer was obviously struggling to come up with a plausible fourth option, which is why they threw in option 3 – an amusing throw-away, but certainly not a genuine distractor.

This second example is just carelessness. Only the word ‘elephant’ fits grammatically with the stem of the question.

Here’s another frequent mistake. The absolute answers (‘no men’ and ‘all men’) are clearly less likely than the softer ‘some men’. No learner is going to be fooled by these distractors.

Another lazy cop out is to use ‘none of the above’. It’s a way of telling the learner that you’re having trouble phrasing a correct answer. And option 3 is another throw-away.

Confusing questions

Learners don’t mind obvious answers, because it makes their job easier. But they certainly get annoyed if they are presented with questions that they don’t understand. Take these examples …

It’s bad enough having a negative in the question stem (“Which of the following is NOT …”), but a double negative such as “never unnecessary” makes the question really hard to fathom out.

There is no grammatical consistency in these options. Each one should be phrased in the same way and flow nicely from the question stem.

The problem here is that the learner could enter their answer in so many different ways: “CERN”, “CERN in Geneva”, “Switzerland”, even “Centre Européen de Recherches Nucléaires”. Text input questions are fine, but you must make it absolutely clear what sort of answer you are expecting, for example, “In what city was the research establishment where the World Wide Web was invented?” Even then you be prepared to accept both “Geneva” and “Genève”.

The problem here is that it is not clear whether you are looking for a single response or multiple responses. Experienced computer users will recognise that the use of check boxes implies you can pick any number of options, but that won’t be obvious to everyone. Better to say “You can choose more than one option.”

Chances are you won’t see your own obvious answers and confusing questions. You have no option really but to have your quiz questions tested by typical learners. Believe me, you will learn lots from what they have to tell you.

Coming in part 4: Quizzes as games

A practical guide to creating quizzes: part 2

Practical guidesIn part 1, we looked at the characteristics of online quizzes and explored how they could be used to assist or assess learning. In this instalment, we look at the various question formats and the types of learning for which they are suited.

Factual knowledge

In an adult learning context, factual information is usually supplemental to the core learning objective and more often than not just for general interest. However, some facts really do need to be known by heart: When was … ? What is … ? Who is … ?

If it is essential that the learner can recall the information without prompting, then you have little choice than to ask a question that requires them to type the answer in. If it is only necessary that they are able to recognise the right answer, then various forms of multiple choice will do.

Testing factual knowledge

In the example on the left, the learner is tested for recall of a date, which they must type in to the text box. The example on the right also checks for recall, in this case of a chronological sequence.

Conceptual knowledge

Concepts provide a common language for understanding a subject. Generally the aim is for the user to be able to identify the class or category to which given objects belong, whether these are tangible (like types of computer) or abstract (like schools of thought). The most common way of checking this knowledge is to provide the learner with examples and ask them to place these in the correct categories, as in the examples below:

Testing conceptual knowledge

On the left is a typical matching exercise in which learners place example foods into categories. The alternative format on the right has learners select those examples which belong to a given category.

Process knowledge

A process explains how something works as a chain of cause and effect relationships. To check understanding of a process, you can ask questions about causes or about effects, as shown below:

Testing process knowledge

In the left hand example the learner identifies a probable cause. On the right, the learner looks at possible effects.

Spatial knowledge

In this instance our aim is for the learner to be able to identify the locations of parts of an object, device, physical space or system. The easiest way to check this knowledge is with a question that has the learner click on a given part as shown below:

Testing spatial knowledge

The example on the left has the learner identify a particular bone on a picture of a skeleton. The task on the right is similar but in this case the object is a software interface.

Procedural knowledge

Procedural knowledge is tougher because in many cases what you really want to test is whether the learner can actually carry out the procedure rather than just answer questions about it. However procedural knowledge is a first step and you can use a variety of questions to check learning:

Testing procedural knowledge

Rules and principles determine how a procedure is implemented in specific cases. The example on the left explores how different principles could be applied to a particular situation. The example on the right checks that the learner knows the correct order in which procedural steps should be applied.

These examples were created in Articulate QuizMaker, although many quiz tools could do a similar job. In the next instalment we look at the principles underlying the writing of quiz questions.

A practical guide to creating quizzes: part 1

Practical guidesWe all know what a quiz is. It’s a test of knowledge, typically accomplished by asking a series of questions.

Quizzes are popular in the digital environment, not least because computers find it so easy to deliver the questions and score the answers. In fact, if you were in your first week of a programming course, you’d probably have a go at putting together a multiple choice quiz. Quizzes are an entertaining diversion, particularly when delivered within the context of a game, with rules, levels, competition and prizes, but they can also play a useful role within a learning solution. A function that is often abused, perhaps, but the potential is there.

Quiz shows

TV quiz shows make for good popular entertainment and similar formats can be used within computer-based quizzes

Media elements

Although many quizzes are primarily textual, the possibility is there to use every media element. Images can provide the basis for questions that test for recognition of people, objects or places or to locate elements within interfaces and other spaces. Video can be used to portray situations that test the learner’s ability to make critical judgements. Audio can be employed to check for recognition of voices or pieces of music. A variety of media can also be used to introduce questions and provide feedback.

Interactive capability

Quizzes are essentially interactive. They serve their function in testing knowledge only by eliciting responses from learners. Just about any input device imaginable can be used as the basis for that interaction – key presses, mouse clicks, touches, the lot.

Applications

The most common application for a quiz is as a test of mastery. This is fine in principle as long as it really is possible for the knowledge and skills in question to be assessed by the sort of questions that a computer can deliver. To state the obvious, you might be able to check that a pilot understands the principles of aerodynamics using a quiz, but you can’t check they can fly the plane. Some caution also needs to be taken in terms of when a quiz is delivered. If the quiz comes right after the delivery of content (and the learner knows it’s coming), it is all too easy for the learner to hold on to enough of the information to get them through the quiz, but then forget it all the day after. We can probably all remember how possible it was to cram in information before an exam, only to see that evaporate almost as soon as we committed it to paper. A much more valid test of knowledge comes weeks, months or years after original exposure to the information.

Math quiz creator

Tools like this low-cost Math Quiz Creator can be used not only for assessment, but to provide loads of valuable practice

Although their potential is rarely exploited to the full, quizzes can actually play a useful role at just about every stage in the learning process:

  • As a way, right up front, for the learner to find out how much they already know and how much they need to know. This sort of diagnostic pre-test not only demonstrates the need for learning, it helps to direct the learner to content that is likely to be most useful.
  • As a vehicle for delivering the learning content itself. One way to create an engaging lesson is to use a series of quiz questions to challenge and then build on the learner’s prior knowledge. Every question alerts the learner to a gap to fill and all you have to do is oblige.
  • As a means for repetitive drill and practice. Unlike teachers, computers never get bored asking questions and they don’t lose their patience when the learner takes a little longer than expected to get the point. In the classroom, most knowledge is under-rehearsed and most skills under-practised. Quizzes represent a good way to remedy that.

So how do I get started?

There is no shortage of tools for creating quizzes. Most cover the usual range of questions types – multi-choice, multi-answer, free text response, sequencing, matching, selecting hotspots and all sorts of variations. All e-learning authoring tools come with a quiz making capability, plus there are specialist stand-alone tools, including ones that you can use for high-stakes assessments or for quiz games.

Articulate Quizmaker

You will find a quiz-building capability in just about any authoring tool, including this one which forms part of Articulate Studio

In practice, it’s likely that tools will be the least of your problems. Writing the questions is a much more challenging task, and that’s where we’ll be directing our attention next.

Coming in part 2: using the correct question for the job