IMAT 3712 Human Computer Interaction Assignment Two

Usability Evaluation of an Interactive System

Deadline: 12:00 Friday 31 January 2025 (Week 18)

Learning Outcomes

This assignment is designed to provide practical experience of carrying out an analysis of usability requirements and priorities, performing a systematic usability evaluation using a standard method, and producing a presentation reporting the findings.

It assesses module learning outcomes 1 and 5.

1. Be able to apply key general principles of usability, and a comprehensive understanding of different aspects of user experience, both to guide effective design and to evaluate existing systems.

2. Be able to apply a user centred approach to the design of an interactive system, employing appropriate prototyping techniques.

3. Be able to specify requirements for and propose a suitable system design that aligns with the cognitive capabilities of its target human stakeholders and fits the needs of different users for different tasks and environments.

4. Be able to investigate and analyse ethical issues involved in the design or use of an interactive system, drawing on theoretical and practical knowledge of computer ethics.

5. Be able to undertake a sophisticated analysis and appraisal of the suitability of a range of different techniques for evaluating the usability of interactive systems for particular systems, situations and purposes, and apply the evaluation techniques to produce usability evaluations.

Submission and Marking Procedure

This is an individual assignment. The assignment is worth 40% of the total mark for the module. The mark for the assignment will be out of 100.

The intention is to mark the assignment though a live presentation by each student, so the marking will not be anonymous. These presentations will be held as early as possible after the submission deadline for the assignment at the end of January. The presentation should be about 15 to 20 minutes presenting plus about 10 minutes of questions. We are currently planning to hold the presentations live in person but may decide to do them via MS Teams.

There will be a submission to hand in as a Word document via Turnitin. The submission should comprise the written documentation you should produce anyway as part of doing the work for the assignment. You need to produce clear and comprehensive documentation of your procedure, your data gathering and your findings. You will be expected to be able to discuss it.

This is an individual assignment, and we are expecting you to work alone (apart from test subjects if you do a user trial). However some activities might be done better with more than one person doing them. You may recruit assistance, but if you have help, you need to describe this clearly in your documentation and presentation. If you do a user trial, you need to take your informed consent process seriously, including getting your subjects to sign a consent form.

Submission

The deadline for the submission of the documentation is 12:00 on Friday 31 January 2025. The submission will need to be submitted electronically to Turnitin via LearningZone. The target date for the completion of marking and the return of results is Monday 24 February 2025 (Week 22).

You should also submit your PowerPoint presentation slides, or equivalent, via Turnitin. The deadline for submitting your PowerPoint presentation is Tuesday 4 February 2025 (Week 23). However this is mainly for moderation purposes.

Writing

The assignment should be written entirely by you and should give a true reflection of your competence in English. It needs to be written in clear, comprehensible English; assignments in murky or unintelligible English with misused words will get fail marks.

Getting any human or machine help with producing your assignments that you have not clearly and honestly acknowledged is serious academic misconduct; this will result in severe penalties that can include expulsion from the university. So is plagiarism: using text, ideas or information from other sources that you have not honestly, clearly and accurately cited. Copying other people’s text by paraphrasing it sentence by sentence constitutes plagiarism; do not do this.

Fabrication of results (claiming to have collected data or experimental results that you haven’t collected, or are different from what you collected) is also very serious academic misconduct.

If you need help with producing assignments written in good English, you can get help, but you need to (1) detail exactly what the help was and who or what provided it, and (2) provide copies of your original versions of texts, so we can evaluate exactly what is yours and what isn’t, so we can arrive at fair marks. Using Grammarly in its standard mode (provided by the free version) to find and correct grammar mistakes is allowed, but using the Grammarly AI feature (in the paid version) to improve your work constitutes cheating.

If you are in doubt about what to do, you should consult your tutor.

Task

Your firm of interaction design consultants is trying to build up a portfolio of impressive work, to enable it to pitch for business convincingly in the future.

Your task is to produce a fully documented usability evaluation of an interactive system, plus a presentation of your results, by applying a systematic evaluation methodology. You have a completely free choice of what interactive system you evaluate.

The Usability Evaluation

Producing the usability evaluation will involve

1. Choosing an interactive system to study. This needs to be a real, existing system. In your documentation you need to tell us where and how you got access to it.

2. Identifying the use cases or aspects of the functioning of the system to be considered, and briefly describing them in your documentation. (These don’t need to be a complete set of use cases; for very complicated systems focusing on one part of what they do is just fine. However you should give a clear indication of what subset of the functionality of the system you are considering, and what you are not considering. If in doubt, cover less functionality in more detail.)

3. Choosing an evaluation methodology. You should apply a standard evaluation methodology such as user testing, cognitive walkthrough, or heuristic evaluation. (If you want to do something non-standard, ask advice from your tutor.)

4. Defining an evaluation procedure. This will include stating one or several user tasks to be tested or considered with exact descriptions of the scenario and the goal the user is trying to achieve, as well as what the evaluator will do to collect results and produce an evaluation. The evaluation procedure needs to be described in full, separately from the description of the results.

5. Carrying out the evaluation. This will involve applying the procedure and documenting what happens, and what the procedure finds. (If applying your procedure looks like an excessive amount of work, or producing an excessively large volume of documentation, ask advice; we would prefer an evaluation giving detailed insight into part of the functionality to an evaluation with broad coverage but a thinner or more superficial analysis.)

6. Deriving findings about the usability of the interactive system from the results of the usability evaluation. This should include consideration of how strong and how general the conclusions are.

Guidance

You are expected to apply a systematic evaluation method. That is, you need to do a user trial, or a heuristic evaluation, or a cognitive walkthrough. (Make sure you know what a heuristic evaluation or a cognitive walkthrough actually is, before claiming to be doing one.) If you want to use a different approach to doing a usability evaluation, ask advice.

We are looking for thorough and detailed evaluations, and especially findings about exactly where there are actual or potential usability problems.  If you think you are doing a disproportionate amount of work, or writing an enormous amount, then you should aim to be thorough and detailed, and compromise on how much of the system you cover.

Your evaluation procedure needs to be planned before you try to follow it. You need to (1) describe what your evaluation procedure is, separately from (2) describing what happened and what you found when you applied it, and (3) describing your evaluation of the usability of the system, drawing from what you found from your systematic evaluation procedure.

User Trials

If you do a user trial, you should aim to observe carefully what your subjects do and where they make mistakes or find things confusing and awkward, and report sources of problems as exactly as possible. Timings for tasks and subjective satisfaction ratings in debriefing are valuable but less interesting than actual usability problems. Remember that having carefully designed, realistic tasks is important, and that unless you want to look at exploration or browsing the tasks should have clear end points and success criteria. Don’t be over-directive: provide clear goals and enough information about the scenario, but don’t tell people what to do. The exact wording of instructions matters. So the instructions need to be described exactly in your documentation and reported in your presentation.

If you do a user trial, you need to get your test subjects to consent to participate and sign a consent form. You will need to prepare a consent form. for your subjects to sign; we suggest customizing the consent form. template provided with the assignment. Please don’t have subjects under 18 unless they are your family members, as this severely compromises our ethics approval.

You need to say who your subjects were, and how they were recruited, and where you carried out the user trial. This won’t require reporting names unless we need to ask for them. What relevant education and experience people have had is likely to be more important than age or sex.

Heuristic Evaluations

If you do a heuristic evaluation, you should state what principles and/or guidelines the evaluation is considering. While using Jakob Nielsen’s ten broad categories of usability problems (AKA his ten design principles) is okay, and popular, you shouldn’t make the mistake of thinking that heuristic evaluation necessarily involves using them, or only them. Heuristic evaluations are likely to be more successful when using more detailed and concrete sets of guidelines than just Nielsen’s ten broad categories of usability problems. We recommend Nielsen’s 113 design guidelines for web homepages for doing heuristic evaluations of websites.

In a heuristic evaluation, aim to be as exact as possible about which heuristic is violated, and exactly where, and exactly how. Including severity ratings is good.

Cognitive Walkthroughs

For cognitive walkthroughs, you need to describe the procedure including the cognitive walkthrough questions to be considered at each step, as there are different variants. It’s a good idea to describe the happy path (or paths, if there’s more than one way to do it) for successfully performing the task, as you need to specify this before you do the cognitive walkthrough. You need to show evidence that the questions have been systematically used in the evaluation. If the answer to a question is ‘no problem’ you don’t need more than a tick or a ‘yes’ as documentation.

Choice of Interactive System

The assignment gives you a completely free choice of what interactive system you consider; but it needs to be a real, existing interactive system that you have access to and can study. You need to tell us how and where you got access to it.

Possibilities include software applications such as programming language development environments or case tools or games or photo editing systems; e-commerce websites or museum websites or government websites; one of DMU’s web-based systems for students or staff; electronic devices such as remote controls for televisions or DVD players, or digital cameras, or car radios; or control panels for appliances such as microwave ovens or home heating systems; or a self-service system such as an automatic train ticket vending machine. You may, if you wish, choose to evaluate two very similar and directly competing products, and assess ways in which one is superior to the other. This is often a very good way to produce a good assignment.

It’s perfectly okay to decide to evaluate a part of a big or complicated system, or consider a limited set of use cases. When in doubt, go into more detail about less of the system.

The one piece of advice we can give is to choose something that is complicated or difficult to use, or is used to carry out complicated tasks, and preferably has obvious usability problems. Studying more complicated and less frequently used features of a system is likely to be more fruitful than focusing on the standard functions people use all the time. Think about use cases and scenarios, and pick ones that will make the tasks complicated. Standard features of highly optimised systems that large numbers of people use, like Amazon, don’t make for interesting evaluations, and won’t give you much to say – please avoid.

You may choose to interpret ‘interactive system’ very broadly and present a usability evaluation of a static information display, but this would require a sophisticated and detailed analysis of how people use it for practical tasks, and these tasks would need to be complicated enough to give you something to analyse. Ask advice if you consider this.

Written Submission

Your report is primarily a collection of your documentation – what you should produce anyway in the course of carrying out your evaluation – with enough supporting information to understand it. This is not an essay – produce terse lists not long paragraphs, and don’t bother with non-essential introductions.

We want evidence that you have conducted a systematic application of a well-defined evaluation procedure and have documented the results thoroughly.

Produce full documentation of your usability evaluation. This should include

1. A brief statement of what the interactive system is, including version number if applicable, and what it does. Do not write any unnecessary general introduction. Keep this to the minimum we need to understand what the assignment is about. You also need to state clearly how and where you got access to the system to evaluate it.

2. Brief accounts of the use cases considered, in just enough detail to make the rest of the documentation comprehensible, plus a statement of what you are not considering, if you are only looking at part of the system.

3. An exact description of the evaluation procedure to be followed, including what the methodology is, exact descriptions of user tasks being considered, exact wording of the instructions to be given to users in user testing, or the set of guidelines used in heuristic evaluation, or the exact questions being considered at each step in a cognitive walkthrough.

4. The results of the evaluation procedure, including notes made during observations of user trials, while conducting a heuristic evaluation, etc.

5. The findings of your evaluation about the usability of the interactive system. Include notes on how the findings relate to the results of the evaluation, and ideally about how strong the evidence is.

6. Your notes from your evaluation procedure should be included as an appendix. Handwritten notes should be scanned or photocopied or photographed.

DO NOT write more than you need to. Do not bother with unnecessary introductions or generalities about usability or human computer interaction. This is just unwelcome extra work for both you and your tutor. Brief means brief. Terse is good. However you do need to be detailed and exact about your procedure and your results and findings. Include word counts for sections.

2-3000 words should be plenty, if you don’t waffle.

Presentation

The assignment will be assessed largely from your presentation, including how you answer questions and can support what you are saying from evidence in the documentation of your usability evaluation. However, we will look at how thorough, detailed, precise, and clear your methods and results are in your documentation.

Your presentation should be 15 to 20 minutes long. It should

· Briefly introduce the interactive system, and explain what aspects of the functionality are being covered by the evaluation.

· Explain what methodology is being used and (if appropriate) how it has been customized for the needs of this particular evaluation.

· Describe the evaluation procedure exactly.

· Describe the application of the procedure and what results it produced. If there is too much to describe, it’s better to be thorough about some of the evaluation rather than more superficial about all of it.

· Describe the findings about the usability of the system.

We want to know exactly what you’ve done and what you’ve found, so don’t describe textbook knowledge or the structure of the assignment, and be precise, concrete and detailed when reporting your procedure and the findings of your evaluation. We expect a PowerPoint presentation – if you would prefer to present your work another way, please discuss this with your tutor.

You may include a demonstration of the system in use to illustrate your points, but we don’t expect this. Pictures help.

You should be prepared to point to and discuss the content of your hand-in, if and when asked about it during or after your presentation.

We recommend planning your presentation, and if possible, getting a friend or two to give you feedback and advice on a live runthrough. At the least, having done a runthrough will give you confidence.