Concerning evaluations and user studies

Evaluation

There are several ways of gathering data when developing a prototype.
Below is a general description of some of them;

Quick and dirty evaluation - Quick feedback, not very carefully documented. Done in a short space of time. Inexpensive and therefore quite attractive to companies.

Usability testing - Observe the user. Record and process every action. This type of test tests the general efficiency of the prototype.

Field studies - Natural setting. Observe the user's natural actions. Good for identifying certain needs and determining certain requirements.

Different techniques for evaluating

Observing users - Don't disturb or interfere, just observe.

Asking users - What do they want, how do they think and why? How many users will be asked?

Asking experts - Cheap and easy. Experts often have solutions to certain problems.

User testing - Conducted in controlled environment. Well-defined tasks. Data collected mainly revolves around the time to complete a certain task, amount of errors made and how easy it is to use the prototype.

Framework to guide evaluation

Determine goals - Why evaluate? Who needs data? Why?

Explore questions to be answered - What to ask? Can questions be divided into sub-questions?

Identify practical uses - Think of this before evaluation. Budget, equipment, facilities?

Users - Are users chosen for evaluation relevant to the study?

Ethical issues

Tell users goal of evaluation. Ensure no personal information is used without permission.

Pilot study 

Small study or evaluation to see if the real study is viable.

Asking users and experts

Interviews

Planning - avoid long questions; hard to remember. Split broad questions into more specific ones. Speak in a way that the interviewee is comfortable with.

Unstructured interviews

• Make sure the interviewee is at ease.
• Respond carefully and with sympathy. Do not change the interviewees' opinion.
• Analyse data as soon as possible.

Structured interviews

• Short, well-defined and clearly worded questions.
• "Closed" questions, requires precise answers.

Semi-structured interviews

• Both closed and open questions.
• Observe body language.
• Do not prompt answers.

Group interviews

• Interviewees develop opinions by talking to each other.
• Facilitator guides and prompts the discussion.

Testing and modeling users

User testing 

Observe how the prototype is used. What errors occur? How long does it take to perform a certain task?

Doing user testing

Plan the testing thoroughly. Are the conditions the same for every participant?

Try to avoid environments with much noise and/or disturbances. If possible, modify the testing space to match a relevant environment.

Be sure to inform the participants about the presence of cameras, microphones, etc.

Question for the second reading seminar; What would be our best choice of gathering data?