Continual technological evolution has led to an explosion of new techniques in Human-Computer Interaction (HCI) research. Research Methods in Human-Computer Interaction is a thoroughly comprehensive guide to performing research and is essential reading for both quantitative and qualitative methods. Chapters cover a broad range of topics relevant to the collection and analysis of HCI data, going beyond experimental design and surveys, to cover ethnography, time diaries, physiological measurements, case studies, and other essential elements in the well-informed HCI researcher's toolkit. "This book is a must read for anyone in the field of Human-Computer Interaction. The multi-disciplinarian approach, housed in the reality of the technological world today, makes for a practical and informative guide for user interface designers, software and hardware engineers and anyone doing user research."
Dr. Mary Czerwinski, Research Area Manager, Microsoft Research, USA
"Research Methods in HCI is an excellent read for practitioners and students alike. It discusses all the must-know theory, provides detailed instructions on how to carry out the research, and offers great examples. I loved it!"
Professor Vanessa Evers, Professor, Human Computer Studies Lab, University of Amsterdam, the Netherlands
"The book is superb: comprehensive, clear, and engaging! This is a one-stop HCI methods reference library. If you can only buy one HCI methods book, this is the one!"
Dr. Clare-Marie Karat, IBM TJ Watson Research, USA, and recipient of the 2009 ACM SIGCHI Lifetime Service Award
"A much needed and very useful book, covering important HCI research methods overlooked in standard research methods texts."
Professor Gilbert Cockton, School of Design, Northumbria University, United Kingdom
About the Authors xvii
Acknowledgments xviii
Preface xix
1 Introduction 1
1.1 Changes in topics of HCI research over time 3
1.2 Shifts in measurement in HCI 5
1.3 Inherent conflicts in HCI 9
1.4 Interdisciplinary nature of HCI research 11
1.5 Communicating your ideas 13
1.6 Research and usability testing 14
Summary of Chapters 14
Discussion Questions 15
Research Design Exercise 16
References 16
2 Experimental Research 19
2.1 Types of behavioral research 20
2.2 Research hypotheses 22
2.2.1 Null hypothesis and alternative hypothesis 23
2.2.2 Dependent and independent variables 25
2.2.3 Typical independent variables in HCI research 25
2.2.4 Typical dependent variables in HCI research 26
2.3 Basics of experimental research 27
2.3.1 Components of an experiment 27
2.3.2 Randomization 28
2.4 Significance tests 30
2.4.1 Why do we need them? 30
2.4.2 Type I and Type II errors 32
2.4.3 Controlling the risks of Type I and Type II errors 34
2.5 Limitations of experimental research 34
Summary 36
Discussion Questions 37
Research Design Exercises 38
References 38
3 Experimental Design 41
3.1 What needs to be considered when designing experiments? 43
3.2 Determining the basic design structure 44
3.3 Investigating a single independent variable 45
3.3.1 Between-group design and within-group design 46
3.3.2 Choosing the appropriate design approach 49
3.4 Investigating more than one independent variable 53
3.4.1 Factorial design 53
3.4.2 Split-plot design 54
3.4.3 Interaction effects 55
3.5 Reliability of experimental results 57
3.5.1 Random errors 57
3.5.2 Systematic errors 57
3.6 Experimental procedures 63
Summary 64
Discussion Questions 65
Research Design Exercises 65
References 66
4 Statistical Analysis 69
4.1 Preparing data for statistical analysis 70
4.1.1 Cleaning up data 70
4.1.2 Coding data 71
4.1.3 Organizing data 73
4.2 Descriptive statistics 73
4.2.1 Measures of central tendency 73
4.2.2 Measures of spread 74
4.3 Comparing means 74
4.4 T tests 76
4.4.1 Independent-samples t test 76
4.4.2 Paired-samples t test 76
4.4.3 Interpretation of t test results 77
4.4.4 Two-tailed t tests and one-tailed t tests 78
4.5 Analysis of variance 78
4.5.1 One-way ANOVA 79
4.5.2 Factorial ANOVA 80
4.5.3 Repeated measures ANOVA 82
4.5.4 ANOVA for split-plot design 83
4.6 Assumptions of t tests and F tests 86
4.7 Identifying relationships 86
4.8 Regression 89
4.9 Nonparametric statistical tests 91
4.9.1 Chi-square test 92
4.9.2 Other non-parametric tests 94
Summary 94
Discussion Questions 95
Research Design Exercises 96
Team Exercises 96
References 96
5 Surveys 99
5.1 Introduction 100
5.2 Benefits and drawbacks of surveys 101
5.3 Goals and targeted users for survey research 102
5.4 Probabilistic sampling 103
5.4.1 Stratification 105
5.4.2 Response size 106
5.4.3 Errors 106
5.5 Non-probabilistic sampling 107
5.5.1 Demographic data 107
5.5.2 Oversampling 108
5.5.3 Random sampling of usage, not users 109
5.5.4 Self-selected surveys 109
5.5.5 Uninvestigated populations 109
5.6 Developing survey questions 111
5.6.1 Open-ended questions 111
5.6.2 Closed-ended questions 112
5.6.3 Common problems with survey questions 113
5.7 Overall survey structure 113
5.8 Existing surveys 115
5.9 Paper or online surveys? 116
5.10 Testing the survey tool 118
5.11 Response rate 119
5.12 Data analysis 120
Summary 121
Discussion Questions 121
Research Design Exercise 122
References 122
6 Diaries 125
6.1 Introduction 126
6.2 Why do we use diaries in HCI research? 127
6.3 Participants for a diary study 130
6.4 What type of diary? 132
6.4.1 Feedback diary 132
6.4.2 Elicitation diary 133
6.4.3 Hybrid feedback and elicitation diary 134
6.5 Data collection for the diary study 134
6.6 Letting participants know when to record a diary entry 136
6.7 Analysis of diaries 137
Summary 138
Discussion Questions 138
Research Design Exercise 138
References 139
Appendix A Frustration Experience Form (Time Diary) 140
Appendix B Excel Time Diary Form 141
7 Case Studies 143
7.1 Introduction 144
7.2 Observing Sara: a case study of a case study 145
7.3 What is a case study? 147
7.3.1 In-depth investigation of a small number of cases 147
7.3.2 Examination in context 147
7.3.3 Multiple data sources 148
7.3.4 Emphasis on qualitative data and analysis 149
7.4 Goals of HCI case studies 150
7.4.1 Exploration 150
7.4.2 Explanation 151
7.4.3 Description 152
7.4.4 Demonstration 154
7.5 Types of case study 156
7.5.1 Intrinsic or instrumental 156
7.5.2 Single case or multiple cases 156
7.5.3 Embedded or holistic 160
7.6 Research questions and hypotheses 161
7.7 Choosing cases 163
7.8 Data collection 164
7.8.1 Data sources and questions 164
7.8.2 Collecting data 165
7.9 Analysis and interpretation 167
7.10 Writing up the study 168
7.11 Informal case studies 170
Summary 172
Discussion Questions 174
Research Design Exercises 174
References 175
8 Interviews and Focus Groups 177
8.1 Pros and cons of interviews 178
8.2 Applications of interviews in HCI research 180
8.2.1 Initial exploration 180
8.2.2 Requirements gathering 184
8.2.3 Evaluation and subjective reactions 186
8.3 Who to interview 187
8.4 Interview strategies 189
8.4.1 How much structure? 189
8.4.2 Focused and contextual interviews 191
8.5 Interviews vs focus groups 192
8.6 Types of question 194
8.7 Conducting an interview 197
8.7.1 Preparation 197
8.7.2 Recording the responses 198
8.7.3 During the interview 199
8.8 Electronically mediated interviews and focus groups 203
8.8.1 Telephone 204
8.8.2 Online 204
8.9 Analyzing interview data 206
8.9.1 What to analyze 207
8.9.2 How to analyze 208
8.9.3 Validity 212
8.9.4 Reporting Results 212
Summary 213
Discussion Questions 214
Research Design Exercises 214
References 215
9 Ethnography 217
9.1 Introduction 218
9.2 What is ethnography? 219
9.3 Ethnography in HCI 221
9.4 Conducting ethnographic research 224
9.4.1 Selecting a site or group of interest 225
9.4.2 Participating: choosing a role 227
9.4.3 Building relationships 230
9.4.4 Making contact 231
9.4.5 Interviewing, observing, analyzing, repeating, and theorizing 232
9.4.6 Reporting results 236
9.5 Some examples 237
9.5.1 Home settings 237
9.5.2 Work settings 238
9.5.3 Educational settings 239
9.5.4 Ethnographies of mobile and ubiquitous systems 240
9.5.5 Virtual ethnography 241
Summary 246
Discussion Questions 246
Research Design Exercises 247
References 248
10 Usability Testing 251
10.1 What is usability testing? 252
10.2 How does usability testing relate to traditional research? 254
10.3 Types of usability testing or usability inspections 256
10.3.1 Expert-based testing 256
10.3.2 Automated usability testing 258
10.4 User-based testing 260
10.4.1 Types of usability testing 260
10.4.2 Stages of usability testing 262
10.4.3 How many users are sufficient? 263
10.4.4 Locations for usability testing 264
10.4.5 Task list 268
10.4.6 Measurement 270
10.4.7 The testing session 271
10.4.8 Making sense of the data 274
10.5 Other variations on usability testing 275
Summary 276
Discussion Questions 276
Research Design Exercise 277
References 277
11 Analyzing Qualitative Data 281
11.1 Introduction 282
11.2 Stages of qualitative analysis 282
11.3 Grounded theory 283
11.4 Content analysis 285
11.4.1 What is content? 286
11.4.2 Why do we need to collect text or multimedia information? 286
11.4.3 Questions to consider before content analysis 287
11.5 Analyzing text content 289
11.5.1 Procedure 289
11.5.2 Identifying coding categories 290
11.5.3 Coding the text 292
11.5.4 Ensuring high-quality analysis 294
11.6 Analyzing multimedia content 300
Summary 301
Discussion Questions 302
Research Design Exercise 303
References 303
12 Automated Data Collection Methods 307
12.1 Exploiting existing tools 308
12.1.1 Web logs 309
12.1.2 Stored application data 315
12.2 Using software to observe and record 317
12.2.1 Web proxies 317
12.2.2 Instrumented software 321
12.2.3 Custom-built software 324
12.2.4 Handling stored data 327
12.2.5 Keystroke and activity loggers 328
12.2.6 Analyzing log files 329
12.3 Hybrid data collection methods 330
12.4 Automated interface evaluation 333
12.5 Challenges of computerized data collection 333
Summary 336
Discussion Questions 337
Research Design Exercises 338
References 339
13 Measuring the Human 343
13.1 Eye tracking 344
13.2 Physiological tools 350
13.2.1 Physiological data 351
13.2.2 Challenges in data collection and interpretation 356
13.3 Examples of physiological research in HCI 359
Summary 361
Discussion Questions 362
Research Design Exercise 363
References 363
14 Working with Human Subjects 367
14.1 Identifying potential participants 368
14.1.1 Which subjects? 369
14.1.2 How many subjects? 371
14.1.3 Recruiting participants 373
14.2 Care and handling of research participants 376
14.2.1 Protecting participants 376
14.2.2 Informed consent 381
14.2.3 Institutional review boards 384
14.2.4 Potentially deceptive research? 387
14.2.5 General concerns 388
14.3 Online research 389
14.3.1 Appropriate topics for online research 389
14.3.2 Recruiting 389
14.3.3 Study design 391
14.3.4 Ethical concerns 391
14.3.5 Data collection 392
Summary 393
Discussion Questions 394
Research Design Exercises 395
References 396
15 Working with Research Participants with Impairments 399
15.1 Introduction 400
15.2 How many participants? 401
15.2.1 Small sample sizes 401
15.2.2 Distributed research 401
15.2.3 In-depth case studies 402
15.3 Proxy users 403
15.4 Multi-Population Studies 404
15.5 Recruiting users through community partners 405
15.6 Pilot studies 407
15.7 Scheduling users with impairments 408
15.8 Documentation for users with impairments 409
15.8.1 Human subjects forms 409
15.8.2 Research documentation 410
15.9 Differing levels of ability 412
15.10 Bringing extra computer parts 413
15.11 Payment 415
Summary 415
Discussion Questions 415
Research Design Exercise 416
References 416
Index 419
