Need Assessment & Usability Evaluation

"Samsung Harman's In-vehicle Infotainment System (SHIIS)"

 

Overview

Researcher     Seungjoo David Baek, Raden Tonev, Kaiwen Sun, and Cathy Chow
My Role         UX Research Consultant
Duration        Jan 2017 - Apr 2017 (15 weeks)

Goal               To conduct needs assessment & usability evaluation of
                       Samsung Harman's in-vehicle infotainment system (SHIIS)
Method         Interview, Comparative Analysis, Survey, Heuristics, and Lab-usability Test

Supervisor     John Hanna (Senior UX designer at Samsung Harman)
Location        University of Michigan School of Information / Samsung Harman International

Keyword        Car Infotainment System, Interaction Map, Comparative Analysis, Survey, Interview,
                     Heuristic Evaluation, and Onsite Usability Test


                                                               (Key findings are not presented due to NDA with Samsung Harman)


1. Background
2. Introduction
3. Research Methods

  • Interaction Map

  • Qualitative Interview
  • Comparative Analysis

  • Quantitative Survey

  • Heuristic Evaluation

  • Onsite Usability Test

4. Finding (NDA-protected)
5. Final Presentation (Samsung Harman International)


1. Background

I dedicate all of this research to my deceased friend, Hong-jun, whom I lost in a Moroccan dessert in 2007 due to his unfamiliarity of the in-vehicle UX while driving at night. I made best use of all my knowledge and experience that I acquired as an amateur circuit racer while conducting this UX research. 

Mobility is an everlasting human need. And driving a vehicle is a 150-years' user experience for us, Homo Velocidepes. We are getting into a new age where mobile software is being incorporated with machinery hardware in personal vehicle UX. I believe this research helped me glimpse over that future.

 

2. Introduction

Brief Description of SHIIS

0_infotainment3.jpg

Samsung Harman's In-vehicle Infotainment System (SHIIS) allows users to interact with in-vehicle information and entertainment functions. Thus, target users of the system are typically drivers and passengers. User can connect their smartphone to the system to access functions.

 

 

Research Goal & Question

The purpose of this UX research is to conduct Needs Assessment and Usability Evaluation of SHIIS. 

  • What is SHIIS? & How it works?
  • To which extent does SHIIS satisfy UX design theories?
  • To which extent does SHIIS provide user-friendly interaction?
  • How can we improve and re-design SHIIS through the UX research data?

 

Research Process

To summarize, we examined SHIIS interaction onsite, and compared the product with those of competitors.   Based on this, we extracted some usability research point to further examine. 

First, we listened to the voice of potential user group. This data helped us generate quantitative question.   The synthesis of qualitative and quantitative was followed.   We also examined the SHIIS on our own based on Nielsen's Heuristic Criteria.   All of the five research data help us how to design and conduct the final Usability Test.

At last, the six research outcome went through a set of thorough and iterative validations across each research, and in doing so, we finally made a design improvement suggestion.

 

My Role

I conducted all the six user research on a team-basis effort. And I specifically led the whole process and analysis of qualitative interview. 

 


3. Research Methods
 

3.1. Interaction Map

In the beginning, we visited Samsung-Harman International, and thoroughly analyzed SHIIS' interaction and created the whole interaction map.

(The map partially disclosed and intentionally blurred here.)

Screenshot 2018-03-04 15.26.50.png

 

3.2. Qualitative Interview


Introduction

We conducted in-depth semi-structured interviews for two groups: 

1) Stakeholder : Samsung Harman UX Team
: Onsite visit, in-person
: To discover what UX values does the UX team deliver in their car infotainment system.
 

2) Potential User Group
: E-mail recruitment, Onsite visit, in-person
: To explore users’ general smartphone experiences with car infotainment system, including driving behaviors, goals, motivations, and problems, and phone behaviors in the car.


The purpose of this approach was to reveal:

  • Frame of user experience with In-vehicle infotainment system.
  • Dimension of current design of in-vehicle system.
  • Focal usability issues for further investigation.

 

Method

  • Two different sets of interview questions
     
  • Semi-structured & 60 minute for each session
     
  • Field note-taking & Audio-recording for analysis
     
  • Thematic Analysis & Affinity Diagram
     
  • Interviewer Attitude
    Interview is to mutually "generate" the relevant qualitative data within the specific scope of research topic and question, and this practice requires active and voluntary participation to create in-depth communication between interviewer and interviewees.
     
  • Power Relation
    The interview is conducted between two parties on equal power, which signifies that interviewer are not to pose any interrogative attitude towards the interviewees, and that interviewees' the right to avoid, reject, and discard are protected.
     
  • Pre-conception & Anti-objectification
    Interviewers are required to write down any pre-conceptive flow of thoughts regarding participants. This practice will help the interviewer cease per-judgement on the interviewees and their statements, and not limit their answers, and not produce any leading questions to the topics and the inquiries set by the interviewer

 

Interview Question (Samsung Harman UX Team)

Interview_Question_UXTeam.png

 

Interview Question (Potential Users Group)

 

Participant Recruitment (Potential Users Group)

An E-mail was sent to all employees at Samsung Harman International requesting interviewees. Four interviewees were selected after consideration of job titles, demographic characteristics and our meeting schedule. We sought to interview a variety of drivers across gender, age, and job title.

This specifically-targeted recruitment was due to the two main issues.

  • Car Infotainment, a Mundane Daily UX
    Before conducting interview, we did a pilot test with our interview questions. However, it turned out that car infotainment UX is a mundane daily UX, so that lacking users' attention and interest in general. All the pilot participants did not even sketch how their own car infotainment system looked like.
     
  • Non-Disclosure Agreement Issue

 

Data Analysis

The interview data was analyzed with a thematic analysis strategy, which is widely used in the field of qualitative data analysis (Braun & Clarke, 2006).

We held interpretation sessions where each interviewer used their notes and transcripts to share the content of the interview to the rest of the team.

Team members asked questions and extracted meaningful statements, taking caution to be true to the interviewee’s words to avoid the potential error of manipulating the data (Lincoln & Guba, 1985).

 

Findings

(Key findings are not presented due to NDA with Samsung Harman)

 

Persona & Scenario

We synthesized the interview data and created three personas to represent the potential users.

 

3.3. Comparative Analysis

Introduction

We synthesized the data from 1) SHIIS interaction map analysis, 2) the Interviews with Samsung Harman UX, and 3) the Interview with Potential User Group.   Then, we framed the design and the UX requirements of the current in-vehicle infotainment systems pursue.

 
3_ComAnalysis_1.png
 

 


Upon reflection to the design and UX requirements, we listed down SHIIS's competitors in three tier groups as shown below.

 
3_ComAnalysis_2_TierGroup.png
 

 

Research Question

  • Who are the potential competitor to SHIIS? and Why?
     
  • The are the similarities and the differences between them?
     
  • How do competitors differ from Harman in terms of functionality, usability, safety, and target population? 
     
  • How can Orinoco be improved, so that it can compete with other products? 

 

 
Screenshot 2018-03-06 20.59.24.png
 

 

Findings

(Findings are not presented due to NDA with Samsung Harman)

 

 

3.4. Survey

Introduction

After completing interviews and a comparative analysis, our team conducted a survey to explore users’ phone and driving behaviors, demographics, and infotainment system preferences.

  • Target Audience : People who interact with infotainment systems in the car

  • Tool : Qualtrics

  • Responses : 154 total people

  • Recruitment : Social media and E-mail

 

Research Question

Using our findings from our previous data synthesis, we developed research questions that would:

  • Determine the prevalence of the interview findings,
     
  • Answer questions that the interviews could not answer in interview,

 

Method

  • Brainstorming Questions
    After defining our research questions, we generated 20 questions related to our research questions. Finally we created four categories to logically organize them:
     
    • Information about their car and driving
    • Phone use in the car
    • Infotainment system use in the car
    • Demographics
       
  • Pilot Questionnaire & Refining Phrasing
    We decided phrasing, answer formats, and choices each question should contain. We created our initial survey using Qualtrics, and had two friends take this pilot questionnaire to make sure if the questions made sense, and we used their feedback when creating our final questionnaire.
     
  • Distributing the survey
    We primarily targeted people over the legal driving age, which is 16. We then distributed our surveys through friends, family, and Samsung Harman on social media and In-mail.
     
  • Analyzing Survey Results
    154 total people completed the survey, and we compared the results to our interview findings. Using Qualtrics data analysis tools, we did cross tabulations and compared graphs for similar questions in order to generate our findings.

 

Findings

(Findings are not presented due to NDA with Samsung Harman)

 
4_Survey1.png
4_Survey2.png
 
4_Survey3.png
4_Survey4.png

 

3.5. Heuristic Evaluation

Introduction

Keeping in mind our findings from our interviews, comparative analysis and survey, we conducted a heuristic evaluation to point out the system’s major flaws and usability issues on not only the overall system interface design, but also specific functions such as media, phone, and projection mode.

 

Research Question

  • What aspects of the system already function well?
     
  • What aspects of the system could use improvement?
     
  • How can we apply what we learned about mobile design elements and people’s phone usage habits when evaluating and making recommendations for SHIIS?

 

Method

In order to answer our research questions, our team used Nielsen’s (1994) usability heuristics as fundamental principles to conduct this heuristic evaluation on SHIIS's interface and functions.  

 
Picture1.png
 

 

The scale of severity that we used to rate various usability issues is as follows:

  • 0 = Don’t agree that this is a usability problem
  • 1 = Cosmetic problem
  • 2 = Minor usability problem
  • 3 = Major usability problem; important to fix
  • 4 = Usability catastrophe; imperative to fix

 

Findings

(Findings are not presented due to NDA with Samsung Harman)

 

3.6. Onsite Usability Test

Introduction

We conducted onsite usability test to assess users’ understanding of the system and identify any usability issues, at Samsung Harman International.

 

Research Question

We designed the usability tests with the following research questions in mind:

  • Are users able to navigate the infotainment system quickly and locate settings and music?
     
  • Are users able to enable projection mode easily?
     
  • Is the connection process between phone and system intuitive?

These questions are important because if the most frequently used infotainment features are not intuitive, drivers are likely to pay more attention to the infotainment system than the road, resulting in an unnecessarily unsafe driving experience.

 

Method

  • Participant Recruitment
    To honor our non-disclosure agreement, we recruited solely from Samsung Harman’s Novi office.
     
    • U1 : a 57-year-old male senior quality engineer
    • U2 : a 49-year-old male senior software engineer
    • U3 : a 52-year-old female director of engineering quality and operations
    • U4 : a 57-year-old male security requirements manager
    • U5 : a 40-year-old male HMI senior software engineer
       
  • Pre-Task Questionnaire
    We created a brief Pre-Task Questionnaire to learn about users’ demographic details and experience level with the infotainment systems, and mobile operating systems since this could influence how easily users would complete the tasks.
     
  • Task Instructions
    We then created five Task Instructions. From our interview and survey data, we had learned that the most-used infotainment system were music, navigation, and phone calls, so we designed three tasks around these three functions.
     
  • Consent Form
    For the user to read and sign so they understood the study, their rights, and to obtain written consent before participating.
     
  • Conducting the onsite Usability Tests
    We printed our pre-test materials and conducted all usability tests at Harman’s Novi office.
 
Picture1.png
 

 

  • Data Logging Spreadsheets
    We identified the steps needed to complete each task and made them the success criteria for each task, putting them into each task’s data logging form.
     
  • Post-Task Questionnaire
    We quantitatively assessed user impressions of the system after finishing the tasks.
     
  • Debriefing Protocol
    The moderator and user could have a brief discussion about the user’s thoughts, difficulties, and satisfaction.
     
  • Analysis
    Each data logger and moderator explained to other team members what happened during their test and what the users said during the debriefing.

    We also aggregated the data from the post-task questionnaires. From these sessions, we found patterns between users, and noted when users made the same errors. Using this information, we extracted key findings which are detailed in the next section.

 


4. Finding

(Findings are not presented due to NDA with Samsung Harman)


5. Final Presentation

The final UX research analysis of Samsung Harman In-vehicle Infotainment System was presented before 12 UX professionals who developed the system.