Recap: User First: Amazon Case Study & Value-driven User Research

As User Experience enthusiasts we all care a lot about our users. But how can we further our research and learn what their exact needs are? For this meetup we’ve invited two speakers (local & international) who will help us learn more about how we can put our users first. Independent Senior Interaction Designer and UX Consultant Anouschka Scholten spoke about value-driven user research. Founder of Loop11 and pioneer of UX Research Shefik Bey shared findings of their Amazon case study.

Everyone who saw this month’s venue for the first time was amazed by the authenticity and beauty of the ballroom at Rockstart! The experience was even better with the royal snacks provided by our sponsor Loop11. We had a chance to make pretty good selfies in that room;)

Even surrounded by this breathtaking beauty and smells of the amazing food we haven’t forgotten why we’ve gathered and the evening was opened by Nadia who gave a warm welcome to everyone. This month we wanted to remind our members that we are looking for volunteers who would be willing to help us with our social media, writing blog posts or maybe you could suggest a company who could host our next event?

Anouschka Scholten independent Senior Interaction Designer and UX Consultant with 20 years experience showed us the importance of value-driven research for any kind of company: a startup or a big corporation. Lack of time, budget or tools are not excuses not to perform user research, because talking to users can help you not only to understand what are user needs, but also helps to find solutions to the problems and expose opportunities for improvement.  

“Observing and asking why makes you find out what people really do and need.”

Focus groups, surveys, remote tracking are tests where it’s very hard to ask “why”. Whereas usability testing is about observing behaviour, that’s why it is called a user-driven testing. It is very important to have empathy while observing user’s behaviour as it is the key function of usability testing to really see, hear and find out what moves your user.

Anouschka stressed that usability testing must be done in an agile way. The traditional usability test has a lot of drawbacks: no retest, done one time at the end of the design process to name a few. She proposed to do Iterative testing: to watch real 2-4 users to do real tasks with design, then revise, redesign and retest again in 1-4 weeks.

Also, it is important to remember that user research gives the best value when it is done by yourself and it is good to involve as many team members as possible. That’s why Anouschka organised mini-workshop so that the audience would learn the basics of user research methods: user checks, a hands-on agile usability test method.

During the workshop, Lauren kindly volunteered to be a user so that Anouschka could perform a user research session. Afterwards, the audience was split into 4 groups and had to brainstorm the problems that they have identified while watching user testing session. After voting on the main problems, teams had to think of the solutions to the problems, prioritise the solutions using Effort&Impact matrix and present to the crowd their result.

In the end, Anouschka shared that from her experience of work with corporations, it really helps to present any kind of findings, even user research results, using measurable metrics. PURE presentation of research results helps her to convince management to work on certain tasks that are critical.

Anouschka shared her slides with us.

After a short break, we remotely connected with Shefik Bey, founder of Loop11, who spoke about the time when they challenged status quo – think-aloud user testing.

Shefik started off by emphasising that empathy (why) is a key to creating products that resonate. Alongside the empathy, there should be target acceptance (what). He also explained the importance of user research by giving an example of a marathon runner who ran too fast and missed the directions where to run hence lost the marathon. The same analogy goes for product development – user research provides guidelines where the product should go or confirms if the team works towards the correct direction.

Loop11 started questioning the established status quo – incorporation of thinking aloud in user testing – because the preexisting research shows influence on natural behaviour. When users are asked to think aloud during user research sessions, they fixate on a screen by scanning everything and telling what they see, naturally they don’t do that. Also, they do fewer tasks and it takes longer to complete them, because people have very different abilities to multitask. Furthermore, the research showed that when people have to think aloud it increases mental workload which makes people do things that they wouldn’t normally do or they started missing things.

Because of the above Shefik with the team decided to test Amazon prime video using 3 different usability testings: no think aloud, think aloud and video recording study think aloud. The results showed very similar ratings of success between think aloud and no think aloud tests.

Also, interesting results were found when comparing net promoter score NPS (advocacy for the interface) and system usability scale SUS (series of questions to measure usability) as it showed the inverse relationship between the two. This could’ve been impacted by a several contributing factors such as incentive (generous incentive could influence user’s opinion) or branding (people recognised this famous brand hence gave more positive feedback that they’d normally do).

The main takeaway of the presentation is to be mindful of data and sentiment, encourage further research and augment with natural testing behaviour.

Also, Shefik suggested to listen to Charlie Munger’s speech on the psychology of human misjudgement. Well worth a listen!

After the presentation there were a couple of announcements:

  • Codemotion is organising the 1st Google & Codemotion Diversity Initiative! This helps to emphasize the goal of fostering a technology community that accepts and promotes diversity by providing 30 complimentary tickets to attend the Codemotion Amsterdam Conference. It includes (but is not limited to): people of colour, LGBTQIA+ people, women and disabled people: if you are eligible to apply. Please drop a line to and explain why do you want to attend to Codemotion!
  • Our member Celeste, who is UX Consultant at Backbase, advised that her company is looking for a Senior UX Designer. If you are interested feel free to contact her.


We’d like to thank our sponsor Loop11 for helping us to make this month’s event. We’d like to remind that Loop11 is a self-serve, browser-based solution for usability professionals, web managers and web designers to create their own online, usability testing projects to get the metrics needed to optimise website usability.

Also big thanks go to our venue sponsor Rockstart who love startups and give them rock-solid support in their first 1000 days.

Stay tuned for the June event!

We’ll see you soon!

Categories: Blog
Tags: , , , , , ,