Convincing interfaces – merging digital experiences with the physical world

As technology progresses at a rapid rate, we hear a lot about the physical world merging with the digital world, to the point where we struggle to differentiate between the two. Sitting at the centre of this idea is that of interfaces.

Interfaces provide the point at which we interact with digital products, and there are varying thresholds at which we become convinced that these digital products are no longer operating in isolation, rather they have fully meshed with the physical.

However, pinpointing these thresholds can be a tricky business. This was the focus of Haiyan Zhang’s talk at the Thing Monk 2016 conference this week in London. As Innovation Director at Microsoft Research, Zhang is very interested in figuring out the point at which people are convinced that their interaction with a digital product is the same as their interaction with a physical one.

This is particularly important when discussing the idea of the Internet-of-Things – if the plan is to connect up everything around us to the internet, merging the digital and physical worlds.

Zhang said:

What I find is that IoT is all things to all people. It’s become this fuzzy term. So I find it useful to anchor it – IoT references a infrastructure of networks, cloud computing and devices. So when we are thinking about of this network of things, at the endpoints of things are how these things will touch people’s lives. It’s through physical product experiences that merge technology with the physical world.

We are really talking about things, how we use things and how we interface with those things. One of the things I like to talk about is, when talking about things in the real world and how we interact with them, it’s very different to how we interact with technology traditionally. How can we develop technology to approximate the way that we naturally interact with the things around us?

Thresholds of truth

Zhang spoke about the idea that, unlike software engineering which is iterative, digital interfaces often have a distinct point at which they become more convincing to human physical interaction. These ‘thresholds’ are what need to be discovered for interactions with IoT to become perceivably a physical reality.

Zhang said:

This idea of technologies that help us perceive the real world through these thresholds of truth. When we think about interacting with the technology and how that comes close to interacting with things in the real world, we come across these magic numbers, these inflection points. These perceptual cliffs, when the behaviour and response time of a system is altered by just a small degree, but for some reason our perceived quality of that system improves dramatically.

As a software engineer, this goes against the thinking in computer science, where small changes are considered linear. We make small improvements to the system always. But when we come to design interfaces, there are points where you can make a small improvement and the user experience improves dramatically. It’s almost a cliff, where you go from disbelieving a system, to believing a system. Where disbelief is suspended.

For example, if we thinking about film and the rate of frames per second. Zhang explained that when viewing film at a rate lower than 24 frames per second, viewers are not convinced about the reality of what they’re seeing. However, at 24 frames per second, the perceptual cliff, reality is realised. Interestingly though, go to far the other way and the reality again becomes more distant. She said:

In the devices we are creating, in the interfaces we are creating, one of these small improvements that will make users believe in the reality of the object you are creating. In terms of visual of perception, frames per second, that comes around 24 fps. This was determined right at the beginning of cinema.

Peter Jackson’s the Hobbit, he filmed that at 48 fps, and a lot of people had trouble adjusting to this frame rate, because they didn’t believe in the reality of the film itself. Translating this to another aspect of human/computer interaction, is this threshold of belief of humanoid robots or avatars that try to be more lifelike. As we try to create avatars that are more humanlike, we find that people detach from them emotionally when they approximate human reality but don’t quite get there.

So there are all these thresholds for everything we are working on. It’s really a matter of figuring out what the threshold is for your interface.

Voice

chat-bot-robotZhang went on to say that as much as humans like this design concept of morphism, where digital artefacts resemble physical objects, there is a fundamental hurdle that needs to be overcome in order for people to subconsciously perceive technology as real physical things.

And of course, one of the up and coming interfaces in the digital era is that of voice and conversation. The likes of Apple, Google, Microsoft and Amazon have all been working hard to make their voice activated personal assistants resemble natural conversation. They want talking to your machine to be as natural as talking to another person.

However, Zhang points out that even though these systems currently have very high accuracy rates (94-95%), this is still below the threshold to make them convincing/useful for most humans. She said:

Of course there are all sorts of thresholds that we need to overcome in conversation. The first one being speech recognition. How many times have I used my personal assistant on my phone and it just doesn’t recognise the words I’m saying? It’s so frustrating.

In fact even if a system works 95% of the time, that threshold going from 95% to 99% makes all the difference in the world. People underestimate the difference between 95% and 99% accuracy. Once we hit 99% accuracy it will be a game changer.

My take

Interfaces are relatively easy when you’re talking touchscreen on your smartphone. When you’re talking natural language, voice interaction and chatting to bots – it’s a whole different ballgame.

Image credit – Robot customer service operator with headset and speech bubbles © kirill_makarov – Fotolia.com

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Scroll to Top