Like any new medium, Mixed Reality is a minefield of UX, technical and aesthetic gotchas. Some are easily avoided with best practices, others require a generous helping of technical knowledge and some you just plain won’t appreciate until you’ve made the mistakes yourself. In this article we share the top 5 mistakes we’ve found from 2 years experience building and trying out HoloLens experiences – and hopefully how to avoid them yourself.
Mistake #1: Forget to onboard your user
If you’re building a web page you don’t need to explain hyperlinks, if you’re building a mobile app you don’t need explain touch inputs. But here in mid-2017, for 80% of people, their first time on your mixed reality app will be their first time on any mixed reality app, and without some basic orientation a surprisingly large number of people will end up wandering around the room, utterly lost.
The good news is, the vast majority of obstacles first time MR users are likely to face can be overcome with simple reusable UX tools like FoV guides, audio cues and interaction training. A lot of this stuff is covered in Microsoft’s design guidelines, and the stuff that isn’t can be gleaned from the sadly still quite small number of really good apps on the Windows Store like Fragments or Galaxy Explorer.
Mistake #2: Don’t optimise your content
Enterprise users of 3D rendering (CAD, CG Production, Datavis, etc.) have been able to remain blissfully aloof from the grubby world of optimisation. Unfortunately the bright dawn of the AR/VR revolution has brought with it a rude awakening as 1.5 GB CAD models are loaded on to headsets designed for mobility and comfort – and fall over immediately.
This situation has been somewhat exacerbated by the enterprise-first nature of the platform – people with cool problems to solve, but with assets and data that are usually totally inappropriate for mobile hardware (it’s a computer that has to perch on your head, give it a break).
Knowing the hardware, the engine, and the profiling tools that go with it is a good start. There are great first-principles docs on optimising for the Unity engine in general and for optimising for the HoloLens platform specifically.
If you don’t have a 3D background, or you need to go a step further, you can hire a team like Fracture to put their 3D specialist shoulders to the wheel, squeezing every last drop of performance out of the device. You might be here because you’ve seen some videos of our work like those for Helsinki Airport and Transport for London, both those projects are good examples of using tricks of the trade to do more with less. Eventually we’ll get round to doing some technical posts about them.
Finally, Microsoft’s recent acquisition of Simplygon – a middleware company that automates the optimisation of complex 3D models – shows they’re serious about creating an as-a-service pathway for getting the super-heavyweight assets out of their enterprise boxes and in to the head mounted displays of tomorrow.
Mistake #3: Ignore optical limitations
The HoloLens is without doubt an impressive piece of hardware and software engineering. It combines decades of optical display and computer vision research, and is probably a good few years ahead of most of its publicly announced competitors. That said, if you’re not at least aware of its optical limitations it’s surprisingly easy to make an unholy mess of things:
Field of View
All head mounted systems, whether they are mixed-, augmented- or virtual-reality, have a limited field of view (the area in front of your eyes that digital content can be rendered into). In the case of the HoloLens, that limitation becomes more pronounced than in the case of VR, because your eyes can still see all of their natural 130-150 degree range, rather than only seeing what the VR device is showing them.
Unless we start making waveguides out of diamonds, this issue isn’t going anywhere, so what are we supposed to do about it? Well for one thing, follow both Microsoft’s and other’s design guidelines. Think about the scale of objects, their placement in relation to the user, and how to help your user find objects when their peripheral vision doesn’t work as well as usual (see point 1), but most importantly, think about adding objects to the world, not replacing it (see point 5).
HoloLens (and basically any other mixed reality system on the cards in the next 10 years) renders additively, meaning it adds light to what your eyes are already about to receive. It’s therefore impossible to accurately render very dark colours or black tones as this generally corresponds to removing light, rather than adding it. While white or near white objects look solid and crisp, darker tones look washed out and fuzzy.
The easiest way to avoid this is to approach your pallet sensibly. Choose tones with higher values and use white when you absolutely need 2D or 3D objects to look as solid and readable as possible (e.g. text and UI). When non-dark tones are not an option, we can employ an optical illusion known as the checker shadow, where it’s possible to create a perception of ‘black’ merely by contrast with the presence of closeby white.
Distance and Realism
A last point, which gets less attention, is that the HoloLens, like any display that creates the illusion of 3D via stereoscopy, struggles to display things very close, or very far away. Oliver Kreylos’ great series of articles on this subject is far more comprehensive than you’ll find here, as is Microsoft’s high level video (right)
but the key point is this: your user’s lizard brain wants to perceive your holograms at a 1-5m distance, if you try and put them 50m away, things will start to get weird
Mistake #4: Forget about reality
Reality. You know – all this boring stuff around you? In mixed reality, that stuff is one half of the experience you’re building, the holograms you add to it are the other. So doing a bad job of integrating the two is a guaranteed way to make your app make no sense at all.
Letting holograms that represent real things penetrate through real objects immediately stops your holograms from feeling real. A more subtle point, but just as important, because the HoloLens renders additively, out of the box they will render in front of everything, including real objects that are actually between them and your user’s eye. This is such a ubiquitous problem with mixed reality systems that Microsoft’s built in spatial mapping systems have the ability to run in a mode that lets real objects partially occlude virtual ones. You can also use different visual effects to show the parts of an object being occluded, such as the ones in the image below.
Of course, the occlusion will only be as good as the accuracy of the spatial mapping. HoloLens’ ability to track regular geometry like walls, tables and floors is excellent, but when it comes to more organic forms and especially ones which are dynamic, even specialised hardware like Meta’s struggles to fool our highly attuned human vision:
Eliminating all occlusion is nearly impossible, but structuring your content in a way that is more likely to avoid it is a great start.
The most sophisticated approaches such as those used in Fragments actually use a technique called spatial understanding to analyse the space around the user first, then place content in a way that avoids interpenetration and occlusion, and make the holograms feel even more situated in the real world.
Mistake #5: Pretend Mixed Reality is just Virtual Reality
This last one is sort of a catch-all for the rest. Perhaps the worst mixed reality experience is one that tries to pretend it’s a virtual reality one, in all its 360 immersive glory. MR experiences that try to replace the entire world with a high detail virtual scene:
- Are likely to perform badly due to overdraw, realtime lighting or lightmap memory, etc.
- Are the pathological case for MR’s current optical limitations, particularly FoV and additive rendering of dark tones
- Obviously ignore reality, gleefully drawing a 3D scene over the top of 3D reality
In fact the only one of our previous tips you’re missing here is failing to onboard the user, but hey if you’re doing all this, you’re probably doing that too. In for a penny, in for a pound!
We hope these tips will at the very least make more people aware of the most likely pitfalls when they start developing their own Mixed Reality applications. The medium – like any other – is capable of providing amazing experience and absolutely terrible ones, by following the advice above, we can’t guarantee the first, but we can hopefully help you avoid the latter, and save you a few of your own ‘learning experiences’ along the way.
Don’t forget to subscribe to our mailing list for more articles on MR development.