Mixed reality technology promises to merge our digital and physical worlds, unlocking new realms of experience and design.
Designers must consider how this will impact future products and experiences now, to identify the most desirable solutions and applications of this technology. The capability of software to simulate reality could fundamentally shift the way we produce and experience physical and digital products, allowing for simplified physical products that support digital experiences.
Until recently, mixed reality has primarily been used for entertainment. But it has tremendous potential as a design tool and as the next paradigm of consumer-product interaction. Mixed reality 3Dcreation software allows for rapid 3D design at true scale, which is impossibleto achieve with conventional 2D CAD programs. This can allow an industrial designer to test and prototype physical products faster.
I first read about mixed reality being used by automotive teams at Ford and Audi. Designers can save time by testing digital models at scale, instead of laboring over full-size clay sculpts.
Naturally, I wanted to test this new tech for myself, but I also had a realization. If everyone could see this digital car design, does the physical one need to change? Better yet, could I drastically simplify the physical design of the vehicle and allow software to handle the rest? This eventuality fascinated me far more than the initial goal. Augmented reality headsets may be clunky now, but so were car phones. It’s my job to think ahead.
Design Contexts
Technology defines many parameters in which we design products and experiences for people.
Consider how smartphones have influenced our products and our lives. We interact with the world through our phones. This context impacts how we design products. Designers must anticipate and understand that people expect products to function with/through their mobile devices.
The magic of mobile computing is its ability, through software, to transform into a multitude of different tools and products.
Consider the horde of “smart” products we have today. Physical and digital products are growing ever closer, yet we are still using them in a mutually exclusive way. Overcomplicating physical products with digital integrations in their current state is a result of technology outpacing our understanding of need. Mixed reality could do the opposite, leveraging software to simplify hardware.
As time passes, more of our product experiences can be handled by software. By embracing this, we can reduce the complexity of physical products as software capability increases. In a scenario where mixed reality technology is as ubiquitous as the smartphone is today, how could it free us to design better, more sustainable products with alternative, improved user experiences?
Designing for Mixed Reality
Mixed reality can digitally replace/simulate any 3- dimensional form or shape in real-time. This means we can separate the appearance of a physical product from its function, drastically simplifying it to make it more functional and cheaper to produce. The physical product can be designed to support a mixed reality experience which is superimposed.
In a mixed reality context, the purpose of a physical product is to provide a substrate for the digital layer that is cast over it.
A. Multifunctional hardware - layered digital interactions
If you sit in most cars right now, each function requires a physical control and feedback mechanism, be this the HVAC or headlights. While convenient, this demands a huge amount of design, manufacturing, and materials so that your favorite radio channel can have its own button... And people only ever want more features, which results in increasingly complex products. This is the classical industrial design model.
Tesla catapulted car interiors into the mobile age by redistributing all functions into software on a tablet. This is an efficient but inadequate replacement for the physical controls we’re used to, especially while driving.
With mixed reality, you get the best of both worlds. For example, you could design a physical interior with minimal inputs, say two buttons and a dial. A software interface between your mixed reality system and the vehicle determines the function of these inputs. Swiping a hand over the console changes modes and with it the response to physical input. Feedback and visual changes in overlaid UI graphics communicate the change in function. This solution has all the efficiencies of customizable software while remaining tangible!
B. Customizable visual product experience
The appearance of a product is no longer determined by its physical shape and materials, but by a software layer. The exterior of a car can look however you want, it doesn’t have to obey the laws of physics and look good, it can just look good. And the cost of creating this masterpiece is merely the time and effort of producing this digital asset. You don’t have to produce an actual Bugatti, you just need the skin of a Bugatti to wrap around your current vehicle or a blank slate. Suddenly a Bugatti is more attainable. The physical vehicle can communicate with mixed reality hardware to convincingly align with a digital avatar.
There is clearly a downside to not owning or driving an actual Bugatti, but the ownership pool for such a vehicle is so small and the resources it consumes so large that it’s worth asking do we really need them? Is the idea of one, the semblance of one enough? Is this too great a sacrifice for our future? We could make some impossibly cool virtual Bugattis to replace them…
C. The evolution of wearables - digital tangibility
How will we interact with the 3D digital world? Ideally, it should mimic the way we interact with our physical one. There are two present solutions.
The first is a physical controller, which provides satisfying tactile feedback and can use sensors to track inputs more closely. The downside is the user is forced to hold this controller at all times. For everyday life, we need our hands free.
The second solution is gesture input – using cameras to track the motion of our hands and interpret an input. This is a much more natural way to interact but loses the haptic feedback to our inputs.
An ideal solution would be a wearable controller that doesn’t impede regular hand use but provides a response when interacting with mixed reality interfaces/objects.
Impact
Increased Personalization
The physical product industry works tirelessly to create choice and stylistically appealing products at the expense of tremendous resources, to create artificial demand. This is excessive and profligate. Where personalization does make sense is in software. Software is easily altered, which is why so many modern products rely on it for personalization. Smartphones are a prime example, with relatively little hardware flexibility and infinite software customization.
Mixed reality applies this flexibility to our 3D world. We can personalize our physical products with software, changing their appearance or function via a connected app and firmware.
From a consumer perspective, this is tremendously appealing; adding new value to basic products, even ones you already own. Mixed reality could help break the needless cycle of physical consumerism without sacrificing the joy of having something new.
An additional benefit is a capacity for users to bring their personalized experiences with them. If the personalized layout of your vehicle interior is digital, your MR system can apply that same digital layout to a completely different vehicle, so long as it’s compatible.
Sustainability and the end of planned obsolescence
Simpler physical products are less expensive, last longer, and consume fewer resources. At a time when human consumption threatens the stability of our world, we must seek alternative methods of satisfying our needs. Planned obsolescence is the practice of designing and manufacturing products to last a limited amount of time, thus encouraging consumers to repeatedly purchase new products. Because mixed reality can superficially alter a product’s appearance and function, there is no need to build things that break or superficially redesign a product to encourage more sales.
Better Design
Mixed reality’s ability to simulate reality can be used by designers to validate their ideas in a reliable and efficient manner. An architect can determine very quickly if a floor plan will fit a property if they can physically see where it will sit in real life. Better visualization equals better realization.
No more screens
Mixed reality renders 2D screen-based interfaces obsolete. Think of the number of digital screens being produced and used that will suddenly be archaic. The quantity of resources that could be saved without every device requiring a physical display is colossal!
Mixed reality will finally allow us to stand up and operate in real space, no longer tethered to a rectangular portal. Apple has said time and again that technology is best when it is invisible. Mixed reality is omnipresent, there when it’s needed, invisible when it’s not.
Better use of existing products and spaces
Because of MR’s capability to alter our perception, it could be used to convert the functionality of objects and spaces for different users. A room’s use is determined by the people and things within it; being able to instantly transform a space for a different group necessitates fewer rooms. The same or greater productivity can be achieved with less proprietary equipment.
Conclusion
We humans can produce almost anything we can imagine, but our incessant cycle of consumerism doesn’t include or respect the impact it has on our environment. Recycling and sustainable production practices do help, but as digital capabilities increase, it’s worth considering how we can leverage them to ease our need for physical goods. Mixed reality offers us a chance to redistribute the way we produce and consume while improving our quality of life. We must think about this now before the technology is ready so we can develop it towards the future we need.
This paper is an extension of my Mixed Reality Design RIT thesis project, initiated in 2018. You can see more assets from the project, or download my book.
RIT Magic Spell Studios director David Long, as well as operations manager Brenda Schlageter and faculty member Mark Reisch were instrumental in providing access to facilities and support with mixed reality equipment. I used a provided license of Gravity sketch to create digital assets in VR and a Microsoft HoloLens for augmented reality experiments. This is an ongoing project. If you have thoughts, comments, applications of this material, or related opportunities, please reach out!