EXA: The Infinite Instrument + Magic Leap One
I’ve never felt quite so powerful sitting on the piano bench. For years, I have accompanied my own piano-playing with rhythms stomped on the floor, melodies sung aloud, and other parts heard only in my imagination. But now, surrounded by EXA’s musical instruments, I can bring all those other parts to life. EXA gives me complete control of a virtual band, floating just above my piano, ready to perform whatever, and whenever, I ask.
Today, I’m proud to announce the launch of “EXA: The Infinite Instrument” for Magic Leap One. My music-making app began its life in virtual reality, but Magic Leap’s Independent Creator Program provided the opportunity to unleash EXA into the Magicverse. This makes EXA more portable and versatile than ever before. Wearing the Magic Leap One, you can now bring your “infinite instrument” almost anywhere — from piano benches and couches to garage bands and live stages.
Building a virtual EXA band around a real-world piano.
There’s a lot you can do with EXA, but the basic idea is this: place musical shapes within your workspace, use the available tools to play those shapes, and capture your performances with the loop recorder. You can move and rearrange your shapes (called “ringers”) into any formation, choose from thousands of different sounds for each one, and assemble your recorded loops into a full song using the sequencer.
To create the “virtual band” experience, EXA features robot-like avatars that can recreate the motions you made while recording each loop. These avatars do a surprisingly good job of conveying the body language and energy of the performer, as they capture the exact ways you moved or danced or otherwise rocked-out during your performance.
Creating a simple song in EXA without ever leaving the couch.
Since 2007, I have worked as a development and design consultant via my company, Aesthetic Interactive. My mission has always been to bring the dev and design worlds together — crafting stunning experiences through exceptional software.
Along the way, I started working with spatial computing, and I haven’t looked back. I love exploring the endless possibilities of this new space, and have a particular interest in user interface, interaction, and experience. There’s something special (and especially challenging) about designing spatial systems — to be most effective, the system must be intuitive, efficient, visually interesting, and feel good to use, all at the same time
My work on EXA began in late 2016, and I have been improving and expanding the app ever since. EXA evolved from one of my earliest spatial projects, VR Guitar, a 30-string virtual instrument performed with 3D hand input. And EXA’s user interfaces are all built using Hover UI Kit, my open-sourced Unity toolkit for creating spatial interfaces.
Into The Magicverse
From the design perspective, converting EXA from VR to Magic Leap wasn’t too difficult. The app’s interfaces and content were already fully movable and customizable, ready for users to fit them into their real-world spaces. Of course, many changes were necessary to accommodate the new headset and the real-world —adjusting interfaces for the field of view, avoiding up-close interactions, keeping track of “floor level”, and so on.
The development side, however, took far more technical effort than I had anticipated. Relative to many other spatial apps, EXA is quite complex and feature-rich. The app also requires high performance and very low latency in order for users to faithfully perform EXA’s virtual instruments in real-time.
A virtual band performs “Robot Blues” in the living room.
Without going into too much detail, my development efforts generally fell into a few categories: building a native audio system for the Magic Leap OS, major refactoring and performance optimization using Unity’s new DOTS architecture, integrating Magic Leap’s various real-world-aware features, implementing a new “Learn” sequence for new users, and working to meet the various requirements of Magic Leap’s app store.
I suspect that much of this work would have been easier (or altogether unnecessary) if I weren’t converting an existing app — not to mention one of EXA’s size and scope. I certainly learned a lot during this process, though, and I’m ready to make use of Magic Leap’s features and Unity’s high-performance DOTS architecture in my future projects.
Stepping Into EXA
I’m excited to see (and hear!) what users create with the Magic Leap version of EXA. As a new user, you can jump-start your EXA experience by going step-by-step through the “Learn” process, which is available when the app first starts. The app also ships with several pre-made layouts, which each include custom instruments, recorded loops, and full songs ready to play — all available for you to watch, study, modify, imitate, and/or reuse.
You can use Magic Leap’s video-capture capabilities to record your EXA experiences and performances. I’ve found that recording directly via your computer (using mldb) works well, and there’s an option that ignores the microphone’s audio while recording.
Don’t be afraid to try unusual or “impossible” playing styles in EXA. You can design instruments to accommodate your preferred style, to prevent playing out-of-key notes, to match up with your dance motions, or even to perform a specific song more easily. EXA allows you to set aside the limitations of physical instruments, so you can let your musical creativity run free.
The virtual band performs “Upsound Jam” in the living room.
Starting today, EXA is available for free on Magic Leap World. I hope you’ll give it a try! I’m always willing to hear any feedback you have about the app, and can’t wait to see and hear your musical creations.
I’d like to thank Magic Leap for this great opportunity, and the team (Tadhg, Shane, Matt, Steve, and others) for their assistance along the way. I’ve learned so much from this project, and gained valuable experience working with all things Magic Leap. I’m looking forward to applying this to all my spatial computing projects in the years ahead.