Pioneering Spatial Design UI/UX and Storytelling
Shipping the first AR enabled NYT apps, and leading groundbreaking new approaches to visual storytelling through augmented reality and 3D web features.
Role
Director of Immersive Storytelling
Team Size
50+ (Cross-functional)
Timeline
Concept to Launch: 6 Months
As director of immersive storytelling at The New York Times, I led a multi-year effort to bring AR to the NYT apps, and produce dozens of AR and 3D web features.
The immersive storytelling team was founded and led by me, and this project went from concept to launch in just 6 months. It involved leading a massive project team of over 50 product designers, engineers, visual designers, journalists, and marketers to define a new grammar for news.
The AR Initiative
Inventing a Spatial Grammar
This initiative was the next chapter in a long tradition of visual storytelling at The Times — carrying decades of innovation in graphics, motion, and interactive design into product, and leaning fully into the future of spatial design and immersive experiences.
The work needed to feel uniquely Timesian, and feel like an evolution from our past work. It needed to respect the user — our readers — and be simultaneously familiar, innovative, and useful. And it needed to live up to the high bar for journalism set by the organization.
On the engineering side, that meant rendering real-time 3D and augmented reality reliably across a huge range of iOS and Android devices, keeping every experience performant and lightweight enough to ship inside a daily news app, and building authoring tools that let journalists and designers create spatial scenes at the pace of the newsroom.
And because these features lived inside the much larger New York Times apps, every interaction had to work within existing constraints — the established design system, navigation patterns, and reading experience — introducing new spatial UI and UX without disrupting the core act of reading the news.
Prototyping
Developing the Prototype
To start, we built a prototype. To pull this off at a legacy institution, we couldn't just use existing structures. Pioneering spatial computing required fundamentally rewiring how we built products—aligning engineering execution with an uncompromising design and editorial vision.
I led the architecture of a specialized, cross-functional unit of engineers, product designers, and journalists.
My mandate for this team was twofold: drive an uncompromising design vision that made complex 3D interactions feel entirely invisible, and establish a new editorial grammar where volumetric data wasn't a gimmick, but a rigorous, foundational tool for reporting the news.
We developed a prototype in Unity that would simulate the NYT app, and let us iterate on how the product would look and feel. I considered the project across three key dimensions:
Narrative Content
Shaping the story itself — what readers experience, and how spatial elements deepen the reporting.
Engineering
Rendering real-time 3D and AR reliably and performantly across a huge range of devices.
Interface Design
Introducing new spatial UI and UX within the constraints of the larger New York Times apps.
Our perspective was to maintain the feeling of a reading experience. It would feel and act like a regular article, but there would immediately be clues that it was new and different.
A frosted-glass look would define the article design, with the camera turning on from the start. This would give an experience of scrolling the words over a window rather than a page.
When an AR moment was scrolled into view, the “window” would suddenly clarify, and prompt a user to project the AR object into their space, and interact with it as if it were there.
Interactive moments would then give the user more interesting ways to learn more about the objects in their space.
Pillar 01 — Narrative Content
Breaking past the flat image
To align the newsroom on the ‘why’ of this project, we had to prove that AR wasn’t a technological novelty—it had to be a rigorous journalistic tool.
A traditional photograph flattens reality.
We realized that by breaking past the physical limitations of a 5-inch screen, we could use true 1:1 scale as a literal tool for understanding. By placing a volumetric object in the reader’s physical space, we transformed a passive graphic into an intuitive, humanistic interaction. We weren’t just working around the constraints of mobile;
We were actively establishing how visual journalism will operate in the era of spatial computing.
Pillar 02 — Engineering
Multi-platform Alignment
We orchestrated a multi-platform consensus—aligning engineering teams across Apple, Google, Unity, and Microsoft to architect a lightweight, performance-optimized 3D pipeline directly into our core native applications.
Lightweight solutions were the priority. We used SceneKit to develop 3D scenes on iOS, and worked with Google to create a custom version of RenderCore on Android. Later, we worked with Microsoft's engineers to layer in their volumetric video SDK.
Pillar 03 — Interface Design
Exploring new territories in interface design
To build an experience where the technology completely vanishes, we first had to navigate the friction. I led our interdisciplinary teams through these rapid prototyping cycles with one relentless objective: reduction. We had to take complex surface-scanning mechanics, permissions, and brutal mobile constraints, and rigorously edit them down until the interaction felt entirely human and intuitive.
In designing the AR window, we needed to balance AR and 3D web versions of the experience.
Initial UI
The inevitable interface
Reduced to the absolute essential. An interface that vanishes, allowing the spatial content to hold the user's complete focus.
Interaction through spatial movement
AR MODE TOGGLE
Our first designs focused on a self-contained AR module that would include a switch for moving between AR and 3D modes for the object, essentially pulling it out of the environment and letting you manipulate it on screen.
SCALE AND MOVE CONTROLS
We also experimented with controls that allowed a user to move and scale the object in space.
FINAL DESIGN
The shipped design was about maximizing simplicity and a UI that mostly got out of the way, while still providing the necessary functionality.
FROSTED WINDOW AESTHETIC
Key elements were getting the frosted window aesthetic dialed in just right, and editing and testing simple prompt language.
OBJECT PLACEMENT FLOW
Design of the object placement flow, which was done through phone movement only, eliminating all onscreen controls.
SCENE REFRESH BUTTON
Creating a simple way to refresh the scene if it was placed poorly, or any other issue with how it loaded.
SCALE TOGGLE
Designing a toggle switch for scale, removing all user scale controls and simplifying it to simply switch between real scale and tabletop mode.
ZONED INTERACTION
We simplified interactivity in the scene from hot spots you needed to reach or tap, as seen in the prototype, into a zone system, where the scene would detect where the user was in space and load different non-diegetic annotations and models to develop the story from that perspective.
The Launch
Hello, World.
Meet the "Honor Box."
We launched the feature with an article that explains to users why we were building an AR capability. This was also our first AR-enabled article.
It had a very simple "example" where users could project what is called an "honor box" or vintage NYT vending machine into their space: a bit of a wink showcasing our newest technology by presenting our oldest technology: print.
This initial launch was very successful, and we saw the honor box appear in living rooms and kitchens across the world, driving many new positive camera permissions in the new app, and generating a lot of organic sharing on social and in the media.
“… by using your smartphone as a ‘window,’ we are extending stories beyond the inches of a screen, by digitally adding objects into your space at real scale.”
Major Features
Following the launch, we produced a series of high-fidelity immersive journalism pieces. These projects set the standard for mobile AR, combining volumetric capture, photogrammetry, and spatial data analysis.
Four of the Best Olympians
The first full spatial experience we shipped was produced for the Olympics — after many rounds of ideation, we arrived at the idea to scan the top athletes competing that year using photogrammetry, creating an experience where you could project them at real scale into your space, and learn about their sport while they were frozen mid performance.
For each AR project we created a parallel 3D web experience that would leverage the same assets and storytelling. This put the user first, understanding that the AR experience only made sense when you had the space and time to engage with it. I feel this struck a good balance between our explorations of spatial storytelling, and the realities of the user experience.
David Bowie in 3 Dimensions
On top of this product development we were then able to create dozens of projects, and 2 solid years of innovative award winning visual storytelling using spatial principles across a variety of topics, like this culture project on David Bowie, looking at a new exhibit of his iconic fashion pieces. Here we could bring the museum experience to users in high fidelity.
We were obsessive about getting every detail right, recreating the experience of the material down to the way it sparkled or shined, developing custom shaders that would look highly realistic on mobile devices. We also created guidelines and a tech stack for capturing and processing real world objects through photogrammetry, which became a key supporting technology for our approach.
One Building, One Bomb
This Emmy award winning visual investigation of an attack in Syria uses a 3D reconstruction of the crime scene in partnership with Forensic Architecture. Here, in what could be perceived as closer to a VR experience, a user can walk around the roof where the bomb landed, and investigate the evidence themselves, navigating to "flags" indicating perspectives of interest.
Fully Immersive Experiences
We explored extending projects from AR mobile onto fully immersive headsets, like in this project that revisited the photography from the moon landing and re-contextualized it from where each photo was shot. You can actually hold the Hasselblad camera they used, and take the photos Buzz and Neil took from the exact spot each was taken.
Spatial Computing / Magic Leap
Designing the Spatial Article
To explore the future of immersive reading, we brought a specialized team down to Magic Leap's headquarters in Florida. Working side-by-side with their engineering and design teams, we investigated what a New York Times article might look like as a fully spatial experience.
We chose to prototype a breaking news event: the devastating 2018 volcanic eruption in Guatemala. We trained a stringer photographer on the ground to capture 700 precise photos of a truck buried in ash, which we then processed into a 1:1 scale 3D model using photogrammetry.
Launching as the very first news story available on the Magic Leap One headset, the design successfully mixed the traditional reading experience with 3D exploration, utilizing the physical space around the reader to convey the immense scale of the disaster.
A glimpse into the future of journalism, using the physical space around you as a canvas.
The Verge
The New York Times is defining what spatial storytelling looks like on Magic Leap.
Fast Company
Mixing breaking news with spatial computing to convey the immense scale of a disaster.
TechCrunch
A creative explosion in spatial storytelling
We visualized the invisible pollution around us, how water might have existed on Mars, used Microsoft's volumetric video SDK to bring celebrities into your space, and many more from which we developed a perspective and tremendous learnings on how to create engaging, effective, and inspiring work, leaning in to the future of spatial platforms.
"The Times displayed innovation in AR reporting under Roberts' leadership."
Advancing the Medium
I have had the privilege of sharing our learnings and strategies for immersive journalism at conferences around the world.
From defining the "grammar" of AR news to building cross-functional teams that can deliver 3D on deadline, the work we did at the Times helped set the stage for the spatial web.
