The Next Interface

When the first motorized vehicles appeared at the turn of the twentieth century, it’s rumored that early builders bolted combustion engines onto wooden horse buggies. They weren’t bad designers; they were exceptional carriage craftsmen who couldn’t yet imagine what an automobile could be, leaving whip holders on the dashboard even after the horse was gone.

Digital design is clinging to that same muscle memory today, caught in the transition to the AI era. We have models capable of real-time vision, voice, code generation, and complex reasoning, yet we funnel that intelligence into plain text boxes that channel the early 2000s.

This tension was the focus of The Next Interface, an afternoon at New York Live Arts where Gemini’s design team partnered with It’s Nice That to weigh in on an uncomfortable question: When software can generate interfaces on demand, what happens to the screen we recognize today — and what is our role in shaping it?

Walk into a room of designers today and bring up generative AI, and the prevailing mood isn’t necessarily one of unbridled awe. Often, it’s characterized by fatigue, guarded skepticism, and a quiet anxiety around finding a place in this new landscape.

Jenny Blackburn, a VP at Google DeepMind, and Google Fellow Matias Duarte recognize that unease. They suggest a simple reset: We aren’t here to design for AI; we’re here to design for people.

For Blackburn, that mandate is personal. Her father lives with an essential tremor and glaucoma, conditions that make touch screens exhausting rather than convenient. When he began using voice interactions with Gemini, however, it gave him back a sense of independence, identity, and control.

That’s what technology should aim to do: restore human agency. Duarte traces his initial awareness of this potential to the first time he touched a computer, realizing that the glowing screen could become literally anything he imagined. Over the next thirty years, he witnessed that infinite possibility get channeled into comfortable, predictable containers: buttons, menus, and form fields. We became so good at perfecting digital tools that we forgot the machine was meant to be boundless.

Wanting to push past that inertia and reclaim the computer as an open invitation, Blackburn and Duarte shared three shifts they are exploring for the next generation of Gemini interfaces

Traditionally, user interfaces box information into cards and grids to maintain order. But when a system understands what it produces in real time, the container can finally fall away.

Instead of dumping an unbroken wall of text into a chat window, such a system can compose dynamic editorial layouts on the fly: balancing typography, hierarchy, imagery, and negative space around a user’s intent. The content is no longer inserted into an interface; the content is the interface.

This fluidity extends to how we interact. In conversational UI today, every mode sits behind a button: Tap a keyboard to type, a microphone to speak, or a camera to see. But human interaction doesn’t work through toggle switches. When talking with a friend, you don’t switch into voice mode. Instead, you speak, gesture, and look where they’re looking.

Future interfaces can move toward that seamless flow, where the camera stops being a passive viewfinder you turn on and off. It becomes a shared canvas, where objects in view turn into clickable prompts and the surrounding environment signals intent without a single tap.

Communicating in real time demands a new visual feedback loop. Motion designer James Gundersen addressed this by asking what machine reasoning actually looks like.

Inspired by the purposeful yet organic momentum of bird murmurations, his team designed fluid particle clusters that pulse and gather around recognized objects. Instead of a mechanical loading spinner, the interface responds with living motion, confirming what the model sees and signaling that it’s actively thinking alongside you.

The best partners anticipate what you need. When software can understand context and take action, it becomes a proactive collaborator.

Imagine a future where, at the gym, raising your phone in front of workout equipment surfaces a rep tracker tailored to your routine. If a flight is delayed, an agentic partner suggests alternate plans to keep your day intact.

Designing for this future means releasing the instinct to choreograph every interaction. Instead of building rigid tools, the role of design shifts toward creating the conditions for genuine collaboration: embedding human judgment, editorial restraint, and cultural context into systems that can adapt in real time.

Translating this philosophy into software requires an intentional approach to teaching machines. With access to vast knowledge, AI models can easily default to excess, from conversational filler to visual overload. If software can generate anything, a designer can decide what the software shouldn’t generate, defining its taste and boundaries.

Gabriella Schwarz, model designer, approaches Gemini’s behavior through the lens of a former political journalist. In 2012, she was on the presidential campaign trail, transcribing speeches to extract what voters truly needed to know. Today, she brings that newsroom discipline to Gemini’s voice.

She and her team log real user prompts, catch failure modes, and personally edit hundreds of “golden responses” that set the standard for clear, empathetic communication. These examples form Gemini’s system instructions, an invisible editorial spine that helps train the model to self-edit and respect a reader’s time before a single word appears on screen.

Designer Justin Marx brought that rigor into Gemini’s Immersive Explorations, a format built for questions text alone can’t answer. When someone asks how a cathedral dome distributes its load or how a jet turbine functions, a vertical feed of text isn’t necessarily the best medium for exploration. Immersive View replaces the chat with an expansive spatial canvas, letting images explain the mechanics while language stays in the margins.

Generative image models produce visually striking output but lack understanding of how to clearly present information. They are natural flatteners, rendering every background blade of grass with the same sharpness as the subject itself. Marx’s team had to teach the system the art of selective focus: using tilt-shift optics, low-angle framing, and shallow focal depth so the eye immediately knows where to look rather than getting lost in uniform detail.

Photorealism presents its own issues. When an image looks too real, users naturally expect forensic accuracy. Instead, the team asked Gemini to build architectural dioramas, which still provide physical clarity without the heightened expectations.

To unify subjects as disparate as microbiology and ancient empires, all Immersive View images are washed in a warm “golden hour” light. This shared visual signature makes Immersive View visuals feel like a unified collection rather than an assortment of random generations.

Stress-testing a variety of prompts exposed interesting edge cases. A historical query created scenes with Athenians strolling past the Parthenon in ruins, nudging the team to write rules to reconstruct ancient environments in their living states. Questions about human anatomy produced uncomfortable visceral gore until the team redirected visuals toward dry surgical resins. And when the model took the golden hour directive so literally that it cast sunset rays in space, the team built in lighting rules that followed the native physics of each environment.

Whether choosing words or composing light, system instructions show that designing with AI isn’t about giving up control. It’s about embedding our judgment so deeply into the machine that it begins to align with our standards of craft.

There’s a running fear that when software generates UI, the designer gets pushed to the perimeter, wrangling prompts or giving visual sign-offs. Inside Google’s design teams, the reality is the exact opposite. Designers are operating as creative directors and makers, shaping the product with more direct influence than ever. When existing tools don’t fit their needs, teams are inventing their own workflows.

Anna Sera Garcia, who works on Gemini’s visual design system, did exactly this while working on typography. Static type specimens didn’t capture how a model generates an unpredictable response in real time. So, she built a set of internal stress-testing tools to visualize headline scales, tracking, and leading against live model output. This allowed the team to tweak optical sizes, weight transitions, and CSS rules on the fly, catching where the type felt off before a single line of code was written.

Designer Brett Barros brought that maker instinct to the browser. Frustrated by the lag between dreaming up an interaction and viewing it on a phone, he built a custom Chrome extension that sits on top of Gemini’s live builds.

Hovering over any element, he could inspect padding and offsets and use standard keyboard shortcuts to nudge spacing, swap icons, or round a corner radius. But rather than updating a disconnected artboard, the extension allowed him to edit the live application, generating clean code for engineers to review. They even included a “point and speak” feature, allowing him to talk directly to the model: tighten this, shift the focus there, pull the palette back.

That proximity to the codebase fundamentally alters how quickly a designer can move. On his commute to the office one morning, designer James Zanoni stumbled upon a Reddit thread venting about an annoying issue in the Gemini mobile app. Previously, fixing such an issue meant logging a ticket, waiting for sprint planning meetings, negotiating priorities, and hoping the update shipped in a month. Instead, Zanoni wrote the interaction fix using AI-assisted development tools and submitted a pull request to engineering before lunch. The fix was completed by the end of the day.

No single extension or custom plug-in completes the whole story, but the example points to a broader transformation unfolding across Google and the design industry at large: The distance between imagining an experience and implementing it is collapsing. Designers no longer have to wait for someone else to build their instruments. When you remove technical friction, design stops being a series of compromises, freeing designers to do what they do best: shaping thoughtful experiences for people.

Given these new ways of working, what happens to the human sitting at the canvas?

The panic over automation often treats generative AI like an image vending machine: insert a prompt and accept a prepackaged artifact. But as designer and artist Sanchit Sawaria demonstrated, leaving the model to its own devices reveals its creative ceiling.

A lazy, generic prompt defaults to predictable tropes, like a smiling cartoon sunflower strolling through sunny meadows. When software art directs itself, it averages out the internet and hands back familiar, unchallenging visuals.

Breakthroughs happen when the designer reclaims the art direction. Sawaria reimagined the same floral subject as an elegant, sculptural tulip. The difference between the two isn’t the rendering engine but the point of view behind the work.

Looking past the novelty of generative tools reveals a reassuring truth: The foundations of great work haven’t changed at all. Composition, color theory, typography, hierarchy, and taste aren’t bound to a single tool. A creative can move from charcoal to an image model without surrendering their artistic identity.

In this light, a prompt isn’t a shortcut to a finished piece. It’s simply an opening sketch and the raw output merely material waiting to be challenged and sculpted. The creative process has never been about shaving seconds off a workflow; it’s about discovering what you’re making in the act of making it.

AI becomes an invitation to push creative pursuit further. It allows the maker to bend the medium until it carries their intent, whether that means coding bespoke tools or building custom brushes. After all, true craft has never been about accepting whatever a tool hands you. What gives creative work meaning is intentionality and discernment, knowing why one variation works while thousands of others fall flat.

Moving past the horseless-carriage era doesn’t require surrendering design craft to algorithms. It asks us to bring our standards into this new frontier. Design has always been the craft of taking raw materials — wood, paper, pixels, and now AI — and shaping them into things that people use and love. There’s never been a more thrilling time to make things and never a more vital time to be a designer.

The images, video clips, and designs featured in this article represent internal research prototypes and early-stage development concepts, shown for illustrative purposes only. Screen visuals and user experiences are simulated, and video sequences may be shortened. Mention of potential features or demos does not guarantee final availability, and final product details may differ significantly.

Words: Maggie Zhang
Art direction: Beth Abrahamson
Project team: Anna Sera Garcia, Brett Barros, Gabriella Schwarz, James Gundersen, James Zanoni, Jenny Blackburn, Justin Marx, Matias Duarte, Sanchit Sawaria