AI Agents x Digital Product Design

The disruptive impact of AI is felt across all domains. It already feels surreal when a computer seems to ‘understand’ what you're prompting. However, the new capabilities of multimodal AI are mind-blowing. The sheer fact that you can natively talk to an AI model, ‘show’ your surrounding to it with a camera, and literally ask what you are looking for is groundbreaking. With such disruptive nature comes the question: how will AI change our interaction paradigms with digital services and products?

While many technical innovations and aspects can be attributed to the disruptive nature of AI, the concept of AI agents has the biggest impact on interaction design paradigms. Put simply, an AI agent is a piece of software that can act autonomously. A key feature is their ability to interact with their environment, meaning they can receive inputs from their surroundings, make decisions, and perform actions to achieve specific goals. In practice, AI agents can range from simple software performing repetitive tasks to complex systems using several AI models to perform a complex set of tasks. For example, ChatGPT by OpenAI is, by definition, an AI agent that makes use of different AI models to understand text, images, or voice. Former Microsoft CEO Bill Gates is so optimistic about the concept of AI agents that he stated at the end of 2023:

„[AI] Agents are not only going to change how everyone interacts with computers. They’re also going to upend the software industry, bringing about the biggest revolution in computing since we went from typing commands to tapping on icons.“ (Gates Notes)

Although that seemed like a pretty bold statement, the transformation is tangible now. The latest flagship models by OpenAI and Google showcase how AI agents are reshaping our interactions with digital products.

Let's start with a quick look at the different levels of current interaction paradigms (Fig. 1):

1. You have a need, like finding a train route, checking your bank account, or planning a physical training session.
2. You pick up your phone.
3. You decide which app or website to use to meet your need.

All of your decisions are highly influenced by the design, functionality, and your personal brand experiences with the digital service. AI agents, however, move right into the sweet spot between you and your services. As AI agents rapidly advance in multimodal comprehension of inputs (text, voice, visual, spatial), the way you interact with your digital services is undergoing profound transformations. Instead of navigating through a predefined user experience by tapping on menus, lists, links, and buttons, you can simply instruct or speak to your device: “Show me train connections from here to Berlin for next Monday, arriving in the late afternoon.” And because AI agents possess enhanced reasoning and creative capabilities, they produce responses that more closely mimic human outputs.

In simpler terms, interacting with an AI agent is becoming increasingly natural and intuitive.

But as these agents demonstrate their capability to autonomously understand, address, and fulfill increasingly complex tasks, people will increasingly rely less on traditional interactions with websites or apps. We will hand over our decision-making and control of the data level to the AI agents. They will autonomously decide which service to use, which information to show, and which decisions to make to satisfy your needs.

Looking at the current landscape of operating systems for consumer mobile devices, desktops, laptops, and tablets, there are only three major tech companies dominating our world: Alphabet, Apple, and Microsoft. And all three are strongly moving towards the implementation of AI agents at the core of their products:

• Google has unveiled its Gemini AI ecosystem with different models, like the efficient 'Nano' model, aimed at enhancing on-device tasks within its existing mobile operating system. Additionally, Google believes AI is the future of search.
• Apple has released only incrementally AI-improved tools, such as photo editing and image recognition. However, rumors hint at a future reveal of its full AI strategy (and a potential cooperation with OpenAI) at the Worldwide Developers Conference on June 10.
• Microsoft has made significant investments in OpenAI and leverages its AI technologies to enhance its own products and services, such as Azure and Office. This collaboration allows both companies to remain leaders in the field of artificial intelligence and to jointly develop innovative solutions.

Despite these advances, OpenAI has started to carve out a unique position within these ecosystems. Adding to its existing apps, OpenAI introduced a new desktop application (as of May, initially for Mac only) that allows users to engage in voice conversations directly from their computers – with promises of new audio and video capabilities in the near future.

It is designed to integrate seamlessly into anything you’re doing on your computer. (OpenAI)

ChatGPT’s versatility and multimodal capabilities position it as a potential gateway to your digital world, seamlessly integrating into daily computing activities and challenging traditional app-based ecosystems. Its natural interaction style not only complements but also extends beyond the existing frameworks, making it feel like a natural portal to many digital experiences.

As all the dominant tech companies push forward with the integration of AI agents, the digital product and service landscape will undergo massive changes. These changes bring about a range of questions and challenges for digital product teams and organisations.