Siri Gets Smarter | Apple Unveils ‘AI for the Rest of Us’ at WWDC 2024
Apple kicked off its annual Worldwide Developers Conference with its usual prominent keynote address highlighting the next major releases of iOS 18 and the rest of its operating systems. However, the high point of the show that was saved for the end was the company’s much-anticipated AI initiative, dubbed Apple Intelligence.
That’s a play on words we were expecting, in keeping with Apple’s penchant for branding things with its unique spin. There have also been enough rumors to give us an idea of where Apple was going with its new AI; however, the presentation highlighted some key points to the company’s strategy and how it’s unique from what most of its competitors are doing in this space.
Apple made it clear that Apple Intelligence is “AI for the rest of us,” emphasizing its focus on privacy and personalization while demonstrating the specific ways in which it will improve Siri, surface and summarize important information, and create fun new interactions. Read on for X things Apple had to say about Apple Intelligence during its WWDC Keynote.
Siri Gets Smarter
Perhaps the most significant part of Apple Intelligence is the one thing we’ve been awaiting for years — significant improvements to Siri.
It’s ironic how far behind the curve Siri has fallen, considering that Apple’s voice assistant beat Amazon’s Alexa and Google’s Assistant to market by at least three years. That’s not to say that Apple’s competitors are fantastic by any stretch of the imagination, but there are some significant ways in which they’ve lapped Siri.
The good news is that Apple seems to have pulled out all the stops this year in its determination to fully realize its original vision for Siri. If today’s demo is anything to go on, the voice assistant will become much more competent at recognizing what we’re saying and letting us engage in more natural conversations.
This includes things like being able to continue a conversation with follow-up questions, such as scheduling a trip somewhere after asking about the weather or getting directions to a location that you’ve just looked up on Wikipedia. However, Apple Intelligence can now dig deeper into your apps to let Siri piece together relevant information from multiple sources, so you could ask Siri when a friend is arriving on a flight, and it will automatically find the travel confirmation in your email and then look it up in a flight tracking app. You could then follow up by asking it to arrange an Uber to take you to the airport.
New “App Intents” will also let you ask Siri to do things inside your apps, such as enhancing a photo in the Photos app or adding someone’s address from Messages to Contacts with a simple and natural request. That last part is thanks to a new on-screen awareness feature that will let Siri analyze and use information that’s already on your screen.
Searching will also become much more natural, with the ability to filter down to things that were previously impossible. For example, you can ask Siri to show you pictures of your daughter outside in a red dress to get just the images you’re looking for.
On-Device Processing and Privacy
A crucial part of Apple’s push into AI is making the experience far more personal than most other AI bots and systems can accomplish. As Apple’s Craig Federighi said, many AI systems know about the world’s information but don’t know much about you. That’s probably a good thing, considering the potential for personal data to be misused. Still, Apple’s goal is to allow Apple Intelligence to leverage as much information about you as it can by doing it with the utmost privacy and security.
As with nearly every other machine learning feature Apple has introduced in the past decade, Apple Intelligence will run its algorithms on-device as much as possible. That means it can pull in data from photos, email messages, iMessage conversations, and just about anything else on your device without any fear of that data leaking out because it never leaves your device.
Of course, even with Apple’s newest and most powerful M4 chip, some things will be beyond the ability of Apple silicon to handle. For those requests, Apple has built “Private Cloud Compute,” a system that securely and temporarily uploads only the information specifically needed to process the request to servers entirely managed and controlled by Apple and independently verified by third-party auditors.
Personal Context
The privacy of Apple Intelligence allows a much greater wealth of personal information to be available to its machine learning algorithms, which enables it to offer an unprecedented level of personal context that will tailor your responses to your needs.
This should solve issues like asking for directions on Apple Maps and getting sent to a location hundreds of miles away since Siri and the Apple Intelligence engine will now be able to use your current location more effectively. However, it goes beyond that to tying together disparate pieces of information scattered around your iPhone to give you the answers you need without having to ferret out those details yourself.
During the presentation, Kelsey Peterson, Apple’s Director of Machine Learning and AI, showed several examples of how Apple Intelligence can respond to requests for finding books that a friend recommended (searching across email and messages), automatically filling a driver’s license number into a form (based on pulling a driver’s license from photos and reading the number from it), and conducting an entire conversation with Siri to find out when her mom’s incoming flight lands, what plans they’ve made for the day, and how to get from the airport to the restaurant where they’re having lunch — a request that tied together information from Mail, Messages, Calendar, flight tracking, Maps, and more.
Improving Your Writing
Apple Intelligence will also help you improve your writing using on-device AI, with the ability to grab a paragraph of text in nearly any app and pop up a set of tools that will let you rewrite it, changing the tone to make it more friendly or professional, proofread it, or even get more creative by making an AI request like asking a block of text to be rewritten to sound like a poem.
It will also be possible to summarize a long block of text to provide a “TL;DR” synopsis or view a summary of a lengthy email. Along the same lines, Apple Intelligence will covert email previews in your inbox into more helpful summaries that let you tell what the messages are about at a glance rather than seeing only the usual introductory preamble.
There’s also a Smart Reply feature that will not help you craft a response to a message but analyze the original message and prompt you with some questions to fill things in. For example, when replying to an invite, Smart Reply will ask whether you’re attending alone or bringing your partner with you and write the response accordingly (e.g., “We’ll be there” instead of “I’ll be there”).
Priority Notifications
In the same way that it can summarize your email inbox, Apple Intelligence will also take care of your notifications by not only helping you see what’s important about them but also filtering them so the critical ones are at the top.
A new “Reduce Interruptions” mode also provides a more intelligent “Do Not Disturb,” allowing those notifications that Apple Intelligence deems essential to get through while muting the rest. This goes a step beyond Focus modes, which let you block or allow specific senders; instead, Apple Intelligence will consider the content and context of the notifications. For example, a partner’s routine “hi” message might be filtered out, while one that reminds you to pick up your child early from daycare would come through.
Genmoji
Seven years ago, Apple gave us “Animoji” with the iPhone X and its TrueDepth camera. Now, it’s taking things up a notch by using generative AI to create custom “Genmoji” for any occasion.
Typing in a descriptive phrase will create a new emoji on-demand that can be shared in Messages or used as a sticker or a tapback. Since these custom emoji won’t be part of the Unicode standard, don’t expect them to work too well beyond Apple devices; they’ll likely work the same way as Memoji does now, which includes using them as stickers or even tapbacks in iOS 18. Like Memoji, it will presumably be possible to send them in graphics via email or other messaging apps.
Image Playground
Apple will also be leveraging its generative Apple Intelligence to help you create unique images through a new feature it’s calling “Image Playground.” This will be built into Apple’s key apps like Messages, Mail, and Notes, but also available as a standalone app for those who want to generate images to save and share elsewhere.
With Image Playground, you’ll be able to combine several ideas to generate an image. These can be abstract concepts like “beach” and “dinner party” or even include specific contacts to create AI-generated images that feature appropriately caricatured depictions of them. Naturally, Apple won’t use this to generate photorealistic images of people; the examples shown were all cartoon-like renders.
The technology behind Image Playground also extends to a new “Image Wand” feature in Apple Notes that will make it even easier to generate a high-quality image from an existing drawing or even the context of the note itself. By circling either a hand-drawn sketch or a blank space, Apple Intelligence will use the image generation features to provide an appropriate illustration of the concept.
Photo Enhancements
Google beat Apple to the punch by about three years, but iPhone users will be happy to know they’ll no longer need to turn to Google Photos to erase background objects from their images. iOS 18 introduces a new “Clean Up” feature that’s basically the same thing built into the photos app.
However, that’s not all that Apple Intelligence will bring to the photos app experience. It will now be possible to enhance pictures with natural language requests, making it easier to get the results you want without fussing with individual editing tools. Memory movies can also be created using natural language, right down to the type of music that accompanies the resulting slideshow.
Search is also being improved, allowing users to find photos based on many more details and also search in videos, with results that take them right to the part of the video they’re looking for.
Record and Summarize Calls
For the first time ever, iPhone users will finally be able to record calls directly on their devices without resorting to awkward hardware workarounds. Apple has always blocked this, such that even third-party apps can’t accomplish it. While it’s unlikely Apple is opening this up to other apps, the Phone app will be able to handle it natively.
The main purpose of this is for Apple Intelligence to summarize your phone calls so you can get a quick synopsis of what was discussed. A full recording will also be available — and transcribed to text — and for privacy purposes, iOS 18 will ensure everyone on the call knows it’s being recorded.
ChatGPT on the Side
Earlier this year, reports surfaced that Apple was exploring partnerships with Google and OpenAI to incorporate their generative AI technology into its system. More recently, it was revealed that Apple has reached a deal with OpenAI to license ChatGPT. However, while that’s turned out to be accurate, the partnership is a bit different than what many folks were expecting.
Specifically, ChatGPT is only being used as an add-on to Apple’s own generative AI features, not as a core component. As Craig Federighi noted, Apple Intelligence is based primarily on providing personalized experiences — something other systems like ChatGPT (thankfully) can’t do. However, Apple’s AI isn’t yet as clued into the broader world as OpenAI’s chatbot, so it’s bringing ChatGPT into the fold to provide those extra capabilities for those who want to take advantage of them.
While ChatGPT is being integrated into Siri and the other Apple Intelligence features, it will be very clear when your iPhone reaches out to the external chatbot. Users will be explicitly asked if they want to use ChatGPT and will be told what will be shared in the process. For example, when asking Siri to help get information based on a photo, iOS 18 will make it clear that you need to upload the image to ChatGPT to continue, allowing you to decline if you’d rather it didn’t leave your iPhone.
ChatGPT will also be integrated into the writing tools, allowing it to help draft emails, notes, stories, and more, including adding photos and refining things. This will all be possible without the need to pay for ChatGPT or even set up an account, although ChatGPT subscribers will also be able to take advantage of their paid services directly within the context of Apple’s integrations.
Supported Devices
As cool as these Apple Intelligence features sound, there’s one big catch. Apple is taking its privacy mandate very seriously, to the point where it wants to do as much on-device as possible without allowing folks with older devices to fall back to cloud-based processing.
This means that Apple Intelligence will be limited to only the latest iPhone models with an A17 Pro or newer chip. Right now, that’s the iPhone 15 Pro, although the entire iPhone 16 lineup will naturally be on board this fall. For the iPad and Mac, any M1 or later chip qualifies.
While older devices could theoretically rely on Apple’s Private Cloud Compute architecture, it seems that the company doesn’t want to bear this burden since it would presumably require everything from older devices to be sent to Apple’s servers. In addition to the matter of scalability, there would also undoubtedly be performance issues here that would make that impractical, creating potentially painful delays while waiting for data to be exchanged with Apple’s cloud servers.
After all, we saw remarkable speed improvements when Apple moved Siri to on-device processing in iOS 15 (for those devices that were capable of it). Considering how much more sophisticated the AI-enhanced Siri will be in iOS 18, it’s fair to say that local processing of requests is crucial to a natural experience.