- Full-time Shazam: The watch can detect music playing and automatically identify it using Shazam, displaying it on widget in the Smart Stack. Apple Watches already do this on demand, but the new thing here is that it does it continuously in the background like Pixel phones.
- Sound recognition: The same feature you can get on HomePods and iPhone now comes to Apple Watch. You can get notified visually when your watch hears sounds such as doorbells, fire alarms, sirens, etc. You used to need your iPhone nearby to use this feature, but the Series 12 and Ultra 4 can do it independently.
- Live Rewind: Double-tap the digital crown to see a text readout of the last 15 seconds of what was said around you. You can then ask Siri about it or save it to the Siri app.
- Siri Recap: Get a high-level summary of an entire conversation in the Siri app, so you don’t forget specifics or need to take notes.
The first two of those features don’t really bother anyone. Those are existing features, and the only new thing is that the Apple Watch can do them in the background without iPhone intervention. The last two will be available later this year as a beta, and those are the ones that have people freaking out. Are people’s Apple Watches going to be recording you all the time?
Apple has issued a Audio Intelligence Security Overview white paper to help users understand how all this works and how it protects everyone’s privacy. It’s worth a read, but here are the highlights.
How Live Rewind works
On the Apple Watch Series 12 and Ultra 4, audio from the microphone is continuously flowing into a little buffer in the Secure Enclave. This actually happens on lots of Apple products; it’s how they know if you’ve said “Hey Siri” or not. This is a rolling buffer, where new data continuously replaces old data.
The Secure Enclave is a completely separate hardware block that is not accessible to the rest of the operating system, or any apps, or even Apple. It’s where your Face ID and Touch ID data is stored, for example.
Live Rewind will show you what you missed—privately.
Apple
When you double-press the Digital Crown, the last 15 seconds of that buffer is sent via an encrypted secure transfer to your iPhone, where an AI model transcribes it. This happens in the iPhone’s Secure Enclave, so the audio cannot be accessed by the rest of iOS, or any app, or the user, or even Apple. This is just a straight voice-to-text transcription; it doesn’t identify or label people who are speaking. The audio is then immediately discarded.
The text of the translation is then sent back to Apple Watch where it appears on the display. It is not saved unless you ask Siri about it or save it to the Siri app to revisit later. The Siri app’s data is end-to-end encrypted and synced between devices, and users can control whether conversations are automatically deleted after 30 days, 1 year, or saved forever. Apple cannot decrypt this data, even with a court order—only the user has the key.
