Developers can now add support for Siri’s Onscreen Awareness feature
iOS 18.2 beta 2 was rolled out to developers and testers on the developer beta channel on Monday, as Apple prepares the next version of its smartphone operating system, expected to release in early December with more Apple Intelligence features. The latest beta versions also include support for a new application programming interface (API) that allows developers to give the system access to on-screen content, allowing Siri and Apple Intelligence to send information to third-party services for processing.
Apple introduces API for Siri’s Onscreen Awareness feature
The company mentioned this on the Apple Developer website documentation (via Macrumors) for the new API entitled ‘Making onscreen content available to Siri and Apple Intelligence’ which is designed to provide access to an app’s screen content, allowing Siri and Apple Intelligence to understand what content the user is accessing.
If a developer adds support for the onscreen content API, their application will pass onscreen content to Siri/Apple Intelligence when a user explicitly requests it, the company said. The information on a user’s screen can then be shared with a third-party service (such as OpenAI’s ChatGPT).
Apple also provided an example of Siri accessing content on the screen. While browsing the web, a user can say or type, “Hey Siri, what is this document about?” to ask Siri to provide a summary of a document.
Developers can also add support for on-screen awareness in browsers, document readers, file management apps, email, photos, presentations, spreadsheets, and word processing apps. Apple says this list is not exhaustive, so more apps should be able to take advantage of the API in the future.
It’s worth noting that iOS 18.2 doesn’t support the new Siri, which is expected to offer greatly improved functionality. This is expected to arrive on iOS 18.4, along with support for in-app actions, which Apple will reportedly release in April 2025, which is plenty of time for developers to integrate support for the API into their apps.
