AI Using Screenshots: AI models are no longer just reading text; they are also reading and interpreting screenshots stored on your device, creating a new privacy risk.
AI Using Screenshots: Just like on your phone, you likely have numerous screenshots on your laptop and desktop. Screenshots of social media posts, favorite recipes, or preferred clothing and footwear are common on almost everyone's devices. While they seem essential at the moment we take them, we often forget about them later. Even if you have forgotten them, they are proving to be highly valuable to AI. AI models do not treat these merely as images; they view them as information that can be searched, organized, and acted upon.
How are AI models using screenshots?
OpenAI’s Computer-Using Agent (CUA) utilizes screenshots to operate software. Similarly, Anthropic’s "Computer Use" and Google’s ScreenAI can comprehend computer screens and interfaces. Microsoft’s "Recall" feature captures screenshots at regular intervals, while the "Mindspace" feature on OnePlus and Oppo devices transform screenshots into a searchable AI memory. This demonstrates that AI models are moving beyond just reading text; they are learning to interact with computers by "seeing" the screen, much like humans do. Multimodal models can now interpret the entire screen to recognize interface elements, dates, locations, prices, menus, and more.
Why are AI models learning to read screens?
Until now, software programs have relied on Application Programming Interfaces (APIs) to communicate with one another; however, many enterprise systems, legacy apps, desktop software, and government portals lack modern API capabilities. To overcome this challenge, AI companies are teaching their models to read screens. This enables the models to see what is displayed on the screen and perform tasks based on that information.
A Major Threat to Privacy
The better AI models become at understanding screens, the greater the risk to privacy. Screens often display highly sensitive information—such as bank balances, medical records, private chats, emails, important work documents, and passwords. Consequently, if AI models can read this content, they could gain access to all your critical information. Experts are also raising warnings about this issue.
