Windows 11 will soon be able to describe images on your screen using AI — and it’ll all be done locally

by

Microsoft has revealed plans for a forthcoming AI function in the recently introduced Click To Do feature on Windows 11, enabling the operating system to verbally describe an image displayed on your screen using artificial intelligence technology. This innovative feature is available only on Copilot+ PCs and can be activated by holding down the Windows key while clicking the mouse.

Since the Copilot+ PC version is unique in using onboard AI models, it implies that the descriptions generated from “Click To Do” requests for image descriptions are provided without needing an internet connection, making it usable even offline.

The blog post announces a new feature in Click to Do: an “image description” action. This allows users to quickly understand the content of images, diagrams, and graphs by providing detailed descriptions directly on their devices. To ensure privacy, the necessary models are automatically set up when you first use this function, and the descriptions are generated locally on your computer, keeping your data secure and private on your PC.

When a user chooses to explain an on-screen image, a box will emerge, providing a description. This box offers the user the chance to generate another description or rate the accuracy of the given description using up and down votes. Additionally, it allows for easy copying of the description into your clipboard. It’s designed to function seamlessly with any app currently open on your device.

The “Image Description” function is one element within a collection of tools and functions embedded in the Click To Do AI interface for Windows 11. This particular tool is connected to Windows Recall and employs local AI models to scrutinize the screen content, offering helpful tasks and actions based on what it identifies.

As a analyst, I can express it this way: So far, Click To Do has shown me its ability to analyze text for summarization, list creation, and rephrasing purposes. It also possesses the capability to recognize images and provides immediate options like blurring or erasing the background of an image on-screen. Additionally, it can conduct web searches using these images to gather more information.

You can customize many actions within Click To Do to suit your needs, and if there are some actions that you never plan on using, you have the option to disable them. Additionally, actions that are passed on to another app can be turned off, so they won’t appear in your Click To Do menu if you prefer not to see them.

Introducing an image description feature significantly enhances the utility of Click To Do, making it even more appealing. It appears that Microsoft aims to establish Click To Do as a counterpart to Google’s Circle of Search, which has gained significant popularity on Android. In certain aspects, Click To Do surpasses its counterpart, offering more capabilities through on-device models for executing actions.

One advantage of employing on-device AI models lies in enhanced privacy and security. For instance, Microsoft emphasizes the safety of its latest image description feature as it operates independently from the cloud. This implies that when you request an AI to describe an image, the picture itself doesn’t get transmitted to Microsoft. Instead, the local AI model on your Copilot+ PC handles the task along with the NPU (Neural Processing Unit) inside your device, ensuring a secure and private experience.

The ability to describe images has been introduced immediately through the Windows 11 Insider Program, more precisely in the Beta and Dev Channels, and it will be broadly released later this year. At present, the preview is limited to Copilot+ PCs with a Snapdragon processor; however, access for Intel and AMD chips is anticipated within the following weeks.

Read More

2025-07-15 02:39