What's new with AI on Windows 11 version 24H2 (2024 Update)

As a seasoned tech enthusiast with years of experience under my belt, I must say that the latest updates from Microsoft are truly revolutionary! The integration of AI in their applications is not just a step forward, but a giant leap towards a future where technology and creativity merge seamlessly.

As an excited fan, I can’t contain my enthusiasm as Windows 11 prepares for its third feature update! This latest edition is unlike any other, offering a groundbreaking array of AI innovations that are truly revolutionary and fresh on our screens.

In the 2024 update (version 24H2), the operating system will include a fresh addition called the Windows Copilot Runtime. This allows several AI functionalities, including Windows Recall, Live Captions, Auto Super Resolution, Voice Clarity, Studio Effects, and more for certain built-in apps, to be operational.

In this article, I’ll explore the anticipated AI capabilities that are set to be introduced with the Windows 11 2024 Update.

What AI changes are coming to Windows 11 2024 Update

In the latest enhancement, Microsoft incorporates close to a dozen AI-driven capabilities. The majority of these functions operate independently, requiring no internet connectivity for their performance.

Windows Copilot Runtime

As I delve into the world of Windows 11, I find myself standing before the core of its AI functionality – the Windows Copilot Runtime. This vital component can be likened to the heart within this operating system, as it seamlessly integrates over forty distinct AI models that can operate directly on the device itself.

Among these AI systems, there’s Phi Silica, a compact language model developed by Microsoft, which shares many advantages typically found in larger language models. Furthermore, you’ll encounter other models such as the Screen Region Detector, Optical Character Recognizer, Natural Language Parser, and Image Encoder. (Alternatively, the models include Phi Silica, a smaller language model from Microsoft that boasts several benefits similar to those found in larger language models, along with options like Screen Region Detector, Optical Character Recognizer, Natural Language Parser, and Image Encoder.)

In the course of this integration, the company is also offering the Windows Copilot Library, an assortment of APIs designed to facilitate the seamless incorporation of AI functionalities within developers’ applications. This library remains active at all times, quietly running on your computer to empower the introduction of innovative AI features.

While AI tools like Copilot do rely on cloud services for certain functionalities, the Windows Copilot Runtime allows a significant portion of these features to operate independently on your local system, keeping data from being sent to the cloud.

Windows Recall

Recall is perhaps the most anticipated and controversial feature coming to Windows 11 (version 24H2). The feature constantly takes screenshots of anything you’re doing and uses multiple AI models running on-device with the help of an NPU (Neural Processing Unit) to record (literally). The feature then analyzes the content and context and stores everything on the device inside the new Windows Semantic Index.

When you’re looking for something, feel free to open the “Recall” app and conduct a search using natural language. Alternatively, you can utilize the timeline slider to move back in history. Once you pick the snapshot, the screenray will facilitate easier content interaction by allowing you to copy text or images.

Additionally, you have the option to utilize the provided controls for launching the application or site, sharing, deleting, or utilizing the Snipping Tool to open a screenshot.

This feature needs space (storage) for saving the snapshots and the index database. By default, it reserves 25GB for a computer that has a 256GB hard drive, but this allotted space can grow based on the size of the storage drive.

From my vantage point, within the Settings menu, a freshly introduced page titled “Recall & Snapshots” has been incorporated. This page empowers me to manage the status, storage, and filtering options related to this feature. Specifically, I can exclude certain websites and applications from participating in the Recall experience by adjusting these settings.

Keep in mind that when utilizing InPrivate or Incognito mode on web browsers like Microsoft Edge and Google Chrome (which are based on Chromium), Windows Recall won’t monitor visited websites. You also have the option to adjust filter settings to prevent specific sites from being tracked by this feature.

While this feature is beneficial in numerous situations, it remains contentious due to the lack of user confidence. There are several doubts about its security and privacy aspects that need resolution. Fortunately, Microsoft has been proactive in addressing these issues.

As per the company, their methodology for Recall adheres strictly to robust security protocols, including the employment of the Microsoft Pluton security processor, Windows Hello verification, and data protection via BitLocker encryption. However, there are certain aspects that seem incongruous.

During its snapshot process, Windows Recall doesn’t filter or censor the data it collects. Consequently, any confidential information is immediately accessible for search.

Initially, the company made the feature available without encrypting the Recall data that was saved directly on the device, leaving it accessible to anyone. However, updates to this feature now involve encryption of the data and necessitate authentication for users to search through the app with Windows Recall.

In this piece I’ve penned down, you’ll find a deep dive into the workings of the Windows Recall feature – a must-read for any tech-savvy individuals like myself who are always eager to expand their knowledge!

Windows Studio Effects

Windows Studio Effects, in simple terms, is a suite of tools that leverages Neural Processing Unit (NPU) technology to add unique visual or audio effects to your webcam or microphone. These features have been around previously, but with the release of version 24H2, Microsoft plans to make them widely accessible on compatible devices for the first time.

Depending on your device’s capabilities, you have the ability to utilize features such as Background Blur, Eye Contact, Automatic Framing, Voice Enhancement, Portrait Lighting, Creative Filters, and Voice Focus.

As a tech enthusiast, I love tinkering with the various features offered by my device. To activate these, simply navigate to the “Studio effects” page within your Quick Settings flyout and toggle the specific feature you’d like to utilize, whether that’s turning it on or off as desired.

Additionally, you have the option to adjust these settings directly under the “Settings” tab, specifically within “Bluetooth & devices” and then “Camera.” Simply choose your camera from the displayed list, and the various options will be accessible under the “Camera Effects” settings.

Effects in the background encompass both standard and portrait blur. The feature called Eye Contact enables a connection as if you are gazing directly into the camera, with options for standard or teleprompter settings.

As a researcher, I’m focusing on developing features that enhance user experience:

The effects available on your device will depend on the peripherals and capability of the NPU.

Live Captions

On Windows 11, the Live Captions feature – available for a while – has been upgraded with the 2024 Update. Now, it’s driven by artificial intelligence and offers real-time translations.

As a tech enthusiast, I can’t help but appreciate the convenience of live translation – a remarkable accessibility tool in Windows 11. This feature is designed to break language barriers, transcribing and translating audio from any source instantly. Whether it’s a YouTube video or a voice call, live translation steps in to ensure understanding.

Voice Clarity

Microsoft is incorporating Voice Clarity, a novel feature powered by artificial intelligence, which works on the fly to filter out ambient noise during discussions or when recording audio.

This function operates on its own, requiring no user adjustments since it’s self-managed. Initially, it was exclusive to Surface devices; however, with the 2024 Update, it’ll be accessible without needing a Neural Processing Unit (NPU).

Because this functionality can be found on devices lacking a Neural Processing Unit (NPU), it implies that not all Artificial Intelligence (AI) capabilities necessarily need specialized hardware in current computer systems.

Auto Super Resolution

Here’s another exciting development I’ve been looking forward to: The integration of Auto Super Resolution (Auto SR), a remarkable feature harnessing onboard AI technology, promises to elevate gaming experiences by boosting frame rates and enhancing visual clarity.

This feature operates in a manner akin to NVIDIA’s Deep Learning Super Sampling (DLSS) Super Resolution, AMD’s FidelityFX Super Resolution, and Intel’s XeSS. However, unlike these technologies that require game developers to make updates, Auto SR can enhance games automatically, without the need for developer intervention.

Activating the feature causes Auto SR to handle the scaling task for NPUs instead of GPUs. This switch leads to improved gaming performance as well as energy savings.

The setting is available through the “Graphics” page, and you can turn it on globally on the computer or per application.

If you start a game that supports Auto SR, the system will display a pop-up message indicating the feature is accessible, and you can enable it by going to the Settings application.

As a researcher, I’ve noticed that Microsoft, in collaboration with Qualcomm, has set up a platform to display games compatible with the ARM variant of Windows 11. This site additionally indicates whether the game is also compatible with Auto Super Resolution. Regrettably, at its inception, only a handful of titles will be supported, such as Borderlands 3, Control (DX11), Dark Souls III, God of War, Kingdom Come: Deliverance, Resident Evil 2, Resident Evil 3, Sekiro Shadows Die Twice, Sniper Ghost Warrior Contracts 2, and The Witcher 3.

It’s important to note that Auto Super Resolution (Auto SR) is not the same as Super Resolution (SR), which usually refers to the upscaling technologies from Intel, AMD, and Nvidia.

Regarding Auto SR, Microsoft, working alongside Intel, AMD, and Nvidia, have developed a novel technology called DirectSR. This innovation enables game creators to incorporate various pre-existing AI upscaling technologies seamlessly by using a unified set of APIs, rather than manually integrating each individual technology.

Restyle Image for Photos

The latest update for the Photos app introduces a fresh “Transform Photo Style” function, leveraging built-in artificial intelligence to allow you to alter the aesthetic of any image by applying diverse photographic methods, much like the filters you’re accustomed to using on smartphones.

You have the ability to explain and guide Photos on how to alter the backdrop and various aspects of an image using the text prompt.

Beyond that, Microsoft plans to transfer the “Image Creator” from the Paint application over to Photos. This feature allows for a text prompt to be transformed into an image by AI. It achieves this through the utilization of Copilot AI technology, which taps into cloud resources to create images using the OpenAI DALL-E model for natural language processing.

Cocreator for Paint

Apart from the existing “Image Creator” function, the updated Paint application will also include a fresh feature called “Co-Creator.” This innovative tool analyzes your sketches and assists in bringing your artwork to life.

With this tool, you can input text and fine-tune the level of creative output by using the slider function.

This distinctive aspect employs a diffusion algorithm, specifically designed for creating a top-tier image that requires minimal input from you.

At least initially, Cocreator will be a feature for Copilot+ PCs.

Copilot app

As a researcher, I’m eager to share some exciting updates about the upcoming Windows 11 2024 Update. Instead of the current method where the Copilot AI is integrated into a side panel that slides in from the right, Microsoft is opting for a new application approach in this update.

The app functions as a modified version of the Microsoft-based AI chatbot, known as Copilot. It’s embedded within an application structure. This version offers a fresh design, resembling the ChatGPT web interface, that includes a panel on the left side for viewing your conversation history, starting new conversations, accessing plugins, and more.

You can still change the conversation style with the option at the top of the page.

Due to these updates, you will no longer find the option to adjust settings using Copilot. The “Copilot” icon has been removed from the System Tray as well. Additionally, Microsoft plans to eliminate the “Copilot” setting from the Settings application.

Additionally, the organization intends to enhance the Copilot’s artificial intelligence by integrating it with OpenAI’s GPT-4 model, allowing for processing of various types of inputs such as text, images, sound, and video, and generating appropriate outputs.

Because Copilot functions using the cloud for analysis and processing, it’s a feature accessible to all users, without any exclusivity towards Copilot+ PC.

Copilot suggestions

Although the Copilot app can no longer be used to change system settings, Microsoft is also introducing “Copilot suggestions.”

In various apps, helpful tips from the chatbot AI will pop up when it’s useful. For instance, when you’re in the Settings app, you might notice suggestions at the top of the screen that can help resolve issues, activate features, and much more.

In File Explorer, when you hover over an image, a Copilot prompt window pops up. This window allows you to inquire about the image and perform actions like erasing the background, altering the image, or adjusting its size.

As a devoted enthusiast, when I find a new email, I might notice a pop-up message suggesting me to quickly summarize the attached document and offer additional insights.

Note: The preliminary release of the Windows 11 2024 Update took place on June 18, 2024, for Copilot+ PCs. However, not all the planned features, such as Windows Recall, have been implemented yet due to a delay. A wider rollout is expected in September or October, making the update more accessible for existing devices. This means that some of the new features and adjustments might not be immediately available to everyone. Additionally, certain features (particularly AI-related ones) may necessitate the availability of new hardware.

More resources

2024-09-18 14:10

What’s new with AI on Windows 11 version 24H2 (2024 Update)