Live media workflows are increasingly embracing the power of AI microservices, significantly enhancing their production capabilities. However, the majority of advanced AI models are still predominantly hosted in cloud environments. This reliance makes it challenging to process high-bitrate, uncompressed media streams, largely due to issues related to network latency, bandwidth limitations, and real-time scalability challenges.
In response to these obstacles, NVIDIA has unveiled groundbreaking AI reference applications designed to simplify the development of AI solutions tailored for live media workflows. These technologies are capable of interfacing with uncompressed ST 2110 streams, enabling the delivery of real-time media effects while maintaining minimal latency.
AI Reference Applications
The latest developments in AI reference applications, available through Holoscan for Media, provide an invaluable starting point for developers looking to create cutting-edge real-time AI solutions specifically tailored for the dynamic requirements of live media workflows.
AI Virtual Cameras
One of the standout applications developed using PyTorch and the NVIDIA DeepStream SDK is the virtual camera solution, which creates individualized virtual camera outputs for each presenter in a video stream. By detecting and tracking individuals in a high-resolution, uncompressed ST 2110 input stream, the reference application can generate multiple cropped camera feeds focused on those detected individuals. This approach enables operators to craft a more engaging and dynamic production shot from a single, static camera rig.
Automatic Speech Recognition
This reference application leverages real-time automatic speech recognition (ASR) on an ST 2110-30 audio source utilizing the NVIDIA Riva Parakeet ASR NIM. By employing a straightforward web user interface, users can monitor transcriptions live, making it easy to search for specific words as they appear in the text. The interface enables users to see real-time captions of the incoming stream alongside a search function to navigate through the transcription efficiently. This simple yet effective front-end serves as a strong foundational platform for developers to refine and customize their implementations further.
How to Get Started
If you’re eager to leverage the power of AI reference applications, here are the prerequisites you’ll want to have in place to facilitate a smooth development process and prevent common setup obstacles:
- An AI workstation equipped with an NVIDIA RTX Pro GPU, along with an NVIDIA ConnectX network interface card (with either loopback cable or switch connectivity), or a certified multi-GPU system.
- A functioning NVIDIA Holoscan for Media environment, which can be established using either a local developer setup with Kubernetes or the platform reference deployment guide featuring a jump node.
- Visual Studio Code or any other compatible IDE for Linux platforms. The GNU Compiler Collection (GCC) is also a viable option.
For installation of v25.4, you can refer to the developer guides provided on the Holoscan for Media collection page. To proceed with installing the AI applications, follow the comprehensive steps outlined on the NGC AI Reference Applications resources page.
Additional Updates
Beyond the AI applications, the Holoscan for Media 25.4 release boasts enhanced monitoring capabilities for both production (OpenShift) and local developer (cloud-native stack) environments. This includes support for the SR-IOV network, PTP, and application-specific Grafana dashboards for the NMOS registry.
Additionally, the update improves automation for single-node OpenShift installations and compact three-node clusters. It now supports a broader range of networking options, including red/blue networking for ST 2022-7 redundancy. Furthermore, automation has been streamlined for local developer setups, which now support Ubuntu 24.04, alongside automated installation of reference applications such as the Helm dashboard, NMOS registry, NMOS controller, and media gateway.
By structuring the content clearly, integrating relevant keywords, and maintaining an informative tone, this article effectively delivers insights into the role of AI in live media workflows, focusing on NVIDIA’s innovations with AI reference applications.
Inspired by: Source
