Editor’s note: This post is part of the AI Decoded series, which demystifies AI by making the technology more accessible, and which showcases new hardware, software, tools and accelerations for RTX PC users.

NVIDIA’s RTX AI platform includes tools and software development kits that help Windows developers create cutting-edge generative AI features to deliver the best performance on AI PCs and workstations.

At GTC — NVIDIA’s annual technology conference — a dream team of industry luminaries, developers and researchers have come together to learn from one another, fueling what’s next in AI and accelerated computing.

This special edition of AI Decoded from GTC spotlights the best AI tools currently available and looks at what’s ahead for the 100 million RTX PC and workstation users and developers.

Chat with RTX, the tech demo and developer reference project that quickly and easily allows users to connect a powerful LLM to their own data, showcased new capabilities and new models in the GTC exhibit hall.

The winners of the Gen AI on RTX PCs contest were announced Monday. OutlookLLM, Rocket League BotChat and CLARA were highlighted in one of the AI Decoded talks in the generative AI theater and each are accelerated by NVIDIA TensorRT-LLM. Two other AI Decoded talks included using generative AI in content creation and a deep dive on Chat with RTX.

Developer frameworks and interfaces with TensorRT-LLM integration continue to grow as Jan.ai, Langchain, LlamaIndex and Oobabooga will all soon be accelerated — helping to grow the already more than 500 AI applications for RTX PCs and workstations.

NVIDIA NIM microservices are coming to RTX PCs and workstations. They provide pre-built containers, with industry standard APIs, enabling developers to accelerate deployment on RTX PCs and workstations. NVIDIA AI Workbench, an easy-to-use developer toolkit to manage AI model customization and optimization workflows, is now generally available for RTX developers.

These ecosystem integrations and tools will accelerate development of new Windows apps and features. And today’s contest winners are an inspiring glimpse into what that content will look like.

Hear More, See More, Chat More

Chat with RTX, or ChatRTX for short, uses retrieval-augmented generation, NVIDIA TensorRT-LLM software and NVIDIA RTX acceleration to bring local generative AI capabilities to RTX-powered Windows systems. Users can quickly and easily connect local files as a dataset to an open large language model like Mistral or Llama 2, enabling queries for quick, contextually relevant answers.

Moving beyond text, ChatRTX will soon add support for voice, images and new models.

Users will be able to talk to ChatRTX with Whisper — an automatic speech recognition system that uses AI to process spoken language. When the feature becomes available, ChatRTX will be able to “understand” spoken language, and provide text responses.

A future update will also add support for photos. By integrating OpenAI’s CLIP — Contrastive Language-Image Pre-training — users will be able to search by words, terms or phrases to find photos in their private library.

In addition to Google’s Gemma, ChatGLM will get support in a future update.

Developers can start with the latest version of the developer reference project on GitHub.

Generative AI for the Win

The NVIDIA Generative AI on NVIDIA RTX developer contest prompted developers to build a Windows app or plug-in.

Submissions were judged on three criteria, including a short demo video posted to social media, relative impact and ease of use of the project, and how effectively NVIDIA’s technology stack was used in the project. Each of the three winners received a pass to GTC, including a spot in the NVIDIA Deep Learning Institute GenAI/LLM courses, and a GeForce RTX 4090 GPU to power future development work.

OutlookLLM gives Outlook users generative AI features — such as email composition — securely and privately in their email client on RTX PCs and workstations. It uses a local LLM served via TensorRT-LLM.

Rocket League BotChat, for the popular Rocket League game, is a plug-in that allows bots to send contextual in-game chat messages based on a log of game events, such as scoring a goal or making a save. Designed to be used only in offline games against bot players, the plug-in is configurable in many ways via its settings menu.

CLARA (short for Command Line Assistant with RTX Acceleration) is designed to enhance the command line interface of PowerShell by translating plain English instructions into actionable commands. The extension runs locally, quickly and keeps users in their PowerShell context. Once it’s enabled, users type their English instructions and press the tab button to invoke CLARA. Installation is straightforward, and there are options for both script-based and manual setup.

Congratulations to the #GenAIonRTX #DevContest winners:

CLARA – Talk with computers in human language – Matthew Yaeger
Rocket League BotChat – Generate in-game chat messages – Brian Caffey
Outlook LLM – Compose emails securely with AI – Francisco Gonzalez Blanch

pic.twitter.com/i52w5Pn1n9

— NVIDIA AI Developer (@NVIDIAAIDev) March 19, 2024

From the Generative AI Theater

GTC attendees can attend three AI Decoded talks on Wednesday, March 20 at the generative AI theater. These 15-minute sessions will guide the audience through ChatRTX and how developers can productize their own personalized chatbot; how each of the three contest winners’ showed some of the possibilities for generative AI apps on RTX systems; and a celebration of artists, the tools and methods they use powered by NVIDIA technology.

In the creator session, Lee Fraser, senior developer relations manager for generative AI media and entertainment at NVIDIA, will explore why generative AI has become so popular. He’ll show off new workflows and how creators can rapidly explore ideas. Artists to be featured include Steve Talkowski, Sophia Crespo, Lim Wenhui, Erik Paynter, Vanessa Rosa and Refik Anadol.

Anadol also has an installation at the show that combines data visualization and imagery based on that data.

Ecosystem of Acceleration

Top creative app developers, like Blackmagic Design and Topaz Labs have integrated RTX AI acceleration in their software. TensorRT doubles the speed of AI effects like rotoscoping, denoising, super-resolution and video stabilization in the DaVinci Resolve and Topaz apps.

“Blackmagic Design and NVIDIA’s ongoing collaborations to run AI models on RTX AI PCs will produce a new wave of groundbreaking features that give users the power to create captivating and immersive content, faster.” — Rohit Gupta, director of software development at Blackmagic Design

TensorRT-LLM is being integrated with popular developer frameworks and ecosystems such as LangChain, LlamaIndex, Oobabooga and Jan.AI. Developers and enthusiasts can easily access the performance benefits of TensorRT-LLM through top LLM frameworks to build and deploy generative AI apps to both local and cloud GPUs.

Enthusiasts can also try out their favorite LLMs — accelerated with TensorRT-LLM on RTX systems — through the Oobabooga and Jan.AI chat interfaces.

AI That’s NIMble, AI That’s Quick

Developers and tinkerers can tap into NIM microservices. These pre-built AI “containers,” with industry-standard APIs, provide an optimized solution that helps to reduce deployment times from weeks to minutes. They can be used with more than two dozen popular models from NVIDIA, Getty Images, Google, Meta, Microsoft, Shutterstock and more.

NVIDIA AI Workbench is now generally available, helping developers quickly create, test and customize pretrained generative AI models and LLMs on RTX GPUs. It offers streamlined access to popular repositories like Hugging Face, GitHub and NVIDIA NGC, along with a simplified user interface that enables developers to easily reproduce, collaborate on and migrate projects.

Projects can be easily scaled up when additional performance is needed — whether to the data center, a public cloud or NVIDIA DGX Cloud — and then brought back to local RTX systems on a PC or workstation for inference and light customization. AI Workbench is a free download and provides example projects to help developers get started quickly.

These tools, and many others announced and shown at GTC, are helping developers drive innovative AI solutions.

From the Blackwell platform’s arrival, to a digital twin for Earth’s climate, it’s been a GTC to remember. For RTX PC and workstation users and developers, it was also a glimpse into what’s next for generative AI.

See notice regarding software product information.

In today’s complex business environments, IT teams face a constant flow of challenges, from simple issues like employee account lockouts to critical security threats. These situations demand both quick fixes and strategic defenses, making the job of maintaining smooth and secure operations ever tougher.

That’s where AIOps comes in, blending artificial intelligence with IT operations to not only automate routine tasks, but also enhance security measures. This efficient approach allows teams to quickly deal with minor issues and, more importantly, to identify and respond to security threats faster and with greater accuracy than before.

By using machine learning, AIOps becomes a crucial tool in not just streamlining operations but also in strengthening security across the board. It’s proving to be a game-changer for businesses looking to integrate advanced AI into their teams, helping them stay a step ahead of potential security risks.

According to IDC, the IT operations management software market is expected to grow at a rate of 10.3% annually, reaching a projected revenue of $28.4 billion by 2027. This growth underscores the increasing reliance on AIOps for operational efficiency and as a critical component of modern cybersecurity strategies.

As the rapid growth of machine learning operations continues to transform the era of generative AI, a broad ecosystem of NVIDIA partners are offering AIOps solutions that leverage NVIDIA AI to improve IT operations.

NVIDIA is helping a broad ecosystem of AIOps partners with accelerated compute and AI software. This includes NVIDIA AI Enterprise, a cloud-native stack that can run anywhere and provides a basis for AIOps through software like NVIDIA NIM for accelerated inference of AI modes, NVIDIA Morpheus for AI-based cybersecurity and NVIDIA NeMo for custom generative AI. This software facilitates GenAI-based chatbot, summarization and search functionality.

AIOps providers using NVIDIA AI include:

• Dynatrace Davis hypermodal AI advances AIOps by integrating causal, predictive and generative AI techniques with the addition of Davis CoPilot. This combination enhances observability and security across IT, development, security and business operations by offering precise and actionable, AI-driven answers and automation.

• Elastic offers Elasticsearch Relevance Engine (ESRE) for semantic and vector search, which integrates with popular LLMs like GPT-4 to power AI Assistants in their Observability and Security solutions. The Observability AI Assistant is a next-generation AI Ops capability that helps IT teams understand complex systems, monitor health and automate remediation of operational issues.
• New Relic is advancing AIOps by leveraging its machine learning, generative AI assistant frameworks and longstanding expertise in observability. Its machine learning and advanced logic helps IT teams reduce alerting noise, improve mean time to detect and mean time to repair, automate root cause analysis and generate retrospectives. Its GenAI assistant, New Relic AI, accelerates issue resolution by allowing users to identify, explain and resolve errors without switching contexts, and suggests and applies code fixes directly in a developer’s integrated development environment. It also extends incident visibility and prevention to non-technical teams by automatically producing high-level system health reports, analyzing and summarizing dashboards and answering plain-language questions about a user’s applications, infrastructure and services. New Relic also provides full-stack observability for AI-powered applications benefitting from NVIDIA GPUs.
• PagerDuty has introduced a new feature in PagerDuty Copilot, integrating a generative AI assistant within Slack to offer insights from incident start to resolution, streamlining the incident lifecycle and reducing manual task loads for IT teams.
• ServiceNow’s commitment to creating a proactive IT operations encompasses automating insights for rapid incident response, optimizing service management and detecting anomalies. Now, in collaboration with NVIDIA, it is pushing into generative AI to further innovate technology service and operations.
• Splunk’s technology platform applies artificial intelligence and machine learning to automate the processes of identifying, diagnosing and resolving operational issues and threats, thereby enhancing IT efficiency and security posture. Splunk IT Service Intelligence serves as Splunk’s primary AIOps offering, providing embedded AI-driven incident prediction, detection and resolution all from one place.

Cloud service providers including Amazon Web Services (AWS), Google Cloud and Microsoft Azure enable organizations to automate and optimize their IT operations, leveraging the scale and flexibility of cloud resources.

• AWS offers a suite of services conducive to AIOps, including Amazon CloudWatch for monitoring and observability; AWS CloudTrail for tracking user activity and API usage; Amazon SageMaker for creating repeatable and responsible machine learning workflows; and AWS Lambda for serverless computing, allowing for the automation of response actions based on triggers.
• Google Cloud supports AIOps through services like Google Cloud Operations, which provides monitoring, logging and diagnostics across applications on the cloud and on-premises. Google Cloud’s AI and machine learning products include Vertex AI for model training and prediction and BigQuery for fast SQL queries using the processing power of Google’s infrastructure.
• Microsoft Azure facilitates AIOps with Azure Monitor for comprehensive monitoring of applications, services and infrastructure. Azure Monitor’s built-in AIOps capabilities help predict capacity usage, enable autoscaling, identify application performance issues and detect anomalous behaviors in virtual machines, containers and other resources. Microsoft Azure Machine Learning (AzureML) offers a cloud-based MLOps environment for training, deploying and managing machine learning models responsibly, securely and at scale.

Platforms specializing in MLOps primarily focus on streamlining the lifecycle of machine learning models, from development to deployment and monitoring. While the core mission centers on making machine learning more accessible, efficient and scalable, their technologies and methodologies indirectly support AIOps by enhancing AI capabilities within IT operations: 

• Anyscale’s platform, based on Ray, allows for the easy scaling of AI and machine learning applications, including those used in AIOps for tasks like anomaly detection and automated remediation. By facilitating distributed computing, Anyscale helps AIOps systems process large volumes of operational data more efficiently, enabling real-time analytics and decision-making.
• Dataiku can be used to create models that predict IT system failures or optimize resource allocation, with features that allow IT teams to quickly deploy and iterate on these models in production environments.
• Dataloop’s platform delivers full data lifecycle management and a flexible way to plug in AI models for an end-to-end workflow, allowing users to develop AI applications using their data.
• DataRobot is a full AI lifecycle platform that enables IT operations teams to rapidly build, deploy and govern AI solutions, improving operational efficiency and performance.
• Domino Data Lab’s platform lets enterprises and their data scientists build, deploy and manage AI on a unified, end-to-end platform. Data, tools, compute, models and projects across all environments are centrally managed so teams can collaborate, monitor production models and standardize best practices for governed AI innovation. This approach is vital for AIOps as it balances the self-service needed by data science teams with complete reproducibility, granular cost tracking and proactive governance for IT operational needs.
• Weights & Biases provides tools for experiment tracking, model optimization, and collaboration, crucial for developing and fine-tuning AI models used in AIOps. By offering detailed insights into model performance and facilitating collaboration across teams, Weights & Biases helps ensure that AI models deployed for IT operations are both effective and transparent.

Learn more about NVIDIA’s partner ecosystem and their work at NVIDIA GTC.

To help mitigate climate change — one of humanity’s greatest challenges — researchers are turning to AI and sustainable computing to accelerate and operationalize their work.

At this week’s NVIDIA GTC global AI conference, startups, enterprises and scientists are highlighting their environmental sustainability initiatives and latest climate innovations. Many are using NVIDIA Earth-2, a full-stack, open platform for accelerating climate and weather simulation and predictions.

Earth-2 comprises GPU-accelerated numerical weather and climate prediction models, including ICON and IFS; state-of-the-art AI-driven weather models, such as FourCastNet, GraphCast and Deep Learning Weather Prediction, offered through the NVIDIA Modulus framework; and large-scale, interactive, high-resolution data visualization and simulation enabled by the NVIDIA Omniverse platform. These capabilities are also available via cloud APIs, or application programming interfaces.

Various members of NVIDIA Inception — a free, global program for cutting-edge startups — are pioneering climate AI advancements with Earth-2. It’s critical work, as extreme-weather events are expected to take a million lives and cost $1.7 trillion per year by 2050.

Tomorrow.io Powers Weather Predictions of Tomorrow

Boston-based Tomorrow.io provides actionable, weather-related insights to countries, businesses and individuals by applying advanced AI and machine learning models to a proprietary global dataset collected from satellites, radar and other sensors. Its weather intelligence and climate adaptation platform delivers high-resolution, accurate weather forecasts across time zones for both short- and long-term projections.

The startup is using Earth-2 to study the potential impacts of its suite of satellites on global model forecasts. By conducting observing-system simulation experiments, or OSSEs, with Earth-2 AI forecast models, Tomorrow.io can identify the optimal configurations of satellites and other instruments to improve weather-forecasting conditions. The work ultimately aims to offer users precision and simplicity, helping them easily understand complex weather situations and make the right operational decisions at the right time.

Learn more about Tomorrow.io’s work with Earth-2 by joining the GTC session, “Global Strategies: Startups, Venture Capital, and Climate Change Solutions,” taking place today, March 19, at 3 p.m. PT, at the San Jose Convention Center and online.

ClimaSens Advances Flood-Risk Management With AI

ClimaSens, based in Melbourne, Australia, and New York, fuses historical, real-time and future climate and weather information using advanced AI models. FloodSens, its upcoming flood risk analysis model, informs clients about the probability of flooding from rainfall, offering high-resolution assessments of flash flooding, riverine flooding and all types of flooding in between.

FloodSens, now in beta, was developed using Earth-2 APIs and the FourCastNet model for high-fidelity, physically accurate representations of future weather conditions, as well as an ensemble of other models for assessing the probabilities of low-likelihood, high-impact flooding events. Through this work, the startup aims to enable a more resilient, sustainable future for communities worldwide.

North.io Garners Ocean Insights With AI and Accelerated Modeling

Based in Kiel, Germany, north.io is helping to map the Earth’s largest carbon sink: oceans. Only about 25% of the ocean floor — a critical source of the world’s renewable energy and food security — has been mapped so far.

North.io is collecting and analyzing massive amounts of data from autonomous underwater vehicles (AUVs) and making it accessible, shareable, visualizable and understandable for users across the globe through its TrueOcean platform.

Using Earth-2 APIs, north.io is developing AI weather forecasts for intelligent operational planning, system management and risk assessment for its AUVs. The combination of high-precision weather modeling and the use of autonomous systems drastically reduces human safety risks in rough, offshore environments.

Learn more about the latest AI, high performance computing and sustainable computing advancements for climate research at GTC, running through Thursday, March 21.

NVIDIA recognized 14 partners in the Americas for their achievements in transforming businesses with AI, this week at GTC.

The winners of the NVIDIA Partner Network Americas Partner of the Year awards have helped customers across industries advance their operations with the software, systems and services needed to integrate AI into their businesses.

NPN awards categories span a multitude of competencies and industries, including service delivery, data center networking, public sector, healthcare and higher education.

Three new categories were created this year. One recognizes a partner driving AI-powered success in the financial services industry; one celebrates a partner exhibiting overall AI excellence; and another honors a partner’s dedication to advancing NVIDIA’s full-stack portfolio across a multitude of industries.

All awards reflect the spirit of the NPN ecosystem in driving business success through accelerated full-stack computing and software.

“NVIDIA’s work driving innovation in generative AI has helped partners empower their customers with cutting-edge technology — as well reduced costs and fostered growth opportunities while solving intricate business challenges,” said Rob Enderle, president and principal analyst at the Enderle Group. “The recipients of the 2024 NPN awards embody a diverse array of AI expertise, offering rich knowledge to help customers deploy transformative solutions across industries.”

The 2024 NPN award winners for the Americas are:

• AI Excellence Partner of the Year — Lambda received this award for its dedication to providing end-to-end AI solutions featuring NVIDIA accelerated computing and NVIDIA AI Enterprise across NVIDIA DGX and Lambda Cloud.
• Enterprise Partner of the Year — World Wide Technology received this newly introduced award for its leadership, dedication and expertise in advancing the adoption of AI with NVIDIA’s portfolio of purpose-built systems, data center networking, software and accelerated computing solutions across machine learning, digital twins, NVIDIA Omniverse and visualization.
• Canadian Partner of the Year — Converge Technology Solutions is recognized for its dedication and expertise in NVIDIA DGX systems and for its Canadian customer support services, leveraging training courses from NVIDIA, to further industry knowledge of the NVIDIA software stack.
• Financial Services Partner of the Year — CDW received this newly introduced award for its ecosystem partnerships, strategic investments and targeted domain expertise serving financial customers seeking HPC solutions and customer experience solutions such as chatbots and agentless routing.
• Global Consulting Partner of the Year — Deloitte is recognized for the fourth consecutive time for its embrace of generative AI and for leveraging the capabilities of NVIDIA DGX Cloud.
• Healthcare Partner of the Year — Mark III is recognized for the second consecutive year for its utilization for the NVIDIA healthcare portfolio, which supports biopharma research, academic medical centers, research institutions, healthcare systems and life sciences organizations with NVIDIA infrastructure, software and cloud technologies.
• Higher Education Partner of the Year — Cambridge Computer is recognized for the fourth consecutive year for its customer service and technical expertise, bringing NVIDIA AI solutions to the life sciences, education and research sectors.
• Networking Partner of the Year — Converge Technology Solutions is recognized for its expertise in NVIDIA high-performance networking solutions to support state-of-the-art accelerated computing deployments.
• Public Sector Partner of the Year — Sterling is recognized for its investment and achievements in developing a robust AI practice. This includes assembling a team of dedicated AI software engineers focused on the full-stack NVIDIA platform, establishing Sterling Labs — an AI briefing center near Washington, D.C. — and collaborating with NVIDIA to launch ARC, a 5G/6G platform targeted for next-gen wireless networks.
• Rising Star Partner of the Year — International Computer Concepts is recognized for its growth in developing AI and machine learning solutions for cloud service providers and financial services customers to power machine learning training, real-time inference and other AI workloads.
• Service Delivery Partner of the Year — Quantiphi is recognized for the third consecutive year for its commitment to driving adoption of NVIDIA software and hardware in the enterprise. Its AI Service Delivery team has demonstrated expertise in using LLMs, information retrieval, imaging and data analytics to solve complex business problems in the telecom, life sciences, retail and energy industries for its global customers.
• Distribution Partner of the Year — TD SYNNEX is recognized for demonstrating its commitment to building its AI business on the NVIDIA AI platform, with year-over-year growth that underscores its operational excellence in distribution.
• Software Partner of the Year — Insight is recognized for its leadership in NVIDIA AI Enterprise deployments, establishing cutting-edge innovation labs and certifications that cultivate expertise while seamlessly embedding generative AI into its operations.
• Solution Integration Partner of the Year — EXXACT is recognized for its commitment and expertise in providing end-to-end NVIDIA AI and high performance computing solutions, including NVIDIA software and data center products across multiple industries.

Learn how to join NPN, or find your local NPN partner.

NVIDIA today was named the world’s most innovative company by Fast Company magazine.

The accolade comes on the heels of company founder and CEO Jensen Huang being inducted into the U.S. National Academy of Engineering.

A team of several dozen journalists at Fast Company — a business media brand launched in 1995 by two Harvard Business Review editors — ranked NVIDIA the leader in its 2024 list of the world’s 50 most innovative companies.

“Putting AI to Work”

“Nvidia isn’t just in the business of providing ever-more-powerful computing hardware and letting everybody else figure out what to do with it,” Fast Company wrote in an article detailing its selection.

“Across an array of industries, the company’s technologies, platforms, and partnerships are doing much of the heavy lifting of putting AI to work,” citing advances in automotive, drug discovery, gaming and retail announced in one recent week.

The article noted the central role of the CUDA compute platform. It also shared an eye-popping experience using NVIDIA Omniverse to interact with a digital twin of a Nissan Z sport coupe.

In a League With Giants

“Even for AI’s titans, building on what Nvidia has created — the more ambitiously, the better — is often how progress happens,” the article concluded.

Last year, OpenAI led the list for ChatGPT, the large language model that started a groundswell in generative AI. In 2021, Moderna and Pfizer-BioNTech topped the ranking for rapidly developing a Covid vaccine.

Fast Company bases its ranking on four criteria: innovation, impact, timeliness and relevance. Launched in 2008, the annual list recognizes organizations that have introduced groundbreaking products, fostered positive social impact and reshaped industries.

NVIDIA invented the GPU in 1999, redefining computer graphics and igniting the era of modern AI. NVIDIA is now driving the platform shift to accelerated computing and generative AI, transforming the world’s largest industries.

Fueling an AI Revolution

Last month, Huang was elected to the National Academy of Engineering (NAE) for contributions in “high-powered graphics processing units, fueling the artificial intelligence revolution.”

Academy membership honors those who have made outstanding contributions such as pioneering new fields of technology. Founded in 1964, the NAE provides a trusted source of engineering advice for creating a healthier, more secure and sustainable world.

“Jensen Huang’s induction into the National Academy of Engineering is a testament to his enduring contributions to our industry and world,” said Satya Nadella, chairman and CEO of Microsoft.

“His visionary leadership has forever transformed computing, from the broad adoption of advanced 3D graphics to today’s GPUs — and, more importantly, has driven foundational innovations across every sector, from gaming and productivity, to digital biology and healthcare. All of us at Microsoft congratulate Jensen on this distinction, and we are honored to partner with him and the entire NVIDIA team on defining this new era of AI,” he added.

“Jensen’s election is incredibly well deserved,” said John Hennessy, president emeritus of Stanford University and an NAE member since 1992.

“His election recognizes both his transformative technical contributions, as well as his incredible leadership of NVIDIA for almost 30 years. I have seen many NAE nominations over the past 30 years, Jensen’s was one of the best!”

Morris Chang, founder of Taiwan Semiconductor Manufacturing Co. and an NAE member since 2002, added his congratulations.

“Jensen is one of the most visionary engineers and charismatic business leaders I have had the pleasure to work with in the last three decades,” he said.

Huang is also a recipient of the Semiconductor Industry Association’s highest honor, the Robert N. Noyce Award, as well as honorary doctorate degrees from Taiwan’s National Chiao Tung University, National Taiwan University and Oregon State University.

Editor’s note: This post is part of our In the NVIDIA Studio series, which celebrates featured artists, offers creative tips and tricks, and demonstrates how NVIDIA Studio technology improves creative workflows. We’re also deep diving on new GeForce RTX 40 Series GPU features, technologies and resources, and how they dramatically accelerate content creation.

Creators are getting a generative AI boost with tools announced at NVIDIA GTC, a global AI conference bringing together the brightest minds in AI content creation and accelerated computing.

Adobe Substance 3D Stager and Sampler via Adobe Firefly, the OBS 30.1 YouTube HDR Beta and NVIDIA Omniverse Audio2Face for iClone 8 will also receive sizable upgrades.

DLSS 3.5 with Ray Reconstruction is coming soon to the NVIDIA RTX Remix Open Beta, enabling modders to upgrade their projects with the power of AI. Sample this leap in high graphical fidelity with the new Portal With RTX update available on Steam with DLSS Ray Reconstruction, which provides enhanced ray-traced imagery. Learn more about the DLSS 3.5 update to Portal With RTX.

The March NVIDIA Studio Driver, optimizing the latest creative app updates, is available for download today.

A March of Creative App Upgrades

The Adobe Substance 3D Stager beta announced a new Generative Background feature — powered by Adobe Firefly — to create backdrops for rendered images. Stager’s Match Image tool uses machine learning to accurately place 3D models within the generated background, optimizing lighting and perspective for greater flexibility and realism.

Meanwhile, Substance 3D Sampler’s announced Text to Texture beta — also powered by Adobe Firefly — gives artists a new way to source texture imagery using only a description. All Text to Texture images are square and tileable with proper perspective, ready for material-creation workflows.

Learn more about both apps in the GTC session “Elevating 3D Concepts: GenAI-Infused Design.” Search the GTC session catalog and check out the “Content Creation / Rendering / Ray Tracing” and “Generative AI” topics for additional creator-focused sessions.

The recently launched OBS 30.1 beta will enable content creators to use Real-Time Messaging Protocol — an Adobe open-source protocol designed to stream audio and video by maintaining low-latency connections — to stream high-dynamic range, high-efficiency video coding content to YouTube. Download OBS Beta 30.1 on the OBS website to get started.

NVIDIA Omniverse Audio2Face for iClone 8 uses AI to produce expressive facial animations solely from audio input. In addition to generating natural lip-sync animations for multilingual dialogue, the latest standalone release supports multilingual lip-sync and singing animations, as well as full-spectrum editing with slider controls and a keyframe editor.

For more information on how RTX is powering premium AI capabilities and performance, check out the new AI Decoded blog series and sign up to receive updates weekly.

Follow NVIDIA Studio on Instagram, X and Facebook. Access tutorials on the Studio YouTube channel and get updates directly in your inbox by subscribing to the Studio newsletter. 

Generative AI promises to revolutionize every industry it touches — all that’s been needed is the technology to meet the challenge.

NVIDIA CEO Jensen Huang on Monday introduced that technology—the company’s new Blackwell computing platform—as he outlined the major advances that increased computing power can deliver for everything from software to services, robotics to medical technology, and more.

“Accelerated computing has reached the tipping point — general purpose computing has run out of steam,” Huang told more than 11,000 GTC attendees gathered in-person — and many tens of thousands more online — for his keynote address at Silicon Valley’s cavernous SAP Center arena.

“We need another way of doing computing — so that we can continue to scale so that we can continue to drive down the cost of computing, so that we can continue to consume more and more computing while being sustainable. Accelerated computing is a dramatic speedup over general purpose computing, in every single industry.”

Huang spoke in front of massive images on a 40-foot tall, 8k screen the size of a tennis court to a crowd packed with CEOs and developers, AI enthusiasts and entrepreneurs, who walked together 20 minutes to the arena from the San Jose Convention Center on a dazzling spring day.

Delivering a massive upgrade to the world’s AI infrastructure, Huang introduced the NVIDIA Blackwell platform to unleash real-time generative AI on trillion-parameter large language models.

Huang presented NVIDIA NIM — a reference to NVIDIA inference microservices — a new way of packaging and delivering software that connects developers with hundreds of millions of GPUs to deploy custom AI of all kinds.

And bringing AI into the physical world, Huang introduced Omniverse Cloud APIs to deliver advanced simulation capabilities.

Huang punctuated these major announcements with powerful demos, partnerships with some of the world’s largest enterprises, and more than a score of announcements detailing his vision.

GTC — which in 15 years has grown from the confines of a local hotel ballroom to the world’s most important AI conference — is returning to a physical event for the first time in five years.

This year’s has over 900 sessions — including a panel discussion on transformers moderated by Huang with the eight pioneers who first developed the technology, more than 300 exhibits, and 20-plus technical workshops.

It’s an event that’s at the intersection of AI and just about everything. In a stunning opening act to the keynote, Refik Anadol, the world’s leading AI artist, showed a massive real-time AI data sculpture with wave-like swirls in greens, blues, yellows and reds, crashing, twisting and unraveling across the screen.

As he kicked off his talk, Huang explained that the rise of multi-modal AI — able to process diverse data types handled by different models — gives AI greater adaptability and power. By increasing their parameters, these models can handle more complex analyses.

But this also means a significant rise in the need for computing power. And as these collaborative, multi-modal systems become more intricate — with as many as a trillion parameters — the demand for advanced computing infrastructure intensifies.

“We need even larger models,” Huang said. “We’re going to train it with multimodality data, not just text on the internet, we’re going to train it on texts and images, graphs and charts, and just as we learned watching TV  there’s going to be a whole bunch of watching video.”

The Next Generation of Accelerated Computing

In short, Huang said “we need bigger GPUs.” The Blackwell platform is built to meet this challenge. Huang pulled a Blackwell chip out of his pocket and held it up side-by-side with a Hopper chip, which it dwarfed.

Named for David Harold Blackwell — a University of California, Berkeley mathematician specializing in game theory and statistics, and the first Black scholar inducted into the National Academy of Sciences — the new architecture succeeds the NVIDIA Hopper architecture, launched two years ago.

Blackwell delivers 2.5x its predecessor’s performance in FP8 for training, per chip, and 5x with FP4 for inference. It features a fifth-generation NVLINK interconnect that’s twice as fast as Hopper and scales up to 576 GPUs.

And the NVIDIA GB200 Grace Blackwell Superchip connects two Blackwell NVIDIA B200 Tensor Core GPUs to the NVIDIA Grace CPU over a 900GB/s ultra-low-power NVLink chip-to-chip interconnect.

Huang held up a board with the system. “This computer is the first of its kind where this much computing fits into this small of a space,: Huang said. “Since this is memory coherent they feel like it’s one big happy family working on one application together.”

For the highest AI performance, GB200-powered systems can be connected with the NVIDIA Quantum-X800 InfiniBand and Spectrum-X800 Ethernet platforms, also announced today, which deliver advanced networking at speeds up to 800Gb/s.

“The amount of energy we save, the amount of networking bandwidth we save, the amount of wasted time we save, will be tremendous,” Huang said. “The future is generative…which is why this is a brand new industry. The way we compute is fundamentally different. We created a processor for the generative AI era.”

To scale up Blackwell, NVIDIA built a new chip called NVLINK Switch. Each  can connect four NVLinks at 1.8 terabytes per second and eliminate traffic by doing in-network reduction.

NVIDIA Switch and GB200 are key components of what Huang described as “one giant GPU,” the NVIDIA GB200 NVL72, a multi-node, liquid-cooled, rack-scale system that harnesses Blackwell to offer supercharged compute for trillion-parameter models, with 720 petaflops of AI training performance and 1.4 exaflops of AI inference performance in a single rack.

“There are only a couple, maybe three exaflop machines on the planet as we speak,” Huang said of the machine, which packs 600,000 parts and weighs 3,000 pounds. “And so this is an exaflop AI system in one single rack. Well let’s take a look at the back of it.”

Going even bigger, NVIDIA today also announced its next-generation AI supercomputer — the NVIDIA DGX SuperPOD powered by NVIDIA GB200 Grace Blackwell Superchips — for processing trillion-parameter models with constant uptime for superscale generative AI training and inference workloads.

Featuring a new, highly efficient, liquid-cooled rack-scale architecture, the new DGX SuperPOD is built with NVIDIA DG GB200 systems and provides 11.5 exaflops of AI supercomputing at FP4 precision and 240 terabytes of fast memory — scaling to more with additional racks.

“In the future, data centers are going to be thought of…as AI factories,” Huang said. “Their goal in life is to generate revenues, in this case, intelligence.”

The industry has already embraced Blackwell.

The press release announcing Blackwell includes endorsements from Alphabet and Google CEO Sundar Pichai, Amazon CEO Andy Jassy, Dell CEO Michael Dell, Google DeepMind CEO Demis Hassabis, Meta CEO Mark Zuckerberg, Microsoft CEO Satya Nadella, OpenAI CEO Sam Altman, Oracle Chairman Larry Ellison, and Tesla and xAI CEO Elon Musk.

Blackwell is being adopted by every major global cloud services provider,  pioneering AI companies, system and server vendors, and regional cloud service providers and telcos all around the world.

“The whole industry is gearing up for Blackwell,” which Huang said would be the most successful launch in the company’s history.

A New Way to Create Software

Generative AI changes the way applications are written, Huang said.

Rather than writing software, he explained, companies will assemble AI models, give them missions, give examples of work products, review plans and intermediate results.

These packages — NVIDIA NIMs, a reference to NVIDIA inference microservices — are built from NVIDIA’s accelerated computing libraries and generative AI models, Huang explained.

“How do we build software in the future? It is unlikely that you’ll write it from scratch or write a whole bunch of Python code or anything like that,” Huang said. “It is very likely that you assemble a team of AIs.”

The microservices support industry-standard APIs so they are easy to connect, work across NVIDIA’s large CUDA installed base, are re-optimized for new GPUs, and are constantly scanned for security vulnerabilities and exposures.

Huang said customers can use NIM microservices off-the-shelf, or NVIDIA can help build proprietary AI and co-pilots, teaching a model specialized skills only your company would know to create invaluable new services.

“The enterprise IT industry is sitting on a goldmine,” Huang said. “They have all these amazing tools (and data) that have been created over the years. If they could take that goldmine and turn it into copilots, these copilots can help us do things.”

Major tech players are already putting it to work. Huang detailed how NVIDIA is already helping Cohesity, NetApp, SAP, ServiceNow, and Snowflake build co-pilots and virtual assistants. And industries are stepping in, as well.

In telecoms, Huang announced the NVIDIA 6G research cloud, a generative AI and Omniverse-powered platform to advance the next communications era. It’s built with NVIDIA’s Sionna neural radio framework, NVIDIA Aerial CUDA-accelerated radio access network and the NVIDIA Aerial Omniverse Digital Twin for 6G.

In semiconductor design and manufacturing, Huang announced that, in collaboration with TSMC and Synopsys, NVIDIA is bringing its breakthrough computational lithography platform, cuLitho, to production. This platform will accelerate the most compute-intensive workload in semiconductor manufacturing by 40-60x.

Huang also announced the NVIDIA Earth Climate Digital Twin. The cloud platform — available now — enables interactive, high-resolution simulation to accelerate climate and weather prediction.

The greatest impact of AI will be in healthcare, Huang said, explaining that NVIDIA is already in imaging systems, in gene sequencing instruments and working with leading surgical robotics companies.

NVIDIA is launching a new type of biology software. NVIDIA today launched more than two dozen new microservices that allow healthcare enterprises worldwide to take advantage of the latest advances in generative AI from anywhere and on any cloud. They offer advanced imaging, natural language and speech recognition, and digital biology generation, prediction and simulation.

Omniverse Brings AI to the Physical World

The next wave of AI will be AI learning about the physical world, Huang said.

“We need a simulation engine that represents the world digitally for the robot so that the robot has a gym to go learn how to be a robot,” he said. “We call that virtual world Omniverse.”

That’s why NVIDIA today announced that NVIDIA Omniverse Cloud will be available as APIs, extending the reach of the world’s leading platform for creating industrial digital twin applications and workflows across the entire ecosystem of software makers.

The five new Omniverse Cloud application programming interfaces enable developers to easily integrate core Omniverse technologies directly into existing design and automation software applications for digital twins, or their simulation workflows for testing and validating autonomous machines like robots or self-driving vehicles.

To show how this works, Huang shared a demo of a robotic warehouse — using multi-camera perception and tracking — watching over workers and orchestrating robotic forklifts, which are driving autonomously with the full robotic stack running.

Hang also announced that NVIDIA is bringing Omniverse to Apple Vision Pro, with the new Omniverse Cloud APIs letting developers stream interactive industrial digital twins into the VR headsets.

Some of the world’s largest industrial software makers are embracing Omniverse Cloud APIs, including Ansys, Cadence, Dassault Systèmes for its 3DEXCITE brand, Hexagon, Microsoft, Rockwell Automation, Siemens and Trimble.

Robotics

Everything that moves will be robotic, Huang said. The automotive industry will be a big part of that, NVIDIA computers are already in cars, trucks, delivery bots and robotaxis.

Huang announced that BYD, the world’s largest AV company, has selected NVIDIA’s next-generation computer for their AV, building its next-generation EV fleets on DRIVE Thor.

To help robots better see their environment, Huang also announced the Isaac Perceptor software development kit with state-of-the-art multi-camera visual odometry, 3D reconstruction and occupancy map, and depth perception.

And to help make manipulators, or robotic arms, more adaptable, NVIDIA is announcing Isaac Manipulator — a state-of-the-art robotic arm perception, path planning and kinematic control library.
Finally, Huang announced Project GR00T, a general-purpose foundation model for humanoid robots, designed to further the company’s work driving breakthroughs in robotics and embodied AI.

Supporting that effort, Huang unveiled a new computer, Jetson Thor, for humanoid robots based on the NVIDIA Thor system-on-a-chip and significant upgrades to the NVIDIA Isaac robotics platform.

In his closing minutes, Huang brought on stage a pair of diminutive NVIDIA-powered robots from Disney Research.

“The soul of NVDIA — the intersection of computer graphics, physics, artificial intelligence,” he said.“It all came to bear at this moment.”

With nearly two dozen world records to its name, NVIDIA cuOpt now holds the top spot for 100% of the largest routing benchmarks in the last three years. And this means the route optimization engine allows industries to hop on board for all kinds of cost-saving efficiencies.

Kawasaki Heavy Industries and SyncTwin are among the companies that are riding cuOpt for logistics improvements.

Today at GTC 2024, NVIDIA founder and CEO Jensen Huang announced that cuOpt is moving into general availability.

“With cuOpt, NVIDIA is reinventing logistics management and operations research. It is NVIDIA’s pre-quantum computer, driving transformational operational efficiencies for deliveries, service calls, warehouses and factories, and supply chains,” he said.

The NVIDIA cuOpt microservice, part of the NVIDIA AI Enterprise software platform, makes accelerated optimization for real-time dynamic rerouting, factory optimization and robotic simulations available to any organization.

Companies can embed cuOpt into the advanced 3D tools, applications and USD-based workflows they develop with NVIDIA Omniverse, a software platform for developing and deploying advanced 3D applications and pipelines based on OpenUSD.

Implemented together, cuOpt, Omniverse and NVIDIA Metropolis for Factories can help optimize and create safe environments in logistics-heavy facilities that rely on complex automation, precise material flow and human-robot interaction, such as automotive factories, semiconductor fabs and warehouses.

cuOpt has been continuously tested against the best-known solutions on the most studied benchmarks for route optimization, with results up to 100x faster than CPU-based implementations. With 15 records from the Gehring & Homberger vehicle routing benchmark and eight from the Li & Lim pickup and delivery benchmark, cuOpt has demonstrated the world’s best accuracy with the fastest times.

AI promises to deliver logistics efficiencies spanning from transportation networks to manufacturing and much more.

Delivering Cost-Savings for Inspections With cuOpt

Kawasaki Heavy Industries is a manufacturing company that’s been building large machinery for more than a hundred years. The Japanese company partnered with Slalom and used cuOpt to create routing efficiencies for the development of its AI-driven Kawasaki Track Maintenance Platform.

Railroad track maintenance is getting an AI makeover worldwide. Traditionally, track inspections and maintenance are time-consuming and difficult to manage to keep trains running on time. But track maintenance is critical for safety and transportation service. Railway companies are automating track inspections with AI and machine learning paired with digital cameras, lasers and gyrometric sensors.

Kawasaki is harnessing the edge computing of NVIDIA Jetson AGX Orin to develop track inspections on its Track Maintenance Platform for running onboard trains. The platform enables customers to improve vision models with the data collected on tracks for advances in the inspection capability of the edge-based AI system.

The platform provides maintenance teams data on track conditions that allows them to prioritize repairs, creating increased safety and reliability of operations.

According to Kawasaki, it’s estimated that such an AI-driven system can save $218 million a year for seven companies from automating their track inspections.

Creating Manufacturing Efficiencies With cuOpt and Omniverse

A worldwide leader in automotive seating manufacturing has adopted SyncTwin’s digital twin capability, which is driven by Omniverse and cuOpt, to improve its operations with AI.

The global automotive seating manufacturer has a vast network of loading docks for the delivery of raw materials, and forklifts for unloading and transporting them to storage and handling areas to ensure a steady supply to production lines. SyncTwin’s connection to cuOpt delivers routing efficiencies that optimize all of these moving parts — from vehicles to robotic pallet jacks.

As the SyncTwin solution was developed on top of Omniverse and USD, manufacturers can ensure that their various factory planning tools can contribute to the creation of a rich digital twin environment. Plus, they eliminate tedious manual data collection and gain new insights from their previously disconnected data.

Attend GTC to explore how cuOpt is achieving world-record accuracy and performance to solve complex problems. Learn more about cuOpt world records in our tech blog. Learn more about Omniverse.

All eyes across the auto industry are on GTC — the global AI conference running in San Jose, Calif., and online through Thursday, March 21 — as the world’s top automakers and tech leaders converge to showcase the latest models, demo new technologies and dive into the remarkable innovations reshaping the sector.

Attendees will experience how generative AI and software-defined computing are advancing the automotive landscape and transforming the behind-the-wheel experience to become safer, smarter and more enjoyable.

Automakers Adopting NVIDIA DRIVE Thor

NVIDIA founder and CEO Jensen Huang kicked off GTC with a keynote address in which he revealed that NVIDIA DRIVE Thor, which combines advanced driver assistance technology and in-vehicle infotainment, now features the newly announced NVIDIA Blackwell GPU architecture for transformer and generative AI workloads.

Following the keynote, top EV makers shared how they will integrate DRIVE Thor into their vehicles. BYD, the world’s largest electric vehicle maker, is expanding its ongoing collaboration with NVIDIA and building its next-generation EV fleets on DRIVE Thor. Hyper, a premium luxury brand owned by GAC AION, is announcing it has selected DRIVE Thor for its new models, which will begin production in 2025. XPENG will use DRIVE Thor as the AI brain of its next-generation EV fleets. These EV makers join Li Auto and ZEEKR, which previously announced they’re building their future vehicle roadmaps on DRIVE Thor.

Additionally, trucking, robotaxis and goods delivery vehicle makers are announcing support for DRIVE Thor. Nuro is choosing DRIVE Thor to power the Nuro Driver. Plus is announcing that future generations of its level 4 solution, SuperDrive, will run on DRIVE Thor. Waabi is  leveraging DRIVE Thor to deliver the first generative AI-powered autonomous trucking solution to market. WeRide, in cooperation with tier 1 partner Lenovo Vehicle Computing, is creating level 4 autonomous driving solutions for commercial applications built on DRIVE Thor.

And, DeepRoute.ai is unveiling its new smart driving architecture powered by NVIDIA DRIVE Thor, scheduled to launch next year.

Next-Generation Tech on the Show Floor

The GTC exhibit hall is buzzing with excitement as companies showcase the newest vehicle models and offer technology demonstrations.

Attendees have the opportunity to see firsthand the latest NVIDIA-powered vehicles on display,  including Lucid Air, Mercedes-Benz Concept CLA Class, Nuro R3, Polestar 3, Volvo EX90, WeRide Robobus, and an Aurora truck. The Lucid Air is available for test drives during the week.

A wide array of companies are showcasing innovative automotive technology at GTC, including Foretellix, Luminar and MediaTek, which is launching its Dimensity Auto Cockpit chipsets at the show. The new solutions harness NVIDIA’s graphics and AI technologies to help deliver state-of-the-art in-vehicle user experiences, added safety and security capabilities.

Also Announced at GTC: Omniverse Cloud APIs, Generative AI

• Omniverse Cloud APIs, announced today at NVIDIA GTC, are poised to accelerate the path to autonomy by enabling high-fidelity sensor simulation for AV development and validation. Developers and software vendors such as CARLA, MathWorks, MITRE, Foretellix and Voxel51 underscore the broad appeal of these APIs in autonomous vehicles.
• Generative AI developers including Cerence, Geely, Li Auto, NIO, SoundHound, Tata Consulting Services and Wayve announced plans to transform the in-vehicle experience by using NVIDIA’s cloud-to-edge technology to help develop intelligent AI assistants, driver and passenger monitoring, scene understanding and more.

AI and Automotive Sessions Available Live and on Demand

Throughout the week, the world’s foremost experts on automotive technology will lead a broad array of sessions and panels at GTC, including:

• How LLMs and Generative AI Will Enhance the Way We Experience Self-Driving Cars
  Tuesday, March 19, 9 a.m. PT
• Accelerating the New Era of Autonomous Vehicles With Generative AI
  Tuesday, March 19, 10 a.m. PT
• Generative AI and Industrial Digitalization in the Automotive Industry
  Tuesday, March 19, 2 p.m. PT
• Accelerating Automotive Workflows With Large Language Models
  Tuesday, March 19, 3 p.m. PT
• Accelerating the Shift to AI-Defined Vehicles
  Thursday, March 21, 8 a.m. PT

On DRIVE Developer Day, taking place Thursday, March 21, NVIDIA’s engineering experts will highlight the latest DRIVE features and developments through a series of deep-dive sessions on how to build safe and robust self-driving systems.

See the full schedule of automotive programming at GTC and be sure to tune in.

Cars of the future will be more than just modes of transportation; they’ll be intelligent companions, seamlessly blending technology and comfort to enhance driving experiences, and built for safety, inside and out.

NVIDIA GTC, running this week at the San Jose Convention Center, will spotlight the groundbreaking work NVIDIA and its partners are doing to bring the transformative power of generative AI, large language models and visual language models to the mobility sector.

At its booth, NVIDIA will showcase how it’s building automotive assistants to enhance driver safety, security and comfort through enhanced perception, understanding and generative capabilities powered by deep learning and transformer models.

Talking the Talk

LLMs, a form of generative AI, largely represent a class of deep-learning architectures known as transformer models, which are neural networks adept at learning context and meaning.

Vision language models are another derivative of generative AI, that offer image processing and language understanding capabilities. Unlike traditional or multimodal LLMs that primarily process and generate text-based data, VLMs can analyze and generate text via images or videos.

And retrieval-augmented generations allows manufacturers to access knowledge from a specific database or the web to assist drivers.

These technologies together enable NVIDIA Avatar Cloud Engine, or ACE, and multimodal language models to work together with the NVIDIA DRIVE platform to let automotive manufacturers develop their own intelligent in-car assistants.

For example, an Avatar configurator can allow designers to build unique, brand-inspired personas for their cars, complete with customized voices and emotional attributes. These AI-animated avatars can engage in natural dialogue, providing real-time assistance, recommendations and personalized interactions.

Furthermore, AI-enhanced surround visualization enhances vehicle safety using 360-degree camera reconstruction, while the intelligent assistant sources external information, such as local driving laws, to inform decision-making.

Personalization is paramount, with AI assistants learning driver and passenger habits and adapting its behavior to suit occupants’ needs.

Generative AI for Automotive in Full Force at GTC 

Several NVIDIA partners at GTC are also showcasing their latest generative AI developments using NVIDIA’s edge-to-cloud technology:

• Cerence’s CaLLM is an automotive-specific LLM that serves as the foundation for the company’s next-gen in-car computing platform, running on NVIDIA DRIVE. The platform, unveiled late last year, is the future of in-car interaction, with an automotive- and mobility-specific assistant that provides an integrated in-cabin experience. Cerence is collaborating with NVIDIA engineering teams for deeper integration of CaLLM with the NVIDIA AI Foundation Models. Through joint efforts, Cerence is harnessing NVIDIA DGX Cloud as the development platform, applying guardrails for enhanced performance, and leveraging NVIDIA AI Enterprise to optimize inference. NVIDIA and Cerence will continue to partner and pioneer this solution together with several automotive OEMs this year.
• Wavye is helping usher in the new era of Embodied AI for autonomy, their next-generation AV2.0 approach is characterized by a large Embodied AI foundation model that learns to drive self-supervised using AI end-to-end —from sensing, as an input, to outputting driving actions. The British startup has already unveiled its GAIA-1, a generative world model for AV development running on NVIDIA; alongside LINGO-1, a closed-loop driving commentator that uses natural language to enhance the learning and explainability of AI driving models.
• Li Auto unveiled its multimodal cognitive model, Mind GPT, in June. Built on NVIDIA TensorRT-LLM, an open-source library, it serves as the basis for the electric vehicle maker’s AI assistant, Lixiang Tongxue, for scene understanding, generation, knowledge retention and reasoning capabilities. Li Auto is currently developing DriveVLM to enhance autonomous driving capabilities, enabling the system to understand complex scenarios, particularly those that are challenging for traditional AV pipelines, such as unstructured roads, rare and unusual objects, and unexpected traffic events. This advanced model is trained on the NVIDIA GPUs and utilizes TensorRT-LLM and NVIDIA Triton Inference Server for data generation in the data center. With inference optimized by NVIDIA DRIVE and TensorRT-LLM, DriveVLMs perform efficiently on embedded systems.
• NIO launched its NOMI GPT, which offers a number of functional experiences, including NOMI Encyclopedia Q&A, Cabin Atmosphere Master and Vehicle Assistant. With the capabilities enabled by LLMs and an efficient computing platform powered by NVIDIA AI stacks, NOMI GPT is capable of basic speech recognition and command execution functions and can use deep learning to understand and process more complex sentences and instructions inside the car.
• Geely is working with NVIDIA to provide intelligent cabin experiences, along with accelerated edge-to-cloud deployment. Specifically, Geely is applying generative AI and LLM technology to provide smarter, personalized and safer driving experiences, using natural language processing, dialogue systems and predictive analytics for intelligent navigation and voice assistants. When deploying LLMs into production, Geely uses NVIDIA TensorRT-LLM to achieve highly efficient inference. For more complex tasks or scenarios requiring massive data support, Geely plans to deploy large-scale models in the cloud.
• Waabi is building AI for self-driving and will use the generative AI capabilities afforded by NVIDIA DRIVE Thor for its breakthrough autonomous trucking solutions, bringing safe and reliable autonomy to the trucking industry.
• Lenovo is unveiling a new AI acceleration engine, dubbed UltraBoost, which will run on NVIDIA DRIVE, and features an AI model engine and AI compiler tool chains to facilitate the deployment of LLMs within vehicles.
• SoundHound AI is using NVIDIA to run its in-vehicle voice interface — which combines both real-time and generative AI capabilities — even when a vehicle has no cloud connectivity. This solution also offers drivers access to SoundHound’s Vehicle Intelligence product, which instantly delivers settings, troubleshooting and other information directly from the car manual and other data sources via natural speech, as opposed to through a physical document.
• Tata Consulting Services (part of the TATA Group), through its AI-based technology and engineering innovation, has built its automotive GenAI suite powered by NVIDIA GPUs and software frameworks. It accelerates the design, development, and validation of software-defined vehicles, leveraging the various LLMs and VLMs for in-vehicle and cloud-based systems.
• MediaTek is announcing four automotive systems-on-a-chip within its Dimensity Auto Cockpit portfolio, offering powerful AI-based in-cabin experiences for the next generation of intelligent vehicles that span from premium to entry level. To support deep learning capabilities, the Dimensity Auto Cockpit chipsets integrate NVIDIA’s next-gen GPU-accelerated AI computing and NVIDIA RTX-powered graphics to run LLMs in the car, allowing vehicles to support chatbots, rich content delivery to multiple displays, driver alertness detection and other AI-based safety and entertainment applications.

Check out the many automotive talks on generative AI and LLMs throughout the week of GTC.

Register today to attend GTC in person, or tune in virtually, to explore how generative AI is making transportation safer, smarter and more enjoyable.