Posts in category: nVidia AI

The world’s largest democracy is poised to transform itself and the world, embracing AI on an enormous scale.

Speaking with the press Friday in Bengaluru, in the context of announcements from two of India’s largest conglomerates, Reliance Industries Limited and Tata Group, NVIDIA founder and CEO Jensen Huang detailed plans to bring AI technology and skills to address the world’s most populous nation’s greatest challenges.

“I think this is going to be one of the largest AI markets in the world,” said Huang, who was wrapping up a week of high-level meetings across the nation, including with Prime Minister Narendra Modi, leading AI researchers, top business leaders, the press and the country’s 4,000-some NVIDIA employees.

The companies will work together to create an AI computing infrastructure and platforms for developing AI solutions. It will be based on NVIDIA technology like the NVIDIA GH200 Grace Hopper Superchip and NVIDIA DGX Cloud.

GH200 marks a fundamental shift in computing architecture that provides exceptional performance and massive memory bandwidth, while DGX Cloud, an AI supercomputing service in the cloud, makes it easier for enterprises to train their employees in AI technology, access the technology internally and provide generative AI services to customers.

In his exchange with more than a dozen of India’s top tech journalists following the announcement, Huang said computer science expertise is a core competency for India, and that with access to technology and capital India is poised to build AI to be able to solve challenges at home and abroad.

“You have the data, you have the talent,” Huang said. “We are open for business and bring great expertise on building supercomputers.

During the freewheeling back and forth with the media, Hiuang emphasized India’s strength in information technology and the potential for AI to accelerate the development of India’s IT industry.

“IT is one of your natural resources. You produce it at an incredible scale. You’re incredibly good at it. You export it all over the world,” Huang said.

India’s “AI Moment”

Earlier, after meeting with many of the region’s top technology leaders — including startup pioneers, AI proponents, and key players in India’s digital public infrastructure — Huang hailed “India’s moment,” saying the nationis on the cusp of becoming a global AI powerhouse.

While India has well-known technical capabilities — distinguished technical universities, 2,500 engineering colleges and an estimated 1.5 million engineers — many of its 1.4 billion people, located across sprawling metropolitan areas and some 650,000 villages, collectively speaking dozens of languages, have yet to fully benefit from this progress.

Applied in the Indian context, AI can help rural farmers interact via cell phones in their local language to get weather information and crop prices. It can help provide, at a massive scale, expert diagnosis of medical symptoms and imaging scans where doctors may not be immediately available. It can better predict cyclonic storms using decades of atmospheric data, enabling those at risk to more quickly evacuate and find shelter.

Reliance Industries and Tata Communications will build and operate state-of-the-art AI supercomputing data centers based on such technology, utilizing it for internal AI development and infrastructure-as-a-service for India’s AI researchers, companies and burgeoning AI startup ecosystem.

That effort, Huang said, during his conversation with the Indian technology press, promises to be part of a process that will turn India into a beacon for AI technology.

“AI could be built in India, used in India, and exported from India,” Huang said.

Thanks to rapid technological advances, consumers have become accustomed to an unprecedented level of convenience and efficiency.

Smartphones make it easier than ever to search for a product and have it delivered right to the front door. Video chat technology lets friends and family on different continents connect with ease. With voice command tools, AI assistants can play songs, initiate phone calls or recommend the best Italian food in a 10-mile radius. AI algorithms can even predict which show users may want to watch next or suggest an article they may want to read before making a purchase.

It’s no surprise, then, that customers expect fast and personalized interactions with companies. According to a Salesforce research report, 83% of consumers expect immediate engagement when they contact a company, while 73% expect companies to understand their unique needs and expectations. Nearly 60% of all customers want to avoid customer service altogether, preferring to resolve issues with self-service features.

Meeting such high consumer expectations places a massive burden on companies in every industry, including on their staff and technological needs — but speech AI can help.

Speech AI can understand and converse in natural language, creating opportunities for seamless, multilingual customer interactions while supplementing employee capabilities. It can power self-serve banking in the financial services industry, enable food kiosk avatars in restaurants, transcribe clinical notes in healthcare facilities or streamline bill payments for utility companies — helping businesses across industries deliver personalized customer experiences.

Speech AI for Banking and Payments

Most people now use both digital and traditional channels to access banking services, creating a demand for omnichannel, personalized customer support. However, higher demand for support coupled with a high agent churn rate has left many financial institutions struggling to keep up with the service and support needs of their customers.

Common consumer frustrations include difficulty with complex digital processes, a lack of helpful and readily available information, insufficient self-service options, long call wait times and communication difficulties with support agents.

According to a recent NVIDIA survey, the top AI use cases for financial service institutions are natural language processing (NLP) and large language models (LLMs). These models automate customer service interactions and process large bodies of unstructured financial data to provide AI-driven insights that support all lines of business across financial institutions — from risk management and fraud detection to algorithmic trading and customer service.

By providing speech-equipped self-service options and supporting customer service agents with AI-powered virtual assistants, banks can improve customer experiences while controlling costs. AI voice assistants can be trained on finance-specific vocabulary and rephrasing techniques to confirm understanding of a user’s request before offering answers.

Kore.ai, a conversational AI software company, trained its BankAssist solution on 400-plus retail banking use cases for interactive voice response, web, mobile, SMS and social media channels. Customers can use a voice assistant to transfer funds, pay bills, report lost cards, dispute charges, reset passwords and more.

Kore.ai’s agent voice assistant has also helps live agents provide personalized suggestions so they can resolve issues faster. The solution has been shown to improve live agent efficiency by cutting customer handling time by 40% with a return on investment of $2.30 per voice session.

With such trends, expect financial institutions to accelerate the deployment of speech AI to streamline customer support and reduce wait times, offer more self-service options, transcribe calls to speed loan processing and automate compliance, extract insights from spoken content and boost the overall productivity and speed of operations.

Speech AI for Telecommunications    

Heavy investments in 5G infrastructure and cut-throat competition to monetize and achieve profitable returns on new networks mean that maintaining customer satisfaction and brand loyalty is paramount in the telco industry.

According to an NVIDIA survey of 400-plus industry professionals, the top AI use cases in the telecom industry involve optimizing network operations and improving customer experiences. Seventy-three percent of respondents reported increased revenue from AI.

By using speech AI technologies to power chatbots, call-routing, self-service features and recommender systems, telcos can enhance and personalize customer engagements.

KT, a South Korean mobile operator with over 22 million users, has built GiGa Genie, an intelligent voice assistant that’s been trained to understand and use the Korean language using LLMs. It has already conversed with over 8 million users.

By understanding voice commands, the GiGA Genie AI speaker can support people with tasks like turning on smart TVs or lights, sending text messages or providing real-time traffic updates.

KT has also strengthened its AI-powered Customer Contact Center with transformer-based speech AI models that can independently handle over 100,000 calls per day. A generative AI component of the system autonomously responds to customers with suggested resolutions or transfers them to human agents for more nuanced questions and solutions.

Telecommunications companies are expected to lean into speech AI to build more customer self-service capabilities, optimize network performance and enhance overall customer satisfaction.

Speech AI for Quick-Service Restaurants

The food service industry is expected to reach $997 billion in sales in 2023, and its workforce is projected to grow by 500,000 openings. Meanwhile, elevated demand for drive-thru, curbside pickup and home delivery suggests a permanent shift in consumer dining preferences. This shift creates the challenge of hiring, training and retaining staff in an industry with notoriously high turnover rates — all while meeting consumer expectations for fast and fresh service.

Drive-thru order assistants and in-store food kiosks equipped with speech AI can help ease the burden. For example, speech-equipped avatars can help automate the ordering process by offering menu recommendations, suggesting promotions, customizing options or passing food orders directly to the kitchen for preparation.

HuEx, a Toronto-based startup and member of NVIDIA Inception, has designed a multilingual automated order assistant to enhance drive-thru operations. Known as AIDA, the AI assistant receives and responds to orders at the drive-thru speaker box while simultaneously transcribing voice orders into text for food-prep staff.

AIDA understands 300,000-plus product combinations with 90% accuracy, from common requests such as “coffee with milk” to less common requests such as “coffee with butter.” It can even understand different accents and dialects to ensure a seamless ordering experience for a diverse population of consumers.

Speech AI streamlines the order process by speeding fulfillment, reducing miscommunication and minimizing customer wait times. Early movers will also begin to use speech AI to extract customer insights from voice interactions to inform menu options, make upsell recommendations and improve overall operational efficiency while reducing costs.

Speech AI for Healthcare

In the post-pandemic era, the digitization of healthcare is continuing to accelerate. Telemedicine and computer vision support remote patient monitoring, voice-activated clinical systems help patients check in and receive zero-touch care and speech recognition technology supports clinical documentation responsibilities. Per IDC, 36% of survey respondents indicated that they had deployed digital assistants for patient healthcare.

Automated speech recognition and NLP models can now capture, recognize, understand and summarize key details in medical settings. At the Conference for Machine Intelligence in Medical Imaging, NVIDIA researchers showcased a state-of-the-art pretrained architecture with speech-to-text functionality to extract clinical entities from doctor-patient conversations. The model identifies clinical words — including symptoms, medication names, diagnoses and recommended treatments — and automatically updates medical records.

This technology can ease the burden of manual note-taking and has the potential to accelerate insurance and billing processes while also creating consultation recaps for caregivers. Relieved of administrative tasks, physicians can focus on patient care to deliver superior experiences.

Artisight, an AI platform for healthcare, uses speech recognition to power zero-touch check-ins and speech synthesis to notify patients in the waiting room when the doctor is available. Over 1,200 patients per day use Artisight kiosks, which help streamline registration processes, improve patient experiences, eliminate data entry errors with automation and boost staff productivity.

As healthcare moves toward a smart hospital model, expect to see speech AI play a bigger role in supporting medical professionals and powering low-touch experiences for patients. This may include risk factor prediction and diagnosis through clinical note analysis, translation services for multilingual care centers, medical dictation and transcription and automation of other administrative tasks.

Speech AI for Energy

Faced with increasing demand for clean energy, high operating costs and a workforce retiring in greater numbers, energy and utility companies are looking for ways to do more with less.

To drive new efficiencies, prepare for the future of energy and meet ever-rising customer expectations, utilities can use speech AI. Voice-based customer service can enable customers to report outages, inquire about billing and receive support on other issues without agent intervention. Speech AI can streamline meter reading, support field technicians with voice notes and voice commands to access work orders and enable utilities to analyze customer preferences with NLP.

Minerva CQ, an AI assistant designed specifically for retail energy use cases, supports customer service agents by transcribing conversations into text in real time. Text is fed into Minerva CQ’s AI models, which analyze customer sentiment, intent, propensity and more.

By dynamically listening, the AI assistant populates an agent’s screen with dialogue suggestions, behavioral cues, personalized offers and sentiment analysis. A knowledge-surfacing feature pulls up a customer’s energy usage history and suggests decarbonization options — arming agents with the information needed to help customers make informed decisions about their energy consumption.

With the AI assistant providing consistent, simple explanations on energy sources, tariff plans, billing changes and optimal spending, customer service agents can effortlessly guide customers to the most ideal energy plan. After deploying Minerva CQ, one utility provider reported a 44% reduction in call handling time, a 12.5% increase in first-contact resolution and average savings of $2.67 per call.

Speech AI is expected to continue to help utility providers reduce training costs, remove friction from customer service interactions and equip field technicians with voice-activated tools to boost productivity and improve safety — all while enhancing customer satisfaction.

Speech and Translation AI for the Public Sector

Because public service programs are often underfunded and understaffed, citizens seeking vital services and information are at times left waiting and frustrated. To address this challenge, some federal- and state-level agencies are turning to speech AI to achieve more timely service delivery.

The Federal Emergency Management Agency uses automated speech recognition systems to manage emergency hotlines, analyze distress signals and direct resources efficiently. The U.S. Social Security Administration uses an interactive voice response system and virtual assistants to respond to inquiries about social security benefits and application processes and to provide general information.

The Department of Veterans Affairs has appointed a director of AI to oversee the integration of the technology into its healthcare systems. The VA uses speech recognition technology to power note-taking during telehealth appointments. It has also developed an advanced automated speech transcription engine to help score neuropsychological tests for analysis of cognitive decline in older patients.

Additional opportunities for speech AI in the public sector include real-time language translation services for citizen interactions, public events or visiting diplomats. Public agencies that handle a large volume of calls can benefit from multilingual voice-based interfaces to allow citizens to access information, make inquiries or request services in different languages.

Speech and translation AI can also automate document processing by converting multilingual audio recordings or spoken content into translated text to streamline compliance processes, improve data accuracy and enhance administrative task efficiency. Speech AI additionally has the potential to expand access to services for people with visual or mobility impairments.

Speech AI for Automotive 

From vehicle sales to service scheduling, speech AI can bring numerous benefits to automakers, dealerships, drivers and passengers alike.

Before visiting a dealership in person, more than half of vehicle shoppers begin their search online, then make the first contact with a phone call to collect information. Speech AI chatbots trained on vehicle manuals can answer questions on technological capabilities, navigation, safety, warranty, maintenance costs and more. AI chatbots can also schedule test drives, answer pricing questions and inform shoppers of which models are in stock. This enables automotive manufacturers to differentiate their dealership networks through intelligent and automated engagements with customers.

Manufacturers are building advanced speech AI into vehicles and apps to improve driving experiences, safety and service. Onboard AI assistants can execute natural language voice commands for navigation, infotainment, general vehicle diagnostics and querying user manuals. Without the need to operate physical controls or touch screens, drivers can keep their hands on the wheel and eyes on the road.

Speech AI can help maximize vehicle up-time for commercial fleets. AI trained on technical service bulletins and software update cadences lets technicians provide more accurate quotes for repairs, identify key information before putting the car on a lift and swiftly supply vehicle repair updates to commercial and small business customers.

With insights from driver voice commands and bug reports, manufacturers can also improve vehicle design and operating software. As self-driving cars become more advanced, expect speech AI to play a critical role in how drivers operate vehicles, troubleshoot issues, call for assistance and schedule maintenance.

Speech AI — From Smart Spaces to Entertainment

Speech AI has the potential to impact nearly every industry.

In Smart Cities, speech AI can be used to handle distress calls and provide emergency responders with crucial information. In Mexico City, the United Nations Office on Drugs and Crime is developing a speech AI program to analyze 911 calls to prevent gender violence. By analyzing distress calls, AI can identify keywords, signals and patterns to help prevent domestic violence against women. Speech AI can also be used to deliver multilingual services in public spaces and improve access to transit for people who are visually impaired.

In higher education and research, speech AI can automatically transcribe lectures and research interviews, providing students with detailed notes and saving researchers the time spent compiling qualitative data. Speech AI also facilitates the translation of educational content to various languages, increasing its accessibility.

AI translation powered by LLMs is making it easier to consume entertainment and streaming content online in any language. Netflix, for example, is using AI to automatically translate subtitles into multiple languages. Meanwhile, startup Papercup is using AI to automate video content dubbing to reach global audiences in their local languages.

Transforming Product and Service Offerings With Speech AI

In the modern consumer landscape, it’s imperative that companies provide convenient, personalized customer experiences. Businesses can use NLP and the translation capabilities of speech AI to transform the way they operate and interact with customers in real time on a global scale.

Companies across industries are using speech AI to deliver rapid, multilingual customer service responses, self-service features and information and automation tools to empower employees to provide higher-value experiences.

To help enterprises in every industry realize the benefits of speech, translation and conversational AI, NVIDIA offers a suite of technologies.

NVIDIA Riva, a GPU-accelerated multilingual speech and translation AI software development kit, powers fully customizable real-time conversational AI pipelines for automatic speech recognition, text-to-speech and neural machine translation applications.

NVIDIA Tokkio, built on the NVIDIA Omniverse Avatar Cloud Engine, offers cloud-native services to create virtual assistants and digital humans that can serve as AI customer service agents.

These tools enable developers to quickly deploy high-accuracy applications with the real-time response speed needed for superior employee and customer experiences.

Join the free Speech AI Day on Sept. 20 to hear from renowned speech and translation AI leaders about groundbreaking research, real-world applications and open-source contributions.

GeForce NOW brings expanded support for PC Game Pass to members this week. Members can stream eight more games from Microsoft’s subscription service, including four titles from hit publisher Focus Entertainment.

Play A Plague Tale: Requiem, Atomic Heart and more from the GeForce NOW library at up to 4K resolution and 120 frames per second with a GeForce NOW Ultimate membership.

Plus, time’s almost up to take on the Ultimate KovaaK’s Challenge. Get on the leaderboard today — the challenge ends on Thursday, Sept. 21.

Laser-Focused 

Four games from Focus Entertainment’s PC Game Pass catalog join GeForce NOW this week. Members signed up with Microsoft’s subscription service can now stream titles like A Plague Tale: Requiem, Atomic Heart and more titles at stunning quality across their devices — without additional purchases.

Embark on a heartrending journey into a brutal, breathtaking world in the critically acclaimed A Plague Tale: Requiem or explore an alternate history of the 1950s Soviet Union in Atomic Heart. Go off road in SnowRunner or salvage among the stars in Hardspace: Shipbreaker. Members can even bring the squad together for military battles in Insurgency: Sandstorm. There’s something for everyone.

Experience it all with a PC Game Pass subscription, best paired with a GeForce NOW Ultimate membership, which provides up to 4K streaming or up to 240 fps for the ultimate cloud gaming experience.

Endless Adventures

A new week, a new batch of games. Catch the 16 new games supported in the cloud this week:

• Chants of Sennaar (New release on Steam, Sept. 5)
• SYNCED (New release on Steam, Sept. 7)
• Void Crew (New release on Steam, Sept. 7)
• Deceive Inc. (Steam)
• A Plague Tale: Requiem (Xbox)
• Airborne Kingdom (Epic Games Store)
• Atomic Heart (Xbox)
• Call of the Wild: The Angler (Xbox)
• Danganronpa V3: Killing Harmony (Xbox)
• Death in the Water (Steam)
• Hardspace: Shipbreaker (Xbox)
• Insurgency: Sandstorm (Xbox)
• Monster Sanctuary (Xbox)
• Saints Row (Steam)
• Shadowrun: Hong Kong – Extended Edition (Xbox)
• SnowRunner (Xbox)
• War for the Overworld (Steam)

What are you planning to play this weekend? Let us know on Twitter or in the comments below.

Editor’s note: This is part of a series profiling researchers advancing science with high performance computing. 

Before she entered high school, Ge Dong wanted to be a physicist like her mom, a professor at Shanghai Jiao Tong University.

“She said clean energy was really important for sustaining humanity, she talked about it a lot,” said Ge Dong (above at age two with her mom).

At 32, she’s following that dream at a startup that hopes to find — with the help of HPC and AI — a commercial path to nuclear fusion.

Pioneering AI in Physics

In 2014, her life’s work took her more than 7,000 miles from her Shanghai home to Princeton University’s prestigious plasma physics lab, where she earned a Ph.D.

Her doctoral thesis was built on advances by Princeton colleagues. They were the first to use AI to predict plasma disruptions that could cause a fusion reactor to fail.

Ge Dong’s work shed light on how the edges of plasma, hotter than the surface of the sun, behave inside a prototype fusion reactor, a donut-shaped enclosure called a tokamak.

Later, she spent more than a year working with her colleagues and NVIDIA experts to create with NVIDIA Omniverse a digital twin to show how plasma circles inside a tokamak. Using AI, the effort slashed the costs of a simulation based on traditional number-crunching methods.

The results may help engineers build controls that keep superheated plasma safely inside tomorrow’s power plants, speeding the arrival of the clean energy source.

A Pivotal Conversation

During the Covid lockdown, Ge Dong returned to Shanghai to work from home. There, in 2021, a pivotal conversation with a friend, Zhou Yang, led to the decision to co-found Energy Singularity, a startup with an ambitious plan.

Yang said he wanted to build a tokamak. When she dismissed the multibillion-dollar idea, he gave a detailed breakdown of a plan that would cost far less.

Then he explained why he wanted to take an approach, popular among researchers, of using high-temperature superconducting magnets to control the plasma. Even though he studied a separate branch of physics, he could explain the rationale down to its fundamental equations.

After their talk, “I was so excited, I didn’t sleep the whole night,” she said of the bold plan.

A few months later, they joined three others to launch the company.

A Fresh Challenge for AI

Learning how to build and control the powerful, but fragile magnets is the startup’s chief technical challenge. The team is turning to HPC and AI to find its way.

“It’s a whole new area of research that’s ripe for the kind of statistical analysis AI can accelerate to deliver the most effective and lowest cost approach,” she said.

The startup is already designing its prototype on an NVIDIA-accelerated server in its office.

“We’ve been using NVIDIA GPUs for all our research, they’re one of the most important tools in plasma physics these days,” she said.

The Next Generation

The work can be all-consuming. No one on the team has had time to check out the free gym in their building. And it’s been a while since Ge Dong had a good game of badminton, a favorite pastime.

But she remains upbeat. Within a decade, someone will show the way to harnessing nuclear fusion, and it could be her company, she said.

Ge Dong is sure her five-year-old daughter will see her intense passion for plasma physics. But when it comes time to choose a career, she may hear a different calling in a fusion-powered world.

Check out other profiles in this series:

• Scientist Sharpens Weather Forecasts With AI
• Scientist Explores Quantum Worlds With HPC, AI
• AI, HPC Help Fuel Breakthrough in Nuclear Fusion
• Scientist Speeds CERN’s HPC With GPUs, AI

Editor’s note: This post is part of our weekly In the NVIDIA Studio series, which celebrates featured artists, offers creative tips and tricks and demonstrates how NVIDIA Studio technology improves creative workflows. We’re also deep diving into new GeForce RTX 40 Series GPU features, technologies and resources and how they dramatically accelerate content creation.

Rafi Nizam is an award-winning independent animator, director, character designer and more. He’s developed feature films at Sony Pictures, children’s series and comedies at BBC and global transmedia content at NBCUniversal.

He’s also the creator of Arteana’s Art Squad — a computer graphics animated series featuring vibrant characters who use the power of art to solve the world’s problems. They come together in the Junior School art classroom, where each brings unique artistic talents, knowledge and perspective on art history, art therapy and in art-making.

Aimed at children, the series seeks to inspire viewers by portraying the characters’ artistic journeys and the power of creative expression. Their adventures are meant to spark a sense of empathy by exploring the universal themes of self-doubt, social dynamics, success and failure. Underscoring the power of imagination and creative thinking is a common throughline.

Nizam’s creative insight and unique perspective are the subjects of this week’s In the NVIDIA Studio installment.

The artist recently participated in the ASUS ProArt Masters’ Talks sessions program, where he demonstrated how ASUS ProArt solutions, including the NVIDIA Studio-validated ProArt Studiobook Pro 16 OLED laptop with a GeForce RTX 3060 GPU and the Scan 3XS RTX Studio workstation with NVIDIA RTX A6000 graphic cards, helped produce a high-end animated series on an indie budget.

Meet Arteana’s Art Squad 

Meet Arteana, leader of the Art Squad, who possesses a keen interest in historical artists and art movements.

Rivette demonstrates diverse art techniques and is always looking for new ways to express her creativity.

ThreeDee, seen here playing the drums, is a kind and compassionate character who uses art therapy as a means of promoting well-being and healing and to uncover the underlying worries that plague the squad.

Then there’s Figgi, whose spontaneous performance art inspires others to redefine boundaries and embrace individuality.

Rounding out the squad is PuttPupp — a lovable and playful character made of putty erasers — who serves as the class pet.

Art Squad, Assemble

Nizam — matching the demeanor and spirit of his work — is honest. He’s not an expert at 3D modeling, nor is he a visual effects artist, and he’s not the most adept at production pipelines. However, he does love to draw.

His focus has always been on characters, storytelling and world-building. He built Arteana’s Art Squad independently while working in NVIDIA Omniverse, a platform for building and connecting 3D tools and apps.

“As a showrunner, embarking on making a CG animated show without a crew is kind of daunting, but I’m using NVIDIA Omniverse to discover ways to overcome my limitations in this space,” Nizam said.

Nizam began by modeling each squad member and building production assets in Adobe Substance 3D Modeler using VR. He also utilized the VR app Gravity Sketch to create models for the different objects required in each set or scene.

“Designing 3D character models in VR makes pre-production and look dev possible for an artist like me,“  he said.

Nizam imported his character into Autodesk Maya for the rigging process — creating a skeleton for the 3D model so that it can move.

His RTX GPU delivered AI-powered, accelerated denoising with the default Autodesk Arnold renderer, resulting in highly interactive and photorealistic renders.

Nizam then moved to Adobe Substance 3D Painter to create textures and materials, applying them to production assets. NVIDIA RTX-accelerated light and ambient occlusion baking optimized assets in mere seconds.

Next, Nizam deployed Unreal Engine to record motion captures via a Perception Neuron suit, creating scenes and camera sequences in real time. NVIDIA DLSS technology increased the interactivity of the viewport by using AI to upscale frames rendered at lower resolution, while retaining high-fidelity detail.

“Motion capture fosters experimentation and spontaneous collaboration with performers capturing an abundance of movement, a luxury often untenable for indie projects,” said Nizam.

NVIDIA Omniverse’s spatial computing capabilities took Nizam’s creative workflow to the next level. The Omniverse USD Composer’s native VR support enables artists to interactively assemble, light and navigate scenes in real time, individually or collaboratively, in fully ray-traced VR.

Here, Nizam adjusted scene lighting and approved the overall layout in VR. He then moved to desktop to polish and refine the 3D sequences, reviewing final shots before exporting the completed project.

Final Renders, Final Thoughts

Nizam is a big proponent of Omniverse, OpenUSD and its ability to streamline 3D content creation.

“Less time and effort, more productivity, cost savings and simpler real-time workflows — I use Omniverse daily for these reasons,” he said.

The Omniverse platform has at its foundation OpenUSD, an open and extensible framework for describing, composing, simulating and collaborating within 3D worlds. OpenUSD unlocks Omniverse’s potential by enabling movement between 3D apps — artists can transition all individual assets to their desired format with a single click.

“All apps were in sync and updated on the fly while I assembled it, thanks to Omniverse being the backbone of my CG creative and production process,” Nizam said.

“I rely on Omniverse Nucleus and Cache as the USD infrastructure for my production pipeline, allowing for seamless collaboration and facilitating cross-application workflows,” Nizam said. “Additionally, I utilize various software connectors, which help bridge different apps and streamline the creative process.”

Check out Nizam on Instagram.

Follow NVIDIA Studio on Instagram, Twitter and Facebook. Access tutorials on the Studio YouTube channel and get updates directly in your inbox by subscribing to the Studio newsletter. 

Get started with NVIDIA Omniverse by downloading the free standard license or learn how Omniverse Enterprise can connect your team. Developers can get started with Omniverse resources. Stay up to date on the platform by subscribing to the newsletter and follow NVIDIA Omniverse on Instagram, Medium and Twitter. 

For more, join the Omniverse community and check out the Omniverse forums, Discord server, Twitch and YouTube channels.

With coral reefs in rapid decline across the globe, researchers from the University of Hawaii at Mānoa have pioneered an AI-based surveying tool that monitors reef health from the sky.

Using deep learning models and high-resolution satellite imagery powered by NVIDIA GPUs, the researchers have developed a new method for spotting and tracking coral reef halos — distinctive rings of barren sand encircling reefs.

The study, recently published in the Remote Sensing of Environment journal, could unlock real-time coral reef monitoring and turn the tide on global conservation.

“Coral reef halos are a potential proxy for ecosystem health,” said Amelia Meier, a postdoctoral fellow at the University of Hawaii and co-author of the study. “Visible from space, these halo patterns give scientists and conservationists a unique opportunity to observe vast and distant areas. With AI, we can regularly assess halo presence and size in near real time to determine ecosystem well-being.”

Sea-ing Clearly: Illuminating Reef Health

Previously attributed solely to fish grazing, reef halos can also indicate a healthy predator-prey ecosystem, according to researchers’ recent discoveries. While some herbivorous fish graze algae or seagrass near the protective reef perimeter, hunters dig around the seafloor for burrowed invertebrates, laying bare the surrounding sand.

These dynamics indicate the area hosts a healthy food buffet for sustaining a diverse population of ocean dwellers. When the halo changes shape, it signals an imbalance in the marine food web and could indicate an unhealthy reef environment.

In Hot Water

While making up less than 1% of the ocean, coral reefs offer habitat, food and nursery grounds for over 1 million aquatic species. There’s also huge commercial value — about $375 billion annually in commercial and subsistence fishing, tourism and coastal storm protection, and providing antiviral compounds for drug discovery research.

However, reef health is threatened by overfishing, nutrient contamination and ocean acidification. Intensifying climate change — along with the resulting thermal stress from a warming ocean — also increases coral bleaching and infectious disease.

Over half of the world’s coral reefs are already lost or badly damaged, and scientists predict that by 2050 all reefs will face threats, with many in critical danger.

Charting New Horizons With AI

Spotting changes in reef halos is key to global conservation efforts. However, tracking these changes is labor- and time-intensive, limiting the number of surveys that researchers can perform every year. Access to reefs in remote locations also poses challenges.

The researchers created an AI tool that identifies and measures reef halos from global satellites, giving conservationists an opportunity to proactively address reef degradation.

Using Planet SkySat images, they developed ‌a dual-model framework employing two types of convolutional neural networks (CNNs). Relying on computer vision methods for image segmentation, they trained a Mask R-CNN model that detects the edges of the reef and halo, pixel by pixel. A U-Net model trained to differentiate between the coral reef and halo then classifies and predicts the areas of both.

The team used TensorFlow, Keras and PyTorch libraries for training and testing thousands of annotations on the coral reef models.

To handle the task’s large compute requirements, the CNNs operate on an NVIDIA RTX A6000 GPU, boosted by a cuDNN-accelerated PyTorch framework. The researchers received the A6000 GPU as participants in the NVIDIA Academic Hardware Grant Program.

The AI tool quickly identifies and measures around 300 halos across 100 square kilometers in about two minutes. The same task takes a human annotator roughly 10 hours. The model also reaches about 90% accuracy depending on location and can navigate various and complicated halo patterns.

“Our study marks the first instance of training AI on reef halo patterns, as opposed to more common AI datasets of images, such as those of cats and dogs,” Meier said. “Processing thousands of images can take a lot of time, but using the NVIDIA GPU sped up the process significantly.”

One challenge is that image resolution can be a limiting factor in the model’s accuracy. Course-scale imagery with low resolutions makes it difficult to spot ‌reef and halo boundaries and creates less accurate predictions.

Shoring Up Environmental Monitoring

“Our long-term goal is to transform our findings into a robust monitoring tool for assessing changes in halo size and to draw correlations to the population dynamics of predators and herbivores in the area,” Meier said.

With this new approach, the researchers are exploring the relationship between species composition, reef health, and halo presence and size. Currently, they’re looking into the association between sharks and halos. If their hypothesized predator-prey-halo interaction proves true, the team anticipates estimating shark abundance from space.

Creating 3D scans of physical products can be time consuming. Businesses often use traditional methods, like photogrammetry-based apps and scanners, but these can take hours or even days. They also don’t always provide the 3D quality and level of detail needed to make models look realistic in all its applications.

Italy-based startup Covision Media is tapping into AI and NVIDIA RTX to enhance 3D scanning processes and 3D-based content creation.

Covision Media develops AI-based 3D scanners that allow customers to create digital twins of any product, including footwear, eyeglasses, sports equipment, toys, tools and household items. The company is a member of NVIDIA Inception, a free program that provides startups with access to the latest resources and technologies.

Using Covision’s technology, customers can quickly create 3D scans and automatically preserve detailed textures, materials, colors, geometry, and more to make images look as realistic as possible.

The technology runs on NVIDIA RTX, which allows users to create high-quality, detailed, photorealistic 3D models. Covision Media is also using neural radiance fields (NeRFs) to increase the quality of 3D models while tackling typical challenges like accurately capturing lighting, reflections and transparent surfaces.

adidas and its partner NUREG, a content creation studio, are among the first to use Covision Media’s 3D scanning technology for automating and scaling e-commerce content production.

Unlocking New Possibilities in 3D With RTX and AI 

Covision’s 3D scanners are connected to several workstations that run on NVIDIA RTX A5000 and RTX A6000 GPUs, both of which provide high ray-tracing performance and powerful AI capabilities.

The ray-tracing performance of the NVIDIA OptiX framework, coupled with the NVIDIA RT Cores, enables Covision to precisely measure the lighting of a scanned object. This is one of the biggest unique factors that allows customers to put their scanned products into any kind of virtual environment. Covision also harnesses NVIDIA’s software infrastructure to develop state-of-the-art AI solutions for its neural texture approach.

“Without NVIDIA RTX GPUs, it would simply not be possible to achieve the level of accuracy and performance that we need,” said Dr. Burkhard Güssefeld, tech lead at Covision Media. “NVIDIA’s hardware and software capabilities are indispensable in pushing the boundaries of our technology.”

Covision’s technology allows 3D models to be fully relightable, meaning users can adjust and manipulate the lighting in the scene. Users can also merge partial scans together to build a 360-degree scan of the product, which can be used in extended reality (XR) environments.

The core technology uses computer vision and machine learning. Covision’s strong expertise in NeRFs has enabled them to integrate it into existing pipelines to overcome traditional challenges like transparencies and reflections. This allows Covision Media to quickly reconstruct 3D shapes and appearances with just a few images.

The company has very high requirements for quality, millimetric precision, material separation and relightability. So the team adapted and expanded the capabilities of NeRF technology using data from elements such as precise light poses, controlled environments and accurate geometric cues.

NeRFs allow the team to create high-quality 3D images from the start of the process. This lets them increase throughput while reducing the amount of post-processing work required.

“Our 3D scanner automatically delivers the highest quality assets at mass production while at the same time helping customers to create value and save costs,” said Franz Tschimben, CEO of Covision Media. “Furthermore, our scanning device will help companies create high-quality 3D assets needed to populate applications and worlds on new spatial computing devices and mixed reality headsets, like Apple’s Vision Pro and Meta’s Quest.”

Covision is looking to integrate additional NVIDIA products and research projects into its solutions, such as Nvdiffrast for high-performance differentiable rendering and Tiny CUDA as a fast neural network framework. The team is also‌ deploying a custom NeRF implementation into its system, which will make use of the APIs provided by NVIDIA’s Instant-NGP.

The Brand With Three Stripes Brings 3D to Life

adidas scans thousands of items a year using Covision’s technology for its online websites and apps, where they’re compatible on both desktop and mobile.

The 3D models have helped enhance adidas’ Virtual Try-On feature, which allows customers to virtually try on shoes before buying them. adidas also uses the 3D models to automatically create 2D virtual product photos and videos, replacing the need for traditional product photography.

According to adidas, Covision’s scanning technology has helped the team take a quantum step forward in quality while maintaining its scaled scanning production. With the highly realistic scans, adidas has experienced time and cost efficiencies by switching from traditional content production, such as photo and film, to computer-generated content production.

To scale production of 3D assets, adidas relies on Covision’s technology and works with an important set of partners. NUREG is an essential partner in creating and preparing the 3D assets to go live on adidas’ platforms. In addition to NUREG’s expertise in logistics, styling and scanning, the studio provides its own software tools, as well as specialties in 2D and 3D production, which enable the 3D workflows to be scalable for thousands of assets every year.

“The unparalleled quality and relightability of 3D scans allows our global team of 3D and photo specialists to leverage the 3D models for all final applications we are creating,” said Tommy Lenssen, head of the adidas team at NUREG. “I am furthermore happy with the success of our post-production platform that allows lean collaboration and quality control.”

And for post-production workflows, Covision and NUREG work with The Kow Company, one of the leading image and video editing companies for businesses all over the world.

Customers can buy Covision Media’s 3D scanners to start production in their own content creation studios, or they can get products scanned through Covision’s partners in Europe or North America.

Learn more about Covision Media and NVIDIA RTX.

Underscoring NVIDIA’s growing relationship with the global technology superpower, Indian Prime Minister Narendra Modi met with NVIDIA founder and CEO Jensen Huang Monday evening.

The meeting at 7 Lok Kalyan Marg — as the Prime Minister’s official residence in New Delhi is known — comes as Modi prepares to host a gathering of leaders from the G20 group of the world’s largest economies, including U.S. President Joe Biden, later this week.

“Had an excellent meeting with Mr. Jensen Huang, the CEO of NVIDIA,” Modi said in a social media post. “We talked at length about the rich potential India offers in the world of AI.”

The event marks the second meeting between Modi and Huang, highlighting NVIDIA’s role in the country’s fast-growing technology industry.

The meeting with Modi comes just a week after India became the first nation to successfully land on the Moon’s south pole, highlighting the expanding technological capabilities of the world’s largest democracy.

Following Huang’s meeting with Modi, Huang met with several dozen researchers from global powerhouses of science and technology, such as the Indian Institute of Science and the various campuses of the Indian Institute of Technology, for an informal dinner.

The attendees represented a dazzling collection of some of the top minds in fields as diverse as large language models, astrophysics, medicine, quantum computing, and natural language processing.

The evening’s discussions ranged across topics from the use of technology to address language barriers, improve agriculture yields, bridge gaps in health care services and transform digital economies — as well as addressing some of the grand scientific challenges of our time.

NVIDIA has deep ties to India.

NVIDIA began operations in India in 2004 in Bangalore, almost two decades ago. India is now home to four engineering development centers in India — in Gurugram, Hyderabad, Pune and Bengaluru  — and there are now more than 3,800 NVIDIANs in India.

In addition, there are more than 320,000 India-based developers in NVIDIA’s developer program. NVIDIA’s CUDA parallel programming platform is downloaded roughly 40,000 times a month in India, and NVIDIA estimates there are 60,000 experienced CUDA developers in India.

That growth comes as India’s government continues to expand the nation’s information technology infrastructure.

For example, a compute grid is expected to link 20 cities across the country soon, helping researchers and scientists collaborate and share data and computing resources more efficiently.

That effort, in turn, promises to help support India’s ambitious development goals in the years to come.

Modi has set a target of 2030 for India to become the world’s third-largest economy. It’s currently the fifth largest.

And Modi has set a target of 2047, the hundredth anniversary of India’s independence, for the South Asian nation to join the ranks of developed economies.

Entrepreneurs are cultivating generative AI from the west coast of Africa to the eastern edge of the Arabian Desert.

Gen AI is the latest of the big plans Kofi Genfi and Nii Osae have been hatching since they met 15 years ago in high school in Accra, Ghana’s capital that sits on the Gulf of Guinea.

“We watched this latest wave of AI coming for the last few years,” said Osae, a software engineer who discovered his passion for machine learning in college.

So, late last year, they expanded Mazzuma — their mobile-payments startup that’s already processed more than $150 million in transactions — to include MazzumaGPT.

The large language model (LLM) trained on two popular blockchain languages so it can help developers quickly draft smart contracts, a Web3 market that International Data Corp. projects could hit $19 billion next year.

Thousands of Hits

In its first month, 400 developers from 70 countries used the LLM that sports 175 billion parameters, a rough measure of a model’s size and strength.

It’s the latest success for the pair that in 2018 made Forbes’ list of 30 top entrepreneurs in Africa under 30.

“Given the high growth and large demographics, there are big opportunities in this region,” said Genfi, who started his first company, an Apple device reseller, when he was 19.

Osae nurtures that potential as founder and chair of the 100+ member AI Association of Ghana. “I think we’re on a trajectory to leapfrog progress elsewhere,” he said.

LLMs Speak Arabic

About two years ago and 6,000 miles to the northeast, another pair of entrepreneurs launched a generative AI business in the Persian Gulf emirate of Dubai, home of the Burj Khalifa, the world’s tallest building.

Yakov Livshits already had about a dozen active startups when AI researcher Eli Braginskiy, a friend with family ties, came to him with the idea for MetaDialog. The startup built the first LLM to support both Arabic and English, a 7-billion-parameter model trained on one of the world’s largest Arabic/English datasets.

“We call it Baby, because we’re proud of it, and we’re building a larger, 40-billion parameter model now,” said Braginskiy.

“Our Baby LLM is currently integrated in one of the biggest governments in the region, and we’re talking with three other governments interested in using it, too,” said Livshits.

With more than 3 million people in just 13 square miles, Dubai is a vibrant hub for the region.

“The way governments in the Middle East think about AI and advanced tech in general is very bold — they want to move fast, so we’re training custom models in different languages and will present them at the GITEX conference” said Livshits, who lived in Russia, Israel and the U.S. before moving to Dubai.

In February, Saudi Arabia alone announced a $2.4 billion startup fund to help diversify the nation’s economy.

Corporations Want Custom LLMs

In Abu Dhabi, just a hundred miles down the coast, Hussein Al-Natsheh leads a team of engineers and data scientists at Beyond Limits training and fine tuning LLMs. One is already drafting documents for a large energy company and verifying they comply with its standards.

Beyond Limits also works on models for energy companies, utilities and other customers that will index and search corporate documents, draft marketing materials and more.

“Companies need their own LLMs trained on their own data which is confidential, so we have machines reading their data, not us,” said Al-Natsheh, a native of Amman, Jordan, who, prior to joining Beyond Limits, worked on Salma, one of the first Arabic speech assistants.

Drilling for Data

Now that data is the new oil, Beyond Limits is developing toolkits to extract it from unstructured files — corporate emails, PowerPoint and other sources — so it can help companies train custom LLMs approaching 70 billion parameters in size.

The toolkits can help address the lack of data samples from the many Arabic dialects. Indeed, a report from the UAE government on 100 top gen AI uses called for more work on Arabic, a language spoken by nearly half a billion people.

The good news is governments and large companies like G42, a regional cloud service company, are pouring resources into the problem. For example, Beyond Limits was able to create its regional headquarters in Dubai thanks to its last funding round, much of which came from G42.

A Big Boost from Inception

All three companies are members of NVIDIA Inception, a free program that helps startups working on cutting-edge technologies like generative AI.

As part of Inception, Beyond Limits had access to libraries in NVIDIA NeMo, a framework for building massive gen AI models, and which cut training time in one case from five days to one.

“NVIDIA software makes our work much easier, and our clients trust NVIDIA technology,” said Al-Natsheh.

As part of Inception, Mazzuma got access to cloud GPU services to accelerate its experiments and introductions to potential investors.

“That really gave us a boost, and there’s a lot of assurance that comes from working with the best people and tools,” said Genfi.

Treating Partners Well

For its part, MetaDialog trained its Baby LLM on 440 NVIDIA A100 Tensor Core GPUs using a service operated by MosaicML, an Inception member recently acquired by Databricks.

“I’ve built many startups, and no company treats its partners as well as NVIDIA,” said Livshits.

At top: From left to right, Nii Osae, Hussein Al-Natsheh, Eli Braginskiy, Yakov Livshits and Kofi Genfi.

Academics Mory Gharib and Alireza Ramezani in 2020 were spitballing a transforming robot that is now getting a shot at work that’s literally out of this world: NASA Mars Rover missions.

Caltech has unveiled its multi-talented robot that can fly, drive, walk and do eight permutations of motions through a combination of its skills. They call it the Multi-Modal Mobility Morphobot, or M4, which is enabled by the NVIDIA Jetson platform for edge AI and robotics.

“It grew in the number of functions that we wanted to do,” said Gharib, a professor of aeronautics and bioinspired engineering at Caltech. “When we proposed it to our design team, at first they all said, ‘no.’”

Caltech funded its initial research, and NASA and its Jet Propulsion Lab (JPL) funded its next phase and brought in Ramezani, an assistant professor of electrical and computer engineering at Northeastern University, as a faculty researcher at JPL last summer to develop it further.

Its M42 version is now under development at NASA as a Mars Rover candidate and has interest from the U.S. Department of Transportation, Gharib said.

“At NASA, we’re being tested right now for transforming while landing,” he said.

And since recently releasing a paper on it in Nature Communications, Gharib says he’s been inundated with proposals.

“We’re kind of dizzy about how it suddenly got so much attention,” he said. “Different organizations want to do different things and are coming to approach us.”

Firefighting, Search and Rescue Operations 

The Caltech team behind the paper — Gharib and Ramezani, as well as Eric Sihite, a postdoctoral scholar research associate in aerospace at Caltech; Arash Kalantari, from JPL; and Reza Nemovi, a design engineer at CAST — said the M4 is designed for diverse mission requirements in search and rescue, among other areas.

For example, when it’s not feasible to roll or walk into areas — like fire zones —  it can fly and do reconnaissance to assess situations using its cameras and sensors.

According to Gharib, multiple fire departments in the Los Angeles area have contacted Gharib with interest in the M4.

“For first responders, this is huge because you need to land in a safe area and then drive into the situation,” he said.

Versatile Drone Deliveries to Get the Job Done

Caltech’s team also aims to solve complications with drone deliveries using the M4. Drone deliveries are the “low hanging fruit,” for this robot, said Gharib.

Traditional drones for deliveries are problematic because nobody wants drones landing near their home or business for safety reasons, he said. The M4 can land somewhere isolated from people and then drive to finish deliveries, making it a safer option, he added.

The M4 can also fly into areas where truck deliveries might have a difficult time getting into or can’t offer delivery service at all.

“There are a lot of places where truck deliveries can’t go,” he said.

Right now, the M4 is capable of traveling as fast as 40 mph, and its battery can last up to 30 minutes on a charge. But the team is working to design larger drones with longer flight times, bigger payloads and increased travel distances.

The sky’s the limit.

Learn about NVIDIA Jetson Nano.