Big news in the generative AI world, Nvidia has released NVLM 1.0, a family of multimodal large language models (LLMs) that can compete with GPT-4. Released in early October 2024, these new AI models are getting attention not just for how well they perform but for the bigger picture. NVLM 1.0 is meant to put Nvidia in the AI game by having a model that matches and in some cases beats the existing generative AI benchmarks.
Nvidia is known for making high performance GPUs that power many of the world’s top AI systems including OpenAI’s ChatGPT. But with NVLM 1.0, the company is moving beyond hardware to get into the AI software game. The model family, which Nvidia says can match GPT-4 in many areas, is for developers to build their own AI applications. Unlike OpenAI and Google who have released AI chatbots like ChatGPT and Gemini, Nvidia is releasing a platform for others to build new tools and systems on top of their models.
Multimodal: A Step Beyond Text
One of the key features of NVLM 1.0 is multimodal. While ChatGPT and others are text only, NVLM can take in text and visual input. This means it can handle tasks that require both types of input, making it especially powerful for vision-language tasks. In testing the model was able to identify people, animals and objects in images and provide contextually relevant answers. It even understood memes—a sign it can grasp complex humor and cultural references, which puts it ahead of the competition in certain use cases.
Benchmarks: Beats ChatGPT and Google Gemini
Nvidia’s announcement was accompanied by benchmarks that show NVLM 1.0 can beat GPT-4 and Google’s Gemini Pro in some tasks. For example, NVLM is very strong in vision-language understanding which puts it in the running for computer vision and autonomous systems. In math problem solving Nvidia’s AI is on par with Meta’s Llama AI which is impressive for a first gen model.
GPT-4 is a powerful general AI but NVLM is strong in areas that fit into Nvidia’s broader AI vision for healthcare, automotive and gaming. This is a different path in the AI landscape.
Nvidia’s Open-Source Strategy: A New Approach
Unlike its competitors, Nvidia is going open with its AI model. The company is open-sourcing the NVLM 1.0 models, including the code and training weights. This is a departure from OpenAI and Google’s more closed models.
In a research paper accompanying the announcement, Nvidia said: “We introduce NVLM 1.0, a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks, matching the leading proprietary models (e.g., GPT-4) and open-access models.” This openness will encourage innovation, allow developers to experiment and build their own applications on top of Nvidia’s foundation and position Nvidia as the AI leader by building a community of researchers and developers who can build on top of their tech.
But this open-source approach has its pros and cons. While it will accelerate innovation, it also raises questions about how Nvidia will ensure responsible use of its models. As AI gets more accessible, misuse and ethical considerations become more important. Nvidia hasn’t outlined comprehensive safeguards or ethical guidelines for its models yet, leaving open the question of how they will manage these risks.
Implications for the AI Landscape
Nvidia’s entry into the generative AI space with NVLM 1.0 changes the competitive landscape. The company is known for being a hardware supplier to AI companies but with this release Nvidia is going head to head with the software side of AI. By open-sourcing an alternative to GPT-4, Nvidia is not only a competitor to OpenAI but also a catalyst for the broader AI ecosystem.
The open-sourcing of NVLM may also force other companies to re-think their strategy. In an industry that has been all about proprietary models and closed ecosystems, Nvidia’s move may lead to more collaboration and sharing of resources. We’ll see how OpenAI and Google react but the introduction of a powerful open-source model will definitely change the AI landscape.
For developers, NVLM 1.0 means new opportunities to build AI systems that go beyond text generation and build more complex and multimodal applications. Nvidia’s AI will find use cases in robotics, autonomous driving and medical imaging where the ability to process text and visual inputs will be particularly valuable.
