MiniGPT-4 is an open-source chatbot that can interpret images and respond to questions about their contents. Its release makes image-based conversation available through a project whose code, demos, and training instructions are shared publicly.
What MiniGPT-4 can do with images
The model can produce descriptions of images and answer questions about what they show. Given a prepared dish, for example, it may suggest a matching recipe. It can also generate descriptions intended to help visually impaired people understand an image.
Image understanding can support other kinds of work, too. MiniGPT-4 may draw ideas or prompts from an image, and its researchers say it can create a website from a hand-written draft. These examples point to a chatbot that works with visual inputs as well as text.
The researchers describe its abilities as similar in some respects to capabilities shown by GPT-4, including detailed image descriptions and website creation from drafts. The source notes that OpenAI introduced image understanding with GPT-4, but had not released that part of the model outside the Be my Eyes app.
How the model was trained
MiniGPT-4 combines the Vicuna-13B language model with the BLIP-2 Vision Language Model. Both are described as open-source software that can be trained and fine-tuned with comparatively limited money and without massive data and computational overhead.
The team’s training process had two stages. First, it used about five million image-text pairs and trained for ten hours on four Nvidia A100 cards. Then the researchers refined the model with 3,500 high-quality text-image pairs produced through an interaction between MiniGPT-4 and ChatGPT.
In that second stage, ChatGPT corrected image descriptions that MiniGPT-4 had generated inaccurately. The refinement took seven minutes of training on a single Nvidia A100 and, according to the source, improved the model’s reliability and usability. The researchers said they were surprised by the efficiency.
A smaller model and an open release
The team also announced a smaller version designed to run on a single Nvidia 3090 graphics card. Along with that version, making the code, demos, and training instructions available on Github gives other people material to examine and work with.
The project follows an approach in which an existing language model and vision-language model are joined and then refined for the task. That makes the training recipe part of the story: the results are presented alongside a description of the data and computing used to produce them.
What the release may signal
MiniGPT-4 is one example of the rapid progress described in the open-source AI community. The article argues that this progress may make it harder for companies focused on AI models alone to maintain a large advantage, especially when open-source components can be adapted with relatively limited resources.
It also places MiniGPT-4 alongside OpenAssistant, an open-source chatbot launched the previous day. OpenAssistant was trained with instructional data collected from volunteers and was intended to become an open ChatGPT alternative. Together, the projects suggest a growing effort to make conversational AI available beyond a small number of closed systems.
The source proposes that OpenAI could focus on building a partner ecosystem through ChatGPT plugins for GPT-4 rather than immediately training GPT-5. Its reasoning is that developing a new model may demand more research and training effort than the competitive head start it provides, while a chat ecosystem can be difficult to build and may encourage users to stay. That is the article’s analysis of the competitive landscape, rather than a demonstrated outcome of MiniGPT-4 itself.