MiniGPT4-The Future of Language Understanding With Vision AI
Uploaded by
My SocialMiniGPT4-The Future of Language Understanding With Vision AI
Uploaded by
My SocialIntroduction
GPT-4 is the latest Large Language Model that OpenAI has released. Its
multimodal nature sets it apart from all the previously introduced LLMs.
GPT-4 has shown tremendous performance in solving tasks like
producing detailed and precise image descriptions, explaining unusual
visual phenomena, developing websites using handwritten text
instructions, and so on. The reason behind GPT-4’s exceptional
performance is not fully understood. Experts believe that GPT-4’s
advanced abilities may be due to the use of a more advanced Large
Language Model. Which is mostly not present in smaller models and
that's where Mini GPT comes into picture. Mini GPT4 is developed by a
team of Ph.D. students from King Abdullah University of Science and
Technology, Saudi Arabia.
What is Mini GPT4?
Mini GPT4 is a new advanced large language model that can
understand both images and text. It is an open-source project that can
understand images, generate recipes for images, identify problems in
the images and give potential solutions, and even create working code
for websites just from an image. The project combines a pre-trained
language model Vicuna (built upon LLaMA and is reported to achieve
90% of ChatGPT’s quality as evaluated by GPT-4) with a visual encoder
(BLIP 2) to get amazing results.
How Does Mini GPT4 Work?
The Frozen input visual encoder from BLIP2 processes the input image
to generate a fixed-length vector representation of the image. This vector
is then combined with the text input and processed through the Frozen
language model of Vicuna 13B. This generates a sequence of hidden
states that represent the language understanding. The hidden states
from the language model are then fixed-length vector representations of
the image which are connected and fed through a single projection layer.
This projection layer transforms a concentrated-like representation into a
final output, generating a text caption of the overall image.
Training Process
The training process was done in two stages. In the first stage, they
used roughly 5 million text-image pairs and trained the model for 10
hours using four A100 GPUs. In the second stage, they only used 3500
high-quality text-image pairs generated by the model itself using Chat
GPT. This fine-tuning took only seven minutes on a single A100 GPU.
Model Abilities
The model can generate very detailed descriptions of images based on
human prompts or questions. The output from this model is simple but
impressive. For example, it can describe logos or designs in detail. The
data set used for fine-tuning the model is available as well.
Data Collection
During the initial stage of training, an enormous number of text-image
pairs, approximately 5 million, were collected. For the next phase, a
selection of 3500 top-quality text-image pairs created by the ChatGPT
model were used for fine-tuning. These pairs are accompanied by image
descriptions stored in a separate Json file, while the images themselves
are kept in a separate folder.
Image Description and Analysis (examples)
● Detailed Image Descriptions - The language model is able to
generate detailed descriptions of images, including the presence of
motorcycles on the side of the road, people walking down the
street, a clock tower with Roman numerals and a small spiral on
top, blue skies with clouds in the distance, and a cactus plant
standing in the middle of a frozen lake surrounded by large ice
crystals.
● Understanding Humor in Memes - The language model is able to
explain why a meme featuring a tired dog with the caption
"Monday just Monday" is funny by recognizing that many people
dislike Mondays and can relate to feeling tired or sleepy like the
dog.
● Identifying Unusual Contents in Images - The language model
is able to identify unusual contents in images such as a cactus
plant standing in the middle of a frozen lake and recognize that it is
not common in real life. It can also identify problems from photos
such as brown spots on leaves caused by fungal infections or
overflowing soap from washing machines.
● Providing Solutions to Image-Related Problems - The system
analyzes images using encoders and information from large
language models to objectify problems and provide solutions.
Click here to read more.
