Generate Content With The Gemini Enterprise API - Generative AI On Vertex AI - Google Cloud
Generate Content With The Gemini Enterprise API - Generative AI On Vertex AI - Google Cloud
The Gemini model family includes models that work with multimodal prompt requests. The
term multimodal indicates that you can use more than one modality, or type of input, in a
prompt. Models that aren't multimodal accept prompts only with text. Modalities can
include text, audio, video, and more.
To start using the Vertex AI API for Gemini, create a Google Cloud account
(https://console.cloud.google.com/freetrial?redirectPath=/marketplace/product/google/
cloudaicompanion.googleapis.com)
.
After creating your account, use this document to review the Gemini model request body
(#request), model parameters (#parameters), response body (#response), and some sample
requests (#sample-requests).
When you're ready, see the Vertex AI API for Gemini quickstart
(/vertex-ai/generative-ai/docs/start/quickstarts/quickstart-multimodal) to learn how to send a
request to the Vertex AI Gemini API using a programming language SDK or the REST API.
Supported models
Model Version
1 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
Model Version
Example syntax
Non-streaming
(#curl)
(#python)
gemini_model = GenerativeModel(MODEL_ID)
generation_config = GenerationConfig(...)
Streaming
(#curl)
(#python)
gemini_model = GenerativeModel(MODEL_ID)
model_response = gemini_model.generate_content([...], generation_config,
Parameter list
2 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
Request body
{
"cachedContent": string,
"contents": [
{
"role": string,
"parts": [
{
∕∕ Union field data can be only one of the following:
"text": string,
"inlineData": {
"mimeType": string,
"data": string
},
"fileData": {
"mimeType": string,
"fileUri": string
},
∕∕ End of list of possible types for union field data.
"videoMetadata": {
"startOffset": {
"seconds": integer,
"nanos": integer
},
"endOffset": {
"seconds": integer,
"nanos": integer
}
}
}
]
}
],
"systemInstruction": {
"role": string,
"parts": [
{
"text": string
}
]
},
"tools": [
{
"functionDeclarations": [
3 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
{
"name": string,
"description": string,
"parameters": {
object ( (https://spec.openapis.org/oas/v3.0.3#schema) (
}
}
]
}
],
"safetySettings": [
{
"category": enum (HarmCategory),
"threshold": enum (HarmBlockThreshold)
}
],
"generationConfig": {
"temperature": number,
"topP": number,
"topK": number,
"candidateCount": integer,
"maxOutputTokens": integer,
"presencePenalty": float,
"frequencyPenalty": float,
"stopSequences": [
string
],
"responseMimeType": string,
"responseSchema": (/vertex-ai/docs/reference/rest/v1/Schema),
"seed": integer,
"responseLogprobs": boolean,
"logprobs": integer,
"audioTimestamp": boolean
},
"labels": {
string: string
}
}
Parameters
Optional:
4 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
Required:
Optional:
Optional:
Enforced on .
Optional:
Optional:
Metadata that you can add to the API call in the format of key-value
pairs.
5 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
This class consists of two main properties: and . The property denotes the
individual producing the content, while the property contains multiple elements, each
representing a segment of data within a message.
Parameters
Optional:
The identity of the entity that creates the message. The following
values are supported:
The value is used to insert messages from the model into the
conversation during multi-turn conversations.
To compute the number of tokens in your request, see Get token count
(/vertex-ai/generative-ai/docs/multimodal/get-token-count).
Parameters
Optional:
6 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
Optional:
Optional:
Optional: .
Optional: .
Optional:
For video input, the start and end offset of the video in Duration
(https://protobuf.dev/reference/protobuf/google.protobuf/#duration)
format. For example, to specify a 10 second clip starting at 1:00, set
and
.
Parameters
7 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
For Gemini 1.5 Pro and Gemini 1.5 Flash, the maximum length of an
audio �le is 8.4 hours and the maximum length of a video �le (without
audio) is one hour. For more information, see Gemini 1.5 Pro media
requirements
(/vertex-ai/generative-ai/docs/multimodal/send-multimodal-
prompts#media_requirements)
.
Text �les must be UTF-8 encoded. The contents of the text �le count
toward the token limit.
8 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
FileData
Parameters
The URI or URL of the �le to include in the prompt. Acceptable values
include the following:
• HTTP URL: The �le URL must be publicly readable. You can specify
one video �le and up to 10 image �les per request. Audio �les and
documents can't exceed 15 MB.
A predicted returned from the model that contains a string representing the
9 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
Parameters
Parameters
Parameters
Optional:
10 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
Optional:
Safety settings.
Parameters
Optional:
Optional:
Optional:
11 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
Parameters
Parameters
Block none.
Parameters
12 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
Parameters
Optional:
If the model returns a response that's too generic, too short, or the
model gives a fallback response, try increasing the temperature.
Optional:
Top-P
(/vertex-ai/generative-ai/docs/multimodal/content-generation-
parameters#top-p)
changes how the model selects tokens for output. Tokens are
selected from the most (see top-K) to least probable until the sum of
their probabilities equals the top-P value. For example, if tokens A, B,
and C have a probability of 0.3, 0.2, and 0.1 and the top-P value is ,
then the model will select either A or B as the next token by using
temperature and excludes C as a candidate.
Specify a lower value for less random responses and a higher value
for more random responses.
13 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
• Range:
• Default for :
• Default for :
• Default for :
• Default for :
Optional: Top-K
(/vertex-ai/generative-ai/docs/multimodal/content-generation-
parameters#top-k)
changes how the model selects tokens for output. A top-K of means
the next selected token is the most probable among all tokens in the
model's vocabulary (also called greedy decoding), while a top-K of
means that the next token is selected from among the three most
probable tokens by using temperature.
For each token selection step, the top-K tokens with the highest
probabilities are sampled. Then tokens are further �ltered based on
top-P with the �nal token selected using temperature sampling.
Specify a lower value for less random responses and a higher value
for more random responses.
Range:
Supported by only.
Default for :
Optional:
Optional: int
14 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
Specify a lower value for shorter responses and a higher value for
potentially longer responses.
Optional:
Speci�es a list of strings that tells the model to stop generating text if
one of the strings is encountered in the response. If a string appears
multiple times in the response, then the response truncates where it's
�rst encountered. The strings are case-sensitive.
Optional:
Positive penalties.
Supported by and .
Optional:
15 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
Supported by and .
Optional:
The schema that generated candidate text must follow. For more
information, see Control generated output
(/vertex-ai/generative-ai/docs/multimodal/control-generated-output).
Optional:
When seed is �xed to a speci�c value, the model makes a best effort
to provide the same response for repeated requests. Deterministic
output isn't guaranteed. Also, changing the model or parameter
settings, such as the temperature, can cause variations in the
response even when you use the same seed value. By default, a
random seed value is used.
16 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
Optional:
If true, returns the log probabilities of the tokens that were chosen by
the model at each step. By default, this parameter is set to .
Optional:
Optional:
Response body
{
"candidates": [
17 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
{
"content": {
"parts": [
{
"text": string
}
]
},
"finishReason": enum (FinishReason),
"safetyRatings": [
{
"category": enum (HarmCategory),
"probability": enum (HarmProbability),
"blocked": boolean
}
],
"citationMetadata": {
"citations": [
{
"startIndex": integer,
"endIndex": integer,
"uri": string,
"title": string,
"license": string,
"publicationDate": {
"year": integer,
"month": integer,
"day": integer
}
}
]
},
"avgLogprobs": double,
"logprobsResult": {
"topCandidates": [
{
"candidates": [
{
"token": string,
"logProbability": float
}
]
}
],
"chosenCandidates": [
{
"token": string,
"logProbability": float
18 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
}
]
}
}
],
"usageMetadata": {
"promptTokenCount": integer,
"candidatesTokenCount": integer,
"totalTokenCount": integer
},
"modelVersion": string
}
The reason why the model stopped generating tokens. If empty, the model
has not stopped generating the tokens. Because the response uses the
prompt for context, it's not possible to change the behavior of how the model
stops generating tokens.
• : Candidates were
19 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
A boolean �ag associated with a safety attribute that indicates if the model's
input or output was blocked.
The date a citation was published. Its valid formats are , , and
.
20 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
Generative AI models break down text data into tokens for processing, which
can be characters, words, or phrases.
A log probability value that indicates the model's con�dence for a particular
token.
Examples
import vertexai
from vertexai.generative_models import GenerativeModel
model = GenerativeModel("gemini-1.5-flash-002")
response = model.generate_content(
"What's a good name for a flower shop that specializes in selling bouquets of dr
)
print(response.text)
# Example response:
# **Emphasizing the Dried Aspect:**
# * Everlasting Blooms
# * Dried & Delightful
# * The Petal Preserve
# ...
21 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
Generate a non-streaming model response from a multi-modal input, such as text and an
image.
import vertexai
model = GenerativeModel("gemini-1.5-flash-002")
response = model.generate_content(
[
Part.from_uri(
"gs:∕∕cloud-samples-data∕generative-ai∕image∕scones.jpg",
mime_type="image∕jpeg",
),
"What is shown in this image?",
]
)
print(response.text)
# That's a lovely overhead shot of a rustic-style breakfast or brunch spread.
# Here's what's in the image:
# * **Blueberry scones:** Several freshly baked blueberry scones are arranged on par
# They look crumbly and delicious.
# ...
import vertexai
22 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
model = GenerativeModel("gemini-1.5-flash-002")
responses = model.generate_content(
"Write a story about a magic backpack.", stream=True
)
Generate a streaming model response from a multi-modal input, such as text and an image.
import vertexai
vertexai.init(project=PROJECT_ID, location="us-central1")
model = GenerativeModel("gemini-1.5-flash-002")
responses = model.generate_content(
[
Part.from_uri(
"gs:∕∕cloud-samples-data∕generative-ai∕video∕animals.mp4", "video∕mp4"
),
Part.from_uri(
"gs:∕∕cloud-samples-data∕generative-ai∕image∕character.jpg",
"image∕jpeg",
23 of 24 11/10/24, 01:37
Generate content with the Gemini Enterprise API | ... https://cloud.google.com/vertex-ai/generative-ai/docs...
),
"Are these video and image correlated?",
],
stream=True,
)
Model versions
What's next
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0
License (https://creativecommons.org/licenses/by/4.0/), and code samples are licensed under the Apache
2.0 License (https://www.apache.org/licenses/LICENSE-2.0). For details, see the Google Developers Site
Policies (https://developers.google.com/site-policies). Java is a registered trademark of Oracle and/or its
a�liates.
24 of 24 11/10/24, 01:37