Skip to content

Commit cb6bd17

Browse files
feat(dataflow): update the api
#### dataflow:v1b3 The following keys were added: - schemas.DataSamplingConfig (Total Keys: 4) - schemas.DataSamplingReport (Total Keys: 16) - schemas.DebugOptions.properties.dataSampling.$ref (Total Keys: 1) - schemas.Environment.properties.useStreamingEngineResourceBasedBilling (Total Keys: 2) - schemas.SdkBug (Total Keys: 8) - schemas.SdkVersion.properties.bugs (Total Keys: 3) - schemas.WorkerMessage.properties.dataSamplingReport.$ref (Total Keys: 1)
1 parent e70656f commit cb6bd17

8 files changed

+388
-1
lines changed

docs/dyn/dataflow_v1b3.projects.html

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,15 @@ <h3>Method Details</h3>
142142
&quot;location&quot;: &quot;A String&quot;, # The [regional endpoint] (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints) that contains the job.
143143
&quot;workerMessages&quot;: [ # The WorkerMessages to send.
144144
{ # WorkerMessage provides information to the backend about a worker.
145+
&quot;dataSamplingReport&quot;: { # Contains per-worker telemetry about the data sampling feature. # Optional. Contains metrics related to go/dataflow-data-sampling-telemetry.
146+
&quot;bytesWrittenDelta&quot;: &quot;A String&quot;, # Optional. Delta of bytes written to file from previous report.
147+
&quot;elementsSampledBytes&quot;: &quot;A String&quot;, # Optional. Delta of bytes sampled from previous report.
148+
&quot;elementsSampledCount&quot;: &quot;A String&quot;, # Optional. Delta of number of elements sampled from previous report.
149+
&quot;exceptionsSampledCount&quot;: &quot;A String&quot;, # Optional. Delta of number of samples taken from user code exceptions from previous report.
150+
&quot;pcollectionsSampledCount&quot;: &quot;A String&quot;, # Optional. Delta of number of PCollections sampled from previous report.
151+
&quot;persistenceErrorsCount&quot;: &quot;A String&quot;, # Optional. Delta of errors counts from persisting the samples from previous report.
152+
&quot;translationErrorsCount&quot;: &quot;A String&quot;, # Optional. Delta of errors counts from retrieving, or translating the samples from previous report.
153+
},
145154
&quot;labels&quot;: { # Labels are used to group WorkerMessages. For example, a worker_message about a particular container might have the labels: { &quot;JOB_ID&quot;: &quot;2015-04-22&quot;, &quot;WORKER_ID&quot;: &quot;wordcount-vm-2015…&quot; &quot;CONTAINER_TYPE&quot;: &quot;worker&quot;, &quot;CONTAINER_ID&quot;: &quot;ac1234def&quot;} Label tags typically correspond to Label enum values. However, for ease of development other strings can be used as tags. LABEL_UNSPECIFIED should not be used here.
146155
&quot;a_key&quot;: &quot;A String&quot;,
147156
},

docs/dyn/dataflow_v1b3.projects.jobs.html

Lines changed: 91 additions & 0 deletions
Large diffs are not rendered by default.

docs/dyn/dataflow_v1b3.projects.locations.flexTemplates.html

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -234,6 +234,11 @@ <h3>Method Details</h3>
234234
&quot;clusterManagerApiService&quot;: &quot;A String&quot;, # The type of cluster manager API to use. If unknown or unspecified, the service will attempt to choose a reasonable default. This should be in the form of the API service name, e.g. &quot;compute.googleapis.com&quot;.
235235
&quot;dataset&quot;: &quot;A String&quot;, # The dataset for the current project where various workflow related tables are stored. The supported resource type is: Google BigQuery: bigquery.googleapis.com/{dataset}
236236
&quot;debugOptions&quot;: { # Describes any options that have an effect on the debugging of pipelines. # Any debugging options to be supplied to the job.
237+
&quot;dataSampling&quot;: { # Configuration options for sampling elements. # Configuration options for sampling elements from a running pipeline.
238+
&quot;behaviors&quot;: [ # List of given sampling behaviors to enable. For example, specifying behaviors = [ALWAYS_ON] samples in-flight elements but does not sample exceptions. Can be used to specify multiple behaviors like, behaviors = [ALWAYS_ON, EXCEPTIONS] for specifying periodic sampling and exception sampling. If DISABLED is in the list, then sampling will be disabled and ignore the other given behaviors. Ordering does not matter.
239+
&quot;A String&quot;,
240+
],
241+
},
237242
&quot;enableHotKeyLogging&quot;: True or False, # When true, enables the logging of the literal hot key to the user&#x27;s Cloud Logging.
238243
},
239244
&quot;experiments&quot;: [ # The list of experiments to enable. This field should be used for SDK related experiments and not for service related experiments. The proper field for service related experiments is service_options.
@@ -253,6 +258,7 @@ <h3>Method Details</h3>
253258
],
254259
&quot;shuffleMode&quot;: &quot;A String&quot;, # Output only. The shuffle mode used for the job.
255260
&quot;tempStoragePrefix&quot;: &quot;A String&quot;, # The prefix of the resources the system should use for temporary storage. The system will append the suffix &quot;/temp-{JOBNAME} to this resource prefix, where {JOBNAME} is the value of the job_name field. The resulting bucket and object prefix is used as the prefix of the resources used to store temporary data needed during the job execution. NOTE: This will override the value in taskrunner_settings. The supported resource type is: Google Cloud Storage: storage.googleapis.com/{bucket}/{object} bucket.storage.googleapis.com/{object}
261+
&quot;useStreamingEngineResourceBasedBilling&quot;: True or False, # Output only. Whether the job uses the new streaming engine billing model based on resource usage.
256262
&quot;userAgent&quot;: { # A description of the process that generated the request.
257263
&quot;a_key&quot;: &quot;&quot;, # Properties of the object.
258264
},
@@ -388,6 +394,13 @@ <h3>Method Details</h3>
388394
},
389395
],
390396
&quot;sdkVersion&quot;: { # The version of the SDK used to run the job. # The SDK version used to run the job.
397+
&quot;bugs&quot;: [ # Output only. Known bugs found in this SDK version.
398+
{ # A bug found in the Dataflow SDK.
399+
&quot;severity&quot;: &quot;A String&quot;, # Output only. How severe the SDK bug is.
400+
&quot;type&quot;: &quot;A String&quot;, # Output only. Describes the impact of this SDK bug.
401+
&quot;uri&quot;: &quot;A String&quot;, # Output only. Link to more information on the bug.
402+
},
403+
],
391404
&quot;sdkSupportStatus&quot;: &quot;A String&quot;, # The support status for this SDK version.
392405
&quot;version&quot;: &quot;A String&quot;, # The version of the SDK used to run the job.
393406
&quot;versionDisplayName&quot;: &quot;A String&quot;, # A readable string describing the version of the SDK.

docs/dyn/dataflow_v1b3.projects.locations.html

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,15 @@ <h3>Method Details</h3>
120120
&quot;location&quot;: &quot;A String&quot;, # The [regional endpoint] (https://cloud.google.com/dataflow/docs/concepts/regional-endpoints) that contains the job.
121121
&quot;workerMessages&quot;: [ # The WorkerMessages to send.
122122
{ # WorkerMessage provides information to the backend about a worker.
123+
&quot;dataSamplingReport&quot;: { # Contains per-worker telemetry about the data sampling feature. # Optional. Contains metrics related to go/dataflow-data-sampling-telemetry.
124+
&quot;bytesWrittenDelta&quot;: &quot;A String&quot;, # Optional. Delta of bytes written to file from previous report.
125+
&quot;elementsSampledBytes&quot;: &quot;A String&quot;, # Optional. Delta of bytes sampled from previous report.
126+
&quot;elementsSampledCount&quot;: &quot;A String&quot;, # Optional. Delta of number of elements sampled from previous report.
127+
&quot;exceptionsSampledCount&quot;: &quot;A String&quot;, # Optional. Delta of number of samples taken from user code exceptions from previous report.
128+
&quot;pcollectionsSampledCount&quot;: &quot;A String&quot;, # Optional. Delta of number of PCollections sampled from previous report.
129+
&quot;persistenceErrorsCount&quot;: &quot;A String&quot;, # Optional. Delta of errors counts from persisting the samples from previous report.
130+
&quot;translationErrorsCount&quot;: &quot;A String&quot;, # Optional. Delta of errors counts from retrieving, or translating the samples from previous report.
131+
},
123132
&quot;labels&quot;: { # Labels are used to group WorkerMessages. For example, a worker_message about a particular container might have the labels: { &quot;JOB_ID&quot;: &quot;2015-04-22&quot;, &quot;WORKER_ID&quot;: &quot;wordcount-vm-2015…&quot; &quot;CONTAINER_TYPE&quot;: &quot;worker&quot;, &quot;CONTAINER_ID&quot;: &quot;ac1234def&quot;} Label tags typically correspond to Label enum values. However, for ease of development other strings can be used as tags. LABEL_UNSPECIFIED should not be used here.
124133
&quot;a_key&quot;: &quot;A String&quot;,
125134
},

0 commit comments

Comments
 (0)