Proto Plus Python Readthedocs Io en Stable
Proto Plus Python Readthedocs Io en Stable
Luke Sneeringer
1 Installing 3
2 Table of Contents 5
2.1 Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3 Type Marshaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.4 Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.5 Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Index 21
i
ii
Proto Plus for Python Documentation
Contents 1
Proto Plus for Python Documentation
2 Contents
CHAPTER 1
Installing
This library carries a dependency on the official implementation (protobuf), which may install a C component.
3
Proto Plus for Python Documentation
4 Chapter 1. Installing
CHAPTER 2
Table of Contents
2.1 Messages
The fundamental building block in protocol buffers are messages. Messages are essentially permissive, strongly-typed
structs (dictionaries), which have zero or more fields that may themselves contain primitives or other messages.
syntax = "proto3";
message Song {
Composer composer = 1;
string title = 2;
string lyrics = 3;
int32 year = 4;
}
message Composer {
string given_name = 1;
string family_name = 2;
}
The most common use case for protocol buffers is to write a .proto file, and then use the protocol buffer compiler
to generate code for it.
However, it is possible to declare messages directly. This is the equivalent message declaration in Python, using this
library:
import proto
class Composer(proto.Message):
given_name = proto.Field(proto.STRING, number=1)
(continues on next page)
5
Proto Plus for Python Documentation
class Song(proto.Message):
composer = proto.Field(Composer, number=1)
title = proto.Field(proto.STRING, number=2)
lyrics = proto.Field(proto.STRING, number=3)
year = proto.Field(proto.INT32, number=4)
2.1.2 Usage
Instantiate messages using either keyword arguments or a dict (and mix and matching is acceptable):
One of the goals of proto-plus is to make protobufs feel as much like regular python objects as possible. It is possible
to update a message’s field by assigning to it, just as if it were a regular python object.
song = Song()
song.composer = Composer(given_name="Johann", family_name="Bach")
a = Album()
songs = [Song(title="Canon in D"), Song(title="Little Fugue")]
a.songs = songs
Note: Assigning to a proto-plus message field works by making copies, not by updating references. This is necessary
because of memory layout requirements of protocol buffers. These memory constraints are maintained by the protocol
buffers runtime. This behavior can be surprising under certain circumstances, e.g. trying to save an alias to a nested
field.
proto.Message defines a helper message, copy_from() to help make the distinction clear when reading code.
The semantics of copy_from() are identical to the field assignment behavior described above.
2.1.4 Enums
import proto
class Genre(proto.Enum):
GENRE_UNSPECIFIED = 0
CLASSICAL = 1
JAZZ = 2
ROCK = 3
class Composer(proto.Message):
given_name = proto.Field(proto.STRING, number=1)
family_name = proto.Field(proto.STRING, number=2)
class Song(proto.Message):
(continues on next page)
2.1. Messages 7
Proto Plus for Python Documentation
All enums must begin with a 0 value, which is always the default in proto3 (and, as above, indistuiguishable from
unset).
Enums utilize Python enum.IntEnum under the hood:
>>> song.genre = 2
>>> song.genre
<Genre.JAZZ: 2>
>>> song.genre = 'CLASSICAL'
<Genre.CLASSICAL: 1>
2.1.5 Serialization
Serialization and deserialization is available through the serialize() and deserialize() class methods.
The serialize() method is available on the message classes only, and accepts an instance:
serialized_song = Song.serialize(song)
The deserialize() method accepts a bytes, and returns an instance of the message:
song = Song.deserialize(serialized_song)
JSON serialization and deserialization are also available from message classes via the to_json() and
from_json() methods.
json = Song.to_json(song)
new_song = Song.from_json(json)
Similarly, messages can be converted into dictionaries via the to_dict() helper method. There is no
from_dict() method because the Message constructor already allows construction from mapping types.
song_dict = Song.to_dict(song)
new_song = Song(song_dict)
Note: Although Python’s pickling protocol has known issues when used with untrusted collaborators, some frame-
works do use it for communication between trusted hosts. To support such frameworks, protobuf messages can be
pickled and unpickled, although the preferred mechanism for serializing proto messages is serialize().
Multiprocessing example:
import proto
from multiprocessing import Pool
class Composer(proto.Message):
name = proto.Field(proto.STRING, number=1)
genre = proto.Field(proto.STRING, number=2)
with multiprocessing.Pool(2) as p:
def add_genre(comp_bytes):
composer = Composer.deserialize(comp_bytes)
composer.genre = "classical"
return Composer.serialize(composer)
updated_composers = [
Composer.deserialize(comp_bytes)
for comp_bytes in p.map(add_genre, (Composer.serialize(comp) for comp in
˓→composers))
2.2 Fields
Fields are assigned using the Field class, instantiated within a Message declaration.
Fields always have a type (either a primitive, a message, or an enum) and a number.
import proto
class Composer(proto.Message):
given_name = proto.Field(proto.STRING, number=1)
family_name = proto.Field(proto.STRING, number=2)
class Song(proto.Message):
composer = proto.Field(Composer, number=1)
title = proto.Field(proto.STRING, number=2)
lyrics = proto.Field(proto.STRING, number=3)
year = proto.Field(proto.INT32, number=4)
For messages and enums, assign the message or enum class directly (as shown in the example above).
Note: For messages declared in the same module, it is also possible to use a string with the message class’ name if
2.2. Fields 9
Proto Plus for Python Documentation
the class is not yet declared, which allows for declaring messages out of order or with circular references.
Some fields are actually repeated fields. In protocol buffers, repeated fields are generally equivalent to typed lists. In
protocol buffers, these are declared using the repeated keyword:
message Album {
repeated Song songs = 1;
string publisher = 2;
}
class Album(proto.Message):
songs = proto.RepeatedField(Song, number=1)
publisher = proto.Field(proto.STRING, number=2)
class Row(proto.Message):
values = proto.RepeatedField(proto.MESSAGE, number=1, message=struct.Value,)
class Row(proto.Message):
values = proto.RepeatedField(proto.MESSAGE, number=1, message=struct.Value,)
self._pb.MergeFrom(self._meta.pb(**{key: pb_value}))
TypeError: Value must be iterable
Similarly, some fields are map fields. In protocol buffers, map fields are equivalent to typed dictionaries, where the
keys are either strings or integers, and the values can be any type. In protocol buffers, these use a special map syntax:
message Album {
map<uint32, Song> track_list = 1;
string publisher = 2;
}
Protocol buffers allows certain fields to be declared as mutually exclusive. This is done by wrapping fields in a oneof
syntax:
import "google/type/postal_address.proto";
message AlbumPurchase {
Album album = 1;
oneof delivery {
google.type.PostalAddress postal_address = 2;
string download_uri = 3;
}
}
When using this syntax, protocol buffers will enforce that only one of the given fields is set on the message, and setting
a field within the oneof will clear any others.
Declare this in Python using the oneof keyword-argument, which takes a string (which should match for all fields
within the oneof):
from google.type.postal_address import PostalAddress
class AlbumPurchase(proto.Message):
album = proto.Field(Album, number=1)
postal_address = proto.Field(PostalAddress, number=2, oneof='delivery')
download_uri = proto.Field(proto.STRING, number=3, oneof='delivery')
Warning: oneof fields must be declared consecutively, otherwise the C implementation of protocol buffers will
reject the message. They need not have consecutive field numbers, but they must be declared in consecutive order.
Warning: If a message is constructed with multiple variants of a single oneof passed to its constructor, the last
keyword/value pair passed will be the final value set.
This is consistent with PEP-468, which specifies the order that keyword args are seen by called functions, and with
the regular protocol buffers runtime, which exhibits the same behavior.
Example:
import proto
class Song(proto.Message):
name = proto.Field(proto.STRING, number=1, oneof="identifier")
database_id = proto.Field(proto.STRING, number=2, oneof="identifier")
2.2. Fields 11
Proto Plus for Python Documentation
All fields in protocol buffers are optional, but it is often necessary to check for field presence. Sometimes legitimate
values for fields can be falsy, so checking for truthiness is not sufficient. Proto3 v3.12.0 added the optional keyword
to field descriptions, which enables a mechanism for checking field presence.
In proto plus, fields can be marked as optional by passing optional=True in the constructor. The message class
then gains a field of the same name that can be used to detect whether the field is present in message instances.
class Song(proto.Message):
composer = proto.Field(Composer, number=1)
title = proto.Field(proto.STRING, number=2)
lyrics = proto.Field(proto.STRING, number=3)
year = proto.Field(proto.INT32, number=4)
performer = proto.Field(proto.STRING, number=5, optional=True)
>>> s = Song(
... composer={'given_name': 'Johann', 'family_name': 'Pachelbel'},
... title='Canon in D',
... year=1680,
... genre=Genre.CLASSICAL,
... )
>>> Song.performer in s
False
>>> s.performer = 'Brahms'
>>> Song.performer in s
True
>>> del s.performer
>>> Song.performer in s
False
>>> s.performer = "" # The mysterious, unnamed composer
>>> Song.performer in s
True
Under the hood, fields marked as optional are implemented via a synthetic one-variant oneof. See the protocolbuffers
documentation for more information.
Proto Plus provides a service that converts between protocol buffer objects and native Python types (or the wrapper
types provided by this library).
This allows native Python objects to be used in place of protocol buffer messages where appropriate. In all cases, we
return the native type, and are liberal on what we accept.
Note: Protocol buffers include well-known types for Timestamp and Duration, both of which have nanosec-
ond precision. However, the Python datetime and timedelta objects have only microsecond precision. This
library converts timestamps to an implementation of datetime.datetime, DatetimeWithNanoseconds, that in-
cludes nanosecond precision.
If you write a timestamp field using a Python datetime value, any existing nanosecond precision will be overwritten.
Note: Setting a bytes field from a string value will first base64 decode the string. This is necessary to preserve
the original protobuf semantics when converting between Python dicts and proto messages. Converting a message
containing a bytes field to a dict will base64 encode the bytes field and yield a value of type str.
import proto
from google.protobuf.json_format import ParseDict
class MyMessage(proto.Message):
data = proto.Field(proto.BYTES, number=1)
# Note: the value is the base64 encoded string of the bytes field.
# It has a type of str, NOT bytes.
assert type(msg_dict['data']) == str
Additionally, every Message subclass is a wrapper class. The creation of the class also creates the underlying protocol
buffer class, and this is registered to the marshal.
The underlying protocol buffer message class is accessible with the pb() class method.
2.4 Status
2.5 Reference
Below is a reference for the major classes and functions within this module.
• The Message and Field section (which uses the message and fields modules) handles constructing mes-
sages.
• The Marshal module handles translating between internal protocol buffer instances and idiomatic equivalents.
• The Datetime Helpers has datetime related helpers to maintain nanosecond precision.
2.5. Reference 15
Proto Plus for Python Documentation
Parameters
• instance – An instance of this message type, or something compatible (accepted by the
type’s constructor).
• use_integers_for_enums (Optional(bool)) – An option that determines
whether enum values should be represented by strings (False) or integers (True). Default
is True.
• preserving_proto_field_name (Optional(bool)) – An option that deter-
mines whether field name representations preserve proto case (snake_case) or use low-
erCamelCase. Default is True.
• including_default_value_fields (Optional(bool)) – An option that de-
termines whether the default field values should be included in the results. Default is True.
Returns
A representation of the protocol buffer using pythonic data structures. Messages and
map fields are represented as dicts, repeated fields are represented as lists.
Return type dict
classmethod copy_from(instance, other)
Equivalent for protobuf.Message.CopyFrom
Parameters
• instance – An instance of this message type
• other – (Union[dict, ~.Message): A dictionary or message to reinitialize the values for
this message.
class proto.fields.Field(proto_type, *, number: int, message=None, enum=None, oneof: str =
None, json_name: str = None, optional: bool = False)
A representation of a type of field in protocol buffers.
descriptor
Return the descriptor for the field.
name
Return the name of the field.
package
Return the package of the field.
pb_type
Return the composite type of the field, or the primitive type if a primitive.
class proto.fields.MapField(key_type, value_type, *, number: int, message=None, enum=None)
A representation of a map field in protocol buffers.
class proto.fields.RepeatedField(proto_type, *, number: int, message=None, enum=None,
oneof: str = None, json_name: str = None, optional: bool
= False)
A representation of a repeated field in protocol buffers.
class proto.enums.Enum
A enum object that also builds a protobuf enum descriptor.
class proto.enums.ProtoEnumMeta
A metaclass for building and registering protobuf enums.
2.5.2 Marshal
2.5. Reference 17
Proto Plus for Python Documentation
p
proto.datetime_helpers, 17
proto.enums, 16
proto.fields, 16
proto.marshal, 17
19
Proto Plus for Python Documentation
C proto.enums (module), 16
copy_from() (proto.message.Message class method), proto.fields (module), 16
16 proto.marshal (module), 17
ProtoEnumMeta (class in proto.enums), 16
D
DatetimeWithNanoseconds (class in R
proto.datetime_helpers), 17 RepeatedField (class in proto.fields), 16
descriptor (proto.fields.Field attribute), 16 replace() (proto.datetime_helpers.DatetimeWithNanoseconds
deserialize() (proto.message.Message class method), 17
method), 15 rfc3339() (proto.datetime_helpers.DatetimeWithNanoseconds
method), 17
E
Enum (class in proto.enums), 16 S
serialize() (proto.message.Message class method),
F 15
Field (class in proto.fields), 16
from_json() (proto.message.Message class method),
T
15 timestamp_pb() (proto.datetime_helpers.DatetimeWithNanoseconds
method), 17
from_rfc3339() (proto.datetime_helpers.DatetimeWithNanoseconds
class method), 17 to_dict() (proto.message.Message class method), 15
from_timestamp_pb() to_json() (proto.message.Message class method), 15
(proto.datetime_helpers.DatetimeWithNanoseconds
class method), 17 W
wrap() (proto.message.Message class method), 14
M
MapField (class in proto.fields), 16
Marshal (class in proto.marshal), 17
Message (class in proto.message), 14
N
name (proto.fields.Field attribute), 16
nanosecond (proto.datetime_helpers.DatetimeWithNanoseconds
attribute), 17
P
package (proto.fields.Field attribute), 16
pb() (proto.message.Message class method), 14
pb_type (proto.fields.Field attribute), 16
proto.datetime_helpers (module), 17
21