04/08/2020 HedEx Startpage

HUAWEI NetEngine 8000 M14 and M8 Series Router Product Documentation

Product Version: V800R012C00
Library Version: 04
Date: 2020-04-30

For any question, please contact us.

Copyright © Huawei Technologies Co., Ltd. 2020. All rights reserved.

Key Telemetry Technologies

Contents
3.3.17.5.2.3 Key Telemetry Technologies

3.3.17.5.2.3 Key Telemetry Technologies

For the Telemetry device side, Telemetry indicates the original data, data model, encoding format, and transmission protocol. For the Telemetry NMS side, Telemetry indicates
the data collection system, storage system, and application analysis system.

This document describes key technologies only on the Telemetry device side.

Original Data
Telemetry collects data from the forwarding, control, and management planes of network devices. Currently, the data that can be collected includes interface traffic statistics,
CPU usage, and memory usage.

Data Model
Telemetry collects data based on the YANG model.
YANG is a data modeling language developed to define configuration data modules, status data models, RPC models, and notification mechanisms for transmission protocols.

For more information, see YANG Model Introduction.

Encoding Format
Support for the Google Protocol Buffers (GPB) Encoding Format
Google Protocol Buffers (GPB) encoding is a language-neutral, platform-neutral, and extensible mechanism for serializing structured data of communications protocols and data
storage.
Telemetry uses the encoding format GPB (the file name extension of a file encoded in GPB format is .proto) and provides a mechanism for serializing structured data flexibly,
efficiently, and automatically. The GPB format is binary encoding and has good performance and high efficiency.
GPB uses the .proto file to describe a dictionary for encoding, which is the data structure description. You can use a tool such as Protoc (for example, the protoc-3.0.2-windows-
x86_64.exe file) to automatically generate code (for example, Java code) based on the .proto file. Then, perform secondary development based on the automatically generated
code to encode and decode GPB, thereby implementing device interworking. Table 1 compares GPB encoding and decoding.

Table 1 Comparison between GPB encoding and decoding

GPB Encoding GPB Decoding

localhost:7890/printtopics.html?time=Tue Aug 04 2020 14:21:55 GMT-0300 (Horário Padrão de Brasília) 1/6

04/08/2020 HedEx Startpage

GPB Encoding GPB Decoding

{ {
1:"HUAWEI" "node_id_str":"HUAWEI",
2:"s4" "subscription_id_str":"s4",
3:"huawei-ifm:ifm/interfaces/interface" "sensor_path":"huawei-ifm:ifm/interfaces/interface",
4:46 "collection_id":46,
5:1515727243419 "collection_start_time":"2018/1/12 11:20:43.419",
6:1515727243514 "msg_timestamp":"2018/1/12 11:20:43.514",
7{ "data_gpb":{
1[{ "row":[{
1: 1515727243419 "timestamp":"2018/1/12 11:20:43.419",
2 { "content":{
5{ "interfaces":{
1[{ "interface":[{
5:1 "ifAdminStatus":1,
16:2 "ifIndex":2,
25:"Eth-Trunk1" "ifName":"Eth-Trunk1"
}] }]
} }
} }
}] }]
} },
8:1515727243419 "collection_end_time":"2018/1/12 11:20:43.419",
9:10000 "current_period":10000,
10:"OK" "except_desc":"OK",
11:"NetEngine 8000" "product_name":"NetEngine 8000",
12:0 "encoding":Encoding_GPB
} }

Support for the JavaScript Object Notation (JSON) Encoding Format

JavaScript object notation (JSON) is a lightweight data exchange format. Based on a subset of the ECMAScript (JS specification developed by the ECMA), it uses a text format
independent of the programming language to store and express data with a simple and clear structure, so that the data can be easily read or compiled by human and parsed or
generated by computers. Table 2 shows the JSON encoding format.

Table 2 JSON encoding format

JSON Encoding

{
"node_id_str":"HUAWEI",
"subscription_id_str":"s4",
"sensor_path":"huawei-ifm:ifm/interfaces/interface",
"collection_id":46,
"collection_start_time":"2018/1/12 11:20:43.419",
"msg_timestamp":"2018/1/12 11:20:43.514",
"collection_end_time":"2018/1/12 11:20:43.419",
"current_period":10000,
"except_desc":"OK",
"product_name":"NetEngine 8000",
"encoding":Encoding_JSON,
"data_str":{
"row":[{
"timestamp":"2018/1/12 11:20:43.419",
"content":{
"interfaces":{
"interface":[{
"ifAdminStatus":1,
"ifIndex":2,
"ifName":"Eth-Trunk1"
}]
}
}
}]
}
}

For data transmitted between devices and collectors, the data encoded in the GPB format has a higher information load capability than that encoded in other formats
(JSON or XML). GPB encoding ensures Telemetry data throughput capabilities and reduces the CPU usage and bandwidth.
If telemetry uses the JSON encoding format, the data packets sent to the collector do not contain the typesetting characters such as carriage return or space to reduce the
size of JSON encoding data.

Transport Protocol
Support for Google Remote Procedure Call Protocol (gRPC)
gRPC is a high-performance general RPC open-source software framework running over HTTP2 protocols. Both communication parties perform secondary development based
on the framework, so that they focus on services and do not need to pay attention to bottom-layer communication implemented by the gRPC software framework.
Telemetry uses the gRPC protocol to report the data encoded in GPB format to the collector.

The gRPC protocol applies to telemetry static subscription and dynamic subscription.

Figure 1 shows the gRPC protocol stack layers.

localhost:7890/printtopics.html?time=Tue Aug 04 2020 14:21:55 GMT-0300 (Horário Padrão de Brasília) 2/6

04/08/2020 HedEx Startpage
Figure 1 Description of gRPC protocol stack layers

Table 3 describes the layers.

Table 3 Layers
Layer Description

TCP layer This is a bottom-layer communication protocol, which is based on TCP connections.

TLS layer This layer is optional. It is based on the TLS–encrypted channel.

HTTP2 layer The HTTP2 protocol carries gRPC, using HTTP2 features such as bidirectional streams, flow control,
header compression, and multiplexing request of a single connection.

gRPC layer This layer defines the protocol interaction format for remote procedure calls.

Data model layer RPC layer Telemetry static subscription: defined in the huawei-grpc-dialout.proto file.
NOTE: Telemetry dynamic subscription: defined in the huawei-grpc-dialin.proto file.
The data model
layer is divided Telemetry layer When the encoding format of the data to be sent to the collector is GPB, it is defined in the
into three huawei-telemetry.proto file.
layers, which
are defined in
When the encoding format of the data to be sent to the collector is JSON, you do not need to use
different .proto the .proto files at the telemetry layer for decoding.
files. To
establish a gRPC Service data layer If the encoding format of the data to be sent is GPB, the encoding field in the huawei-
connection
between the telemetry.proto file is Encoding_GPB (with the value of 0), the data_gpb field carries
collector and a sampling data in GPB format, and the data_str field is empty.
network device,
use the .proto NOTE:
files for The data_gpb field needs to be decoded by the corresponding service .proto file. The
interconnection. sensor_path field in the huawei-telemetry.proto file identifies the specific
The .proto files service .proto file. For example, if the value of the sensor_path field is huawei-
on the two ends ifm:ifm/interfaces/interface , the corresponding data structure is defined in the
must be the
same, which
huawei-ifm.proto file.
facilitates If the encoding format of the data to be sent is JSON, the encoding field in the huawei-
decoding. telemetry.proto file is Encoding_JSON (with the value of 1), the data_str field carries
sampling data in JSON format, and the data_gpb field is empty.
NOTE:
If the JSON encoding format is used, you only need to use the .proto files at the RPC
layer for decoding. The .proto file at the telemetry layer and the corresponding service
.proto file are not required.

The .proto files are described as follows:

huawei-grpc-dialout.proto is the RPC header file. When a device functions as the client to push data, this file defines the RPC interface. The following table
describes the file content and meaning.

huawei-grpc-dialout.proto

syntax = "proto3"; //The .proto file version is defined as v3.

package huawei_dialout; //The package name is huawei_dialout
.
service gRPCDataservice { //The service name is gRPCDataservice
.
rpc dataPublish(stream serviceArgs) returns(stream serviceArgs) {}; //The method name is dataPublish , providing the
}
message serviceArgs { //Message format description.
int64 ReqId = 1; //Request ID, whose index is 1 during GPB coding.
oneof MessageData {
bytes data = 2; //Sampled data in GPB encoding format is carried.
string data_json = 4; //Sampled data in JSON encoding format is carried.
}
string errors = 3; //Error description, whose index is 3 during GPB coding.
}

huawei-grpc-dialin.proto is the RPC header file. When a device functions as the server to push data, this file defines the RPC interface. The following table describes
the file content and meaning.

huawei-grpc-dialin.proto

localhost:7890/printtopics.html?time=Tue Aug 04 2020 14:21:55 GMT-0300 (Horário Padrão de Brasília) 3/6

04/08/2020 HedEx Startpage

huawei-grpc-dialin.proto

syntax = "proto3"; //The .proto file version is defined as v3.

package huawei_dialin; //The package name is huawei_dialin
.
service gRPCConfigOper { //The service name is gRPCConfigOper
.
rpc Subscribe (SubsArgs) returns (stream SubsReply) {}; //The method is Subscribe
, and the server stream mode is used, providing th
rpc Cancel (CancelArgs) returns (CancelReply) {}; //The method is Cancel
, and the query and response mode is used, providing the me
}
message Path { //Path message structure.
string path = 1; //Subscribed sensor-path.
uint32 depth = 2; //Sampling depth of the subscribed sensor-path. The value 1 indicates the current level, and the value 2 i
}
message SubsArgs { //Subscribing request parameter.
uint64 request_id = 1; //Request ID, which is transferred by the invoker.
uint32 encoding = 2; //Encoding type. Currently, the value can only be 0, indicating GPB encoding.
repeated Path path = 5; //Subscribed path structure.
uint64 sample_interval = 6; //Sampling interval.
uint64 heartbeat_interval = 7; //Redundancy suppression interval. This parameter is valid only when suppress_redundant
is set to 1
oneof Suppress {
bool suppress_redundant = 8; //Redundancy suppression. The sampled data is not reported if the data content remains unchanged.
uint64 delay_time = 9; //Delay time, which ranges from 100 to 60000, in ms. After data changes, the new data is reported
}

}
message SubsReply { //Subscribing response parameter.
uint32 subscription_id = 1; //If the subscribing is successful, the subscription ID is returned. If the subscribing fails, 0 is retu
uint64 request_id = 2; //ID of the subscribing request.
string response_code = 3; //Return code. The value 200 indicates a success.
oneof MessageData {
bytes message = 4; //If an error occurs, the error description is returned. If no error occurs, the GPB encoding data is
string message_json = 5; //If no error occurs, the JSON encoding data is sent.

}
}
message CancelArgs { //Unsubscribing request parameter.
uint64 request_id = 1; //Request ID, which is transferred by the invoker.
uint32 subscription_id = 2; //ID of the subscription to be canceled.
}
message CancelReply { //Unsubscribing response parameter.
uint64 request_id = 1; //Request ID, which is transferred by the invoker.
string response_code = 2; //Return code. The value 200 indicates a success.
string message = 3; //Error description.
}

huawei-telemetry.proto is the telemetry header definition file. It defines the data header when telemetry data sampled is sent, including key information, such as the
sampling path and sampling timestamp. The following table describes the file content and meaning.

huawei-telemetry.proto

localhost:7890/printtopics.html?time=Tue Aug 04 2020 14:21:55 GMT-0300 (Horário Padrão de Brasília) 4/6

04/08/2020 HedEx Startpage

huawei-telemetry.proto

syntax = "proto3"; //The .proto file version is defined as v3.

package telemetry; //The package name is . telemetry
message Telemetry { //Telemetry message structure definition.
string node_id_str = 1; //Device name.
string subscription_id_str = 2; //Subscription name during static subscription configuration, whose index is 2 during GPB
string sensor_path = 3; //Subscription path, whose index is 3 during GPB encoding.
string proto_path = 13; //Message path for the sampling path in the proto file.
uint64 collection_id = 4; //Sampling round, whose index is 4 during GPB encoding.
uint64 collection_start_time = 5; //Start time of a sampling round, whose index is 5 during GPB encoding.
uint64 msg_timestamp = 6; //Timestamp when the current message is generated, whose index is 6 during GPB encoding.
TelemetryGPBTable data_gpb = 7; //Indicates that the data carried is defined in TelemetryGPBTable
. The index is 7 durin
uint64 collection_end_time = 8; //End time of a sampling round, whose index is 8 during GPB encoding.
uint32 current_period = 9; //Sampling precision, in milliseconds, whose index is 9 during GPB encoding.
string except_desc = 10; //Exception description, whose index is 10 during GPB encoding. It is used to report exception inform
string product_name = 11; //Product name.
enum Encoding {
Encoding_GPB = 0; //GPB encoding format.
Encoding_JSON = 1; //JSON encoding format.
};
Encoding encoding = 12; //Data encoding format. If the GPB encoding format is used, the data_gpb
field is valid. Otherwise, the da
string data_str = 14; //This field is valid only when a non-GPB encoding format is used.
string ne_id = 15; //Unique ID of an NE. In the gateway scenario, this parameter is used to identify the NE to which dat
string software_version = 16; //Software version number.
}
message TelemetryGPBTable { //TelemetryGPBTable message structure definition.
repeated TelemetryRowGPB row = 1; //Array definition, whose index is 1 during GPB encoding. Its member is TelemetryRowGPB st
repeated DataPath delete = 2; //Delete the data path.
Generator generator = 3; //Data source description. This applies to the OnChange+ service that requires high reliability.
}
message Generator {
uint64 generator_id = 1; //Data source ID. Multiple data sources can provide data concurrently and maintain their own reliabi
uint32 generator_sn = 2; //Message sequence number. The sequence numbers of messages sent by each data source must be consecu
bool generator_sync = 3; //Data source synchronization. Indicates whether to perform OnChange full data synchronization. In a
}
message TelemetryRowGPB {
uint64 timestamp = 1; //Timestamp of the current sampling instance, whose index is 1 during GPB encoding.
bytes content = 11; //Sampling instance data carried, whose index is 11 during GPB encoding. The fi sensor_path
}
message DataPath {
uint64 timestamp = 1; //Timestamp of the current sampling instance, whose index is 1 during GPB encoding.
Path path = 2; // Data tree node, which contains only the data path and key field information.
}
message Path {
repeated PathElem node = 1; //Data tree node, which contains only the data path and key field information.
}
message PathElem {
string name = 1; //Name of the data tree node.
map<string, string> key = 2; //Key field name and value mapping table of the data tree node.
}
message TelemetrySelfDefinedEvent {
string path = 1; //Sampling path that triggers the customized event, which describes the method of parsing the content.
string proto_path = 13; //Message path for the sampling path in the proto file.
uint32 level = 2; //Level of the user-defined event.
string description = 3; //Description of the user-defined event.
string fieldName = 4; //Name of the field that triggers the customized event.
uint32 fieldValue = 5; //Value of the field that triggers the customized event.
TelemetrySelfDefineThresTable data_threshold = 6; //Threshold filter criteria when the customized event is triggered.
enum ThresholdRelation {
ThresholdRelation_INVALID = 0; //The relationship between thresholds is not configured.
ThresholdRelation_AND = 1; //The relationship between thresholds is And.
ThresholdRelation_OR = 2; //The relationship between thresholds is Or.
}
ThresholdRelation thresholdRelation = 7; //Relationship between threshold filter criteria when the customized event is triggered.
bytes content = 8; //Sampled data that triggers the customized event.
}
message TelemetrySelfDefineThresTable {
repeated TelemetryThreshold row = 1; //Multiple thresholds are included.
}
message TelemetryThreshold {
uint32 thresholdValue = 1; //Delivered threshold.
enum ThresholdOpType {
ThresholdOpType_EQ = 0; //The actual value in the data sent equals to the configured data threshold.
ThresholdOpType_GT = 1; //The actual value in the data sent is greater than the configured data threshold.
ThresholdOpType_GE = 2; //The actual value in the data sent is greater than or equals to the configured data threshold.
ThresholdOpType_LT = 3; //The actual value in the data sent is less than the configured data threshold.
ThresholdOpType_LE = 4; //The actual value in the data sent is less than or equals to the configured data threshold.
}
ThresholdOpType thresholdOpType = 2; //Threshold on the device.
}

A service data file describes the detailed service data formats. .

Support for User Datagram Protocol (UDP)

UDP is a datagram-oriented simple transport-layer protocol. It does not provide reliability, flow control, or error recovery functions for IP. UDP is applicable to networks with
low reliability requirements and economical transmission.
Telemetry uses the UDP protocol to report the data collected to the collector.

The UDP protocol applies to telemetry static subscription.

Figure 2 shows the UDP protocol stack layers.

Figure 2 UDP protocol stack layers

localhost:7890/printtopics.html?time=Tue Aug 04 2020 14:21:55 GMT-0300 (Horário Padrão de Brasília) 5/6

04/08/2020 HedEx Startpage
Table 4 describes the layers.

Table 4 Layers
Layer Description

UDP layer UDP-based communications protocol.

Data model layer Message header For the definition of message header architecture, see UDP based Publication Channel for Streaming Telemetry.
NOTE: Telemetry layer When the encoding format of the data to be sent to the collector is GPB, it is defined in the
The data model huawei-telemetry.proto file.
layer is divided
into three When the encoding format of the data to be sent to the collector is JSON, you do not need to use
layers, which the .proto files at the telemetry layer for decoding.
are defined in
different .proto
files except the
Service data layer If the encoding format of the data to be sent is GPB, the encoding field in the huawei-
Message header. telemetry.proto file is Encoding_GPB (with the value of 0), the data_gpb field carries
To establish a sampling data in GPB format, and the data_str field is empty.
UDP connection
between the NOTE:
The data_gpb field needs to be decoded by the corresponding service .proto file. The
collector and a
network device,
sensor_path field in the huawei-telemetry.proto file identifies the specific service
.proto file. For example, if the value of the sensor_path field is huawei-
use the .proto
files for
ifm:ifm/interfaces/interface , the corresponding data structure is defined in the
interconnection.
The .proto files
huawei-ifm.proto file.
on the two ends
must be the
If the encoding format of the data to be sent is JSON, the encoding field in the huawei-
same, which telemetry.proto file is Encoding_JSON (with the value of 1), the data_str field carries
facilitates sampling data in JSON format, and the data_gpb field is empty.
decoding. For
description of NOTE:
the .proto If the JSON encoding format is used, you only need to use the .proto files at the RPC
files, see the layer for decoding. The .proto file at the telemetry layer and the corresponding service
relevant content .proto file are not required.
of gRPC.

Customized Event Reporting Based on Telemetry Static Subscription

You can configure a Telemetry customized event. If a performance indicator of a resource object that Telemetry monitors exceeds the user-defined threshold, the customized
event is reported to the collector in time for service policy determination.

Push Mode
The Telemetry push mode enables network devices to periodically push data to the NMS, avoiding repeated query and improving monitoring performance. Figure 3 shows the
push process.
Figure 3 Sampling process in telemetry push mode

Parent Topic: Understanding Telemetry

Copyright © Huawei Technologies Co., Ltd.
Copyright © Huawei Technologies Co., Ltd.
< Previous topic

localhost:7890/printtopics.html?time=Tue Aug 04 2020 14:21:55 GMT-0300 (Horário Padrão de Brasília) 6/6