KALEIDOSCOPE

Network Control and

Management Automation:
Architecture Standardization Perspective
Ved P. Kafle, Takahiro Hirayama, Takaya Miyazawa, Masahiro Jibiki, and Hiroaki Harai

Abstract applications that require very high bandwidth are

supported by eMBB services, and Internet of Things
Artificial intelligence (AI) and machine learning (IoT) devices and smart utility meters connected to
(ML) techniques are becoming essential to auto- the network in large numbers are supported by an
mate the control and management of complex sys- mMTC service, whereas automated vehicles that
tems, such as 5G and future networks and services, require ultra-high reliability and low latency are sup-
through the collection and timely processing of ported by a URLLC service. These three types of
a large amount of control data from the system. communication services can be offered from the
The standardization activities of AI/ML integrat- same sharable virtualized cloud computing and net-
ed network architectures and frameworks have working infrastructure by configuring network slices
recently been progressing in various standards containing the required VNFs [3].
developing organizations (SDOs). This article Unlike traditional telecommunication infra-
provides a detailed survey of the AI/ML network structures, in which the network functions are
standardization activities of the International Tele- mostly implemented in hardware, the control
communication Union. An overview of the relat- and management of virtualized, programmable,
ed activities of other SDOs, such as the European software-driven, and distributed cloud/edge com-
Telecommunications Standards Institute and the puting infrastructures are challenging because of
3rd Generation Partnership Project, is also provid- the proliferation of vendor-specific platforms and
ed. Finally, a set of related open issues that deserve management technologies. Moreover, the emerg-
further research and standardization are discussed. ing demand for more complex services (e.g., con-
text-aware and personalized services) for a large
Introduction number of user subscriptions has made network
Fifth generation (5G) networks and beyond will control and management extremely challenging.
be built on the convergence of virtualized net- Starting from the network service design,
working and cloud/edge computing infrastruc- resource provisioning, and deployment for perfor-
tures. These networks are configured through mance monitoring, resource adjustment, failure
software using the recently developed technolo- detection, and recovery, network control and man-
gies of software-defined networking (SDN) and agement operations require the analysis of a huge
network function virtualization (NFV) [1]. A virtu- volume of control data collected from various com-
alized computing infrastructure can be segmented ponents of the network. Under this complex situa-
into multiple virtual machines or containers, each tion, the capacity of the human operator does not
of which can be used to deploy and operate one meet the requirements of quickly processing a large
or more cloud-native network functions or virtual volume of data and providing a timely response. To
network functions (VNFs) [2]. SDN enables the address this issue, advanced techniques of artificial
configuration of virtual network slices by orches- intelligence (AI) and machine learning (ML) have
trating VNFs remotely through control functions recently been applied in network control and man-
implemented in the software. Moreover, the agement functions [5, 6].
computational (e.g., CPU and memory), storage, The effectiveness of ML applications for work-
and networking (e.g., buffer size and bandwidth) load and resource utilization prediction, and accord-
resources allocated to each VNF can be adjusted ingly adjusting the computational resources and
dynamically such that its performance can always reconfiguring the network to maintain the required
meet the quality-of-service requirements of the quality of service at all times, was demonstrated in a
delivered communication service [3]. paper presented at the ITU Kaleidoscope Academic
In addition, 5G networks, which are also Conference 2020 [7]. Extending the standardization
referred to as IMT-2020 networks in the Interna- aspects identified in that paper, this article provides a
tional Telecommunication Union (ITU) standards, survey of AI/ML integrated network standardization
have been designed and developed to offer three activities of various standards developing organiza-
types of communication services: enhanced mobile tions (SDOs), mainly focusing on the activities of the
broadband (eMBB), massive machine-type commu- International Telecommunication Union (ITU). ITU’s
nication (mMTC), and ultra-reliable low-latency com- Telecommunication Standardization Sector (ITU-T)
munication (URLLC) [4]. High-definition video (e.g., has recently produced several standard documents
4K video) and augmented/virtual reality (AR/VR) (i.e., ITU-T Recommendations) on the high-lev-
Digital Object Identifier:
10.1109/MCOMSTD.121.2100043 The authors are with the National Institute of Information and Communications Technology, Tokyo.

106 2471-2825/21/$25.00 © 2021 IEEE IEEE Communications Standards Magazine • September 2021

KAFLE_LAYOUT.indd 106 10/7/21 1:40 AM

 el architectures and frameworks of ML-controlled ments are mapped to three groups of functions:
networks. In this article, the latest developments in data collection, data storage and processing, and ITU-T Supplement 55
ITU-T AI/ML high-level architectures of data han- ML models. The requirements are labeled as criti- describes 30 use cases
dling, network intelligence evaluation, network ser- cal, expected, and added values. and their requirements.
vice provisioning, resource and fault management, ITU-T Recommendation Y.3072 specifies the
and quality of service assurance are reviewed. Sim- requirement and high-level architecture for inte- The use cases are clas-
ilarly, the reference architectures of network and grating ML in future networks, including IMT-2020. sified into the following
service management developed by the European It also provides example guidelines for mapping
Telecommunications Standards Institute’s (ETSI’s) this ML-based architecture framework with a stan- five groups: network slice
two Industry Specification Groups (ISGs), Experien- dardized 5G network architecture. As shown in Fig. and service, user plane,
tial Networked Intelligence (ENI) and Zero Touch 1, the architecture framework consists of four sub- applications, signaling
Network and Service Management (ZSM), as well systems: the ML pipeline subsystem, ML sandbox
as Network Data Analytics Function (NWDAF) of subsystem, management subsystem, and ML under- and management, and
the 3rd Generation Partnership Project (3GPP) are lay networks. The ML pipeline subsystem contains security. Similarly, the
reviewed. It also discusses a set of open issues that a set of ML models and execution abstraction. It
deserve further research and standardization for the consists of several functional components, such requirements are mapped
realization of intelligent and autonomous network as the input data collector (C), preprocessor (PP), to three groups of func-
control and management. ML models (M), policy (P), and output result dis- tions: data collection, data
tributor (D). The input data are collected through
storage and processing,
AI/ML Network a source function (SRC), and the ML output results
are distributed to the network controllers through and ML models. The
Standardization in ITU a sink function (SINK). The ML sandbox subsystem
ITU started a study and standardization of AI/ also contains the same functional components as requirements are labeled
ML integration in telecommunication networks in the ML pipeline subsystem. The ML sandbox sub- as critical, expected, and
2017 with the establishment of the Focus Group system uses simulated underlay networks to train added values.
on Machine Learning for Future Networks, includ- and test ML models. Trained ML models are then
ing 5G (FG ML5G) [8]. This section introduces deployed in the ML pipeline subsystem to mon-
the ITU-T Recommendations developed on the itor and control live ML underlay networks. The
basis of FG ML5G deliverables, ITU-T Recommen- management subsystem consists of an ML func-
dations developed from contributions by ITU-T tion orchestration (MLFO) and other management
Study Group 13 (SG13), and an overview of the functions. The MLFO function obtains the input of
activities of the recently established Focus Group the ML intents, and manages and orchestrates the
on Autonomous Networks. functional components of the ML pipeline and ML
sandbox. A detailed description of these functions
ITU-T Recommendations from and interfaces can be found in [9].
FG ML5G Deliverables ITU-T Recommendation Y.3173 starts with a
FG ML5G studied the use cases, requirements, description of the progress in network intelligence
data formats, interfaces, intelligence-level evalua- levels and then specifies methods and an architec-
tion, and ML-enabling network architecture among ture for the network intelligence level evaluation.
three working groups. In its two-and-a-half-year life- The standard methods for evaluating network intel-
time (2018–2020), FG ML5G produced 10 deliver- ligence levels can provide a decision mechanism to
ables (the list is available on the FG ML5G website operators, equipment vendors, and other network
[8]) and submitted them to ITU-T SG13 for the industry participants for planning network technol-
consideration as ITU-T Recommendations. ogy features and product roadmaps. The following
ITU-T SG13 further improved the content of five dimensions are provided for an intelligence level
the FG ML5G deliverables and approved the fol- evaluation: demand mapping, data collection, anal-
lowing documents as one ITU-T Supplement and ysis, decision, and action implementation. Demand
four ITU-T Recommendations: mapping corresponds to the process of converting
• ITU-T Supplement 55: Machine learning in the network configuration or requirements given
future networks including IMT-2020: use cases by a human operator into specific instructions that
• ITU-T Recommendation Y.3172: Architectur- network components can understand and execute.
al framework for machine learning in future Data collection and analysis dimensions are related
networks, including IMT-2020 to the process of collecting network monitoring and
• ITU-T Recommendation Y.3173: Framework control data and conducting an analysis.
for evaluating intelligence levels of future The decision and action implementation
networks, including IMT-2020 dimensions relate to the process of making net-
• ITU-T Recommendation Y.3174: Framework work or service configuration decisions and,
for data handling to enable machine learning accordingly, command executions.
in future networks, including IMT-2020 To evaluate the overall network intelligence
• ITU-T Recommendation Y.3176: Machine level of an entire network, the intelligence levels
learning marketplace integration in future of individual workflows and network subsystems
networks including IMT-2020 need to be evaluated with respect to the five
• ITU-T Recommendation Y.3179: Architectur- dimensions. The workflows constitute the tasks of
al framework for ML model serving in future planning, deployment, maintenance, optimization,
networks, including IMT-2020 and provisioning. Similarly, the network subsystem
ITU-T Supplement 55 describes 30 use cases includes components such as network elements,
and their requirements. The use cases are classi- management subsystems, and application plat-
fied into the following five groups: network slice forms. To determine the overall network intelli-
and service, user plane, applications, signaling and gence level, a basic method is provided, which is
management, and security. Similarly, the require- based on the network intelligence capability level

IEEE Communications Standards Magazine • September 2021 107

KAFLE_LAYOUT.indd 107 10/7/21 1:40 AM

 The 3GPP has defined the
network data analytics
function (NWDAF) in the
5G service-based architec-
ture (SBA) as the enabler
of intelligence and auton-
omous network operation
and management. The
NWDAF collects data from
various modules of the
5G system and provides
analysis results.

FIGURe 1. High-level architecture for integrating machine learning in future networks including IMT-2020.

of each of the five dimensions. Network intelli- three fundamental stages: inference optimization,
gence capability levels are determined based on model deployment, and model inference. In the
whether the function of a dimension is executed inference optimization stage, trained ML models
by a human only, a human and a system, or a sys- are modified to improve the performance when
tem only. Based on the combinations of the intel- executing an inference in a certain deployment
ligence capability levels of individual dimensions, environment according to the requirements of
six intelligence levels are defined, as shown in the use case and the current state of the network.
Table 1. In the lowest network intelligence level of In the model deployment stage, the ready-to-run
L0, which corresponds to a manual network oper- ML model is deployed in a specific deployment
ation, all dimensions are executed by humans. environment. Finally, in the inference stage, the
Similarly, in the highest network intelligence level model inference output result (e.g., prediction or
of L5, which corresponds to full intelligence, all classification) is applied to ML pipelines.
dimensions are autonomously executed by the In addition to the above ITU-T Recom-
system only. In between, there are intermediate mendations produced by enriching FG ML5G
intelligence levels in which the tasks of some deliverables, ITU-T SG13 has also produced Rec-
dimensions require human involvement. ommendations from contributions submitted by
ITU-T Recommendation Y.3074 specifies the its delegates, which are described next.
ML data handling framework to deal with the
diversity of the control data produced by various ITU-T Recommendations from
components in the network. It introduces data SG13 Contributions
models, brokers, and application program inter- From the contributions of its delegates, ITU-T
faces (APIs) in both the user and control planes. SG13 has produced three Recommendations,
It also provides the requirements for input data Y.3175, Y.3177, and Y.3178, which are briefly
collection, processing, and output data. described below.
ITU-T Recommendation Y.3176 describes the ITU-T Recommendation Y.3175 specifies the
challenges, motivations, requirements, and archi- functional architecture of ML-based quality of ser-
tecture for ML marketplace integration in networks. vice (QoS) assurance, including the reference points.
It defines the ML marketplace as a repository of It also describes the procedures for ML-based QoS
interoperable trained AI/ML models. It also spec- assurance in the IMT-2020 network.
ifies a method that uses the ML intent and MLFO By extending the basic ML architecture spec-
to select appropriate ML models from the ML mar- ified in ITU-T Y.3172, ITU-T Recommendation
ketplace, and interfaces to connect the ML market- Y.3177 specifies a high-level architecture of AI/
place with the ML sandbox and the ML pipeline. ML-based network automation for resource and
Similarly, ITU-T Recommendation Y.3179 spec- fault management. Figure 2 shows the architec-
ifies an architectural framework for ML model ture, which is composed of four subsystems. The
serving, that is, the preparation and deployment of management subsystem contains the following
ML models in different deployment environments. three management functions: resource manage-
The ML model service takes place in the following ment, fault management, and other manage-

108 IEEE Communications Standards Magazine • September 2021

KAFLE_LAYOUT.indd 108 10/7/21 1:40 AM

 Dimensions
Network intelligence level
Action implementation Data collection Analysis Decision Demand mapping

L0 Manual network operation Human Human Human Human Human

L1 Assisted network operation Human and system Human and system Human Human Human

L2 Preliminary intelligence System Human and system Human and system Human Human

L3 Intermediate intelligence System System Human and system Human and System Human

L4 Advanced intelligence System System System System Human and System

L5 Full intelligence System System System System System

Table 1. Network intelligence levels.

ment functions. It also contains the ML functional pre-standardization activities on autonomous net-
orchestrator (MLFO), which takes ML intents as works. The terms of reference of FG AN define
input. The AI/ML pipeline consists of six function- autonomous networks as those that possess the
al groups: data collection, fault detection, fault ability to monitor, operate, recover, heal, protect,
recovery, resource prediction, resource adapta- optimize, and reconfigure themselves. FG AN stud-
tion, and controller. The AI/ML sandbox is com- ies the autonomy of various processes or network
posed of an AI/ML pipeline for the purpose of aspects, such as planning, security, audits, invento-
training AI/ML models by using the data obtained ry, optimization, orchestration, and quality of expe-
from the simulated AI/ML underlay network. The rience. The group studies the various approaches
reader is referred to [10] for a detailed descrip- of exploratory evolution, emergent behavior, and
tion of these functional groups and interfaces. real-time responsive experimentations that can
Similarly, ITU-T Recommendation Y.3178 spec- provide a new layer of abstraction for introducing
ifies a functional framework for AI-based network evolution mechanisms leading to the realization
service provisioning. It starts with a description of of autonomous networks. The group also tries to
the business-role-based model for AI-based net- address the questions associated with accountabil-
work service provisioning, then provides a list of ity for non-human decisions that affect customers
high-level requirements for the roles and their inter- and explore the approaches of exploratory evolu-
actions from an AI-based operational perspective, tion, emergent behavior, and real-time responsive
as well as the functional framework showing the experimentation to enable an autonomous net-
components and their interactions for AI-based work.
operations for network service provisioning. Over the course of a one-year period, FG AN
has planned to produce deliverables defining the
Focus Group on Autonomous Networks characteristics, use cases, requirements, proofs of
ITU-T SG13 established the Focus Group on concepts, high-level architecture, standardization
Autonomous Networks (FG AN) [11] in December gap analysis, specification languages, and repre-
2020 to provide an open platform for conducting sentations of autonomous networks.

FIGURe 2. High-level architecture of AI/ML-based network automation for resource and fault management.

IEEE Communications Standards Magazine • September 2021 109

KAFLE_LAYOUT.indd 109 10/7/21 1:40 AM

 analytical feedback may be concerned with many
of these functional domains. TR 23.791 pro-
vides guidelines for the application of NWDAF
to various use cases belonging to one or several
domains. In the description of each use case, the
following six aspects are included: domain (e.g.,
performance, QoS, resilience, and security) and
timescale, nature of input data (e.g., systems logs,
KPI, and events), nature of the output data, types
of NFs consuming NWDAF analytics output data,
examples of actions taken, and benefits (e.g., rev-
enue, resource saving, quality of experience assur-
ance, and reputation enhancement).

ETSI EXperiential
Networked Intelligence
ETSI’s Experiential Networked Intelligence Indus-
try Specification Group (ENI ISG) is targeting
the development of efficient and extensible
standards-based mechanisms to provide con-
text-aware services. It has specified an experiential
architecture that uses AI/ML and other mecha-
nisms to improve its understanding of the network
environment, and thus the operator experience,
over time. It can adapt its functionality based on
contextual changes in user requirements, network
conditions, and business goals [14].
FIGURe 3. General framework of 3GPP 5G network automation using Figure 4 shows the ETSI ENI system architecture,
network data analytics function. which consists of three modules: the input process-
ing module, analysis module, and output generation
AI/ML Network module. The figure also shows the API broker; how-
ever, the ENI system can function with or without
Standardization in Other SDOs the API broker. The API broker serves as a gateway
This section presents a review of network-automa- between different systems. It possesses the transla-
tion-related activities of 3GPP and ETSI. tion mechanisms to translate data communicated
from the external system into a normalized form that
3GPP Network Automation for 5G all ENI functional blocks can understand, as well as
3GPP has defined the network data analytics func- translate recommendations and commands from
tion (NWDAF) in the 5G service-based architecture the normalized form of the ENI system to a form
(SBA) as the enabler of intelligence and autono- that the external system can understand. Thus, it
mous network operation and management. The enables heterogeneous types of external systems
NWDAF collects data from various modules of the such as infrastructure, applications, and users to
5G system and provides analysis results. interoperate with the ENI system.
3GPP TR 23.791 specifies a general frame- The input processing and normalization mod-
work for 5G network automation [12]. As shown ule possesses data ingestion and normalization
in Fig. 3, the NWDAF collects data from the oper- functional blocks to process the input data such
ation, administration, and maintenance (OAM) that the other functional blocks in an ENI system
module, application functions (AFs), 5G core can interpret and understand the data in a unified
network functions (NFs), and data repositories. and consistent manner.
In addition, 3GPP TS 23.288 specifies the refer- The analysis module includes context-aware,
ence architecture and detailed procedures for knowledge management, cognitive processing,
data analytics [13]. It also provides a description situation-aware, model-driven, and policy manage-
of NWDAF discovery and selection by NFs/AFs, ment functional blocks. The context-aware func-
network performance analytics, and user data tional block describes the state and environment
congestion analytics. in which a set of entities in the controlled or assist-
NWDAF analyzes the data by leveraging AI/ ed system (i.e., the system being assisted and/or
ML models. The analytic results are then delivered controlled by the ENI system) exist. The context
to the NFs/AFs that have requested the NWDAF. consists of measured and inferred knowledge that
The NFs/AFs utilize analytical output data to make may change over time. The knowledge manage-
appropriate decisions for network operation and ment functional block includes the mechanisms for
management actions. NWDAF utilizes existing 5G knowledge representation, inference, and reason-
service-based interfaces to collect data from NFs/ ing to represent information about both the ENI
AFs and OAM, as well as deliver analytical output system and the controlled external system.
data to them. The cognition processing functional block
Because the input data of the NWDAF may includes a mechanism to understand normalized
come from multiple sources, such as mobility ingested data and information, as well as the con-
management, session management, QoS man- text that defines how those data were produced.
agement, application layer, security management, Based on data interpretation results, it determines
and NF life cycle management, the resulting whether any action needs to be taken to ensure
actions that an NF or AF takes according to the that the goals and objectives (e.g., improving or

110 IEEE Communications Standards Magazine • September 2021

KAFLE_LAYOUT.indd 110 10/7/21 1:40 AM

 The ENI system architec-
ture was designed based
on the key assumption
that the ENI system func-
tionality evolves over time
to meet emerging func-
tional requirements such
as network and service
planning requirements,
service provisioning,
deployment, optimization,
data collection, modeling,
analysis, policy specifica-
tions, interoperability with
other systems, non-func-
tional requirements of
system performance , and
scalability, among other
factors.

FIGURe 4. High-level ENI system architecture with API broker.

optimizing the performance, reliability, and/or The ENI architecture can be applied to var-
availability) of the ENI system have been met. ious aspects of network management, such as
The situation awareness functional block includes infrastructure management, network operation,
a mechanism to enable the ENI system to be service orchestration, and the management of
aware of events and behaviors that are relevant numerous use cases.
to the controlled system. It has the capacity to
understand how information, events, and recom- ETSI Zero Touch Network and
mendation commands given by the ENI system Service Management
impact the management and operational goals ETSI ISG Zero Touch Network and Service Man-
and behavior in the short and long terms. agement (ZSM) specifies the architecture, function-
The model-driven engineering functional block al, and operational requirements for E2E networks,
contains a set of models that collectively abstract and service automation based on the closed-loop
all important concepts for managing the control control and integration of AI/ML techniques. The
system governed by the ENI system. The policy ZSM architecture aims to address the challenges of
management functional block provides a set of technological and managerial heterogeneity in E2E
rules to manage the system in such a way that the cross-domain network management by defining
system goals and objectives are met. a holistic management framework that can reuse
Similarly, the denormalization functional block the management capabilities available in various
of the denormalization and output generation standard technologies. It follows the principles of
module includes mechanisms to process and modularity, extensibility, scalability, model-driven
translate policies, recommendations, and data open interfaces, closed-loop management automa-
received from other functional blocks of the ENI tion, support of stateless management functions,
system into an intermediate form that can be sub- resilience, intent-driven interfaces, and simplicity,
sequently translated or transcoded by the output among others [15].
generation functional block into a form that the Figure 5 shows the ETSI ZSM reference archi-
controlled systems are able to understand and tecture, which is composed of a two-layer hier-
use. If an API broker exists, the output is sent to archical structure of management domains. In
the API broker; otherwise, it is sent directly to the the lower layer, there are multiple management
controlled system. domains (MDs), each of which is responsible for
The ENI system architecture was designed managing a domain-managed infrastructure; in
based on the key assumption that the ENI system the upper layer, this is an end-to-end (E2E) ser-
functionality evolves over time to meet emerging vice management domain, which orchestrates the
functional requirements such as network and ser- management services provided by individual man-
vice planning requirements, service provisioning, agement domains to realize E2E cross-domain
deployment, optimization, data collection, model- management. Both the individual and E2E man-
ing, analysis, policy specifications, interoperability agement domains expose a set of management
with other systems, non-functional requirements services that they provide.
of system performance (latency, accuracy, and Management domain services can be cate-
efficiency), and scalability, among other factors. gorized into the following groups. Domain data

IEEE Communications Standards Magazine • September 2021 111

KAFLE_LAYOUT.indd 111 10/7/21 1:40 AM

 Data services provide the means of data shar-
ing with authorized consumers within and across
management domains while taking care of infor-
mation security and data privacy regulations.
Data services may store and provide performance
monitoring data, trace data, configuration data,
topology data, and inventory data, among other
data types.
An integrated fabric exists inside each manage-
ment domain. There is also a cross-domain inte-
gration fabric between each management domain
and the E2E service management domain. The
integration fabric provides a set of ZSM services
to facilitate the interoperation of management
functions with regard to the offered and con-
sumed management services. Services offered
by the integration fabric include the registration/
de-registration of management services, discovery
of registered management services and the means
to access them, the means to support the invoca-
tion of management services, and the means to
support synchronous and asynchronous commu-
nication. Similarly, the cross-domain integration
fabric offers the services of registration, discovery,
communication, invocation, and capability expo-
sure configuration of management services.

KeY Architecture Features

As common features of intelligent and autono-
mous network management framework architec-
tures being developed in ITU-T, 3GPP, ETSI ENI,
and ZSM, all are based on data collection (from
various network functions, application functions,
and other repositories) and data processing and
analysis using AI/ML models. The key features of
these architectures are listed in the second col-
FIGURe 5. ETSI ZSM framework reference architecture. umn of Table 2.
The last column mentions the target networks
collection services monitor the domain-managed that fall in the scope of each of these architec-
infrastructure and collect data as demanded by tures. Both ITU-T and 3GPP have considered 5G
other management functions. Domain analytics networks; however, the ITU-T ML architecture
services apply various data analytics to the collect- is focused mainly on fixed wireline networks,
ed data to generate domain-specific knowledge whereas 3GPP SBA 5G architecture also includes
and predict domain-specific behavior. Domain radio access networks in the scope. ETSI does
intelligence services execute closed-loop automa- not assume any specific network type but con-
tion functions of decision support, decision mak- siders a generalized network architecture. In con-
ing, and action planning to enable autonomous trast to the ITU-T, 3GPP, and ETSI ENI framework
management of the domain-managed infrastruc- architectures, which mainly focus on the manage-
ture. Domain orchestration services automate ment of a single administration domain, the ETSI
workflows and processes inside a management ZSM architecture specifies a holistic management
domain to handle the life cycle management of framework to enable the interworking of various
the domain-managed infrastructure. Domain con- domain-specific network management standards
trol services facilitate the control of each manage- to support E2E service management across differ-
ment entity. ent technology and administrative domains.
Similarly, the E2E service management domain
contains the following groups of management Network Automation
services: E2E service orchestration services are
responsible for the coordination of provisioning, Research Issues
configuration, and life cycle management of var- Although several framework architectures have
ious services across management domains. E2E been developed in various SDOs, the detailed
service intelligence services are responsible for specifications of the related protocols, mechanisms,
driving closed-loop automation in E2E networks and technologies remain to be developed. This
and service management. E2E service analytics section discusses a set of related open research
services are responsible for deriving E2E service and standardization issues for intelligent and auton-
insight for managing E2E network service perfor- omous networks and service management.
mance. E2E service data collection services are Heterogeneity in Multiple Domains: The first
responsible for collecting E2E service-related data. issue is the challenges in the E2E management
As in the individual management domains, each of heterogeneous technology and administration
E2E management service is offered through one domains. As stated earlier, the E2E management
or more interfaces. of network services in cloud-native future net-

112 IEEE Communications Standards Magazine • September 2021

KAFLE_LAYOUT.indd 112 10/7/21 1:40 AM

 Architectures Key features Target networks Hierarchical management
• Pipelining of ML models layers have been proposed
• Defining network intelligence levels 5G and beyond, mainly fixed wireline for E2E network and
ITU-T ML architecture
• Integrating ML model marketplace network aspects
• Integrating ML sandbox for model training service management (e.g.,
• NWDAF as key component
the ZSM architecture
3GPP 5G SBA • Using 5G service-based interfaces to access data from OAM module, 5G architecture, including radio contains open interfaces,
architecture AF, NF, and repositories, and deliver analysis results access network aspects model-driven services,
• Applicability of NWDAF in various use cases
and resource abstrac-
• Generalized adaptive architecture capable of learning from the
context and evolving the system functionality over time
tion). However, they lack
• Focused on data processing and analysis, together with knowledge Not specific to 5G, assuming a detailed specifications of
ETSI ENI architecture
management generalized architecture interfaces that can be fol-
• Applicable to various aspects of network management, e.g.,
infrastructure, operation, service orchestration lowed to implement mech-
• Holistic management framework for reusing management capabilities
anisms for the exchange of
Not specific to 5G, focused on
available in various standard technologies management capabilities
ETSI ZSM architecture interworking of multiple technological
• E2E service management through bridging individual network and related data among
and administrative domains
management domains
multiple management
Table 2. Key architecture features.
domains.

works formed by the convergence of network- tion in each management domain is required to
ing and cloud/edge computing infrastructures is expose only the relevant information in the stan-
challenging due to the involvement of different dard form and size.
technologies and administrative domains, such Scalability of Telemetry Data Collection: For
as the domains of mobile operators, edge-com- intelligent and autonomous E2E network and
puting service providers, core network operators, service management, the existence of standard
and cloud-computing service providers. Each of technology for agile monitoring and control of all
these domains may employ different types of net- involved network functions is essential. The moni-
work resource management mechanisms. In a toring and control functions include a process for
multi-vendor E2E communication service deliv- network telemetry data collection, which needs
ery environment, each vendor network may have to be efficient to avoid incurring a high overhead
its own operation and business support system hampering network performance and delaying
(OSS/BSS) with different control interfaces, the execution of control commands. The intelli-
resource virtualization and SDN technologies, gent and autonomous telemetry data collection
AI/ML techniques, and models. To address this may require attaching an AI/ML process to each
issue, the 5G service-based architecture of 3GPP network function to carry out optimal decisions
applies SBI between the NWDAF and other net- regarding the right amount and time of telemetry
work functions; in addition, ZSM has an E2E ser- data collection to ensure the efficiency and scal-
vice management domain, ENI has an API broker, ability of the overall system.
and the ML network architecture of ITU-T has an Data Models: Standard data models are essen-
MLFO. However, they lack detailed specifications tial for sharing cross-domain control data for the
for the design and implementation of an easy-to- realization of autonomous E2E networks and ser-
deploy multi-domain management mechanism vice management. Cross-domain data services in
that autonomously re-composes and reconfigures the ZSM reference architecture and data handling
itself to provide near-optimal performance for framework in ITU-T Y.3174 have been defined
different types of network services. For example, and specified at the conceptual level. The ZSM
the standard representations of ML models, man- framework architecture also provides definitions
aged resources, and management methods to be and requirements for data collection, data stor-
used in each management domain have yet to be age, data persistence, and data processing ser-
defined. Only after having standardized represen- vices. However, they lack detailed specifications
tations can these entities be orchestrated from the of data models that can be followed to implement
overlay E2E service management domain. scalable mechanisms for efficiently exchanging rel-
Abstraction of Hierarchical Management evant data across multiple management domains.
Layers: Hierarchical management layers have AI/ML Pipelining: To realize scalable E2E net-
been proposed for E2E network and service work management, there must be a mechanism
management (e.g., the ZSM architecture con- for collaboration between the AI/ML models
tains open interfaces, model-driven services, used within the same local management domain
and resource abstraction). However, they lack as well as across different management domains.
detailed specifications of interfaces that can The ML pipelining concept has been mentioned
be followed to implement mechanisms for the in the ITU-T ML architecture [9]. Similarly, the
exchange of management capabilities and relat- ZSM framework’s hierarchical management struc-
ed data among multiple management domains. ture also assumes the existence of domain-specif-
Moreover, because the volume of management ic ML or data analytics models [15]. However, the
data produced in each management domain is detailed design of management architecture that
extremely large, an appropriate level of abstrac- can fully leverage AI/ML capabilities for cross-do-

IEEE Communications Standards Magazine • September 2021 113

KAFLE_LAYOUT.indd 113 10/7/21 1:40 AM

 main management is still missing, which deserves [9] ITU-T Rec. Y.3172, “Architectural Framework for Machine
Although framework further research and standardization. Learning in Future Networks Including IMT-2020,” June
2019.
architectures are being [10] ITU-T Rec. Y.3177, “Architectural Framework of Artificial
developed in various SDOs,
Conclusion Intelligence-Based Network Automation for Resource and
This article surveys the status of the standardization Fault Management in Future Networks Including IMT-2020,”
detailed specifications of AI/ML-based network and service management Feb. 2021.
[11] ITU-T Focus Group on Autonomous Networks, https://
of related protocols, architectures in ITU-T, 3GPP, and two ISGs of ETSI: www.itu.int/en/ITU-T/focusgroups/an/Pages/ default.aspx),
ENI and ZSM. The ITU-T architectures are based accessed 15 Apr. 2021.
mechanisms, and tech- on AI/ML model pipelining for the autonomous [12] 3GPP TR 23.791, “Study of Enablers for Network Automa-
nologies are still missing. tion for 5G,” v. 16.2.0, 2019.
operations of control information handling, network [13] 3GPP TS 23.288, “Architecture Enhancements for 5G Sys-
Therefore, future research intelligence evaluation, network service provision- tem (5GS) to Support Network Data Analytics Services,” v.
ing, resource and fault management, and quality 16.6.0, Release 16, 2021.
and standardization of service assurance. Similarly, the 3GPP 5G ser- [14] ETSI GS ENI 005, “Experiential Networked Intelligence
(ENI) System Architecture, v. 1.1.1, 2019.
are required to develop vice-based architecture has the NWDAF function [15] ETSI GS ZSM 002, “Zero Touch Network and Service Man-
that uses 5G service-based interfaces to access data agement (ZSM) Reference Architecture,” v. 1.1.1, 2019.
detailed specifications from OAM modules, application functions, network
that can be followed to functions, and repositories, and delivers analysis Biographies
implement the framework results to them. The ETSI ENI architecture is capable Ved P. Kafle [S’04, M’07, SM’14] received his B.E. degree in
of learning from context and evolving in terms of electronics and communications engineering from Punjab Engi-
architectures. system functionality over time. The ETSI ZSM archi- neering College, Chandigarh, India, his M.S. degree in computer
science and engineering from Seoul National University, South
tecture presents a holistic E2E service management Korea, and his Ph.D. degree in informatics from the Graduate
framework for reusing management capabilities University for Advanced Studies, Japan. He joined the Nation-
available in various standard technologies used in al Institute of Information and Communications Technology
individual network management domains. (NICT), Tokyo, Japan, as a researcher in 2006. He is currently
working as a research manager at NICT, concurrently holding
Although framework architectures are being a visiting associate professor position at the University of Elec-
developed in various SDOs, detailed specifications tro-Communications, Tokyo, Japan. His recent research interests
of related protocols, mechanisms, and technolo- include 5G and beyond network architectures, IoT, network
gies are still missing. Therefore, future research and resource management, network operation automation by AI/
ML, and standardization. He has been serving as a Co-Rap-
standardization are required to develop detailed porteur of ITU-T Study Group 13 since 2014. He received the
specifications that can be followed to implement ITU Association of Japan’s Encouragement Award and Accom-
the framework architectures. For future study, the plishment Award in 2009 and 2017, respectively. He has also
issues of interworking of heterogeneous multiple received four Best Paper Awards from the ITU Kaleidoscope
Academic Conferences in 2009, 2014, 2018, and 2020. He is a
domains, abstraction of management layers, scal- Fellow of ITU-T Study Group 13 and a member of IEICE.
able telemetry, data models, and AI/ML pipelining
are also discussed. To realize the concept of AI/ML Takahiro Hirayama [M’18] received his M.S. and Ph.D. degrees
pipelining, standardization activities are required in information science from Osaka University, Japan, in 2010
and 2013, respectively. He joined NICT as a researcher in 2013,
to specify the design, implementation, and opera- where he is currently working as a senior researcher. His research
tional cycles that allow AI/ML models from differ- interests are in optical networks, software defined networking,
ent domains to be combined in the same way that network function virtualization, and machine learning for network
software libraries and modules are combined in management. He received the Best Paper Award from the IEICE
Communications Society in 2019. He is a member of IEICE.
general-purpose computing platforms.
Takaya Miyazawa [S’03, M’06] received his M.E. and Ph.D.
Acknowledgment degrees in information and computer science from Keio University,
This study was conducted partially as a part of the Yokohama, Japan, in 2004 and 2006, respectively. From April 2006
to March 2007, he was a visiting researcher at the University of
project “Research and development for innova- California, Davis. He joined NICT as a researcher in 2007, where
tive AI network integrated infrastructure technol- he is currently a research manager. From April 2019 to July 2020,
ogies (JPMI00316),” supported by the Ministry of he worked at the Ministry of Internal Affairs and Communications,
Internal Affairs and Communications, Japan. Japan, as a deputy director, and returned to NICT in August 2020.
His research interests include the automation of network control,
management, and virtualization technologies. He served as Secre-
References tary of the IEEE Communications Society Asia-Pacific Board from
[1] ITU-T Rec. Y.3104, “Architecture of the IMT-2020 Network,” 2016 to 2017. He received the Hiroshi Ando Memorial Young
Dec. 2018. Engineer Award in 2007, the Funai Young Researcher Award in
[2] S. Sharma, R. Miller, and A. Francini, “A Cloud-Native 2010, and the Best Paper Award from the ITU Kaleidoscope Aca-
Approach to 5G Network Slicing,” IEEE Commun. Mag., vol. demic Conference in 2018. He is a member of IEICE.
55, no. 8, Aug. 2017, pp. 120127.
[3] V.P. Kafle et al., “Adaptive Virtual Network Slices for Diverse Masahiro Jibiki [M’21] received his Ph.D. degree in systems man-
IoT Services,” IEEE Commun. Stds. Mag., vol. 2, no. 4, Dec. agement from the University of Tsukuba, Japan, in 2003. He
2018, pp. 33–41. joined NEC Corporation in 1992 and worked as a researcher in
[4] ITU-R Rec. M.2083-0, “IMT Vision “Framework and Overall the Central Research Laboratories until 2011. From 2006 to 2009,
Objectives of the Future Development of IMT for 2020 and he was also a visiting professor at the University of Wakayama,
Beyond,” Sept. 2015. Japan. Currently, he is a senior researcher at NICT. His research
[5] V. P. Kafle, P. Martinez-Julia, and T. Miyazawa, “Automation interests include networking, software science, and mathematical
of 5G Network Slice Control Functions With Machine Learn- models. He received two Best Paper Awards from the IEICE Com-
ing,” IEEE Commun. Stds. Mag., vol. 3, no. 3, Sept. 2019, munications Society in 2010 and 2019. He is a member of IEICE.
pp. 54–62.
[6] D. Rafique and L. Velasco, “Machine Learning for Network H iroaki H arai [M’97] received his M.E. and Ph.D. degrees
Automation: Overview, Architecture and Applications,” in information and computer science from Osaka University,
IEEE/OSA J. Opt. Commun. Net., vol. 10, no. 10, Oct. 2018, Japan, in 1995 and 1998, respectively. He is currently a director
[7] V. P. Kafle and A.H.A. Muktadir, “Automation of Compu- general of the Network Research Institute at NICT. His research
tational Resource Control of Cyber-Physical Systems With interests include new network architectures and optical net-
Machine Learning,” Proc. ITU Kaleidoscope Academic Conf., works. He received the Outstanding Young Researcher Award
Dec. 2020 from the IEEE Communications Society Asia-Pacific Region in
[8] ITU-T Focus Group on Machine Learning for Future Net- 2007 and the Young Researcher Award from the Ministry of
works Including 5G; https://www.itu.int/en/ITU-T/focus- Education, Culture, Sports, Science and Technology, Japan, in
groups/ ml5g/Pages/default.aspx, accessed 15 Apr. 2021. 2009. He is a member of IEICE.

114 IEEE Communications Standards Magazine • September 2021

KAFLE_LAYOUT.indd 114 10/7/21 1:40 AM