Cluster Computing (2023) 26:1845–1875

https://doi.org/10.1007/s10586-022-03713-0 (0123456789().,-volV)(0123456789().
,- volV)

Energy efficiency in cloud computing data centers: a survey

on software technologies
Avita Katal1,2 • Susheela Dahiya2 • Tanupriya Choudhury2

Received: 8 February 2022 / Revised: 21 June 2022 / Accepted: 7 August 2022 / Published online: 30 August 2022
 The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2022

Abstract
Cloud computing is a commercial and economic paradigm that has gained traction since 2006 and is presently the most
significant technology in IT sector. From the notion of cloud computing to its energy efficiency, cloud has been the subject
of much discussion. The energy consumption of data centres alone will rise from 200 TWh in 2016 to 2967 TWh in 2030.
The data centres require a lot of power to provide services, which increases CO2 emissions. In this survey paper, software-
based technologies that can be used for building green data centers and include power management at individual software
level has been discussed. The paper discusses the energy efficiency in containers and problem-solving approaches used for
reducing power consumption in data centers. Further, the paper also gives details about the impact of data centers on
environment that includes the e-waste and the various standards opted by different countries for giving rating to the data
centers. This article goes beyond just demonstrating new green cloud computing possibilities. Instead, it focuses the
attention and resources of academia and society on a critical issue: long-term technological advancement. The article
covers the new technologies that can be applied at the individual software level that includes techniques applied at
virtualization level, operating system level and application level. It clearly defines different measures at each level to
reduce the energy consumption that clearly adds value to the current environmental problem of pollution reduction. This
article also addresses the difficulties, concerns, and needs that cloud data centres and cloud organisations must grasp, as
well as some of the factors and case studies that influence green cloud usage.

Keywords Cloud Computing  Containerization  Data center  Load balancing  Workload categorization

1 Introduction interface. The web is the principal interface (medium) via

which cloud computing distributes or makes its services
The last decade internet services like cloud computing and available to everyone. Since time immemorial, the defini-
web 2.0 have changed the entire architecture of the internet tion of the term web has evolved. Now web encompasses a
ecosystem. The web, which began as a worldwide hyper- slew of technologies and services that enable interactive
text system, has developed into a distributed application sharing, collaboration, user-centred design, and application
platform with distinct entities for application logic and user development. As a result, web 2.0 refers to the current state
of internet technology in relation to the early days of the
web, and it includes increased user involvement and
& Avita Katal
[email protected] cooperation, as well as improved communication channels.
In recent years, a new computer paradigm known as cloud
& Tanupriya Choudhury
[email protected] computing has begun to emerge. As we transitioned from
web content to web of apps on the next generation web
Susheela Dahiya
[email protected] platform, web 2.0, the network cloud housed the vast bulk
of user data and applications. Cloud computing is also
1
Research Scholar, School of Computer Science, University of gaining prominence as a low-cost way of software storage
Petroleum and Energy Studies, Dehradun, India and distribution. Data, software and applications no longer
2
School of Computer Science, University of Petroleum and exist on the client side in this environment; instead, they
Energy Studies, Dehradun, India

123
1846 Cluster Computing (2023) 26:1845–1875

are viewed as abstract services and live in cloud. A cloud enables businesses to reduce downtime caused by high
may be defined as a network of server-side nodes. Many demand or other problems. Hyper scaling also allows IT
underlying technologies like utility computing, Service- systems to be restored considerably more quickly. The cost
oriented architecture (SOA) had a direct impact on cloud of cooling and maintaining the facility’s temperature is one
computing. SOA made it possible for several services to of the most significant operating expenditures a data centre
connect with one other via a loose coupling technique in confronts. A hyperscale data center optimizes airflow
order to exchange data or organize an activity. In contrast, throughout the structure. The combination of vertical and
utility computing is a service delivery model in which a horizontal scaling increases the utilization of energy by
service provider makes computational resources, network hyperscale data centres dramatically. Although these
administration accessible to clients on as need basis, and facilities are generally highly energy efficient, their sheer
pays them for the usage patterns rather than a set fee. The scale exerts huge power demands on the world’s energy
utility model, like other types of on demand computing supplies. From 2015 to 2023, hyperscale data center’s
(such as grid computing), seeks to optimize operational energy usage is anticipated to nearly quadruple, making it
efficiencies while minimising associated costs. the world’s highest proportion of data centre energy con-
The word ‘‘data centre’’ is, by definition, an assumption. sumption [1].
It harkens back to a time when a company’s back-office The architecture of the current services of cloud is
computer systems were mainly devoted to data storage and highly centralized meaning that the different types of ser-
were cobbled together in a basement or closet. Nobody was vices can be run through a single site called data centers.
meant to see or notice ‘‘infrastructure,‘‘ such as a sewage The data centers are increasing rapidly due to rapid
system or the foundation of a roadway beneath the pot- advancement in cloud computing. The energy consumption
holes. All of these assumptions have now been debunked. of data centres alone will rise from 200 TWh in 2016 to
The IT infrastructure of a business includes computer 2967 TWh in 2030 [2]. Despite the COVID-19 problem,
power, networking, and data storage. In addition, like the the worldwide market for Internet Data Centers, which was
enterprise, it has a natural propensity to become dispersed. predicted at US$59.3 billion in 2020, is expected to reach
As with many novel notions in the IT industry, there is no US$143.4 billion by 2027, increasing at a CAGR of 13.4%
universally accepted definition of what constitutes a between 2020 and 2027 [3]. In the United States, the
hyperscale data centre. Hyperscale data centres are much Internet Data Centers business is estimated to be valued
larger than corporate data centres, and they outperform US$16 billion by 2020. China, the world ‘s second biggest
them greatly as well, owing to the benefits of economies of industry, is expected to have a data centre industry of
scale and custom engineering. The International Data US$32 billion by 2027, with a 17.5% CAGR between 2020
Corporation (IDC), that offers new tech sector design and and 2027 [3].
consultancy assistance, defines hyperscale as any data- Power Usage Effectiveness (PUE), a unit of analysis for
centre with at least 5000 servers and 10,000 square feet of data centre power consumption efficiency, is a real-time
open field. Nevertheless, Synergy Research Group focuses and annual study of total facility power split by IT hard-
on ‘‘scale-of-business criteria,‘‘ that also assess a com- ware power, or a measurement of power ‘loss’ flowing to
pany’s cloud, e-commerce, and social networking proce- non-IT devices. The ideal PUE is 1.0, which corresponds to
dures instead of physical features [1]. Traditionally, data 100% effectiveness [4]. However, it is nearly difficult to
centres have used one of two techniques to provide this achieve. The average yearly data centre PUE in 2021 was
additional computing capacity. Horizontal scaling is inef- 1.57, a small increase with the average of 1.59 in 2020, and
ficient in terms of energy consumption, particularly for keeping with the overall trend of PUE stagnation over the
complicated workloads. It also introduces a new issue in previous five years. The bulk of the energy consumed is
that each storage unit added necessitates the inclusion of used to power the servers; however, they generate heat and
the appropriate compute and network resources required to must be cooled [5]. The need for data centres is increasing
use them. Data centers require proper cooling systems to due to the exponential rise in data gathering and con-
work properly which leads to increase in expenses. sumption. In general, cloud computing uses a large number
Hyperscale computing aids in lowering the cost of data of data centers and servers to service a big number of
disruptions. Systems that fail due to a lack of hyperscale clients using a pay-per-use model. Such resources cover a
computing lose money, goodwill, and the services of their big area and demand a considerable amount of electricity
IT employees who must find out why the services failed for networking devices, cooling technologies, displays, and
among other business operations losses. Before the system server farms, among other things. Making the resources
can be used again, they may need to fix compliance con- green using green technology has thus become a main goal
cerns and alert consumers. Companies might lose data for of several government and industry organizations. Green
hundreds of thousands or millions of dollars. Hype scaling IT, from an environmental standpoint, and to deal with IT-

123
Cluster Computing (2023) 26:1845–1875 1847

related environmental challenges, offers a broad number of as the policies/standards available for assessing energy
approaches and practices through several green initia- efficiency in data centers. This survey report serves as a
tives.Using energy more effectively is one of the most foundation for academics working in the field of green
straightforward and cost-effective method to save money, computing, as it covers layer-by-layer software modelling
decrease greenhouse gas pollutants, generate employment, of data centres, as well as an emphasis on the many
and satisfy rising power demands. Improved efficiency has research issues to which researchers should target their
the potential to reduce greenhouse gas (GHG) emissions, efforts.
other contaminants, and water usage. Energy efficiency can Thus, four research queries have been answered:
bring long-term advantages by lowering total power con-
RQ1 What are the numerous options used at the software
sumption, minimising the need for new energy generation
level like operating system, virtualization and application
and distribution infrastructure investment.
to reduce the usage of power by data centers?
Cloud providers that host a range of applications must
follow service-level agreements (SLAs), achieve low RQ2 What are the various strategies utilised in data cen-
access latencies, meet task deadlines, and provide secure, tres, both virtualized and non-virtualized systems, to min-
dependable, and effective data management. Low-cost imise power usage?
hardware designs and capacity planning tactics employed
RQ3 What are the major impacts of a data center on the
in back-end data centres for energy savings might often
environment?
conflict with the commercial objectives of cloud providers.
Due to the online assessment of dynamic elements such as RQ4 What are the major software academic difficulties for
workload allocation, resource allocation, cooling plan, inter developing green data centres?
process communication, and traffic conditions, data centre
energy management is a challenging operation. Moreover,
as energy prices rise and the cloud service pricing market
2 Related work
becomes more competitive, cloud providers are being
Despite the fact that there has been a substantial quantity of
obliged to investigate energy-saving alternatives for back-
research on data centre energy usage estimation and fore-
end data centres [6]. Typical workload on the data center is
casting, there have been comparatively few studies in this
usually about 30% and does not necessitate the use of all
sector. The following papers describe software-based
computer resources [7]. As a result, certain unused equip-
technologies for developing energy-efficient green data
ment can be turned off to achieve energy savings while
centres.
meeting data center workload expectations. However,
The authors of [11] presented an analysis on cloud
scheduling data centre resources necessitates careful con-
computing energy usage. The research considered both
sideration of data center traffic patterns [8], client SLAs
public and private clouds, as well as the energy consumed
[9], latency and performance concerns [7], and data repli-
in switching and communication, information computation,
cation [10].
and storage. They demonstrated that power usage in transit
Software level modelling is very important for energy
and switching may account for a sizable portion of total
efficiency in data centers because the different software
energy demand in cloud computing. Their proposed
components require power to process the tasks. The soft-
method regards Cloud Computing (CC) as an equivalent of
ware developed should be able to benefit from advance-
a classic logistics and supply chain issue that takes into
ments achieved at the hardware component level. If the
account the power usage or expense of computing, keep-
software generated is not as efficient as hardware technical
ing, and transporting physical goods. The authors in [12]
advances and consumes a large number of resources,
highlighted the reasons and difficulties associated with
overall energy consumption will remain high, negating the
excessive power / energy usage, as well as presented a
entire goal of building green data centres. The survey
taxonomy of energy-efficient computing system architec-
papers published until date do not include the complete
ture at the OS, hardware, virtualization, and data centre
details of energy efficiency techniques employed at each
levels. They evaluated important contributions in the area
software layer in the data center. They do not include
and linked them to their classification to guide future
different mechanisms employed for modelling the con-
development and research initiatives. They investigated
tainers energy consumption that is one of the emerging
and categorised numerous ways to controlling a system’s
areas in cloud computing domain. This paper explains the
power usage from the OS level using DVFS and other
various techniques employed for energy efficiency in
power-saving strategies and algorithms. Many research
container technology that is first a kind of effort in this
efforts targeted at developing efficient algorithms for reg-
direction as per author’s knowledge. This paper also pro-
ulating CPU power usage have culminated in the
vides information about the environmental effect, as well

123
1848 Cluster Computing (2023) 26:1845–1875

widespread acceptance of DVFS in the form of an imple- establishing green data centres [16]. This paper provides
mentation in a kernel module of the Linux operating sys- the detailed information about the different techniques that
tem. In addition, the authors in [13] highlighted research can be applied at the individual software levels and in-
difficulties connected to the competing needs of enhancing depth information about the power modelling at operating
the quality of services (QoSs) supplied by cloud services system virtualization and data center level along with the
while lowering energy consumption of data centre resour- work done in different problem-solving approaches like
ces. They addressed the idea of creating an energy-efficient VM migration, workload categorization, load balancing
data centre controller suitable of combining data centre and VM placement. The article also addresses the envi-
capabilities while reducing the effect on QoS objectives. ronmental impact of data centers and ends with a discus-
They investigated strategies for controlling and coordinat- sion of the recent research challenges in the construction of
ing data centre resources in order to achieve energy-effi- green data centres.
cient operations. They also offered a central controller The articles in this study were obtained from several
concept and proposed resource controller cooperation. sources, including IEEE, Springer, and Elsevier. Web of
Energy-saving hardware ideas for data centre resources Science and Scopus are the databases used to collect
were also thoroughly examined. The authors in [14] dis- publications. All of the publications included have been
cussed the different mechanism and architectures for the peer reviewed, and the bulk of them were published and
design of energy efficient data centers. They investigated 2015 and 2020. This research includes publications that
the different power models for virtual machines, operating focus on software-based methods for energy efficiency in
systems and software applications. Their systematic tech- data centres. This analysis excludes publications that were
nique enables them to investigate a variety of challenges not peer reviewed and were published before to 2015.
typical in power simulation at different stages of data Studies that are not published in English and do not provide
centre systems, such as: (i) few modelling efforts devoted details about software innovations for energy savings in
at overall data centre power consumption (ii) many cutting- data centres are not evaluated for inclusion.
edge power models rely on a few CPU or server specs; (iii)
the efficacy and accuracy of these power models is still
unknown. They completed the study by identifying 3 Motivation
important obstacles for future studies on building efficient
and optimum data centre power models based on their Data centres are critical, energy-intensive infrastructure
findings. The authors in [15] conducted research and cre- that provide large-scale Internet-based services. Power
ated a taxonomy based on pre-existing energy efficiency utilization models are essential for creating and enhancing
related surveys, i.e., research on energy saving surveys. energy-efficient processes in data centres in order to
Existing surveys were classified into five categories: those decrease excessive energy use. In recent years, the neces-
on the power consumption of all cloud-related processes, sity of energy efficiency in data centres has increased
that on a particular level or component of the cloud, those substantially and has become more complicated. To guar-
on all energy-efficient methodologies, that on a specific antee high availability of data, all elements of the data
energy-efficiency technique, and those on other energy- centre design must perform their given tasks to minimise
efficiency-related studies. A taxonomy and survey on sur- data centre downtime that requires appropriate energy
veys are conducted from the viewpoints of foci, views, support. Power supply, technical cooling, and technical
target system, and years. The survey findings on energy security are all part of the technical infrastructure, which is
consumption savings measures are then examined, laying the foundation of all information technology (IT) infras-
the groundwork for their future work in the subject of tructures. Any physical infrastructure outage, no matter
energy consumption. how slight, has a major impact on the functioning of IT
The survey articles described above are either incom- services. The essential qualities of a green data centre are
plete or having limitations. They have not gone into length energy efficiency and low global impact. A green or sus-
on the issues of power usage at the application, virtual- tainable data centre is a data storage, management, and
ization, and operating system layers of software. Further- dissemination facility in which all systems, especially
more, these survey studies did not give comprehensive mechanical and electrical frameworks, improve energy
information on the solutions that may be deployed at the efficiency. It produces less carbon footprints, saves money
data centre level and containers (operating system virtu- and increases efficiency. These eco-friendly data centres
alization). These survey reports also did not get into spe- help modern enterprises save power and reduce carbon
cifics concerning environmental variables or case studies. emissions. Globally, their use is rising among both major
This article is an extension of the authors’ earlier work, organisations and small and medium-sized businesses
which provides a study of hardware solutions for (SMBs). From data collection through processing,

123
Cluster Computing (2023) 26:1845–1875 1849

assessment, and distribution, such data centres can effi- 4 Energy usage in data centres: a system’s
ciently fulfil the objectives for a plethora of corporate data. perspective
The objective of this manuscript is to look at the current
research on green cloud computing and outline the major This segment analyses the whole data centre to the required
concerns that have been raised which consumes more levels based on electricity use. The data centre model uti-
power in data centers. lised in this study is depicted in Fig. 1 below. Every
The software components require a large amount of computer system is made up of two components: hardware
power to perform their operations in the data centers. To and software. A data centre also has two primary compo-
reduce the energy usage of data center in respect to the nents: software and hardware.
software layer, different techniques can be applied at These layers can be enhanced or optimized so that the
individual software level. The CPU core is the primary power usage by data centers can be minimized. The soft-
resource consumer in computation-intensive tasks (s) and ware layer is categorized into three sublevels: OS layer,
cloud system’s storage resources in data-intensive opera- virtualization layer, application layer. For establishing the
tions. Connected devices such as network cards, routers, green data centre, a taxonomical method for software
switches, and others require a substantial amount of energy approaches is offered, as illustrated in Fig. 2. To fulfil the
when performing communication-intensive operations. The aims of green cloud computing, many strategies at the
operating system (OS) resides between the physical hard- individual programme level might be used. Aside from
ware and the applications layers of the data centre archi- software approaches, external factors to the data centre
tecture. Most of the research is done on the hardware level such as government-imposed laws and policies, organisa-
for power consumption in data centers but the software tions, and renewable energy are also considered to fulfil the
level is equally important in order to reduce the power goals of green cloud computing.
usage. Software developed should be able to exploit the
advancements done at the hardware components level. If
software developed is not as efficient as the hardware 5 Data center power modelling at individual
technology advancements and consumes a large number of software level
resources then overall energy consumption still remains
high defying the whole purpose of developing green data RQ1 What are the various approaches used at the soft-
centers. Physical hardware is the component that consumes ware level like operating system, virtualization and appli-
the IT power, while applications produce the demand for cation to reduce the usage of power by data centers?
resources. Hence, looking into the details of power mod-
elling /energy consumption at the software layer becomes 5.1 Operating system level
equally important. Apart from the various techniques that
are applied at the OS, virtualization and the application The operating system is placed between the two layers: the
level, problem solving approaches like load balancing, application and the hardware. The main role of applications
workload categorization, VM placement and VM migration is to create the resource demand and the OS job is to
helps in minimizing the energy usage by consolidating the manage the resources for all these applications. The main
physical servers and dynamically modifying operations. component that consumes power is the physical hardware
These approaches prove to be effective in lowering energy but it is very essential to keep a check on the events that
usage in high performance cloud data centers. consume power at the operating system level if energy
In this article, the analysis is performed at many levels usage optimization at data centre is to be done at the
such as OS, virtualization, application and data centre to software levels too. The power usage breakdown of the
determine the energy usage by different software layers in operating system functions is shown in Fig. 3. Data-path
data centers. The case studies are also included for better and pipeline topologies that allow for numerous problems
understanding the importance of green cloud data centers. and out-of-order execution were found to squander 50% of
The research challenges are discussed along with their the total power of the OS processes investigated. Further-
solutions for reducing energy consumption in data centres. more, the clock consumes 34% power and different levels
of cache consumes the remaining power.
Operating System Power Management (OSPM) is a
mechanism utilized by OS to manage the power of the
underlying platform and transition of it between different
power modes. OSPM allows a platform or system to adopt
the most efficient power mode and is applicable to all

123
1850 Cluster Computing (2023) 26:1845–1875

Fig. 1 A comprehensive
picture of data centre energy
usage modelling

devices and components inside the platform/system. OSPM issues they encountered when trying to deploy the GRUB-
is also known as OS-directed configuration and Power PA algorithm on a real OS like Linux. Experiment findings
Management. on a multi-core ARM architecture demonstrated the effi-
The trade-off between quality and power efficiency has cacy of their suggested solution.
been intensively examined and analysed, since control over With the advancement of semiconductor and software
running voltage and energy management has been largely technologies, the capabilities of an embedded system have
shifted from the hardware and firmware level to the oper- grown by incorporating new features and performance. In
ating system. Herzog et al. [18] offers PolaR, a method for recent years, the network has also advanced as communi-
automatically determining energy-efficient setups, as well cation infrastructure and contact with server systems has
as a Linux implementation. PolaR proactively chooses become essential. So far, TCP / IP connections between
optimal settings by integrating application profiles and servers and embedded devices have been established by
system level data, and no application adjustments are two methods. The first is a technique that includes a TCP/
required. They take into account bank shots (configuration IP stack in embedded devices. The second is a technique of
settings unrelated to power management) in addition to communicating via a ‘‘gateway’’ (to translate end-device
controlling the system in the proper manner. OS develop- communications). There are several server system com-
ment teams recognised the value of energy as a resource on position options, such as putting a server in-house, estab-
par with time. With energy seen as just another resource lishing a server at a data centre outside of town, and
available to the operating system, operating system inter- utilising cloud computing. Smaller, more widespread and
nals (such as locking mechanisms) were changed to less well known ‘‘embedded data centres’’ consume half of
accommodate for this new perspective in order to produce all data centre energy, or about 1% of all energy generated
energy-aware operating systems. Scordino et al. [19] in the United States. In general, embedded data centres are
illustrates how the deadline scheduler and the cpufreq data centre facilities that have less than 50 kW of IT
subsystem may be changed to relax the restriction that the demand [20]. Server rooms, server closets, localised data
frequency scaling technique used only when no real-time centres, and several mid-tier data centres are among them.
processes are running and to create an energy-aware real- Energy harvesting technologies based on rechargeable
time scheduling approach. They described the architectural batteries are a popular option for addressing the issue of

123
Cluster Computing (2023) 26:1845–1875 1851

separation, memory protection, and efficient memory

governance for dynamic applications and services written
in Rust. Low-power embedded operating systems fre-
quently use the same memory areas for both applications
and the operating system. Merging applications and the
kernel allows them to easily exchange references and gives
efficient procedure call access to low-level functionality.
This monolithic method often necessitates building and
installing or upgrading a device’s apps and operating sys-
tem as a single unit.

5.2 Virtualization level

Virtualization uses software to construct a layer of

abstraction above computer equipment, enabling the actual
features of a single computer, storage, disk, and so on—to
be separated into numerous virtual computers, also called
as virtual machines (VMs). Each virtual machine created
for a user can be allocated an individual operating system
on a single physical machine that makes sure of the per-
formance of the virtual machines and failure isolation
among them. Hence, a Virtual Machine Monitor (VMM) /
Hypervisor is responsible for multiplexing of resources to
the virtual machine and helps in the management of the
power to perform efficient operations. The two ways in
which a virtual machine monitor can take part in the
management of power:
• A VMM acts as a power-aware operating system. It
verifies the entire performance of the system and
applies the DVFS (Dynamic Voltage and Frequency
Scaling) or any DCD (Dynamic Component Deactiva-
tion) techniques to the components of the system.
• The other way is to leverage the policies for the
management of power and knowledge of applications at
Fig. 2 A systematic summary of data centre power demand
prediction at the software level
OS level. Power management calls can be mapped from
different virtual machines. In addition, a coordinated
delivering continuous power to deeply implanted devices system wide limits on the power can be enforced.
such as wireless sensor nodes. However, if the use of a Virtualization technology has regained prominence in
node is not carefully planned, the battery may be depleted computer system architecture during the last few years.
too quickly, making continuous operation of such a device Virtual machines (VMs) provide a development route for
unfeasible. To regulate the flow of energy, an energy- adding new capabilities—for example, server consolida-
management solution is necessary. Buschhof et al. [21] tion, transparent migration, and secure computing—into a
presented an idea that enables the modelling of hardware system while maintaining compatibility with existing
energy usage and the creation of energy-aware device operating systems (OSs) and applications. Multiple VMs
drivers for the embedded OS. Their drivers can account for executing on the same core in contemporary virtualized
the energy usage of each driver function call with greater settings must adhere to a single management of power
than 90% accuracy. Similarly Levy et al. [22] presented controlled by the hypervisor. These settings have different
Tock, a unique embedded OS for low-power systems that limitations. It does not enable users to specify a desired
utilises the limited hardware-protection processes accessi- power control scheme for each virtual machine (or client).
ble on latest microcontrollers and type-safety functionali- Second, it frequently affects the energy efficacy of some or
ties of the Rust programming language to offer a all VMs, particularly when the VMs need competing
multiprogramming ecosystem that provides software fault energy management strategies. For mitigating above

123
1852 Cluster Computing (2023) 26:1845–1875

Fig. 3 Power dissipation of OS

routines [17]

problems, Kang et al. [23] suggested a per-VM power estimated the application’s energy usage. The evaluation
control method that enables each VM’s guest OS to utilise focuses on assessing software efficiency in aspects of
its chosen energy administration strategy and avoiding performance and power consumption per job especially
similar VMs from competing with each other’s energy when there exists shared resources and heterogeneous
control strategy. When compared to the Xen hypervisor’s environments based on profiles of energy, with the objec-
default on demand governor, Virtual performance (VIP) tive of determining the best resource configurations. The
minimises power usage and enhances the completion time applications were divided into two categories: data inten-
of CPU-intensive applications by up to 27% and 32%, sive application and communication intensive application.
respectively, without breaching the SLA of latency-sensi- Data-intensive applications that generate, analyse, and
tive implementations. Furthermore, Xiao et al. [24] transmit enormous volumes of data had been executed with
examined the VM scheduling model and the I/O virtual- minimal regard for energy efficiency. Large amounts of
ization paradigm in terms of energy-efficiency optimiza- energy may be consumed because of issues such as data
tion. They provided a power-fairness credit sequencing management, migration, and storage. A communication-
approach with a novel I/O offset method to achieve speedy intensive application is made up of one or more interde-
I/O performance while simultaneously raising energy pendent services, and the communication traffic between
conservation. Apart from this, Prabhakaran et al. [25] them is typically distinct. The communication traffic
introduced VM resource calibration. They created a system requires a large amount of power and various techniques
to reduce the energy usage of virtual servers by utilising for dynamic power management that can be applied for
controlled feedback architecture as well as power moni- energy efficiency.
toring services. Cloud services are referred to as Software as a Service
(SaaS) on the uppermost layer of cloud computing archi-
5.3 Application level tecture, which is a software delivery technique that offers
on-demand permissions. SaaS providers, in general, pro-
Energy efficiency is always a major concern in cloud vide extra layers of cloud computing, and hence keep client
computing and when it comes to the application level many data and tailor apps to match customer demands. This
recommendations have been made to optimise energy situation reduces the initial cost of obtaining new software
usage at the system level. However, the rising variety of and infrastructure significantly. Customers are under no
contemporary workloads necessitates a better analysis at obligation to maintain or build infrastructures on their sites.
the application level to allow adaptive behaviours and to They only need a fast network to access their apps rapidly.
minimise global energy consumption. For achieving energy SaaS providers service a variety of businesses by utilising
efficiency especially at the application level, Ho et al. [26] the same infrastructure and software [27]. This method is
concentrated on batch applications executing on VMs in clearly more power saving than installing several copies of
data centres. They investigate the application’s character- software on various infrastructure, which can reduce the
istics, computed the energy spent on each job, and requirement for new equipment. The lower the volatility in

123
Cluster Computing (2023) 26:1845–1875 1853

demand, the better the forecast and the bigger the energy Furthermore, Malik et al. [32] have developed ECoST, a
savings. SaaS companies must model and monitor the method for optimising energy efficiency and self-tuning for
energy efficiency of their software design, execution, and data-intensive workloads. They proved that fine-tuning
deployment because they primarily sell software hosted on settings at the application, microarchitecture, and system
their own data centers or resources from IaaS providers. levels simultaneously opens up the possibility of co-lo-
The SaaS provider selects data centres that are not only cating applications at the node level and improving server
power saving but also close to consumers. This is partic- energy efficiency without compromising functionality.
ularly crucial for social networking and gaming applica- Energy efficiency is a critical component in the devel-
tions, because users are often ignorant of their impact on opment of big supercomputers and low-cost data centers.
environmental sustainability. SaaS companies can also However, adjusting a system for energy efficiency is
provide Green Software Services hosted in carbon-efficient challenging due to the competing needs of power and
data centres with less replications. performance. The authors in [33] utilized Bayesian opti-
The authors in [28] introduced a solution for dynamic mization (BO) to optimise a graphics processing unit
software consolidation in order to decrease the number of (GPU) cluster system for the Green500 list, a prominent
VMs utilized. Software consolidation allows dynamically energy-efficiency rating of supercomputers. BO might
collocating different software applications on the same obtain an excellent configuration by defining the search
VM. The proposed method may be used with VM con- space beforehand with minimum information and prior
solidation, which places several VMs on fewer actual experiments. As a result, BO could remove time-consum-
machines. The authors of [29] proposed an energy-aware ing manual tweaking and shorten the system’s occupancy
application element migration technique that calculates the time for benchmarking. Furthermore, because of its influ-
load of data centre servers by taking the number of com- ence on operating costs and processing system rate of
ponents connected to the servers, the number of rental failure, energy efficiency became a crucial component of
people attempting to access the software applications, the high-performance computing. Processors are outfitted with
component strike rate, and various other important factors low-power methods such as DVFS and power capping to
into account when trying to decide which elements to increase the power effectiveness of such devices. These
migrate. To save energy, the server is turned off once all approaches must be tightly managed in relation to the load;
components of the underused servers have been moved. otherwise, considerable productivity loss and/or energy
They used discrete event simulation to test their suggested usage may occur because of system overhead expenditures.
approach. The authors in [34] proposed a workload-aware runtime
power-control strategy for effective V-f control. The pro-
5.3.1 Data intensive applications posed technique incorporates thread synchronisation con-
flict and delay due to Non-Uniform Memory Accesses to
Energy and power consumption are becoming increasingly find an acceptable V-f value (NUMAs).
significant in today’s high-performance computing (HPC) MapReduce is used for data processing in modern data
systems. New cluster systems are planned to be no more centers. It is known as the programming model that can be
than 20 MW in power [30], with the goal of attaining used for the processing and generation of large data items.
exascale performance as quickly as possible. The rise of The MapReduce programming model processes huge
big data and cloud computing has given the globe with amounts of data by executing a series of data-parallel jobs
huge opportunities as well as enormous challenges. How- that work on distinct sections of the data set. MapReduce
ever, the growing trend in cloud energy demand as a result platforms, which are runtime environments, allow cus-
of the fast-expanding volume of data to be delivered and tomers to scale up their programmes fast and easily. In
analyzed has propelled cloud computing, along with the big order to optimize the energy efficiency for MapReduce,
data phenomenon, to become the primary source of energy Tiwari et al. [35] have proposed a configurator based on
consumptions and, hence, CO2 emissions. To decrease the performance and energy models to enhance MapReduce
power usage of data intensive applications in cloud data system energy efficiency. It considers the dependence of
centers, the authors in [31] have presented an adoption the energy consumption and performance of a cluster on
framework for the data intensive applications whose pri- MapReduce parameters. Their proposed solution improves
mary goal is to minimize energy usage. The proposed the energy efficiency of up to 50% in two structurally
framework is driven by the values of data gathered from distinct clusters of typical MapReduce applications.
the data streams or data sets of the applications. The
authors looked at the data from different facets, from its
general to its domain-specific features, and then combined
them to provide a number indicating the data’s importance.

123
1854 Cluster Computing (2023) 26:1845–1875

5.3.2 Communication intensive applications instructions [38]. This gives rise to problems in certain
architectures(x86) as some privileged instructions may
Communication intensive application programs are made silently fail. Hypervisor resorts to a binary translation
up of a series of tasks that share a vast number of messages mechanism where validation is done on the set of
over the process of computing. These applications are instructions that may fail silently to resolve the above-
designed by utilizing the Message Passing Interface (MPI). mentioned problem. The other approach of conventional
Dynamic end-to-end request needs and uneven route power virtualization is Paravirtualization (PV). Paravirtualization
effectiveness, as well as uneven and time-varying link is a kind of CPU virtualization in which instructions are
usage, throughput and delay limits for service needs, all handled at compile time via hyper calls. Instead of trying to
offer challenges to power effective connections. The imitate an entire hardware eco system, PV is a virtualiza-
authors in [36] proposed a multi-constraint optimization tion technology advancement in which a guest OS is
framework for improving energy efficiency in cloud com- reconfigured even before to setup within a VM to allow all
puting technology including geographically dispersed data guest OS inside the scheme to share resources and effec-
centres linked by cloud networks. Their technique tively cooperate.
improves energy savings in both data centres and cloud The other approach to virtualization is containerization
networks. An intelligent heuristic technique is provided to that is also known as the virtualization at OS level. Vir-
handle this model for dynamic request demands among tualization technology utilizes the hypervisor that helps in
data centres as well as among data centres and consumers. emulating the hardware resources in order to run the guest
Furthermore, the authors in [37] established a simultaneous operating systems on top of it. The concept behind this was
optimisation of server power usage, network connectivity, that an application running on the hardware seldom makes
and migration expense with workload and host hetero- use of the entire resources. Virtualization creates copies of
geneity constrained by resource and bandwidth restrictions the functionality of the physical resources that includes the
in VM placement. Although Integer Quadratic Program computational, storage, memory, networking resources that
(IQP) can only be addressed for relatively small systems run an application. Containerization, a new concept intro-
and but it has been decomposed into master and price sub duced lately, is on the verge of development and growth.
problems that can be solved using the column generation Containers also aid in lowering administration expenses.
approach for larger systems. Since they use the same OS, just one needs to be monitored
and fed for security patches, and so on.
Virtualization allows several operating systems on a
6 Power modelling at operating system single physical server’s hardware, while containerization
virtualization enables to install many programs that use the same OS on
the same virtual machine or host. The architectural dif-
Virtualization is regarded as the most important technique ference between the virtualization and containerization is
for initiating modern clouds by sharing the physical as shown in Fig. 5.
resources among applications and the users. Virtualization Containerization is a lightweight virtualization solution
allows the efficient use of resources like software, hard- that facilitates the distribution and operation of application
ware, energy, etc. by consolidating many underutilized services across platforms such as edge/fog, cloud, and IoT.
machines on to a single system. Virtualization is divided Containerization is changing the working of industries
into five categories: application, server, desktop, network, because it is storage and resource efficient, performance
storage and based on the execution environment. The efficient, cost efficient, portable, energy efficient and
detailed classification of the virtualization techniques is extremely quick during boot up. Although the traditional
shown in Fig. 4. VMs enhance the efficiency of the physical servers, they
The conventional virtualization can be further divided incur a fair amount of overhead in costs and effort. A
into two different categories: Para and Full virtualization. container model enables the data center’s owners to simply
Full virtualization can be defined as the creation of virtual deliver the code they need to perform the function of the
processor, storage devices, memory and I/O devices in application without all the extra dependencies. This leads
order to run the various guest operating systems on a single to the efficient use of the resources within the data center.
machine so that the guest OS is not aware about the With the traditional virtual machines, the guest operating
presence of virtualization. In case of full virtualization, the system rather than the actual mission of the application
goal is to run the unmodified binaries of the operating utilizes a major portion of the resources. The lighter foot-
system. The code of the operating system remains print of the containers has many advantages throughout the
unchanged, that is why it is not aware of the fact that it data center. A container model needs fewer racks, less
does not have the required permissions to run privileged energy for cooling and power, less software licenses, less

123
Cluster Computing (2023) 26:1845–1875 1855

Fig. 4 Types of virtualizations

Fig. 5 Architectural difference

between virtualization and
containerization

maintenance. Containers offer a higher level of service Migration of applications running in virtual machines to
quality than other virtualization technologies. Furthermore, another data center or machine/server also needs the whole
because they require fewer resources than virtual machines, OS to be migrated along with it. Virtualization technology
additional entries are anticipated and will be integrated on is developed to exploit the existing resources but the
the same server, decreasing energy usage because fewer operations of the workload in virtualization do not con-
servers are planned to operate the same number of services. sume all the resources available to them, which leads to a
Docker, when configured to a maximum latency of significant wastage of resources also. In addition, the vir-
3000ms, can operate up to 21% more services than KVM. tual machines do not incorporate the leftover resources in
Docker provides this service in this setup while consuming capacity planning and distribution across all the virtual
11.33% less energy than KVM [39]. machines and workloads. On the other hand, container-
In containerization, containers execute onto the shared ization enables the individual functionality of an applica-
operating system kernel in isolation. One of the major tion to run independently making it possible for different
differences between containerization and hypervisor-based workloads to run on the same physical resource. These can
virtualization is that in containerization, the objects that are execute on bare metals or on top of hypervisors or cloud
virtualized are limited to the resources of the global kernel infrastructure too. Containers have the capability to create
that enables containerization to start various virtual envi- isolated OS environments within the same host, different
ronments onto the common host kernel. The created virtual functionalities of the same application run by sharing the
machines are resource intensive and do not allow individ- Linux Kernel in containerization [40]. Table 1 shows the
ual application’s functionalities/components to run in iso- difference between Virtualization and Containerization.
lated environments. The execution of an individual The performance of containers and virtual machines is
component or application in an isolated environment needs compared in experimentation conducted by various
a separate virtual machine. researchers. The experiments are performed on Docker and
KVM. Below are the results of the comparison of the

123
1856 Cluster Computing (2023) 26:1845–1875

Table 1 Difference between virtualization and containerization

Parameters Virtual machine Container

Guest Hypervisor allows multiple and distinct OS to run on the same host. All the guests share the same base OS and its Kernel. The
operating Each VM is allocated with a specified amount of memory on which image of the Kernel is loaded into the physical
system the Kernel functions. memory.
Security Security of the VMs depends on how the hypervisor is implemented. Container software like Docker have built-in security
features that can be leveraged.
Performance The VMs have a small overhead when compared to containers as the There is little to no overhead in using containers as the
translation of machine instructions occurs from the host OS to guest applications are executed in the base OS itself.
OS.
Isolation Hypervisor isolates each VM from host OS as well as from other Each container has its own set of file systems that can be
VMs. This means files, libraries etc. cannot be shared between shared between other applications.
guests and the host.
Startup time VM takes sufficient time to boot. Containers take less time to boot as compared to VMs.
Storage VMs take ample storage as the whole Kernel and the secondary Since the base OS is shared, containers require less
programs associated with the OS need to be installed. storage.

containers and virtual machines on the basis of different the VM can interact directly with the host, bypassing the
parameters [41]: in-between layers. Latency is another network metric that
Throughput metric is used to calculate the output of a may be used to assess performance. Throughput is also
task when the CPU is exposed to a compression High used to assess the efficiency of disc operations. As previ-
Performance Computing (HPC) test. It has been seen that ously stated, Docker and KVM add relatively little over-
native and Docker compression performance is compara- head when compared to native; however, there is a
ble, but KVM is slower. HPC performance is comparable significant performance difference in KVM’s case due to a
on native and Docker but very sluggish on KVM due to potential bottleneck in fibre channel. Docker has no cost
abstraction, which acts as negative in this scenario. The for random read and write operations, while KVM’s per-
CPU schedulers have no effect on the processor in either formance suffers considerably. The system’s I/O scheduler
the native or Docker arrangement; therefore, there is no has an impact on disc performance.
difference in performance. The parameter bandwidth is Containers are gaining popularity and will be a signifi-
used to measure the speed of memory access operations. cant deployment strategy in cloud computing. Consolida-
According to many benchmarks designed to test memory in tion techniques are widely employed in the cloud
linear and random-access approaches, the efficiency of environment to maximise resource usage and minimise
native, Docker, and KVM systems is almost similar for a power consumption. To minimize the power consumption
variety of workloads with very little variation. The testing through container consolidation, Piraghaj et al. [42] have
was carried out on a single node using large datasets. presented the problem of container consolidation and have
Container-based systems returned unused memory to the compared the different algorithms. They evaluated their
host, resulting in more efficient memory utilisation. Vir- performance against parameters like, SLA violations,
tualization systems suffered from double cache since the consumption of energy, average rate of transferring con-
host and virtual computer used the same memory blocks. tainers, and the typical number of VMs generated. The
Bandwidth is used to assess network communication per- consumption of power by data centers at time t can be
formance. Bulk data transfer using a single TCP connec- calculated as:
tion, similar to the client-server architecture, is the X
NS
communication situation. Because the TCP/IP stack has Pdc ðtÞ ¼ Pi ð t Þ ð1Þ
distinct regulations for sending and receiving data, the data i¼1
transfer rate is measured in both directions. The NIC, Pdc(t) denotes the consumption of power by the data
which utilises CPU cycles to measure overhead, is the center during t time, Ns denotes Number of servers and
primary component that causes a bottleneck in perfor- Pi(t) denotes the consumption of power by Server i during
mance. In terms of performance, Docker employs bridging the time t. The metric for SLA is calculated as the fraction
and Network Address Translation (NAT), which lengthens of the difference among the allocated and the requested
the path. Dockers that do not utilise NAT operate similarly CPU for each VM [43].
to native systems. KVM performance can be increased if

123
Cluster Computing (2023) 26:1845–1875 1857

NS X
X N VM X
NV
CPU r ðvmj;i ; tp Þ  CPU a ðvmj;i ; tp Þ The central processing unit (CPU) is the main compo-
SLA ¼ ð2Þ nent that consumes the power in the data center. The
i¼1 j¼1 p¼1
CPU r ðvmj;i ; tp Þ
energy consumed for computation is [45]:-
 
Ns represents Number of servers, Nvm represents Ecomp ¼ Emaxcomp  Eidlecomp  U CPU idlecomp
Ni Ni Ni Ni þ ENi
Number of VMs, Nv represents Number of SLA Viola-
tions, CPUr(vmji, tp) represents the amount of CPU needed ð6Þ
by VM j on server i during tp time, CPUa(vmji, tp) rep-
The container-based cloud-computing concept has
resents the amount of CPU amount assigned to VM j grown through time as a versatile and power efficient
during tp time. resource-use approach. Cloud providers strive to enhance
For the reduction of the consumption of power by data
utilisation of resources and resource use when executing
center that consists of Nvm VMs, M containers, and Ns container aggregation, which includes VM selection and
servers, they designed the equation as: placement. Shi et al. [46] have designed TMPSO, for
X
NS energy aware consolidation of containers. The proposed
minðPdc ðtÞ ¼ Pi ðtÞÞ ð3Þ algorithm integrates the heuristic and greedy optimization
i¼1
mechanism to get the balance between the computation and
In the above equation, Pdc(t) denotes the consumption performance cost.
of power by data center during the t time, the consumption In a cloud, the authors have assumed a set of physical
of power by server i at time t is denoted by Pi(t), and Ns machines PM = {PM1, …, PMm, …, PMc}, a set of virtual
represents Number of servers. machines VM = {VM1, VM2, …, VMi, …, VMv}. Each
Since container-based service aggregation is a tough virtual machine VMi has CPU Ci, memory Mi and oper-
process, the cloud data centre consumes a lot of power due ating system Oi, i.e., VMi (Ci, Mi, Oi). They have assumed
to a lack of management over the data centre systems. that the capacity of each virtual machine is the same so for
Because containers need a smaller resource footprint, each PMm [ PM, it can be demonstrated as PMm(CC,
consolidating them in servers may result in limited CM). A CSP first associates each of the applications to a
resource availability. Nath et al. [44] have designed an container that satisfies their needs of the resource that
energy efficient service based on the consolidation of the includes CPU c, Memory m and operating system o. These
containers. The authors have formulated a service consol- containers are demonstrated by triple Cj (cj, mj, oj). They
idation problem as an optimization problem by considering have assumed that each Cj is assigned to a VM. VMi [ VM
the minimizing the power usage by the data centers. to satisfy the needs of the resources including CPU cj,
In data centers, there are n servers and m containers in Memory mj and operating system oj. They represent allo-
the system. They represent N = {Ni: i [ (1. . n)}as sets of cation kj : Cj ? VMi.
data center servers and Cn = {Cnj : j [ (1,. ., m)} as sets of Finally, each VMi is allocated to a PMm, the allocation
data center containers. They have defined the total con- can be denoted as ci : VMi ? PMm.
sumption of energy by a server as: For the reduction in the consumption of power by data
rcvfrmmanager center, the authors have designed objective function for the
ENi ¼ Erevdata
Ni þ ENi þ ð1 þ a þ bÞ  Ecomp
Ni
revfrmnode consolidation of containers as:
þ b  ðENi þ Esendtonode
Ni Þ þ aðEofflnode
Ni
rcvfrmnode Xc
þ ENi Þ þ Esendtoclient
Ni minimizeE ¼ Pm :zm ð7Þ
ð4Þ m¼1

In the above equation a (0 B a \ 1) denotes the data In the above equation, Pm represents the consumption of
offload percentage to other server (worker) and b (0 B energy by physical machine PMm and zm represents the
b \ 1) denotes the data received percentage from another binary variable, zm [ {0, 1}, indicating whether PMm is
server (worker). active.
Collecting and delivering bits in a host uses energy. It The below equation shows the connection among the
can be defined as: consumption of energy and CPU utilization.
 idle  busy 
Eofflnode ¼ Esend rcv P þ P  Pidle :umcpu ; ifN vm [ 0
Ni Ni ¼ ENi ¼ E bit  B ð5Þ Pm ¼ ð8Þ
0; ifN vm ¼ 0
In the above equation, Ebit denotes the energy con-
sumption by sending one bit of data and B denotes the bits where the CPU utilization is, um
cpu Pidle and Pbusy are the

received or sent by the user. consumption of power by PM during the utilization is 0%

and 100% respectively, Nvm denotes all the VMs allocated
to the PM.

123
1858 Cluster Computing (2023) 26:1845–1875

The utilization of memory by PMm is calculated by order the virtual machine preference list. According to the
Eq. 10, OMi denotes the overhead of memory of VMi. simulation findings, the algorithm may cut energy usage by
X
N vm an average of 12.8% when compared to the First Fit
um
cpu ¼ ðOC i þ C i Þ:ym
i ð9Þ method.
i¼1 In the technique proposed by Chen et al. [49], the energy
X
N vm usage is by Eq. 1:
um
mem ¼ ðOM i þ M i Þ:ym
i ð10Þ They often assess server power utilization utilizing the
i¼1 CPU efficiency ratio, as the CPU is the most commonly
Apart from the literature that is mentioned above, there utilised element of energy expenditures in terms of server
are various other works that help in the placement of utilisation. The CPU utilization ratio of each server is equal
containers, the placement of virtual machines so that the to the
power consumed is less, and resource utilization is efficient X
N vm X
Nc

in the container as a service environment. The authors in U c ðk; j; tÞðtÞ ð13Þ

j¼1 k¼1
[47] have addressed the issues of container placement and
VM placement in the two stages. To solve the container The server energy consumption is estimated using the
placement problem, they merged the two strategies. The Eq. 8:
proposed solution is divided into four decision-making They assume M containers, Nvm VMs, and Ns servers
processes: VM selection and creation, PM selection and in the containerized cloud computing paradigm, and the
creation. A hybrid technique, genetic programming-based energy consumption problem may be stated as Eq. 3.
hyper-heuristics (GPHH) and human-designed rules was They assume M containers, N VMs, and K servers in the
used to tackle the two-level container allocation problem. containerized cloud computing paradigm, and the energy
To assess container allocation, they took into account consumption problem may be stated as Eq. 1:
accumulated power. The energy consumed Ptd of all active Apart from this, Al-Moalmi et al. [50] have addressed
PMs over the time period t1, t2 is added to calculate the the issues related to the placement of container and VM in
accumulated energy (see Eq. 11). In other words, the Container as a Service (CaaS) environment by considering
authors add the energy usage of all PMs at each time the optimization of the power consumption and resource
interval ti. Fan’s [48] energy model of a PM (Eq. 12) is a utilization. They proposed an algorithm based on the
commonly used model. In their energy model, Pidle and Whale Optimization Algorithm (WOA) to solve the two
Pmax represent the energy use while a PM is idle and fully problems that are container placement and VM placement.
utilized. However, cpu(d) indicates a PMd’s CPU use at Each VM and PM in this cloud data centre may be assigned
time t. The purpose of container assignment is to decrease to any single type of container and VM. To put it different
net power consumption, i.e., M in Accumulated Energy way, every type of container may be hosted by a single
(AE). VM, and any type of VM can be hosted by a single PM,
Z t¼t2 XD according to the constraints stated below:
AE ¼ Ptd ð11Þ 8ðConta ; V b Þand ðContc ; V d ÞifConta ¼ Contc ; thenV b ¼ V c
t¼t1 d¼1
 max  t ð14Þ
Ptd ¼ Pidle
d þ Pd  Pidle
d :ucpu ðdÞ ð12Þ
where Conta and Contc are container identifiers and Va and
In the problem, they examined three sorts of restrictions. Vd are VM identifier.
To start, the total capacity use of containers need not sur-
pass the threshold of the target VM. The aggregated energy 8ðV a ; Pb Þand ðV c ; Pd ÞifV a ¼ V c ; thenPb ¼ Pc ð15Þ
needs of VMs must not surpass the target PM’s capacity. where V a and V c are VM identifiers and Pa and Pd are PM
Second, a container may only be assigned once. Third, they identifier
use an affinity restriction to limit container deployment to
X
L
OS-compatible VMs alone. Contcpul  V cpul ; 8cpul 2 C; andV i 2 VÞ ð16Þ
Furthermore, Chen et al. [49] presented many-to-one l¼1
stable matching method and a container placement tech-
X
L
nique MLSM. They started with an early container hosting Contraml  V raml ; 8Contl 2 C; andV i 2 VÞ ð17Þ
technology to shorten migration durations by the use of a l¼1
trustworthy matching mechanism. This programme utilises
resemblance algorithms as a finding choice strategy for
containers and VMs. The resource usage rate is used to

123
Cluster Computing (2023) 26:1845–1875 1859

X
M They investigated the energy savings provided by suspend-
V cpui  Pcpucapacityj ; 8V i 2 V; andPj 2 PÞ ð18Þ to-disk and suspend-to-RAM approaches, as well as the
i¼1 influence of shutdown processes on the energy consump-
X
M tion of future models with various CPUs. Furthermore, the
V rami  Pramcapacityj ; 8V i 2 V; andPj 2 PÞ ð19Þ authors in [52] presented different shutdown models that
i¼1
may be utilised under current and future supercomputer
Equations (14) and (15) specify the prerequisites for limitations, considering the impact of closing down and
presenting containers to VMs and VMs to PMs, respec- waking up networks as well as the idle and off states seen
tively, while Eqs. (16)–(19) ensure that a collection of after such procedures as they influence power usage of the
containers’ total consumed capacity does not surpass the resources.
host VM’s CPU and memory capacities. Similarly, the
overall resources used by a group of VMs should not 7.2 Virtualized systems
exceeding the host PM’s Memory and CPU capabilities.
Virtualization and Cloud computing are enabling innova-
tions for the creation of resource planning algorithms that
7 Dynamic power management at data are energy-aware in virtualized data centres. Indeed, one of
center level the primary problems for large data centres is to reduce
power usage, both to save money and to reduce environ-
RQ2 What are the various strategies utilised in data mental effects. The authors in [53] developed a one-of-a-
centres, both virtualized and non-virtualized systems, to form of combined server and network reduction model that
minimise power usage? considers the power consumption of both switches capable
of transmitting traffic and servers hosting virtual machines.
When developing higher-level power models for data
Under QoS constraints, it shuts access points and sends
centres, it is important to understand the intricacies of the
information to the least energy-consuming host over the
lower-level elements that account for the total power use of
most energy-efficient route. Due to the model’s complex-
the data centre. A technique is often focused on workload
ity, a quick Simulated Annealing-based Resource Consol-
reduction among physical nodes in data centres. The goal is
idation (SARC) approach is provided. Furthermore, the
to assign as little physical resources as feasible to requests /
authors in [54] developed the energy-aware fault-tolerant
virtual machines while shutting off or placing unused
dynamic scheduling system (EFDTS), a dynamic task
resources to sleep / hibernate. The allocation difficulty is
assignment and scheduling method that uses a fault tolerant
twofold: first, new requests must be allotted; second, the
mechanism to maximise resource usage and minimize
performance of present services / VMs must be constantly
energy consumption. In the task assignment scheme, a task
checked, and if required, the allocation must be altered to
classification approach is designed to divide incoming
give the best possible power-performance trade-off con-
tasks into separate classes and then redistribute them to the
cerning stated QoS. This section delves into data centre
most appropriate VMs based on their classes in order to
power models.
minimize mean response time while minimizing energy
usage. Apart from this, the authors in [55] presented an
7.1 Non-Virtualized
energy-conscious management method for virtualized data
centres that is based on dynamically adjusting and scaling
Power consumption is increasing in large-scale systems
computer capability to workload factors. They created a
such as data centers and supercomputers. These systems
new ontological model for describing the energy and per-
are typically measured based on peak demand. Since the
formance aspects of data center operations. To address the
power consumption of these devices is not proportional,
issue of energy usage, the authors of [56] proposed the
their energy usage stays high even when the workload is
Energy and Performance-Efficient Task Scheduling Algo-
low. Shutdown processes have been established to match
rithm (EPETTS) in a heterogeneous virtualized cloud. The
the number of servers that are actively engaged in the
proposed algorithm is divided into two stages: initial
workload processing. However, because of the potential
scheduling, that aims to minimize completion time and
influence on performance and hardware concerns, data
meet task deadlines while not keeping in account power
centre administrators are cautious to utilise such tactics.
consumption. The second stage is task reassignment plan-
Furthermore, the energy advantage is usually overesti-
ning, which enables for the best execution location within
mated. The authors in [51] have evaluated the potential
the timeline limit while using the least amount of energy.
benefits of shutdown procedures by accounting for shut-
down and boot up costs in terms of both time and energy.

123
1860 Cluster Computing (2023) 26:1845–1875

8 Problem solving approaches process of migrating a functioning VM or application

among PMs without interrupting the client or service.
Virtual machine migration, load balancing and workload Consolidation of virtual machines (VMs) is a typical
categorization are the various problem-solving techniques technique for lowering energy usage based on peak, off-
that can be used for the reduction in the power consump- peak, or average CPU use of VMs in order to execute them
tion. These techniques are used to migrate the virtual on the least number of servers while preserving service
machine when the threshold is attained for the particular quality (QoS). There are different techniques for the live
server, balance the load among the different VMs and migration like pre copy migration [57], post copy migration
categorize workload type before placing them onto the [58], hybrid VM migration [59], dynamic self-ballooning
server. The various ML algorithms are used on top of these [60], Adaptive Worst Fit Decreasing [61], Check pointing/
approaches to efficiently manage the power consumption in recovery and trace/replay technology [62], Composed
data centers. Image Cloning (CIC) methodology [63], Memory man-
agement based live migration [64], Stable Matching [65],
8.1 Virtual machine Migration Matrix Bitmap Algorithm [66], Time Series based Pre-
Copy Approach [67], Memory Ballooning [68], WSClock
Virtual machine migration can be defined as sending the Replacement Algorithm [69], Live Migration using LRU
VM from one host to another by remaining connected with and Splay Tree [70]. Apart from these, the various
the application or client. The Virtual Machine Migration machine-learning approaches are also used to migrate the
(VMM) can be categorized as live migration and non-live VM from one host to other. The techniques like autore-
migration of virtual machine as shown in Figs. 6 and 7. gressive integrated moving average [71], support vector
Live VM migration refers to the moving of VMs from regression [72], linear regression, SVR with bootstrap
one server to another when the host system stays active. aggregation [73] were also used for VM migration. These
There are two forms of live virtual machine migration: pre- approaches are used to forecast and manage resources
copy live VMM and post-copy live VMM. Non-live virtual effectively in the data center, as well as to calculate the
machine migration is defined as migrating a VM from one energy consumption. Moreover, metaheuristics are also
server to another by turning off the virtual machine on the used for the migration of virtual machines. The techniques
host server. Non-live migration stops or shuts down the like Firefly Optimization [74], Particle Swarm Optimiza-
VM prior to transfer, depending on whether it wants to tion [75], Ant Colony Optimization [76], Biogeography-
continue running services after transfer. When a virtual Based Optimisation [77], Discrete Bacterial Foraging
machine is terminated, its operating states are wrapped and Algorithm [78] are also used for the migration of virtual
transferred to the destination location. Live migration is the machines. These approaches optimise energy usage, QoS,
resource use, or all three.
The purple box in (8) represents a VM that has been shut
down or terminated on the originating host.

8.2 Load balancing

The practice of equally splitting workload in a distributed

environment such that no processor is overloaded, under
loaded, or idle is known as load balancing. Load balancing
assists in the acceleration of various constrained parame-
ters such as execution speed, response time, device relia-
bility, and so on. Load balancers are highly efficient where
huge workloads will quickly overload a single computer or
SLAs need high levels of service efficiency and response
times for certain business processes. The users deliver
multiple requests and load balancer that is installed prior to
the cloud server handles these requests. Load balancer
distributes the incoming workload to the different cloud
servers. Figure 8 shows the mechanism of load balancing.
Load balancing (LB) gives a well-organized solution to
a wide range of difficulties in a cloud environment. LB
Fig. 6 Non-Live VM migration plays an essential factor in the system’s efficiency and

123
Cluster Computing (2023) 26:1845–1875 1861

Fig. 7 Live VM migration

Fig. 8 Load Balancing mechanism in cloud computing environment

robustness. LB in cloud computing is one of the most workload among VMs is required. Load balancing strate-
difficult and valuable research topics for spreading work gies come in a variety of forms that balance the requests of
across virtual machines in data centres. As a result, a the resources. These are Round Robin [79], Equally Spread
method for improving system efficiency by balancing Current Execution Algorithm [80], Throttled Load

123
1862 Cluster Computing (2023) 26:1845–1875

Balancing Algorithm [81], Biased Random Sampling [82], extensibility, and administration. Moreover, the cloud must
Min-Min Algorithm [83], Max-Min Algorithm [84] and provide capabilities that meet the best-in-class demands of
Token Routing [85]. The above-mentioned techniques the organisation, such as privacy, realistic reliability, and
were not able to dynamically balance the workload in a economy. Computer equipment are organised differently in
cloud-computing environment; therefore, the machine different computing environments, and they share data
learning approaches were introduced. There are various between themselves to analyze and solve problems. One
machine learning algorithms like K-Nearest Neighbors computing ecosystem is made up of different computa-
[86], deep neural networks [87], multi-layer perceptron tional resources, software, and networks that help with
[88], Simulated Annealing [89] that were used for LB in computation, sharing, and problem solving. Workload can
cloud environments. These approaches enable accurate and be classified into three types based on the generation:
practical decision making the resource allotment to Synthetic, Real and Cloud. Furthermore, based on the
inbound requests, resulting in the selection of the most application, the workload is classified into four categories:
relevant applications to finish. There are two types of Web, Social Network, Video Service etc.
metaheuristics algorithms: nature-inspired algorithms and The load-balancing problem falls under the category of
evolution-based algorithms. There are various nature NP-complete. As a result, application developers fre-
inspired algorithms like Swarm behaviour-based algorithm quently employ heuristic or stochastic approaches to solve
[90]: Ant colony optimization [91], Particle Swarm Opti- it. Initially, the characterization of workload was done
mization (PSO) [92], Artificial bee colony (ABC) opti- using statistical methods like mean and standard deviation,
mization [93], modified Particle swarm optimization Auto Correlation Function, Pearson Coefficient of Corre-
(MPSO) and improved Q-learning algorithm [94], Bat lation, Coefficient of Variation, and Peak to Mean Ratio
Algorithm [95], Crow inspired metaheuristic algorithm [100]. These approaches were utilised for a thorough
[96], Shuffled Frog Leaping Algorithm [97], Honey bee characterisation of both requested and actually used
behaviour [98]. Depending on task levels, these strategies resources, including data relating to CPU, memory, disc,
give tasks to VMs while guaranteeing equitable load and network resources. Parameters like, CPU usage,
sharing and generate extra number of available slots in memory consumption, network bandwidth, storage band-
series or parallel mode. Moreover, for complicated and width, and job length were frequently included in data
huge sample space issues with a hazy sample space various center traces. Not all qualities are equally significant in
evolution-based algorithms are employed. Techniques like workload categorization and characterization. When all of
Genetic Algorithm (GA) [99] are also used for LB. the features are considered, the model’s complexity
increases. This being one of the drawbacks of statistical
8.3 Workload categorization and prediction methods, clustering (unsupervised learning) was used to
classify the workloads. For better categorization of work-
Workload is the total amount of effort done by a targeted load, different types of clustering like Hierarchical clus-
server for a fixed period. Workload is classified before tering [101], Density based clustering [102] etc. were used.
being transferred to the virtual machines that minimizes the Apart from the unsupervised clustering techniques there
probability of server overutilization, eliminates the need are various other supervised learning techniques like Sup-
for virtual machine relocation, and thereby improves port Vector Machine (SVM) [103], Stochastic Gradient
energy consumption. Web applications, web servers, dis- Descent (SGD) [104], Logistic Regression (LR) [105],
tributed data storage, containerized microservices, and Random Forest (RF) [106], Multi-Layer Perceptron (MLP)
other workloads that require broad processing capacity are [107], Backpropagation neural network [108] were used.
prevalent. In cloud data centers, workload classification After categorising the workload, the prediction is per-
and characterisation are used for resource planning, formed in order to forecast the future workload. Forecast-
application performance management, capacity sizing, and ing of workload, management of resources dynamically
projecting future resource requirements. An accurate esti- and scaling proactively may all aid in the achievement of a
mate of future resource demand assists in meeting QoS variety of essential objectives. Accurate forecasting of
requirements and guaranteeing effective resource utilisa- near-term workload, for example, has a direct influence on
tion. Workloads are categorised based on computational response time, SLA violations over-provisioning, and
paradigms, technology stack, resources, and applications, under-provisioning concerns. Effective workload manage-
as seen in Fig. 9. Based on the processing methodology, ment improves system scalability and throughput. Fur-
workloads are divided into two types: batch workloads and thermore, by limiting over-provisioning of virtual
interactive workloads. Based on resource requirement, the resources, cloud DC power consumption, cost, and the
workload is classified as Memory, CPU, IO and database. number of unsuccessful requests may be reduced, and
These requirements include scalability, flexibility, satisfaction of customer can be enhanced.

123
Cluster Computing (2023) 26:1845–1875 1863

Fig. 9 Classification of workload

There are many techniques that can be utilized to predict employed to achieve a suitable VM-PM mapping: Con-
the future workload. These include: regression-based straint Programming [128], Stochastic Integer Program-
schemes that includes ARIMA-based schemes [109], ming [129]. The above-mentioned approaches are not
Support vector regression-based schemes [110]; Classifier- suitable for today’s scenario, as these approaches cannot
based schemes that includes SVM-based schemes [111], predict the future based on the previous history. For the
Random forest-based schemes [106], Artificial neural net- suitable placement, machine-learning approaches can be
work-based schemes [112], Bayesian-based schemes [113], used for the placement of virtual machines. Reinforcement
Deep learning-based schemes [114]; Stochastic-based Learning [130], Artificial Neural Network [131], and Fuzzy
workload prediction schemes that includes Hidden Markov reinforcement learning [132] are some of them. Population
model-based schemes [115], Queuing model-based based techniques begin with a collection of single solutions
schemes [116]. Apart from this there are various other that grow from one generation to another. This category is
approaches like Grey predicting-based schemes [117], centred on exploration and provides for greater variety in
Autocorrelation clustering-based schemes [118], Chaos- the search process. In order to discover the Pareto optimum
based schemes [119], Kalman filter model-based schemes solutions, population-based techniques employ the idea of
[120], Wavelet-based schemes [121], Collaborative filter- dominance in their screening process. The techniques used
ing-based schemes [122] and Ensemble-based schemes are Genetic [133], Ant Colony Optimization (ACO) [134],
[123] that can be used to predict the workload. The com- Memetic [135], Firefly [136], Whale optimization [137],
bination of the above-mentioned techniques are also used Sine-Cosine Algorithm and the Salp Swarm Algorithm
to predict the future workload. These include SVR and [138]. Single solution–based algorithms begin with a single
Kalman filter [124], ARIMA and RNN [125], ARIMA and solution, which is then modified and transformed
wavelet decomposition [126] (Fig. 9). throughout the optimization process. These algorithms are
exploitation-focused, which means they try to enhance the
8.4 VM placement search strength in certain locations. The techniques include
EAGLE algorithm [139], Imperialist competitive algorithm
Virtual machine placement refers to the process of deter- [140], Krill herd algorithm [141].
mining the appropriate PM for a certain VM. As a conse-
quence, a VM placement algorithm finds the ideal VM to
PM connection, whether it is a new VM placement or a VM 9 Environment
migration for placement re-optimization. A VM placement
method may be roughly classified into two categories based RQ3 What are the major impacts of a data center on the
on the aim of placement: Power-based [127] and QoS- environment?
based [127]. VM Placement strategies are primarily cate-
gorised as under, based on the type of principal strategy

123
1864 Cluster Computing (2023) 26:1845–1875

9.1 Renewable energy Most common e-waste includes LCD monitors, LCD
and Plasma televisions, and computers with Cathode Ray
In the last few years, it can be seen that there is an expo- tubes. However, this does not mean that other electrical and
nential rise in data centers developed by different compa- electronic equipment does not fall into this category.
nies to provide services like cloud computing. They Actually, any piece of electronic equipment is e-waste.
consume very large amounts of electricity for normal Almost all technology-based industries produce e waste.
functioning. Apart from the high consumption of electric- However, the footprint of the data center in this domain is
ity, the increase in the consumption of energy by cloud data relatively small. Data center consists of components like
centers results in adverse effects on the environment. In generators, Uninterrupted Power Supply (UPS) etc. These
many parts of the world, the electricity is produced by components have a long primary lifecycle of about 5–10
burning coal that leads to negative results like increased years and are also repurposed into non 24 9 7 roles before
carbon dioxide emission and increase in pollution. they can be recycled.
The growing use of renewable energy plants, in partic-
ular, presents a tremendous potential for more effective 9.3 Carbon footprint (Greenhouse gas
administration of dispersed data centres. Dynamic work- emissions)
load allocation and migration across data centres might
help to save costs by shifting workload to regions where Cloud data centers consist of a huge number of rows of
energy is cleaner or cooling costs are lesser. After electricity consuming servers having network, storage,
obtaining the user’s query, the cloud hosting has the option power supply systems along with gigantic HVAC (heating,
of selecting the target region depending on a variety of ventilation and air conditioning) units that avoid over-
factors. The authors in [142] have worked upon the already heating. However, these data centers appear to be clean but
proposed technique called EcoMultiCloud. They looked at they are not contributing to the green initiative.
the example of a complex network made up of data centres Total Greenhouse Gas Emissions (GHG) attributed to
(DCs) spread around the country, with renewable energy data centres in 2018 were 3.15 9 107 tons CO2-eq,
producers co-located with cloud services to reduce the accounting for about 0.5% of total GHG emissions in the
amount of electricity purchased by the power grid. Since United States [144]. A little more than half (52%) of total
renewable energy sources are infrequent, infrastructure data centre emissions is attributable to the Northeast,
load control solutions must be customised to the intermit- Southeast, and Central United States, which have a high
tent nature of the sources. Furthermore, the authors in [143] concentration of thermoelectric power plants as well as a
have addressed the problem of reducing energy costs for big number of data centres. Almost 30% of the emissions
geographically distant data centres while maintaining from the data centre sector occur in the Central United
assured service quality (i.e., service latency) under time- States, which depends significantly on coal and natural gas
varying system dynamics. They proposed a green geo- to satisfy its energy needs.
graphical load balancing (GreenGLB) online solution for Looking at these estimates, it becomes quite evident that
interactive and indivisible work distribution based on the companies need to reduce GHG emissions. Many compa-
greedy algorithm design approach. An indivisible job is nies and countries are already trying in this direction and
something that cannot be divided further and must be have introduced improvements to the existing data centers.
allocated to a single data centre. It is estimated that most of the company’s servers are
having a utilization of only 10–15% and 30% of these
9.2 E-Waste corporate servers are zombies in the sense that they are
inactive yet use electricity while doing so (data centres
The rapid escalation in the use of electronic devices at the play a key role in reducing GHG emissions) [145].
consumer level along with the growth of enterprise-class
and hyperscale computing has added to the issue of 9.4 Case studies
e-waste. It is a combined responsibility of consumers,
manufacturers, enterprises and governments to ensure that Zero carbon, zero emissions, and zero waste are the goals
this waste is being minimized, reused and recycled prop- of the next generation of sustainable data centers. It all
erly. Waste Electrical and Electronic Component (WEEE) starts with sustainable energy. Solar energy has no carbon
passes regulations at country, state or province level aiming impact, generates no pollutants, and may grow fast. Solar-
to promote the reuse and recycling of e waste leading to powered data centers are not only environmentally green,
reduction of consumption of such resources and the amount but also financially effective. It reduces the energy expense
of e-waste going to landfill. by about 70%, a benefit that is immediately passed on to
customers, resulting in significant cost savings [146]. The

123
Cluster Computing (2023) 26:1845–1875 1865

relative stability of solar power expenses vs the ever-in- 9.4.2 Project natick by microsoft [147]
creasing cost of regular energy provides organisations with
significant headroom to engage in innovation. Scalability is Nearly 50% of the world’s people lives within a few miles
also not a problem. The below mentioned case studies uses of the shore. By locating data centers near coastal cities,
the renewable energy to power the data centers. The CtrilS data transit times to coastal towns would be reduced,
data center make use of the solar energy to power the data resulting in faster and smoother online surfing, video
center and it is the first data center in the world which is streaming, and gaming. Furthermore, not only do the
awarded with the LEED Platinum’ certification from Uni- obstacles connected with creating data centres on land
ted States Green Building Council whereas the project disappear, but the ocean also delivers a rather stable envi-
Natick by Microsoft is the first underwater data center ronment for these underwater pods. Finally, even at inter-
which is 100% powered by solar and wind. The project mediate depths of 10–200 m, the water maintains a
proves that the underwater data center is feasible as well as temperature range of 14–18 degrees Celsius, making it
logistically, environmentally and economically practical. ideal for cooling data centres. As a result, cooling costs are
lowered.
9.4.1 CtrilS data center [146] Microsoft project Natick is a research project whose
main aim is to determine the credibility of the subsea data
CtrilS data centre is one of the data centers in Mumbai that centers that can be powered by offshore renewable energy.
has taken initiative in the building of green data centers and On the seafloor off the coast of Scotland, Microsoft’s
has been awarded with the highest rating - platinum - under underwater data centre project used wave energy and post-
LEED V4 existing Building (O ? M) category. It is the quantum encryption. The concept of an underwater data
first data centre in the entire world to get this recognition center was put forward at Microsoft in the year 2014 during
from the U.S Green Building Council (USGBC). the ThinkWeek that is related to the employees to discuss
CtrilS has identified and worked upon many key areas the extraordinary ideas. The project aims to provide the
like operations, design and technology. Some of them are lightning quick services of the cloud to the coastal popu-
as: lation and save energy. The cool surface of the sea enables
• Many of the enhancements done like consumption of the energy efficient design of data centers. The experi-
power by cooling system, enhancing the pumping sys- mental green energy solutions under research at the Euro-
tems efficiency, cooling towers automation and effec- pean Marine Energy Centre and the wind and solar energy
tiveness, helped them to gain savings of 15 lakhs kWh/ used to power the grid are some of the main reasons for
annum which means saving of approximately 1.5 crore Northern Isles installation by the Project Natick team at the
rupees on the entire operations. ‘The Orkney Islands’.
• Mercury free lights used resulted in saving of 0.5 lakh
kWh/annum (cost saving of 5 lakhs/year). 9.5 Policies and standards
• Proper centralized monitoring system was used to
monitor the electricity utilization and report anomalies Energy is one of the most essential components of a data
when there is excessive use. center’s running expenses, yet rising energy costs and
• Water saving fixtures were used leading to the saving of associated operational costs pose hurdles to corporate
5KL of water per day that was about 24% of water competitiveness. As a result, it is vital to minimise energy
reduction in the overall water requirement. consumption in data centres, and improved energy effi-
• Rainwater treatment for storing rainwater during the ciency has been identified as an acceptable tool for this
rainy season was also used. aim. There are several policies in place to address energy
• Green seal products were used for cleaning purposes in usage in data centres [148]. Some of these are mentioned
order to avoid toxic substances. below in Table 2 with their key features.
• CFC-Free Refrigerants in cooling systems were used to
minimize greenhouse gas emissions.
• Strategic ventilation points were introduced in order to 10 Research challenges
address the quality of indoor air and increase in
productivity. These were created to get the levels RQ4 What are the major software academic difficulties for
above the baseline of Ashrae Standard 62.1 for the developing green data centres?
needs of the fresh air.
Green data centers have been an active area of research
• Extensive Training Programs were held to aware the
and the major challenges to achieve green data center
employees about the importance of incorporating the
includes the challenges in container technology, VM
green compliance features.

123
1866 Cluster Computing (2023) 26:1845–1875

Table 2 Key Features of various policies and standards for data centre energy consumption
Policy name Country Year Key features

ENERGY STAR Rating by Environment USA 1992 The ENERGY STAR score for data centers is applicable to facilities that
Protection Agency (EPA) are specially constructed and equipped to fulfil the demands of high-
density computer equipment, such as server racks, which are utilised for
data storage and processing. The goal of the ENERGY STAR score is to
offer a fair assessment of a facility’s power efficiency in compared to
peers, while considering climate, weather, and business operations at the
location into account. It entails measuring and optimizing energy usage
on a consistent basis.
American Society of Heating, Refrigerating USA 1895 This standard establishes the minimal energy saving requirements for
and Air-Conditioning Engineers (ASHRAE) structures such as data centres and telecommunications buildings, in
terms of architecture, development, service, and maintenance, as well as
the use of on-site or off-site renewable energy supplies.
National Australian Built Environment Rating Australia 1998 This policy measures the performance related to the environment on a
System (NABERS) scale of 1 to 6 stars. The three categories decide the final rating.
Green Mark Singapore 2005 This rating system is based in Singapore that gives points features like
efficiency of energy and best practices.
Green Building Index (GBI) Malaysia 2009 This is based on current ranking tools, such as the Green Mark, but it has
been heavily updated to account for Malaysia’s tropical weather,
environmental background, and cultural and social needs.
Certified Energy Efficient data center Audit United NR This rating system is based in UK that assesses the best practices that
(CEEDA) Kingdom should be followed for building energy efficient data centers.
Building Research Establishment United 1990 This rating system is based in the UK which is also followed by Hong
Environmental Assessment for Data Centers Kingdom Kong. It is based on the data center score across the ten categories.
(BREAM)
European Code of Conduct by the European Europe 2008 This aids in goal planning, monitoring energy use, creating attention, and
Commission promoting energy-efficient best practises to improve data center energy
quality.
Blue Angel Eco-Label Germany 1978 This norm is provided to every environmentally aware organization that is
committed to implementing a long-term strategy to improve the resource
and energy usage of its data centre in relation to the IT services to be
distributed, as well as performing daily testing to improve data centre
operations.
International Standards Organization Europe 2011 This international standard outlines the Energy Management Systems
50001:2011 (EMS) standards that data centers must meet in order to formulate and
enforce an energy strategy, as well as set goals, priorities, and action
plans that take legal requirements and facts into account.

migration, virtualization, load balancing and workload with containers when distributed across cloud architectures
categorization. might be an intriguing future research topic.
Table 3 shows the research challenges in containers. Table 4 shows the research challenges in the domain of
Although operating system virtualization is becoming VM Migration.
more popular, it commonly adds layers of inference and Another critical area of research is VM aggregation and
abstraction beneath the application code to an already resource redistribution via VM migrations, with an
intensively layered software stack that contains assistance emphasis on both power monitoring and network overhead.
for legacy physical protocols and unimportant enhance- Considerations on VM placement that are purely focused
ments. If the current rate of layer inclusion persists (due to on increasing server resource usage and lowering power
the distributed presence of logic across various software usage may result in data centre designs that are neither
elements written in different languages), the problem will traffic-aware nor network efficient, resulting to more SLA
worsen in the coming years, as future software developers breaches. Consequently, VM allocation techniques that
will have to dig through multitudes of layers to debug even take into account both VM resource demands and inter-VM
the relatively simple applications. Developing a security- traffic load may be able to make more precise and eco-
aware scheduler to mitigate security concerns associated nomical placement decisions [151].

123
Cluster Computing (2023) 26:1845–1875 1867

Table 3 Research challenges in containers

Domain Problems & Solutions

Containers Containers pose a number of security threats like attack through the untrusted images. These images may contain backdoors. In
addition, the images may have configuration defects that may provide them unnecessary privileges that will allow the application
to have control over the container. The DoS attack on the containers can also create a serious security threat in which a vulnerable
application attacks the neighbouring containers and uses the large number of resources that affects the performance of the
container.
Using the trusted images and verifying the images with the signatures is one such solution available to counter security threats.
Periodic scanning of applications/ images and running applications with minimum privileges are some other measures that can be
taken.
The industry requires a comprehensive container orchestration architecture. Aside from those, there are a few other issues to
consider when it comes to container deployment and orchestration: During stateful container deployment and orchestration,
dynamic life-cycle management is a major concern. This is because stateful distributed services often have many dependencies
and dynamic implementation stages (for example: Kubernetes deployment has 21 steps). Furthermore, orchestration operators
must handle the orchestrator in real time, as well as deal with numerous updates, crashes, interface adjustments, scaling, and
other issues. Because of the following basic issues, multitenancy becomes even more difficult when multiple tenants with their
own conditions, isolation, and protection are included. It is critical to keep an eye on/resist any single tenant from monopolizing a
cluster’s services.
Advanced automation and orchestration systems are the most significant technologies for aiding IT professionals in operating
large-scale cloud data centres. Many off-the-shelf solutions are already accessible as open-source software, but it has been seen
that almost every organisation is creating its own bespoke orchestration solution to meet its specific use cases. Because of this
strategy, the container orchestrator market is extremely fragmented, making it difficult for new small IT businesses to implement
them in production.
The container networking poses a number of problems. In the containerized environment, there is no static IP as containers spin up
and down continuously. Traditional IP networking is inefficient and difficult to automate. Static addresses need to be manually
configured. Use of DHCP or anything similar must be done in order to address the issues automatically but in this case, service
discovery and address assignment might take several seconds or even minutes. That is insufficient in a containerized environment
that is constantly changing. Overlay networks, service discovery, IDs, and labels enable traffic to be routed across containerized
networks without the need for traditional IP networking

When apps like CPU and memory intensive are co-lo- and utilising the host as the client. As there are many VMs
cated on the same physical server, resource competition for in virtualization, proper isolation is critical for security.
some capabilities may occur, while others may be under- The isolation approach is commonly used in huge software
utilised. Furthermore, such resource contention will have a stacks. Under the direct supervision of cloud providers, it
significant impact on application performance, resulting in may be vulnerable to VMM vulnerability exploitation and
SLA breaches and economic loss. As a result, it is critical insider assaults. Utilization of verified thin virtualization
to comprehend the host’s behaviour and resource utilisa- instead of employing a conventional framework, a Trusted
tion patterns. Overhead for VM relocation and reconfigu- Computing Base (TCB) can solve this challenge.
ration might have a negative impact on scalability and data Table 6 shows the research challenges in the domain of
centre bandwidth consumption, as well as performance of load balancing.
the application. Because of this, VM strategies for place- Due to the sheer exponential development in demand for
ment and scheduling that are uninformed about VM cloud services, efficient utilisation of energy and process-
migration and reconfiguration overhead may significantly ing resources has become a major challenge. Load bal-
clog the network, induce unnoticed SLA breaches. Incor- ancing improves resource efficiency, quality, and energy
poration considering the expected migration overhead savings by optimally distributing the load in the data center
using placement techniques and VM placement optimiza- among diverse computing units. It has been observed that
tion and migration by balancing utilisation in terms of the methods under consideration frequently function to
network resources, migration overhead, and so on, aspects improve QoS, resource consumption, and energy conser-
of energy consumption are still to be explored. vation. Current LB algorithms have a number of flaws,
Table 5 shows the research challenges in virtualiza- such as energy and resource waste, inadequate frequencies
tion.The entire virtualization process increases the total management, and static impediments. As a result, there is a
complexity of the machine, making testing, compilation, lot of space for growth. More efficient and adaptable LB
and management difficult. This may be avoided by running algorithms should be developed to offer clients with
the emulated server solution on a desktop virtual machine excellent services at the lowest possible cost in order to

123
1868 Cluster Computing (2023) 26:1845–1875

Table 4 Research challenges in the domain of VM Migration

Domain Problems & Solutions

VM One of the important problems is to enhance the efficiency and effectiveness of the VM migration because it is widely used in the
Migration enterprise environments. Most of the work done in this domain is application-oblivious. It could be possible to design more
flexible migration strategies that reduce migration costs by disclosing the application’s characteristics and efficiency goals.
Research should be done for the different types of resources and the different types of the workloads.
Virtualized servers are intended to separate the underlying hardware from the applications that run on it. Workload migration is
made possible by this concept. Although hardware interruptions between the source and destination servers are uncommon,
they can inhibit a successful transfer.
It is critical to begin troubleshooting by assessing the server hardware and configuration. For example, in order for a workload
transfer to be successful, the source and destination servers must have the same CPU.
One of the main issues that the virtualization architecture must overcome is security. While Xen lacks adequate protection
countermeasures against tampering and exploitation threats, several techniques exist, that can be used to mitigate these
problems [149]. To provide a more stable platform for supporting VM migration, more sophisticated and collaborative defences
are needed.
50% of all x86 data center workloads are in the form of virtual machine (Gartner research) [150], and this percentage will
continue to rise in the coming years. This would result in a growth in the number of virtual machines (VMs) housed in data
centers, which will increase operational complexity. In order to identify reasonable trade-offs between costs and advantages of
VM migration, multiple competing goals and criteria related to efficiency and energy usage should be taken into account.
Finding a flexible approach for tracking and determining appropriate policies to guide migration decisions is an issue that
should be overcome. To resolve the issues of complexity and scalability, automated VM migration management techniques
must be specified. Large-scale VM implementation necessitates the creation of sophisticated migration management techniques.
If a task lacks the appropriate computing capacity, it cannot be migrated to a destination server. When the destination node does
not have enough CPU cores, memory space, or NIC ports, or when storage is limited, and cannot reserve resources for the new
demand, migration issues might arise. This is becoming a more prevalent issue as physical server numbers are decreasing and
workload consolidation levels are rising.
Workload migration between physical servers is a necessary element of a virtualized system, but the procedure is laden with
potential pitfalls. Factors such as hypervisor flaws, migration settings, unanticipated hardware requirements, network
connectivity difficulties and configuration errors, resource limitations, and SAN configurations can all work together to hinder
effective workload transfers.

Table 5 Research challenges in virtualization

Domain Problems & Solutions

Virtualization Virtualization consolidates multiple clients on a single physical machine making it a Single Point of Failure (SPOF). However,
along with this, the hypervisor supporting these multiple clients also becomes a SPOF. A bug in the hypervisor may affect
some or all of the clients. To ensure that individual component failures do not result in service loss, highly available systems
are configured without Single Points of Failure (SPOF). The basic approach to avoiding SPOFs is to offer duplicate
components for each required resource, allowing service to continue even if a component fails.
As multiple clients are on the same machine, side channel attacks are common between VMs. Attacks can be performed to
extract sensitive information. Performance degradation by resource exhaustion is some of the common security attacks seen in
this domain. Many solutions have been provided which can work at hardware or application level. Mitigation at the hypervisor
level, on the other hand, may be more advantageous because it covers all clients and needs no interaction between them.
The capacity to create as many virtual machines as needed may result in the development of more VMs than the business
requires. VM sprawl may appear to be innocuous, but it can worsen resource distribution issues by diverting resources to VMs
that aren’t even being utilised, while those that are being used and required suffer from reduced functionality. Businesses may
avoid VM sprawl by restricting the number of VMs that are genuinely needed and adding more as needed.
Nested Virtualization refers to running a hypervisor inside another. It finds many applications in the domain of testing, security
and fault tolerance. However, many of the hypervisors do not support this feature. The hypervisors face stability and
performance issues while supporting nested virtualization.
Congestion is a common and well-known problem with VMs. Prior to virtualization, a standalone executable on a central
computer would often only consume a small portion of the computer’s bandwidth usage. However, several VMs on a
virtualized server, each requiring network bandwidth, gradually exceeds the virtual environment’s NIC port (typically one in a
server). One way to resolve the congestion issue is by adding more NIC’s to the VM server. Another is by utilizing solutions
that balance VMs across multiple servers. One such example being VMware DRS cluster of ESXi hosts.

123
Cluster Computing (2023) 26:1845–1875 1869

optimise resource efficiency, energy conservation, and workload must be operational. As a result, cloud workload
production. Adaptive LB will enable for traffic control security and app service workload security are essentially
between quick activities, efficient resource consumption, different from desktop application safety. Workload safety
and a combination of centralised and distributed control is especially challenging in hybrid data centre designs that
mechanisms. Energy conservation is a key aspect in pro- use any from physical on-premises desktops to various
moting economic growth in situations when greater public cloud infrastructure as a service (IaaS) setting to
resource demand results from decreased resource container-based application architectures. Cloud workload
acquisition. security is particularly difficult because, when workloads
Table 7 shows the research challenges in workload travel between providers and hosts, responsibility for
categorization. securing the workload must be pooled. One of the signifi-
Workload characterisation is a trivial research issue due cant future research objectives is the integration of auto-
to the prevalence of workloads with distinctive, nontrivial, scaling techniques with IDS and IPS systems to better
and poorly understood qualities. The workloads related to manage Distributed Denial-of-Service (DDoS) and Yo–Yo
fog and vehicular clouds, big data, anonymous social net- attacks. To deal with malicious behaviour, auto scaling
works, and sensor networks, to mention a few, are systems often transform DDoS attacks to Economic Denial
addressed in open research problems. of Sustainability (EDoS) attacks. Recognizing DDoS
The practise of safeguarding workloads that migrate workload from user workload is an unresolved topic that
among cloud environments is known as cloud workload should be addressed in future study.
protection. For a cloud-based application to function cor-
rectly and without posing security problems, the entire

Table 6 Research challenges in the domain of load balancing

Domain Problems & Solutions

Load One of the most difficult challenges in the load-balancing algorithm is determining which technique is utilised to assess the
balancing workload of a certain node. The total number of processes operating on computer are used as a measurement of workload in the
majority of load balancing algorithms. Load depends on many parameters like: no of processes, demands of the processes
running, instruction mix, architecture/speed of the processor, CPU utilization etc. Research should focus on all the parameters
before proceeding in the direction of designing and developing load-balancing algorithms. Some other factors that play an
important role in research in this domain include location policy for load-balancing algorithms (Threshold based, bidding
based, pairing etc.), process transfer policy for load-balancing algorithms (static threshold vs. dynamic threshold) etc.
Many load balancing algorithms targeting a single and multiple objectives have been proposed by researchers. However, many
of these do not consider algorithm complexity. Similarly, the load balancing process in data centers requires VM migrations,
where migration cost is involved. Migration cost has not been considered as an important metric in many proposed load-
balancing algorithms. The following metrics have been considered more frequently: latency, resource efficiency, computation
time, interoperability, and communication cost. The less considered metrics include: throughput, overhead, fault tolerance,
degree of balance, migration time etc.

Table 7 Research challenges in workload categorization

Domain Problems & Solutions

Workload Workload is categorised based on many factors such as submitting time, completion time, inter-session interval, and so
categorization on, however there are several types of characteristics that need to be addressed during workload classification.
Due to the huge magnitude and complexity of the workload, in-depth statistical analysis and workload classification
inside a large-scale production cloud is difficult.
Due to the behavioural characteristics of workload in the setting of cloud computing, there is a shortage of techniques to
define the workload.

123
1870 Cluster Computing (2023) 26:1845–1875

11 Conclusion Declarations

Data centres’ energy usage is becoming a significant Conflict of interest The authors have no conflicts of interest to declare
that are relevant to the content of this article.
problem. Advances in server equipment technologies and
growing demand for processing power have resulted in Ethical approval This paper does not contain any studies with human
higher workload and hence has resulted in higher data participants or animals performed by any of the authors.
centre power usage. Data centers are the core of today’s
Informed consent Informed consent was obtained from all individual
Internet and cloud computing networks. Data centers’ participants included in the study.
growing demand for electrical energy necessitates
accounting for the immense amount of energy they con-
sume. In this context, data center energy modelling and References
prediction are critical. This survey article conducted a
thorough examination of the present status of software 1. Fiona, B., Ballarat, C.: International Review of Energy Effi-
ciency in Data Centres Acknowledgements. (2021)
solutions that aids in data centre power consumption
2. Koot, M., Wijnhoven, F.: Usage impact on data center electricity
reduction. The paper is divided into five separate sections. needs: A system dynamic forecasting model. Appl. Energy. 291,
First, approaches for applying software virtualization are 116798 (2021)
described, followed by ways for applying operating system 3. Analysts, G.I.: I. Internet Data Centers - Global Market Tra-
jectory & Analytics. (2021)
virtualization. Furthermore, methodologies used in data
4. Chester, S.: What Is Power Usage Effectiveness (PUE)? (2019).
centres as well as other problem-solving approaches for https://www.colocationamerica.com/blog/what-is-pue
energy saving have been examined. Environmental factors 5. The future of: data center power consumption – 5 essential facts
have also been studied in order to reduce electricity use. | Danfoss. https://www.danfoss.com/en/about-danfoss/insights-
for-tomorrow/integrated-energy-systems/data-center-power-
The article also covers the significance of containerization
consumption/
in decreasing data centre power consumption and con- 6. US20080086731A1 - Method: and system for managing
cludes with research challenges for sustainable data centre resources in a data center - Google Patents. https://patents.goo
building. The container technology is rising and requires gle.com/patent/US20080086731
7. Kliazovich, D., Bouvry, P., Khan, S.U. DENS: Data center
more research in the domain of energy efficiency. Con-
energy-efficient network-aware scheduling. Proceedings – 2010
tainers, when properly built, allow a host to utilise nearly IEEE/ACM International Conference on Green Computing and
all available resources. Isolated containers can function Communications, GreenCom 2010 IEEE/ACM International
independently of other containers, allowing a single host to Conference on Cyber, Physical and Social Computing, CPSCom
2010 69–75 (2010) https://doi.org/10.1109/GREENCOM-
perform many functions. The future work includes devel-
CPSCOM.2010.31
opment of an efficient technique for placement of con- 8. Heller, B., et al. ElasticTree: Saving Energy in Data Center
tainers/tasks onto physical machines considering: CPU Networks. IN NSDI (2010)
multicores, memory, storage & network together. Con- 9. Khargharia, B., et al. Autonomic power & performance man-
agement for large-scale data centers. Proceedings – 21st Inter-
tainer migration technique will also be developed to reduce
national Parallel and Distributed Processing Symposium,
energy usage while preserving needed service quality IPDPS 2007; Abstracts and CD-ROM (2007). https://doi.org/10.
(QoS). 1109/IPDPS.2007.370510
10. SRCMap: Energy Proportional Storage Using Dynamic Con-
solidation. https://www.researchgate.net/publication/
Author contributions All authors contributed to the study conception 221353706_SRCMap_Energy_Proportional_Storage_Using_
and design. Avita Katal, Susheela Dahiya and Tanupriya Choudhury Dynamic_Consolidation
performed material preparation, data collection and analysis. Avita 11. Baliga, J., Ayre, R.W.A., Hinton, K., Tucker, R.S. Green cloud
Katal wrote the first draft of the manuscript and all authors com- computing: Balancing energy in processing, storage, and trans-
mented on previous versions of the manuscript. All authors read and port. Proceedings of the IEEE 99, 149–167 (2011)
approved the final manuscript. 12. Beloglazov, A., Buyya, R., Lee, Y.C., Zomaya, A.A.: Taxon-
omy and survey of energy-efficient data centers and cloud
Funding No funding was received to assist with the preparation of computing systems. Adv. Computers 82, 47–111 (2011)
this manuscript. 13. Shuja, J., et al.: Survey of techniques and architectures for
designing energy-efficient data centers. IEEE Syst. J. 10,
Data Availability Data sharing not applicable to this article as no 507–519 (2016)
datasets were generated or analyzed during the current study. 14. Dayarathna, M., Wen, Y., Fan, R.: Data center energy con-
sumption modeling: A survey. IEEE Commun. Surv. Tutorials.
Code Availability Not Applicable. 18, 732–794 (2016)

123
Cluster Computing (2023) 26:1845–1875 1871

15. You, X., Li, Y., Zheng, M., Zhu, C., Yu, L.: A survey and Performance Computing and Simulation, HPCS 292–299 (2018)
taxonomy of energy efficiency relevant surveys in cloud-related https://doi.org/10.1109/HPCS.2018.00057
environments. IEEE Access. 5, 14066–14078 (2017) 35. Tiwari, N., Bellur, U., Sarkar, S., Indrawan, M.: Optimizing
16. Katal, A., Dahiya, S., Choudhury, T.: Energy efficiency in cloud MapReduce for energy efficiency. Software: Pract. Experience.
computing data center: a survey on hardware technologies. 48, 1660–1687 (2018)
Cluster Comput. 2021, 1–31 (2021). https://doi.org/10.1007/ 36. Jiang, D., Wang, Y., Lv, Z., Wang, W., Wang, H.: An Energy-
S10586-021-03431-Z Efficient Networking Approach in Cloud Services for IIoT
17. LiTao, Kurian, J.: Run-time modeling and estimation of oper- Networks. IEEE J. Sel. Areas Commun. 38, 928–941 (2020)
ating system power consumption. ACM SIGMETRICS Perfor- 37. Vakilinia, S.: Energy efficient temporal load aware resource
mance Evaluation Review. 31, 160–171 (2003) allocation in cloud computing datacenters. J. Cloud Comput. 7,
18. Herzog, B., Hügel, F., Reif, S., Hönig, T., Schröder-Preikschat, 1–24 (2018)
W.: Automated selection of energy-efficient operating system 38. Barrett, D., Kipper, G.: How virtualization happens. Virtualiz.
configurations. Energy (2021). https://doi.org/10.1145/3447555. Forensics. (2010). https://doi.org/10.1016/B978-1-59749-557-8.
3465327 00001-1
19. Scordino, C., Abeni, L., Lelli, J.: Energy-aware real-time 39. Cuadrado-Cordero, I., Orgerie, A.C., Menaud, J.M. Comparative
scheduling in the linux kernel. Proc. ACM Sympos. Appl. experimental analysis of the quality-of-service and energy-effi-
Comput. (2018). https://doi.org/10.1145/3167132.3167198 ciency of VMs and containers’ consolidation for cloud appli-
20. Embedded Data Centers: | Products | ENERGY STAR. https:// cations. 25th International Conference on Software,
www.energystar.gov/products/office_equipment/data_center_ Telecommunications and Computer Networks, SoftCOM 2017
storage/data_center_energy_efficiency/embedded_data_centers (2017) https://doi.org/10.23919/SOFTCOM.2017.8115516
21. BuschhoffMarkus, F.R., SpinczykOlaf: Energy-aware device 40. Huang, D., Wu, H.: Virtualization. Mob. Cloud Comput.
drivers for embedded operating systems. ACM SIGBED (2018). https://doi.org/10.1016/B978-0-12-809641-3.00003-X
Review. 16, 8–13 (2019) 41. Ramchandra Desai, P.A. Survey of Performance Comparison
22. Levy, A., et al. Multiprogramming a 64 kB Computer Safely and between Virtual Machines and Containers. Int. J. Comput. Sci.
Efficiently. Proceedings of the 26th Symposium on Operating Eng. (2016)
Systems Principles (2017) https://doi.org/10.1145/3132747 42. Piraghaj, S.F., Dastjerdi, A.V., Calheiros, R.N., Buyya, R.A.
23. Kang, D.G.I.S.T., Alian, K.-D., Kim, M., Huh, D.G.I.S.T.D. Framework and Algorithm for Energy Efficient Container
KAIST, J. & Sung Kim, N. VIP: Virtual Performance-State for Consolidation in Cloud Data Centers. Proceedings – 2015 IEEE
Efficient Power Man-agement of Virtual Machines. Proceedings International Conference on Data Science and Data Intensive
of the ACM Symposium on Cloud Computing ’18 (2021) Systems; 8th IEEE International Conference Cyber, Physical
24. Xiao, P., Ni, Z., Liu, D., Hu, Z.: Improving the energy-efficiency and Social Computing; 11th IEEE International Conference on
of virtual machines by I/O compensation. J. Supercomputing. Green Computing and Communications and 8th IEEE Inte
77, 11135–11159 (2021) 368–375 (2015). https://doi.org/10.1109/DSDIS.2015.67
25. Prabhakaran, G., Selvakumar, S.: An diverse approach on virtual 43. Beloglazov, A., Buyya, R.: Optimal online deterministic algo-
machines administration and power control in multi-level rithms and adaptive heuristics for energy and performance
implicit servers. J. Ambient Intell. Humaniz. Comput. (2021). efficient dynamic consolidation of virtual machines in Cloud
https://doi.org/10.1007/S12652-021-03013-2 data centers. Concurrency Comput. Pract. Experience. 24,
26. Ho, T.T.N., Gribaudo, M., Pernici, B.: Characterizing Energy 1397–1420 (2012)
per Job in Cloud Applications. Electron. 2016. 5, 90 (2016) 44. Nath, S.B., Addya, S.K., Chakraborty, S., Ghosh, S.K. Green
27. Kumar, S., Buyya, R.: Green cloud computing and environ- Containerized Service Consolidation in Cloud. IEEE Interna-
mental sustainability harnessing green. Principles Practices tional Conference on Communications (2020)
(2012). https://doi.org/10.1002/9781118305393.CH16 45. Ferdaus, M.H., Murshed, M., Calheiros, R.N., Buyya, R. Virtual
28. Tchana, A., et al. Software consolidation as an efficient energy machine consolidation in cloud data centers using ACO meta-
and cost saving solution for a SaaS/PaaS cloud model. Lecture heuristic. Lecture Notes in Computer Science (including sub-
Notes Comput. Sci. 9233, 305–316 (2015) series Lecture Notes in Artificial Intelligence and Lecture Notes
29. Samrajesh, M.D., Gopalan, N.P. Component based energy aware in Bioinformatics) 8632, 306–317 (2014)
multi-tenant application in software as-a service. 15th Interna- 46. Shi, T., Ma, H., Chen, G. Energy-Aware Container Consolida-
tional Conference on Advanced Computing Technologies, tion Based on PSO in Cloud Data Centers. IEEE Congress on
ICACT 2013 (2013). https://doi.org/10.1109/ICACT.2013. Evolutionary Computation, CEC 2018 - Proceedings (2018).
6710502 https://doi.org/10.1109/CEC.2018.8477708
30. Czarnul, P., Proficz, J., Krzywaniak, A. Energy-Aware High- 47. Tan, B., Ma, H., Mei, Y.A., Hybrid Genetic Programming
Performance Computing: Survey of State-of-the-Art Tools, Hyper-Heuristic Approach for Online Two-level Resource
Techniques, and Environments. Scientific Programming (2019) Allocation in Container-based Clouds. 2019 IEEE Congress on
31. Ho, T.T.N., Pernici, B.: A data-value-driven adaptation frame- Evolutionary Computation, CEC 2019 - Proceedings 2681–2688
work for energy efficiency for data intensive applications in (2019). https://doi.org/10.1109/CEC.2019.8790220
clouds. IEEE Conf. Technol. Sustainabil. (2015). https://doi.org/ 48. Fan, X., Weber, W.D., Barroso, L.A. Power provisioning for a
10.1109/SUSTECH.2015.7314320 warehouse-sized computer. Proceedings - International Sympo-
32. Malik, M., et al. ECoST: Energy-efficient co-locating and self- sium on Computer Architecture 13–23 (2007). https://doi.org/
tuning mapreduce applications. ACM International Conference 10.1145/1250662.1250665
Proceeding Series (2019). https://doi.org/10.1145/3337821. 49. Chen, F., Zhou, X., Shi, C. The container deployment strategy
3337834 based on stable matching. IEEE 4th International Conference on
33. Miyazaki, T. Bayesian Optimization of HPC Systems for Energy Cloud Computing and Big Data Analytics, ICCCBDA 2019
Efficiency. Lecture Notes Comput. Sci. 10876: 44–62 (2018) 215–221 (2019). https://doi.org/10.1109/ICCCBDA.2019.
34. Reddy Basireddy, K., Wachter, E.W., Al-Hashimi, B.M., Mer- 8725707
rett, G. Workload-Aware runtime energy management for HPC
Systems. Proceedings – 2018 International Conference on High

123
1872 Cluster Computing (2023) 26:1845–1875

50. Al-Moalmi, A., Luo, J., Salah, A., Li, K., Yin, L.: A whale 67. Ruchi, T. & Avita Katal. An Optimized Time Series based Two
optimization system for energy-efficient container placement in Phase Strategy Pre-Copy Algorithm for Live Virtual Machine
data centers. Expert Syst. Appl. 164, 113719 (2021) Migration. Internat. J. Eng. Res. V6, (2017)
51. Raı̈s, I., Orgerie, A.-C., Quinson, M., Lefèvre, L.: Quantifying 68. Chashoo, S.F., Malhotra, D. VM-Mig-framework: Virtual
the impact of shutdown techniques for energy-efficient data machine migration with and without ballooning. PDGC
centers. Concurrency and Computation: Practice and Experi- 2018–2018 5th International Conference on Parallel, Distributed
ence. 30, e4471 (2018) and Grid Computing 368–373 (2018). https://doi.org/10.1109/
52. Benoit, A., Lefèvre, L., Orgerie, A.-C., Raı̈s, I. Reducing the PDGC.2018.8745993
energy consumption of large-scale computing systems through 69. Sagana, C., Geetha, M., Suganthe, R.C. Performance enhance-
combined shutdown policies with multiple constraints (2017). ment in live migration for cloud computing environments. Int.
https://doi.org/10.1177/1094342017714530 Conf. Informat. Commun. Embedded Syst, ICICES 2013
53. Marotta, A., Avallone, S., Kassler, A.A.: Joint power efficient 361–366 (2013). https://doi.org/10.1109/ICICES.2013.6508339
server and network consolidation approach for virtualized data 70. Rajapackiyam, E., Subramanian, A.V., Arumugam, U.: Com-
centers. Comput. Netw. 130, 65–80 (2018) mons Attribution (CC-BY) 3.0 license. J. Comput. Sci. (2020).
54. Marahatta, A., et al.: Classification-based and energy-efficient https://doi.org/10.3844/jcssp.2020.543.550
dynamic task scheduling scheme for virtualized cloud data 71. Patel, M., Chaudhary, S., Garg, S. Machine learning based sta-
center. IEEE Trans. Cloud Comput. 1–1 (2019). https://doi.org/ tistical prediction model for improving performance of live
10.1109/TCC.2019.2918226 virtual machine migration. J. Eng. (United Kingdom) (2016)
55. Cioara, T., Anghel, I., Salomie, I.: Methodology for energy 72. Tseng, F.H., Chen, X., Chou, L., Chao, H.C., Chen, S.: Support
aware adaptive management of virtualized data centers. Energ. vector machine approach for virtual machine migration in cloud
Effi. 10, 475–498 (2017) data center. Multimedia Tools Appl. 74, 3419–3440 (2015)
56. Hussain, M., et al.: Energy and performance-efficient task 73. Jo, C., Cho, Y., Egger, B. A machine learning approach to live
scheduling in heterogeneous virtualized cloud computing. Sus- migration modeling. 14, (2017)
tainable Computing: Informatics and Systems. 30, 100517 74. Kansal, N.J., Chana, I.: Energy-aware virtual machine migration
(2021) for cloud computing a firefly optimization approach. J. Grid
57. Shukla, R., Gupta, R.K., Kashyap, R.A.: Multiphase pre-copy Comput. 14, 327–345 (2016)
strategy for the virtual machine migration in cloud. Smart Innov. 75. Rodrigues, T.G., Suto, K., Nishiyama, H., Kato, N. A PSO
Syst. Technol. 104, 437–446 (2019) model with VM migration and transmission power control for
58. Jalaei, N., Safi-Esfahani, F. VCSP: virtual CPU scheduling for low Service Delay in the multiple cloudlets ECC scenario. IEEE
post-copy live migration of virtual machines. International International Conference on Communications (2017). https://
Journal of Information Technology 2020 13:1 13, 239–250 doi.org/10.1109/ICC.2017.7996358
(2020) 76. Hossain, M.K., Rahman, M., Hossain, A., Rahman, S.Y., Islam,
59. Kaur, R.A.: Hybrid approach for virtual machine migration in M.M. Active Idle Virtual Machine Migration Algorithm-a new
cloud computing environment. Int. J. Adv. Res. Comput. Sci. Ant Colony Optimization approach to consolidate Virtual
Softw. Eng. 7, 30 (2017) Machines and ensure Green Cloud Computing. ETCCE -
60. Hines, M.R., Gopalan, K. Post-copy based live virtual machine International Conference on Emerging Technology in Comput-
migration using pre-paging and dynamic self-ballooning. Pro- ing, Communication and Electronics (2020). https://doi.org/10.
ceedings of the 2009 ACM SIGPLAN/SIGOPS International 1109/ETCCE51779.2020.9350915
Conference on Virtual Execution Environments, VEE’09 51–60 77. ZhengQinghua, et al.: Virtual machine consolidated placement
(2009). https://doi.org/10.1145/1508293.1508301 based on multi-objective biogeography-based optimization.
61. Nashaat, H., Ashry, N., Rizk, R. Smart elastic scheduling Future Generation Comput. Sys. 54, 95–122 (2016)
algorithm for virtual machine migration in cloud computing. 78. Sha, J., et al.: A method for virtual machine migration in cloud
The Journal of Supercomputing 75, 3842–3865 (2019) computing using a collective behavior-based metaheuristics
62. Liu, H., Jin, H., Liao, X., Yu, C., Xu, C.Z.: Live virtual machine algorithm. Concurr. Comput. 32, e5441 (2020)
migration via asynchronous replication and state synchroniza- 79. Ghosh, S., Banerjee, C. Dynamic time quantum priority based
tion. IEEE Trans. Parallel Distrib. Syst. 22, 1986–1999 (2011) round robin for load balancing in cloud environment. Proceed-
63. Celesti, A., Tusa, F., Villari, M., Puliafito, A. Improving virtual ings – 2018 4th IEEE International Conference on Research in
machine migration in federated cloud environments. Proceed- Computational Intelligence and Communication Networks,
ings – 2nd International Conference on Evolving Internet, ICRCICN 33–37 (2018) https://doi.org/10.1109/ICRCICN.
Internet 1st International Conference on Access Networks, 2018.8718694
Services and Technologies, Access 2010 61–67 (2010). https:// 80. Falisha, I.N., Purboyo, T.W., Latuconsina, R.: Experimental
doi.org/10.1109/INTERNET.2010.20 model for load balancing in cloud computing using equally
64. Bloch, T., Sridaran, R., Prashanth, C.: Understanding Live spread current execution load algorithm. Int. J. Appl. Eng. Res.
Migration Techniques Intended for Resource Interference Min- 13, 1134–1138 (2018)
imization in Virtualized Cloud Environment. Adv. Intell. Syst. 81. Patel, D., Rajawat, A. Efficient Throttled Load Balancing
Comput. 654, 487–497 (2018) Algorithm in Cloud Environment.International Journal of
65. Kella, A., Belalem, G.: A stable matching algorithm for VM Modern Trends in Engineering and Research (2015)
migration to improve energy consumption and QOS in cloud 82. Manakattu, S.S., Kumar, S.D.M. An improved biased random
infrastructures. Cloud Technology: Concepts, Methodologies, sampling algorithm for load balancing in cloud based systems.
Tools, and Applications. 2, 606–623 (2014) ACM International Conference Proceeding Series 459–462
66. Hu, B., Lei, Z., Lei, Y., Xu, D., Li, J. A time-series based (2012). https://doi.org/10.1145/2345396.2345472
precopy approach for live migration of virtual machines. Pro- 83. Chen, H., Wang, F., Helian, N., Akanmu, G. User-priority gui-
ceedings of the International Conference on Parallel and Dis- ded min-min scheduling algorithm for load balancing in cloud
tributed Systems - ICPADS 947–952 (2011). https://doi.org/10. computing. National Conference on Parallel Computing Tech-
1109/ICPADS.2011.19 nologies, PARCOMPTECH (2013). https://doi.org/10.1109/
PARCOMPTECH.2013.6621389

123
Cluster Computing (2023) 26:1845–1875 1873

84. Hung, T.C., Hy, P.T., Hieu, L.N., Phi, N.X. MMSIA: Improved Optimization, ICMSAO 2019 (2019) doi: (2019). https://doi.org/
max-min scheduling algorithm for load balancing on cloud 10.1109/ICMSAO.2019.8880305
computing. ACM International Conference Proceeding Series 102. Yousif, S.A., Al-Dulaimy, A. Clustering Cloud Workload Tra-
60–64 (2019). https://doi.org/10.1145/3310986.3311017 ces to Improve the Performance of Cloud Data Centers. Pro-
85. Ananthakrishnan, B. An-Efficient-Approach-for-Load-Balanc- ceedings of the World Congress on Engineering (2017)
ing-in-Cloud-Environment.doc. Int. J. Sci. Eng. Res. 6, (2015) 103. Zhao, X., Yin, J., Chen, Z., He, S. Workload classification
86. Banerjee, A., Chatterjee, G., Chakraborty, D., Majumder, S.: model for specializing virtual machine operating system. IEEE
Cluster based intelligent load balancing algorithm applied in International Conference on Cloud Computing, CLOUD
cloud computing using KNN. SSRN Electron. J. (2019). https:// 343–350 doi: (2013). https://doi.org/10.1109/CLOUD.2013.144
doi.org/10.2139/SSRN.3503518 104. Li, S., Ben-Nun, T., Girolamo, S., di, Alistarh, D., Hoefler, T.
87. Kaur, A., Kaur, B., Singh, P., Devgan, M.S., Toor, H.K.: Load Taming Unbalanced Training Workloads in Deep Learning with
balancing optimization based on deep learning approach in Partial Collective Operations. (2020)
cloud environment. Int. J. Inform. Technol. Comput. Sci. 12, 105. Mathematics, K.K.-T.J. of C. and & undefined. Forecasting of
8–18 (2020) Cloud Computing Services Workload using Machine Learning.
88. Chen, J. Machine learning for load balancing in the linux kernel. turcomat.org 12, 4841–4846 (2021). (2021)
Proceedings of the 11th ACM SIGOPS Asia-Pacific Workshop 106. Cetinski, K., Juric, M.B.: AME-WPC: Advanced model for
on Systems 20 efficient workload prediction in the cloud. J. Netw. Comput.
89. Mondal, B., Choudhury, A. Simulated annealing (SA) based Appl. 55, 191–201 (2015)
load balancing strategy for cloud computing. Int. J. Comput. Sci. 107. Shekhawat, V.S., Gautam, A., Thakrar, A. Datacenter Workload
Informat. Technologies (2015) Classification and Characterization: An Empirical Approach.
90. Singhal, U., Jain, S.: An analysis of swarm intelligence based 13th International Conference on Industrial and Information
load balancing algorithms in a cloud computing environment. Systems, ICIIS 2018 - Proceedings 1–7 (2018) doi: (2018).
Int. J. Hybrid Inform. Technol. 8, 249–256 (2015) https://doi.org/10.1109/ICIINFS.2018.8721402
91. Gupta, A., Garg, R. Load Balancing Based Task Scheduling 108. Sun, Q., Tan, Z., Zhou, X.: Workload prediction of cloud
with ACO in Cloud Computing. International Conference on computing based on SVM and BP neural networks. J. Intell.
Computer and Applications, ICCA 2017 174–179 (2017) doi: Fuzzy Syst. 39, 2861–2867 (2020)
(2017). https://doi.org/10.1109/COMAPP.2017.8079781 109. Kumar, A.S., Mazumdar, S. Forecasting HPC workload using
92. Acharya, J., Mehta, M., Saini, B. Particle swarm optimization ARMA models and SSA. Proceedings – 2016 15th International
based load balancing in cloud computing. Proceedings of the Conference on Information Technology, ICIT 294–297 (2017)
International Conference on Communication and Electronics doi: (2016). https://doi.org/10.1109/ICIT.2016.52
Systems, ICCES (2016) doi: (2016). https://doi.org/10.1109/ 110. Barati, M., Sharifian, S.: A hybrid heuristic-based tuned support
CESYS.2016.7889943 vector regression model for cloud load prediction. J. Super-
93. Ullah, A., Nawi, N.M., Uddin, J., Baseer, S., Rashed, A.H.: computing. 71, 4235–4259 (2015)
Artificial bee colony algorithm used for load balancing in cloud 111. Zhong, W., Zhuang, Y., Sun, J., Gu, J. A load prediction model
computing: review. IAES Int. J. Artif. Intell. (IJ-AI). 8, 156–167 for cloud computing using PSO-based weighted wavelet support
(2019) vector machine. Applied Intelligence 2018 48:11 48, 4072–4083
94. Jena, U.K., Das, P.K., Kabat, M.R.: Hybridization of meta- (2018)
heuristic algorithm for load balancing in cloud computing 112. Yang, Q., et al. Multi-step-ahead host load prediction using
environment. J. King Saud Univ. - Comput. Inform. Sci. (2020). autoencoder and echo state networks in cloud computing. The
doi:https://doi.org/10.1016/J.JKSUCI.2020.01.012 Journal of Supercomputing 2015 71:8 71, 3037–3053 (2015)
95. Sharma, S., Luhach, A., Kr, Sheik Abdhullah, S. An Optimal 113. Tian, C., et al.: Minimizing Content Reorganization and Toler-
Load Balancing Technique for Cloud Computing Environment ating Imperfect Workload Prediction for Cloud-Based Video-
using Bat Algorithm.Indian Journal of Science and Technol- on-Demand Services. IEEE Trans. Serv. Comput. 9, 926–939
ogy9, (2016) (2016)
96. Crow Search based Scheduling Algorithm for Load Balancing in 114. Zhang, Q., Yang, L.T., Yan, Z., Chen, Z., Li, P.: An Efficient
Cloud Environment: Int. J. Innovative Technol. Exploring Eng. Deep Learning Model to Predict Cloud Workload for Industry
8, 1058–1064 (2019) Informatics. IEEE Trans. Industr. Inf. 14, 3170–3178 (2018)
97. Wang, Q., Liu, D. Research on Load Balancing Method in 115. Li, S. A workload prediction-based multi-VM provisioning
Cloud Computing. Proceedings of IEEE 3rd Advanced Infor- mechanism in cloud computing. 1–6 (2013)
mation Technology, Electronic and Automation Control Con- 116. Jiang, J., Lu, J., Zhang, G., Long, G. Optimal cloud resource
ference, IAEAC 2018 1489–1493 (2018) doi: (2018). https://doi. auto-scaling for web applications. Proceedings – 13th IEEE/
org/10.1109/IAEAC.2018.8577591 ACM International Symposium on Cluster, Cloud, and Grid
98. Hashem, W., Nashaat, H., Rizk, R.: Honey bee based load Computing, CCGrid 2013 58–65 doi: (2013). https://doi.org/10.
balancing in cloud computing. KSII Trans. Internet Inf. Syst. 11, 1109/CCGRID.2013.73
5694–5711 (2017) 117. Jheng, J.J., Tseng, F.H., Chao, H.C., Chou, L. der. A novel VM
99. Makasarwala, H.A., Hazari, P. Using genetic algorithm for load workload prediction using grey forecasting model in cloud data
balancing in cloud computing. Proceedings of the 8th Interna- center. International Conference on Information Networking
tional Conference on Electronics, Computers and Artificial 40–45 doi: (2014). https://doi.org/10.1109/ICOIN.2014.6799662
Intelligence, ECAI (2017) doi: (2016). https://doi.org/10.1109/ 118. Kluge, F., Uhrig, S., Mische, J., Satzger, B., Ungerer, T.
ECAI.2016.7861166 Dynamic workload prediction for soft real-time applications.
100. Derdus, K.M., Omwenga, V., Ogao, P. Statistical Techniques for Proceedings – 10th IEEE International Conference on Com-
Characterizing Cloud Workloads: A Survey.International Jour- puter and Information Technology, CIT- 7th IEEE International
nal of Computer and Information Technology2279–0764(2019) Conference on Embedded Software and Systems, ICESS-2010,
101. Ismaeel, S., Al-Khazraji, A., Miri, A. An efficient workload ScalCom-2010 1841–1848 (2010) doi: (2010). https://doi.org/
clustering framework for large-scale data centers. 8th Interna- 10.1109/CIT.2010.317
tional Conference on Modeling Simulation and Applied

123
1874 Cluster Computing (2023) 26:1845–1875

119. Qazi, K., Li, Y., Sohn, A. PoWER - Prediction of workload for Minimizing Resource Wastage. Commun. Comput. Inform. Sci.
energy efficient relocation of virtual machines. Proceedings of 488, 153–164 (2014)
the 4th Annual Symposium on Cloud Computing, SoCC 2013 135. Pires, F.L., Barán, B. Multi-objective virtual machine placement
doi: (2013). https://doi.org/10.1145/2523616.2525938 with service level agreement: A memetic algorithm approach.
120. Hu, Y., Deng, B., Peng, F., Wang, D. Workload prediction for Proceedings – 2013 IEEE/ACM 6th International Conference on
cloud computing elasticity mechanism. Proceedings of IEEE Utility and Cloud Computing, UCC 2013 203–210 doi: (2013).
International Conference on Cloud Computing and Big Data https://doi.org/10.1109/UCC.2013.44
Analysis, ICCCBDA 2016 244–249 (2016) doi: (2016). https:// 136. Li, X.K., Gu, C.H., Yang, Z.P., Chang, Y.H. Virtual machine
doi.org/10.1109/ICCCBDA.2016.7529565 placement strategy based on discrete firefly algorithm in cloud
121. Lyu, H., et al. Load forecast of resource scheduler in cloud environments. 12th International Computer Conference on
architecture. PIC - Proceedings of the 2016 IEEE International Wavelet Active Media Technology and Information Processing,
Conference on Progress in Informatics and Computing 508–512 ICCWAMTIP (2015). https://doi.org/10.1109/ICCWAMTIP.
(2017) doi: (2016). https://doi.org/10.1109/PIC.2016.7949553 2015.7493907
122. Zhang, L., Zhang, Y., Jamshidi, P., Xu, L., Pahl, C. Workload 137. Abdel-Basset, M., Abdle-Fatah, L., Sangaiah, A.K. An
patterns for quality-driven dynamic cloud service configuration improved Lévy based whale optimization algorithm for band-
and auto-scaling. Proceedings – 2014 IEEE/ACM 7th Interna- width-efficient virtual machine placement in cloud computing
tional Conference on Utility and Cloud Computing, UCC 2014 environment. Cluster Computing 22, 8319–8334 (2018)
156–165 doi: (2014). https://doi.org/10.1109/UCC.2014.24 138. Gharehpasha, S., Masdari, M., Jafarian, A. (2020) Power effi-
123. Cao, J., Fu, J., Li, M., Chen, J.: CPU load prediction for cloud cient virtual machine placement in cloud data centers with a
environment based on a dynamic ensemble model. Software: discrete and chaotic hybrid optimization algorithm. Cluster
Pract. Experience. 44, 793–804 (2014) Computing 24, 1293–1315
124. Hu, R., Jiang, J., Liu, G., Wang, L., KSwSVR: A new load 139. Li, X., Qian, Z., Lu, S., Wu, J.: Energy efficient virtual machine
forecasting method for efficient resources provisioning in cloud. placement algorithm with balanced and improved resource uti-
in Proceedings - IEEE 10th International Conference on Ser- lization in a data center. Math. Comput. Model. 58, 1222–1235
vices Computing, SCC 2013 120–127 doi: (2013). https://doi. (2013)
org/10.1109/SCC.2013.67 140. Jamali, S., Malektaji, S., Analoui, M.: An imperialist competi-
125. Janardhanan, D., Barrett, E. CPU workload forecasting of tive algorithm for virtual machine placement in cloud comput-
machines in data centers using LSTM recurrent neural networks ing. J. Experimental Theoretical Artif. Intell. 29, 575–596
and ARIMA models. 12th International Conference for Internet (2017)
Technology and Secured Transactions, ICITST 2017 55–60 141. Baalamurugan, K.M., Vijay Bhanu, S.: A multi-objective krill
(2018) doi: (2017). https://doi.org/10.23919/ICITST.2017. herd algorithm for virtual machine placement in cloud com-
8356346 puting. J. Supercomputing. 76, 4525–4542 (2020)
126. Fard, A.K., Akbari-Zadeh, M.R.: A hybrid method based on 142. Laganà, D., Mastroianni, C., Meo, M., Renga, D. Reducing the
wavelet, ANN and ARIMA model for short-term load fore- operational cost of cloud data centers through renewable energy.
casting. J. Experimental Theoretical Artif. Intell. 26, 167–182 Algorithms 11 (2018)
(2014) 143. Khalil, M.I.K., Ahmad, I., Almazroi, A.A.: Energy Efficient
127. Usmani, Z., Singh, S.A.: Survey of Virtual Machine Placement Indivisible Workload Distribution in Geographically Distributed
Techniques in a Cloud Data Center. Phys. Procedia. 78, 491–498 Data Centers. IEEE Access. 7, 82672–82680 (2019)
(2016) 144. Abu Bakar Siddik, M., Shehabi, A., Marston, L.: The environ-
128. Yu, Y., Gao, Y.: Constraint Programming-Based Virtual mental footprint of data centers in the United States. Environ.
Machines Placement Algorithm in Datacenter. IFIP Adv. Res. Lett. 16, 64017 (2021)
Inform. Communication Technol. 385 AICT, 295–304 (2012) 145. Improving Data Center Power Consumption & Energy Effi-
129. Lin, M.-H., Tsai, J.-F., Hu, Y.-C., Su, T.-H. Optimal Allocation ciency:. https://www.vxchnge.com/blog/growing-energy-
of Virtual Machines in Cloud Computing. Symmetry Vol. 10, demands-of-data-centers
Page 756 10, 756 (2018). (2018) 146. Solar Powered Datacenters Drive Sustainable Growth - CtrlS
130. Long, S., et al. A Reinforcement Learning-Based Virtual Blog:. https://www.ctrls.in/blog/solar-powered-datacenters-
Machine Placement Strategy in Cloud Data Centers. Proceed- drive-sustainable-growth/
ings – 2020 IEEE 22nd International Conference on High 147. Project Natick Phase 2:. https://natick.research.microsoft.com/
Performance Computing and Communications, IEEE 18th 148. Data Center Energy Efficiency Standards in India: : Preliminary
International Conference on Smart City and IEEE 6th Interna- Findings from Global Practices | Energy Technology Area.
tional Conference on Data Science and Systems, HPCC- https://eta.lbl.gov/publications/data-center-energy-efficiency
SmartCity-DSS 2020 223–230 doi: (2020). https://doi.org/10. 149. Mostafavi, M., Kabiri, P.: Detection of repetitive and irregular
1109/HPCC-SMARTCITY-DSS50907.2020.00028 hypercall attacks from guest virtual machines to Xen hypervisor.
131. Shalu, Singh, D. Artificial neural network-based virtual machine Iran. J. Comput. Sci. 2018. 1(2 1), 89–97 (2018)
allocation in cloud computing. (2021). 150. Virtual machines to run 50% of workloads by 2012: :Gartner.
132. Jumnal, A., Dilip Kumar, S.M. Optimal VM placement https://www.computerweekly.com/news/1372216/Virtual-
approach using fuzzy reinforcement learning for cloud data machines-to-run-50-of-workloads-by-2012-Gartner
centers. Proceedings of the 3rd International Conference on 151. Ferdaus, M.H., Murshed, M., Calheiros, R.N., Buyya, R.: Net-
Intelligent Communication Technologies and Virtual Mobile work-aware virtual machine placement and migration in cloud
Networks, ICICV 2021 29–35 doi: (2021). https://doi.org/10. data centers. Emerging Res. Cloud Distrib. Comput. Syst.
1109/ICICV50876.2021.9388424 (2015). https://doi.org/10.4018/978-1-4666-8213-9.CH002
133. Kaaouache, M.A., Bouamama, S.: An energy-efficient VM
placement method for cloud data centers using a hybrid genetic Publisher’s Note Springer Nature remains neutral with regard to
algorithm. J. Syst. Inform. Technol. 20, 430–445 (2018) jurisdictional claims in published maps and institutional affiliations.
134. Tawfeek, M.A., El-Sisi, A.B., Keshk, A.E., Torkey, F.A.: Vir-
tual Machine Placement Based on Ant Colony Optimization for

123
Cluster Computing (2023) 26:1845–1875 1875

Springer Nature or its licensor holds exclusive rights to this article has published several research papers in SCI and Scopus indexed
under a publishing agreement with the author(s) or other journals and conferences.
rightsholder(s); author self-archiving of the accepted manuscript
version of this article is solely governed by the terms of such Tanupriya Choudhury received
publishing agreement and applicable law. his bachelor’s degree in CSE
from West Bengal University of
Technology, Kolkata, India, and
Avita Katal Avita Katal is master’s Degree in CSE from
working as an Assistant Pro- Dr. M.G.R University, Chennai,
fessor in School of Computer India. He has received his Ph.D.
Science, University of Petro- degree in the year 2016. He has
leum and Energy Studies Deh- Ten years’ experience in teach-
radun, Uttarakhand. She has ing as well as in Research.
received her B. E degree from Currently he is working as an
University of Jammu in Com- Associate Professor in dept. of
puter Science Engineering in CSE at UPES Dehradun.
2010 and M. Tech degree in the Recently he has received Global
year 2013. She is currently Outreach Education Award for
pursuing her Ph.D. in the area of Excellence in bestYoung researcher Award in GOECA 2018 His
Cloud Computing. Her research areas of interests include human computing, softcomputing, Cloud
interest is in the area of Cloud computing, Data Mining etc. He has filed 14 patents till date and
Computing, Mobile Ad hoc received 16copyrights from MHRD for his own software. He has been
Networks, IoT, and Artificial Intelligence. She has published various associated with many conferences inIndia and abroad.He has authored
research papers in renowned conferences and journals and has also more than 85 research papers till date. He has delivered invitedtalk
served as a reviewer in various conferences and journals. She is an and guest lecture in Jamia Millia Islamia University, Maharaja
active member of IEEE, IEEE Women in Engineering and ACM. She Agersen College of DelhiUniversity, Duy Tan University Vietnam
is actively involved in all areas of education including research, etc. He has been associated with many conferencesthroughout India
curriculum development, teacher mentoring, student career prepara- as TPC member and session chair etc. He is a lifetime member of
tion and community work with a genuine interest in student’s cog- IETA, memberof IEEE, and member of IET (UK) and other
nitive and social growth. renowned technical societies. He is associated withCorporate and he
is Technical Adviser of DeetyaSoft Pvt. Ltd. Noida, IVRGURU, and
Susheela Dahiya Susheela Mydigital360etc. He is holding the post of Secretary in IETA (Indian
Dahiya is working as Assistant Engineering Teacher’s Association-India),He is also holding the
Professor in School of Compu- Advisor Position in INDO-UK Confederation of Science, Technology
ter Science, University of Pet- andResearch Ltd. London, UK and International Association of Pro-
roleum and Energy Studies fessional and Fellow Engineers-Delaware-USA.
(UPES), Dehradun. She had
completed her M.Tech. and
Ph.D. degree from IIT, Roorkee.
She has more than 8 years of
academic/research/industry
experience. Her research work
is focused on satellite image
processing, video processing,
cyber security, and cloud com-
puting and deep learning. She

123