High Efficiency Video Coding (HEVC) in a Changing World—What Can MSOs Expect?

Mukta Kar, CableLabs; Yasser Syed, Comcast Cable; Munsi Haque, Consultant

Abstract INTRODUCTION

The cable operator world has been Video compression technology coupled
undergoing a sea-change over the last couple with MPEG standardization brought a new era
of years. Content is increasingly being viewed in video delivery and applications. The
in a non-linear fashion. Service providers are deployment of MPEG-2 video in broadcast
not just delivering to the leased set-top box video has been a huge success in terms of
(STB), but to PCs, gaming machines, tablets, improved video quality and increased number
cell phones and other customer owned and of channels. It also increased consumer choice
managed (COAM) devices. The migration of in the area of video services due to emergence
Picture quality is not just from standard of direct broadcast satellite (DBS), DVD and
definition (SD) to high definition (HD), but IP delivery. The MPEG-2 standard is
also to 3D, 4K (also called UltraHD), and broadcast centric and not friendly to other
several resolutions in-between. STB Video video applications, especially to real-time
Processors are changing from dedicated internet delivery. As the internet is a public
hardware processors to general-purpose network, and is based on a best efforts
multicore processors running video delivery protocol, streaming video delivery
processing applications. over the internet lacks in video quality and
resolution compared to broadcast video. But,
Amidst these changes, MPEG High there are two primary advantages of HTTP
Efficiency Video Codec (HEVC, also called internet delivery over broadcast delivery: 1)
H.265), the new video coding standard video/audio content can be delivered to any
released by ISO/IEC & ITU-T has just been receiving device with internet connectivity
released in its version 1 format in January and 2) content can be delivered in a
2013. It brings an additional 2:1 compression personalized manner. To help reduce the
efficiency over it predecessor Advanced Video bandwidth needed for video delivery and to
Codec (AVC) and incorporates several address a wider area of video applications, the
improvements suitable for video deployments Joint Video Team (JVT) of MPEG and ITU-T
in this new environment. This paper will published AVC/H.264 video compression
examine the new improvements of HEVC, standard in 2003 [9]. AVC provides 2:1
including compression performance, and what compression gain over MPEG-2 video. This
areas it may be employed to enhance in this acted as a catalyst for explosive growth in
new and changing service operator video applications, especially video over the
environment. Lastly this paper will conclude internet. Although the internet was generally
with integration/migration strategies to developed as a non-real-time data delivery
introduce HEVC technologies into Cable network, numerous video applications, real-
services, including TV Everywhere services. time and non-real-time, are now using nearly
50% of the internet bandwidth capacity.
Obviously this is impacting other services Again HEVC provides nearly 2:1
delivered over the internet due to real time compression gain over AVC.
bandwidth delivery demands. To mitigate the
negative impact from video delivery over the HEVC: DIFFERENCES FROM AVC
internet, MPEG and ITU-T formed a Joint
Collaboration Team for Video Coding (JCT- The development of HEVC started
VC) and initiated another compression approximately a decade after AVC was
standard known as HEVC/H.265 in 2010 started, but essentially is still an evolution of
which provides significantly better AVC with enhancement and refinement to
compression than that of AVC. After working some AVC tools, and with the addition of a
nearly 2-1/2 years, JCT-VC finalized HEVC few new tools (See Figure 1).
version 1 standard in January 2013 [1, 2].

Input General Coder General

Video Control Control
Signal Data
Transform,
Scaling & Quantized
- Quantization Scaling & Transform
Split into CTUs Inverse Coefficients
Transform

Coded
Header Bitstream
Intra Prediction Formatting &
Data CABAC
Intra-Picture
Estimation
Filter Control
Analysis Filter Control
Data
Intra-Picture
Prediction
Deblocking &
SAO Filters Motion
Motion Data
Intra/Inter Compensation
Output
Selection
Video
Motion Decoded Signal
Estimation Picture
Buffer

Figure 1: Typical MPEG HEVC Video Encoder Structure [10]

The new HEVC compression tools can be (e.g., Prediction Units, Spatial directional
categorized around three main areas while Modes, Adaptive Quantization), and 3)
maintaining the same or better visual quality improvements in informational compaction/
by providing: 1) improvements to reduce symbol rates in bitstreams (simplified
number of bits required for region CABAC, New Scanning Modes). Table 1
representations (Coding Unit, Transform below describes some of the evolution in
Unit), 2) improvements for better prediction common encoding tools from MPEG-2 to
accuracy and reduction of errored residuals AVC to HEVC.
 Coding Tools MPEG-2 AVC HEVC
Intra-prediction None Yes (9 modes) Yes (35 predictions)
Inter-prediction Yes Yes Same as AVC
(No B-picture as (allows hierarchical b-
reference) picture as reference)
CU 16x16 (fixed, known 16x16 MB (same as in Variable, 64x64, 32x32,
(coding unit) size as Macroblock (MB)) MPEG-2 video) 16x16, and 8x8
PU 16x16 16x16, 16x8, 8x16 32x32, 16x16, 16x8, 8x16
(prediction unit) size 8x8, 8x4, 4x8, 4x4
TU 8x8 (DCT floating 8x8 and 4x4 (DCT 32x32, 16x16, 8x8, 4x4
(transform unit size) point) integer) (DCT integer and also 4x4
DST integer)
In-loop filter None One Deblocking filter Two in-loop filters
(deblocking and SAO)
Entropy VLC CAVLC and CABAC CABAC only
Parallel Processing None None Tile and Wavefront
tool
Table 1: Evolution in Common Video Encoding Tools

It is to be noted that the coding unit (CU) HEVC also has three new features (Tiles,
is analogous to the AVC macroblock but in Wavefront Parallel Processing, and
this case the macroblock can change in size Dependent Slice Segments) to enhance
(see Figure 2) which when used parallel processing capabilities or modify
appropriately can lead to bitrate savings. (See slice data structures for packetization
Table 2 [8].) Similarly, compression purposes. Such features help in an encoder or
algorithms adaptively determine the size of decoder implementation to derive benefits in
prediction unit (PU) and transformation unit particular application contexts.
(TU) to achieve savings in bits while the
picture quality is maintained at a desired Lastly HEVC also provides enhanced
level. High Level Syntax bitstream support to
improve operations over a variety of
To aid in quality improvements, AVC applications, network environments and
uses a Deblocking Filter as an In-loop filter; robustness to data losses.
HEVC uses a simpler but comparable
Deblocking Filter and also adds a new
Sample Adaptive Offset (SAO) Filter as the
In-loop Filters. For Inter-picture prediction,
HEVC uses Quarter-sample precision for the
motion-vectors, and 7-tap or 8-tap filters for
interpolation of fractional-sample positions.
Whereas, AVC uses 6-tap filtering of half-
sample positions followed by linear Figure 2: MPEG HEVC Coding Unit (CU)
interpolation for quarter-sample positions. compared to MPEG AVC Macroblock
 PROFILES AND LEVELS

Profiles, tiers and levels specify

conformance points for implementing the
standard in an interoperable way. A profile
defines a set of coding tools or algorithms
that can be used in generating a conforming
bitstream, whereas a level places constraints
on certain key parameters of the bitstream
Table 2: Example of bitrate increase from 64x64 such as maximum picture size, maximum bit
CU size to 32x32/16x16 CU [7]
rate, and a few other parameters which
basically relate to decoder processing load
In brief, enhancements/refinements have and memory capabilities. In the design of
been done to some AVC tools with addition HEVC, it was determined that two distinct
of a few new tools as well. The effect of sets of applications exist that have
these improvements has been the ability to requirements that differ only in terms of
use existing AVC tools in combinations to maximum bit rate and CPB capacities. To
more precisely allocate bits in alignment with resolve this issue, two tiers were specified for
visual perceptual models as indicated by some levels – a “Main” tier is intended for
Figure 3 [6]. This allows for future advances most consumer video applications and a
in compression efficiency as image analysis “High” tier for higher quality delivery that
algorithms improve. In addition, the requires much higher bitrates. A level using
reduction in bitrate and new parallel the “Main” tier only needs to encompass
processing tools align well with battery- levels below with bitrates targeted for the
enabled portable devices. This is done “Main” tier at that level. The decoders
through the use of low-power multi- conforming to a specific profile must support
processors CPUs rather than a single high- all features in that profile.
power processor and helps in new models of
how video is consumed (connectedness, Currently, three profiles, called the
conferencing, portability, and on-demand). “Main”, “Main 10” and “Main Still” have
been specified. Main is intended to be used
with video with pixel depth of 8 bits and
Main 10 is for 10 bit video. Main and Main
10 have the same set of tools, and a Main 10
decoder can decode a Main 8 compliant
bitstream, however, the reverse is not true. It
is possible that some other profiles of the
standard will also be specified. Minimizing
the number of profiles provides a maximum
amount of interoperability among devices,
and across applications such as broadcast,
mobile, conferencing and streaming. Thus, a
Figure 3: Example of recursive quad-tree Main profile compliant device may be used
partitioning for coding block (white) and for more than one application. Main Still is
transform block (red) [3]
intended to be used for still picture decoding.
The objective here is that devices like
cameras, smart phones and other similar
devices can capture or decode both video as
well as still pictures without the need to
support two different codecs as is the case
today. Table 3: MPEG HEVC Performance over Typical
and Anticipated Video Services (Linear/VOD)
The definition of 13 picture levels has
been defined, starting with picture sizes such
as a luma picture size of 176×144
(sometimes called quarter common
intermediate format) to picture sizes as large
as 7680×4320 (often called 8k). There are
two tiers supported for eight of these levels
(levels 4 and higher); Main tier and High tier.
A decoder supporting High tier will be able
Table 4: MPEG HEVC Anticipated Complexity,
to decode Main tier compliant streams, while
Memory, and Memory Bandwidth Performance
the reverse is not true. High tier decoders
support bitstream with higher maximum
bitrate and hence may need higher processing
power. In most cases high bitrate bitstreams
provide higher video quality than the one Real-time software decoding [3, 4, 5] of
provided by a lower bitrate bitstream. HEVC bitstreams is very feasible on current
generation devices — 1080p60 decoding on
PERFORMANCE IMPROVEMENTS laptops or desktops, and 480p30 decoding on
mobile devices.
It is noted earlier that HEVC provides 2:1
better compression gain over AVC. So to As an example-1, HEVC software
deliver content with same quality, a HEVC decoding of 480p to 1080p at 25 or 30 fps is
codec will use approximate half the bitrate of possible with a single core of an ARM
that required by AVC. In other words with processor. Here the player application itself
bitrate close to the AVC bitrate, a HEVC can be multi-threaded, with separate
coded video can provide significantly better decoding and display threads (OpenGL to
video quality. Again the compression gain display video in real time). Also scaling to fit
can vary with content and also picture sizes. screen size and YUV to RGB color
Larger size pictures tend to provide better conversion can be done on the GPU during
compression efficiency than smaller shading.
resolution pictures. The table below shows
the typical bitrates needed by MPEG-2, AVC As an example-2, the software playback
and HEVC bitstreams. In subjective tests of 4K sequences at 60 fps is possible on a
results with equivalent reproduction qualities laptop where bitstreams encoded with
for codecs, Table 3 shows HEVC encoders random access main profile at 12 Mbps bit-
use approximately 50% less bit rate on rate can be decoded by using three parallel
average than AVC encoders. HEVC design is decoding threads with a quad core, 2.7 GHz,
especially effective for low bit rates, high- Core-i7 processor. Up to 100 fps are
resolution video content, and low-delay achieved with four parallel decoding threads
communication applications. on the same laptop.
 MPEG HEVC encoders are expected to be
a few to several times more complex than
AVC encoders, and a subject of research in
years to come. The market availability of
real-time 1080@60P encoders supporting
HEVC Main/Main 10 profile is expected to
be available by 2014.

Service Providers: WHAT’s ALREADY

CHANGING?

Over the last few years, the types of

services being supported have significantly
Figure 4: Household Past and Present/ Future
evolved (see Figure 4). Previously customers Video Device Consumption Models
watched video services through a cable set-
top box and on one of the 2-3 TVs in the
house. Data connections were meant for the So what makes content more valued to the
PC and laptops available in the house. With customer? Is it the material? Partially. But it
the advent of wireless systems, those PCs, also has a lot to do with connection and
laptops, and now phone devices became experience as well (see Figure 5). In terms of
portable, but bandwidth reliability was still content, the value can be affected by how it
causing interruptions and used mainly for can be purchased (subscription, single time,
data services rather than video viewing. With catalog depth), who is excluded from getting
higher performing data devices, and the it, and by when it can be available (first to
advent of MPEG-AVC, and settling some of air, first available). In terms of experience,
the firewall issues, these devices are starting value can be affected by how immersive it is
to favor video viewing more often. At the (large screen/4k/8k/portable screen), where it
same time, television screens are getting can be watched (fixed, mobile), how it can be
bigger, HD is more firmly entrenched in shared (group/individual setting), and lastly
households, and various flavors of non-linear how it can be consumed (languages, CC,
viewing (on-demand / DVR) are increasing Binge viewing). There are some exciting
in popularity. With these establishing trends, things happening in the area of experience.
the service expectation is evolving to view The content can be more valued because it is
video on a plethora of devices (cable box, larger (more resolution), faster (more frame
gaming console, Blu-Ray Player, Boxee Box, rate), and brighter (more contrast).
Roku, Tivo, Tablet, and Smartphone) and not Personal/Portable devices (PC/tablets) in the
just something that is permanently fixed in home are rapidly replacing the 3rd or 4th TV
the living room. The choices made by the set in the household. The value of content
customer then are determined by the within the household increases in this mode
perceived value of the service.
due to added accessibility of the content and devices. The calculations are for the value of
transferability of it (your child can now content changes to include platform
watch anime on his/her laptop). The value portability as well as increasing the
further increases if one can then walk out the weighting of accessibility and availability. In
door and still can watch the content. The the real-time linear domain, this can increase
tradeoff on this is what the risk is to secure the value of sports and news. While in the
the content versus mobility, and a VOD domain, this can help determine if a
compromise to this would perhaps be a cable subscription of aggregate VOD catalog
download-to-go application. In addition to is ordered, or whether an over-the-top (OTT)
increased resolution there is another service account is enough. What is also
dimension of increased frame-rates (60 Fps, interesting is that the infrastructure and
30 Fps and 3:2 pull-down). Lastly with technology can change but the service to the
advancements in cameras, images can be customer could remain the same.
captured with higher contrast due to
improvements in sensors which leads to
shooting of a lot of dark, high-contrast
scenes. In this case the experience has
expanded, but is only allowed if the
implemented technology can support it.

The last component is connection (which

is more hidden to the service) that deals with
more infrastructure and technology aspects.
The value here can be affected by the
availability of the connection (always-on,
managed, and unmanaged), the distribution Figure 5: Content & Service Value Diagram
of the content (broadcast, unicast, DRM, and
link encryption), the host processor
capabilities (STBs, tablets, software based, The service evolution that has been
hardware based), and lastly by request happening has affected the connection value
(linear, non-linear). Note the advent of IP aspects the most. With the new devices and
delivery has opened up a set of new customer new distribution pathways to get content in
owned and managed devices. The choices of front of the viewer, there has been a
using broadcast or IP unicast can limit or subsequent 10x-100x increase of the number
expand the audience size when demand of streams and files to support new devices
requires it. In earlier periods, the and unmanaged networks in the service
infrastructure design deemed all content operator ecosystem. These changes have put
valuable and protected the content all increased pressure on bandwidth and storage
equally, but restricted content to the confines demands in the network due to support of
of the infrastructure. With the advent of adaptive streaming, expanded on-demand
DRM-protected content, the infrastructure catalogs, and PVR services. Until just
restrictions were loosened, all content did not recently a single transcoder would have a
have to be protected equally, and it became single output stream. Presently a single input
more mobile. Lastly technical capabilities, could generate up to 10 streams or files at the
like codec format, can affect capacity of the output. New distribution pathways such as
connection because the codec can affect on-demand MBR (Multi-BitRate) streaming
things like the management of bandwidth, for http adaptive streaming technologies has
the capacity of storage, and the processing on also increased bandwidth and storage
demands to support bandwidth OTT transitioning environment. For OTT-based
congestion management and network storage. services, the number of stream
representations can be reduced since higher
The delivery of video is not only to a STB quality streams at lower bitrates can survive
using an MPEG-2 transport stream over more often through bandwidth-congested
QAM, but now also to connected personal environments. For a present HD OTT
devices through an HTTP protocol over IP. service, the number of streams could be
For individual viewing of real-time content, reduced by 50% and in addition each of the
the IP connection is very suitable to this remaining streams has also a reduced bitrate.
medium. But when it comes to high valued For backhaul, bandwidth distribution
content that is driven real-time high viewer demands for mezzanine/contribution streams
demanded, often the MPEG-TS system has can be reduced by switching from
benefits that are sometimes more AVC/MPEG-2 formats to HEVC (or the
complicated to replicate in the IP domain. To quality of distributed content can be
repeat this experience in the IP domain increased for the same bandwidth costs).
would be an analogous multicast equivalent This also encourages the transition from
similar to the broadcast medium, but in the satellite-based feeds to fiber-based IP
near term it is still replicated using a unicast connections, even further reducing the signal
IP delivery. Presently the storage may be integrity risks encountered with satellite
commonly shared (used in HTTP adaptive distribution and redundancy strategies
streaming) but the delivery is still a unicast associated with it. In the area of non-linear
delivery which works unless one scales up in services (VOD/Cloud DVR) and targeted ad
real-time viewership (e.g., the Super Bowl). insertion services, storage demands can be
reduced by storing in an HEVC format even
The service expectation is already though output may be transcoded to a more
changing to support better displays, more traditional format. This becomes especially
mobile and personal devices, and increased needed in light of supporting unique copy
non-linear viewing. The service expectation services across cable devices and customer
has been moving in this direction because of owned and managed (COAM) devices. The
the increase in value to the customer reduction of storage demands can be greater
experience in the living room, and on the go. than 50% due the decrease in number of
A large amount of improvement and value to streams supported and the decrease in bitrate
the services has already happened with more to support the same quality streams. At the
expected to come. There will be increased granularity of the stream for VOD, efficiency
pressure on the infrastructure due to higher can occur by switching from using multiple
pixel demands, increased number of output trick files to support 2-3 speeds to supporting
streams, and related storage requirements. dynamic trickplay in the stream itself through
Can our current technologies (Bandwidth, the use of picture order count (poc) /
Storage, IP distribution) support the ramp up temporal IDs in HEVC.
of these services as popularity grows? HEVC
can be a good candidate to relieve some of In terms of the device perspective, HEVC
the infrastructure demands as the service can increase the value associated with the
continues to grow. host device by decreasing the service
bandwidth to each device and increasing the
POSSIBLE MPEG HEVC BENEFITS amount of bandwidth available for the
service. This benefit will assist in terms of IP
HEVC can reduce the complexity and unicast replications of expected broadcast
costs of handling multiple streams in this customer experiences (e.g., popular events
like the Super Bowl) while longer term early next year with 4K decoding that will
solutions can then be created (e.g., multicast add value to the larger display based devices
IP). Additionally HEVC does not require a that can be implemented through an ROI
new set of mobile personal devices to be based approach based on 4K and higher
created; existing mobile devices can already services. Bandwidth demands on the plant
play 720p30 HEVC using software-based can be managed with expectations set for
decoders taking advantage of the small new higher quality deployments (4K@60fps,
increase in decoder complexity. Since mobile 10 bit displays, etc.) within the same
devices often have short (18 month) bandwidth of a traditional HD and in
lifetimes, the transition to HEVC in this area combination with other bandwidth saving
can be easier with later introduction of higher strategies. The install base on this would
valued content experiences. 720p30 HD coincide with an increased value in services
streams are already the currently accepted that can be enabled on next generation STBs
deployment of HD streams for OTT services and gaming machines.
and smartphone/tablets devices. Lastly,
battery lifetimes on the device (often
consumed quickly by video viewing) can CONCLUSIONS
increase (HEVC streams with 4-8 hours of
continuous play) by switching to an HEVC Cable services have evolved rapidly from
coding standard designed to take advantage a few years ago. Video services are now
of low-power multi-core processors in these being offered for not just the living room, but
devices. Additionally the reduced bitrate can also on the go for laptops, tablets, and mobile
save on antennae power and processing devices. An IP distribution structure using
power required to decode on the portable adaptive streaming techniques is used for
device. This can also further increase battery these newer devices which enables
lifetime. portability in the managed home
environment, WIFI, and OTT) delivery.
In the areas with more immersive Additionally there is an increase in number
experiences and larger displays such as in the of streams delivered and managed as well as
living room/bedroom, HEVC can greatly storage in cloud DVR and CDN structures.
help out in this area by reducing the bitrate With the increased popularity of these
associated with carrying a higher quality or services, there will be increasing pressure on
larger resolution stream. In terms of the current infrastructure to support these
integrating these new experience streams, services. Higher Efficient Video Coding
HEVC can reduce the bit rate to fit into the (HEVC) can play an important role in
same slot as a present day MPEG-2 HD deploying these new services on a larger
channel (~8-15 Mbps). With higher scale by reducing the number and size of
resolution streams, the gains could increase streams delivered and stored, making the IP
non-linearly (compared to HD) due to use of unicast model more efficient for replication
larger macroblock sizes, and increased of older broadcast services, increasing the
spatial mode. With increased frame rates, battery lifetimes on devices, and creating
HEVC can also reduce bit rate from adding more immersive experiences for larger
additional frames using simplification displays with higher frame rates and
methods/increasing accuracy in motion resolutions. HEVC can increase value by
vectors and use of different options in making the connection more efficient and
prediction units. Hardware based decoders improving the experience which can support
through silicon based chipsets are coming out the expansion of the service market.
 ENDNOTES
[6] Y. Syed and D. Holden, “Strategies
[1] B. Bross, et. al., “High Efficiency for Deploying High Resolution and High
Video Coding (HEVC) text specification Frame Rate Cable Content Leveraging
draft 10 (for FDIS & Last Call)”, JCTVC- Visual Systems Optimizations,” Proceeding
L1003_v34, Geneva, January 2013 of NCTA 2012, May 21, 2012.

[2] G. J. Sullivan, J.-R. Ohm, W.-J. Han, [7] J.-R. Ohm, G. Sullivan, H. Schwarz,
and T. Wiegand, “Overview of the High T.K. Tan and T. Wiegand, “Comparison of
Efficiency Video Coding (HEVC) Standard,” the Coding Efficiency of Video Coding
IEEE Transactions on Circuits and Systems Standards—Including High Efficiency Video
for Video Technology, December, 2012. Coding (HEVC)”, IEEE Transactions on
Circuits and Systems for Video Technology,
[3] F. Bossen, B. Bross, K. Suhring and December, 2012
D. Flynn, “HEVC Complexity and
Implementation Analysis”, IEEE [8] A. Rodriguez and K. Morse, “HEVC-
Transactions on Circuits and Systems for Driving Disruption in Multiscreen
Video Technology, December, 2012. Converged Service Delivery Architecture,”
Proceedings of 2013 NAB Broadcast
[4] T.K. Tan, Y. Suzuki and F. Bossen, Engineering Conference, April 7, 2013, pp
“On software complexity: decoding 4K60p 33-38.
content on a laptop”, JCTVC-L0098,
Geneva, CH, 14–23 January 2013. [9] ITU-T Rec. H.264 | ISO/IEC
14496-10, (2005), “Information
[5] K. Veera, R. Ganguly, B. Zhou, N. Technology – Coding of audio visual
Kamath, S. Chowdary, J. Du, I.-S. Chong, objects – Part 10: Advanced Video
and M. Coban, “A real-time ARM HEVC Coding.”
decoder implementation,” JCTVC- H0693,
San Jose, CA, February 2012. [10] Gary J. Sullivan, HEVC Tutorial in
INCITS meeting at CableLabs, March 2013.