H.-Y. Ko et al.

: Implementation and Evaluation of Fast Mobile VNC Systems 1211

Implementation and Evaluation

of Fast Mobile VNC Systems
Ha-Young Ko, Jae-Hyeok Lee, Jong-Ok Kim, Member, IEEE

Abstract — VNC (virtual network computing) is a popular wireless

tool for sharing an application, allowing users to access
graphic displays remotely. In mobile VNC systems, it has been screen request
challenging to increase screen update rate by fast screen image
encoding. In this paper, we implement a prototype system for screen update
mobile VNC, and several works are done for improving screen
update rate. At first, a number of video encoders are integrated RFB
into a prototype system, and we investigate what is the most VNC Server Protocol VNC Client
suitable codec for mobile VNC. The existing RFB protocol is Fig. 1. Mobile VNC systems.
extended straightforwardly to integrate video codecs. Next, the (The game images courtesy of Olivestudio, EBS and Dream Search.)
overall system architecture is modified from serial operation to
parallel. Finally, we propose a modified region coding to as encoding screen images on the server side, and
further reduce the encoding time of screen images. The decoding and scaling them on the client side. Another
proposed methods are implemented into our prototype mobile challenge is that current mobile network environments
VNC system, and practical performances are widely evaluated. cannot cover huge resolution image data of a screen
We report that JPEG is the most suitable for mobile VNC in enough. Due to these prominent problems, several VNC
terms of both complexity and compression ratio. In addition, the systems have been researched and developed in the past.
proposed modified region coding can decrease encoding time, For details, refer to the next section of “Related Works”.
and consequently increase screen update rate1. In this work, we propose a fast screen sharing method
to improve screen update rate in mobile VNC systems. In
Index Terms — mobile VNC, MJPEG, screen image coding, case of mobile devices, high complexity video
modified region coding
compression techniques cannot be employed due to their
strict computation limit. However, the bandwidth
I. INTRODUCTION
limitation requires a certain level of compression ratio.
In recent years, there have been popularly released a variety of Thus, there exists a trade-off between encoder complexity
multimedia mobile devices such as smartphone and tablet PC. and compression ratio for fast mobile VNC systems. We
The service which today’s mobile devices provide to users is first integrate various video encoders into our prototype
almost comparable to desktops or similar devices. This trend system, and explore their suitability for mobile VNC.
requires the necessity of collaboration among these systems such Also, the existing RFB protocol for VNC is extended to
as sharing multimedia contents and applications. easily integrate video encoders in a backward-compatible
VNC [1-5] has been used as a tool for a multi-platform way. We additionally propose a new modified region
application suite allowing users to access graphic displays coding method which transmits only modified regions
remotely. It is based on the thin-client architecture and uses the between current and previous screen images. It can
RFB (remote frame buffer) protocol for sharing a screen between further reduce encoder computation and resultantly
distinct devices. The client software runs on the local user’s increase screen update rate. We implemented a prototype
machine while the server part operates at the target host. In this mobile VNC system actually, and its practical
way, VNC brings remote display to the local user’s machine. performance is widely evaluated.
Successful application of VNC to mobile devices should The remainder of the paper is organized as follows.
guarantee rapid screen image transfer from server to client. Related works are described in Section II. In Section III,
However, this is considerably restricted by a number of we give explanation on prototype system implementation.
factors. Mobile devices do not provide sufficient computing In Section IV, we describe video codec integration and
performance of processing large amount of data such protocol extension. Section V presents the proposed
modified region detection methods for mobile VNC
1
This research was supported by Seoul R&BD Program (ST100059). systems in detail. The experimental results of our
H.-Y. Ko, J.-H. Lee, and. J.-O. Kim are with School of Electrical prototype system are presented in Section VI. Finally,
Engineering, Korea University in Seoul 136-713, Korea (e-mail: Section VII concludes the paper.
[email protected]).
Contributed Paper
Manuscript received 10/15/12
Current version published 12/28/12
Electronic version published 12/28/12. 0098 3063/12/$20.00 © 2012 IEEE
1212 IEEE Transactions on Consumer Electronics, Vol. 58, No. 4, November 2012

II. RELATED WORKS Server Client

Many of previous works have studied efficient screen image • [S  C] ProtocolVersion
encoding. To transfer screen image data with limited network Handshaking • [S  C] Security
• [S  C] SecurityResult
bandwidth, screen image compression is typically required, and an
appropriate encoder should be carefully selected in terms of
compression ratio and speed. TightVNC [6] developed by Initialization • [C  S] ClientInit
• [S  C] ServerInit
Kaplinsky is an improved version of the original VNC. Original
VNC has only six encoding algorithms where all use lossless
• [C  S] SetPixelFormat
compression. It implemented a new encoding scheme called ‘tight’. • [C  S] SetEncoding
Normal protocol interaction
The ‘tight’ uses data analyzer and a set of data filters as • [C  S] FramebufferUpdateRequest
• [S  C] FramebufferUpdate
preprocessors to improve adaptation ability in encoding. The works
in [7] and [8] classify image blocks into two categories, and Fig. 2. The RFB protocol procedures of VNC systems.
accordingly, more suitable coding technique is used for
compression. In [7], image blocks are divided into high and low handshaking phase, and they agree on the type of security by
motion parts, which are encoded by H.264 and run-length coding, exchanging the security message. After handshaking, the
respectively. Zhang et al [8] proposed a scheme which performs procedure moves to the initialization step as a second phase. If
either lossless compression or JPEG encoding according to the the server receives the ClientInit message, it responds with the
number of colors. Shen et al [9] first detects whether image blocks ServerInit message. This message includes width and height
are modified or not, and only modified blocks are encoded by of the server’s framebuffer, pixel format and the device name.
either JPEG or color index based coding. (This is basically similar The final stage is normal protocol interactions. The client sets
to our work. But the key difference is that [9] deals with how to the pixel format and encoding type by sending the
efficiently code image blocks while our work concentrates on how SetPixelFormat and SetEncoding messages respectively. The
fast we code image blocks using modified region detection.) types of encoding defined in RFB comprise run-length
Some works focus on overcoming the drawbacks of the client- encoding methods such as RRE, Hextile and ZRLE. Once
pull system [10]. VNC is a client-pull system where the server encoding method is agreed, the client sends the
sends screen image data to the client only in response to the screen FramebufferUpdateRequest message to the server. As
update request of the client. Since it prevents surplus updates, it is mentioned above, VNC is a client-pull system where the client
probably suitable for thin-client system such as VNC. In high decides when to update its screen. In other words, the server
latency environment, however, the update request from the client transmits its screen image data only when it explicitly receives
can be delayed, and it badly affects screen update performance. In the request update message from the client. It has an advantage
[11], the message accelerator is employed to mitigate the effects of of preventing surplus updates for the client. Triggered by the
network latency. The message accelerator requests more frequently FramebufferUpdateRequest and FramebufferUpdate messages,
updates to the server while forwarding screen update data to the a couple of internal processes are performed within both server
client. In [12], they proposed a time-driven model for screen and client as depicted in Fig. 3. Gray boxes in Fig. 3 represent
updates by capturing screen data before screen update starts. It process modules in the typical VNC systems. After receiving
minimizes time interval between screen updates. the update request message from the client, the server captures
its screen image data (typically RGB color format) from
III. PROTOTYPE SYSTEM IMPLEMENTATION framebuffer. Note that most of video encoders generally support
We implement a prototype system for mobile VNC in order only YUV format which is easy to compress. So, the color space
to evaluate the proposed methods practically. Our prototype of screen images is converted to YUV before compression. The
system is built on mobile operating system platform which captured screen image data is encoded and is carried from server
has been popularly used for mobile devices such as to client by the FramebufferUpdate message. Meanwhile, the
smartphones and tablet PCs. It consists of VNC server and client receives and decodes bitstreams. The aforementioned
client which are developed, based on the open source ‘droid procedure is repeatedly done until connection is lost.
VNC’. The original ‘droid VNC’ does not contain any video
codecs for compressing screen images. Thus, the open source Framebuffer RFB Display
video codecs, FFmpeg [12] is ported into the platform, and is Protocol
integrated to both the VNC server and the VNC client using Color space Color space
Run-length conversion Run-length conversion
JAVA Native Interface (JNI) [13]. The server and client are encoding decoding
Video Video
connected by a TCP/IP connection over Wi-Fi networks. encoding decoding
screen request
A. Overview of RFB Protocol Procedures Network IF Network IF
The RFB protocol is mainly divided into three stages as screen update
VNC Server VNC Client
illustrated in Fig. 2. After establishing connection first, the
server sends the ProtocolVersion message to the client for Fig. 3. Overall architecture of our mobile VNC systems.
H.-Y. Ko et al.: Implementation and Evaluation of Fast Mobile VNC Systems 1213

0 1 2 4 12 16
encoding-type codecID
message-type padding #-of-rects x y w h encoding-type
number name CODEC_ID_MPEG4
0 Raw CODEC_ID_MJPEG
Header Extended header Payload : encoded stream
1 CopyRect CODEC_ID_MPEG2VIDEO
: CODEC_ID_MPEG1VIDEO
srcX srcY srcW srcH codecID encoded-byte-size 21 VideoMode
0 2 4 6 8 12 16
Fig. 4. Header extension for supporting video encoders in the FramebufferUpdate message.

IV. VIDEO CODEC INTEGRATION AND PROTOCOL needed for decoding at the client such as top-left position
EXTENSION of encoding block, and its size (width and height). If the
client receives the FramebufferUpdate message, it can decode
A. Video Codec Integration successfully with an appropriate encoder by parsing this
A mobile VNC server requires fast encoding of screen information.
images due to its strict resource constraint. When screen As described in section III, VNC is a client-pull system
images are encoded in the server, encoder complexity and which is generally appropriate for thin-client architecture.
compression ratio become important factors to determine the However, the client-pull system has occasionally a
overall VNC performance such as screen update rate. The drawback in high latency network environments. For
encoding method implemented in the original VNC system is instance, consider the screen update flow as illustrated in
run-length encoding. It has low complexity, but exhibits bad Fig. 5 (a). The client waits for the next update message
compression ratio. When sharing gaming or movie playbacks after sending a request to the server. While waiting for the
on a screen, low compression ratio leads to large amount of next update, it remains idle, and does not do anything.
bitstreams which cause network congestion and slow screen Similarly, the server also enters into an idle state after
updates at the client. On the other hand, video encoding sending the screen update. This idle state unnecessarily
techniques such as MPEG exhibit high compression ratio but wastes scarce resource for both server and client. To
computation proportionally increases because of complex remove the redundancy time in the conventional VNC
motion estimation and compensation. In particular, encoder system, we modified protocol operations as shown in Fig.
complexity in the server is more critical for mobile devices. 5 (b). As soon as the server receives the screen request, it
Thus, a method to encode screen images should be carefully immediately sends previous screen image which has been
selected in terms of both encoding time and compression already encoded, and so, the idle time can be reduced.
efficiency. As mentioned in the previous section, we Thus, the client receives screen image data soon after
integrated FFmpeg to our prototype system because it sending the screen request. This allows client to update its
provides a variety of image and video encoders. Among them, screen faster.
we select MPEG2, MPEG4 and motion JPEG as candidate
encoding methods, and their performances are reported in the V. MODIFIED REGION CODING
experimental results.
A. Motivation
B. Protocol Extension and Improvement There may be large regions which have no change between
The RFB protocol implementation of ‘droid VNC’ does not consecutive screen images, depending on applications to be
support any video encoder for screen image encoding. shared. Motivated by this observation, we propose a modified
Integrating video codecs into VNC systems requires us to region coding, which encodes the modified region only as
define a new encoding-type in RFB. That also needs a illustrated in Fig. 6. Note that modified region coding is
mechanism to convey additional information associated with allowed for MJPEG only, and it is not applicable to typical
video encoder. For this, the RFB protocol is extended video coding standards such as MPEG and H.264 due to their
straightforwardly, while preserving backward compatibility inter-frame coding.
with conventional VNC systems. In order to support video In the first step of modified region coding, a screen image is
encoders, we newly define ‘VideoMode’ for encoding-type segmented into unit rectangles which are fixed size blocks.
which is specified at the header of the FramebufferUpdate Then, difference detection between current and previous
message as shown in Fig. 4. In addition, there are several screen images is performed for each unit rectangle. If all pixel
kinds of video codecs for ‘VideoMode’. For identifying video values are identical, the unit rectangle is regarded as a skip
codec in the video encoding mode, ‘codecID’ is additionally block, which does not need to be transmitted. If any difference
defined at the extended header followed by main header. is detected, the unit rectangle is encoded and is transmitted to
Note that theextended header also contains other information the client as usual.
1214 IEEE Transactions on Consumer Electronics, Vol. 58, No. 4, November 2012

Server Client
Receiving

 Video decoding
 CSC
 screen request Display

Framebuffer Compare
Previous Current
CSC
Idle
Video encoding Screen
Sending
screen update update
Receiving time
Video decoding
Modified Region
Idle CSC
screen request Display Fig. 6. Modified region detection.

Framebuffer
CSC
Idle B. Conventional Raster Scan Detection
Video encoding
screen update The conventional modified region detection method
Sending
simply compares each pixel between previous and current
screens in a raster scan order (i.e., from the leftmost top

corner to the rightmost bottom). While performing the
Idle 

raster scan, if even a single modified pixel is detected, the
comparison operation stops, and the corresponding unit
(a) rectangle is regarded as a code block. Then the comparison
Server Client moves to the next unit rectangle. Such early termination is
Receiving devised for minimizing the unnecessary computation load.
 Video decoding Nevertheless, as the modified pixel is located closer to the
 CSC rightmost bottom along the raster scan order, computation
 screen request Display load increases linearly. In the worst case, if the modified
Sending
Receiving pixel is located at the rightmost bottom corner, the
screen update Screen
Framebuffer
Video decoding update algorithm should carry out the comparison for all pixels in
CSC the unit rectangle. This is a very severe overload. The
CSC time
Video encoding screen request Display larger the size of the unit rectangle, the more vulnerable the
Sending
algorithm is to this phenomenon. Motivated by this
Receiving
Framebuffer screen update observation, we propose a hierarchical region detection
Video decoding
CSC
CSC
algorithm.
Video encoding screen request Display
Sending C. Proposed Hierarchical Region Detection
Receiving In modified region coding, the precise prediction of the
screen update
  modified pixel location is crucial in order to reduce time
  consumed for detecting modified regions. However, methods
  such as motion estimation, as used in video coding, require
(b) much more computation rather than gain of computation
Fig. 5. Protocol improvement from (a) serial operation to (b) parallel one. reduction by modified region coding. Thus, we hierarchically
determine the pixel location for comparison instead of
In modified region coding, the unit rectangle size may affect the predicting the modified pixel location. For this, we propose
overall encoding performance sensitively. If it is too large, the two different methods, which will be described in the remains
performance may be worse on the contrary due to both less number of the subsection.
of skip blocks and the overhead taken for detection. On the other
hand, if the unit rectangle size is too small, much overload is 1. Hierarchical region detection algorithm
accompanied because of the induced computation for encoder We first propose a hierarchical 3-step region detection
initialization and additional data transmission per unit rectangle. algorithm, which is divided into three steps.
Modified region coding plays a decisive role for fast screen In ‘step 1’, as shown in Fig. 7 (a), the unit rectangle is
image encoding. However, detecting modified regions requires down-sampled by a factor of 4 in both horizontal and vertical
additional computational overhead, which may cause worse directions. Then, pixel comparison on the low resolution is
performance rather than full region coding. Therefore, we done in a raster scan order. If a modified pixel is detected in
propose a fast modified region detection method in this paper. ‘step 1’, the remaining ‘step 2’ and ‘step 3’ are skipped, and
H.-Y. Ko et al.: Implementation and Evaluation of Fast Mobile VNC Systems 1215

w
Server

1 Server AP
w
2 `
Client
1 h
w
4
1
h
1 2
h Client AP
4
(a) (b) (c)
Fig. 7. Hierarchical 3 step detection method; (a) Step 1, (b) Step 2, and (c) Fig. 9. Experiments over our prototype system.
Step 3. (The game images courtesy of Olivestudio, EBS and Dream Search.)

the corresponding unit rectangle is determined as a code VI. EXPERIMENTAL RESULTS

block. In ‘step 2’, as shown in Fig. 7 (b), the unit rectangle
is down-sampled by a factor of 2 and pixel comparison is A. Testbed Setup
carried out as in ‘step 1’. Likewise ‘step 1’, if a modified Our testbed consists of a smartphone and a tablet PC, which
pixel is detected, the next ‘step 3’ is skipped and the unit are connected by a Wi-Fi network as illustrated in Fig. 9. The
rectangle is regarded as a code block. Finally, in ‘step 3’, VNC server and client are running on a smartphone (480x800
the remaining pixels are compared as in the previous steps resolution and the single core processor with 1.0GHz), and a
and if there is no modified pixel, the corresponding unit tablet PC (1024x600 resolution and the single core processor
rectangle is determined as a skip block. This algorithm is with 1.0GHz), respectively.
very effective if comparison is early terminated at ‘step 1’ For the benchmarks used in this experiment, we used five
and ‘step 2’. sequences, three of which are gaming sequences recorded off-
line, and two of which are natural sequences known as
2. Dual region detection algorithm
‘Akiyo’ and ‘Mother and daughter (MnD)’. During
Another fast detection method is proposed and is so
experiments, the sequences are displayed on a full-screen for
called dual region detection. It compares a subset of the
one minute.
unit rectangle by two steps. In ‘step 1’, the unit rectangle
is down-sampled by a factor of 2 in both horizontal and B. Selecting Optimal Encoding Method for Mobile VNC
vertical directions. This is done by sampling the upper left In the first experiment, we evaluate various encoding
pixel, as in Fig. 8 (a). The down-sampled rectangle is methods for screen image compression. They include MJPEG,
compared in a raster scan order. As in the 3–step MPEG2 and MPEG4 (newly integrated) as well as Zlib (run-
hierarchical region detection algorithm, if a modified length). During experiments, the gaming sequence is played
pixel is detected in ‘step 1’, ‘step 2’ is skipped and the back on a full-screen for one minute. Table I compares the
unit rectangle is determined as a code block. In next ‘step compressed data size and average encoding time per screen
2’, the unit rectangle is down-sampled, similar to ‘step 1’, image. Note that we use the same QP (quantization parameter)
but the lower right pixel is sampled as in Fig. 8 (b). Note which guarantees sufficient image quality. As expected,
that the only difference between ‘step 1’ and ‘step 2’ is MPEG2 and MPEG4 achieve higher compression efficiency,
the location of the sampled pixel. Then, pixels are but it takes longer time to encode a screen image. On the other
compared in a raster scan order. If no modified pixel is hand, MJPEG shows much faster encoding with reasonable
detected, the corresponding unit rectangle is determined compression ratio. As shown in Fig. 10, it achieves the fastest
as a skip block. encoding time per screen image when compared to the other
video encoders and Zlib. Thus, we can see that MJPEG is the
most suitable for mobile VNC systems. For the remaining
experiments, MJPEG is used, and its results are shown.
` `
TABLE I
Performance of video encoders
Encoding Compressed Encoding Quantization
algorithm data size(bytes) time(ms) parameter
Zlib 157960 361.25 -
MPEG4 3557 193.93 8
(a) (b) MPEG2 4224 190.98 8
Fig. 8. Dual detection method; (a) Step 1 and (b) Step 2. MJPEG 19579 138.73 8
1216 IEEE Transactions on Consumer Electronics, Vol. 58, No. 4, November 2012

170 because QP is fixedly set. This means that the proposed

160 modified region coding method can reduce the encoding time
Total Encoding Time (ms)

150 without almost quality degradation. That is easily confirmed

140 by the next experimental result (Fig. 13).
130
MPEG4 130
120

Total Encoding Time (ms)

MPEG2 120 8x8 480x800
110 110
MJPEG
100 100
90 90
80 80
angry mole animal akiyo MnD 70
Sequence 60 160x200
50
Fig. 10. Encoding time comparison of a screen image. Unit Rectangle Size

TABLE II Fig. 11. Decision of unit rectangle size.

Protocol Improvement
Algorithms Idle time(ms) 160
Conventional 163.85 140

Total Encoding Time (ms)

Proposed 9.61
120
100 VNC
C. Protocol Improvement 80 VNC + Raster
Next, we report the experimental results of the protocol 60 VNC + 3 Step

improvement shown in Fig. 5. RFB protocol operation is changed VNC + 2 Step

40
VNC + 1 Step
from serial to parallel in order to remove idle time redundancy. 20
That modification is actually implemented into our prototype 0
system, and the idle time is measured. As shown in Table II, we angry mole animal akiyo MnD
Sequence
can significantly reduce the redundant idle time, and it can
contribute to boost screen update rate. Fig. 12. Modified region detection time.

D. Modified Region Coding TABLE III

In order to apply modified region detection, we need to decide PSNR Comparisons
Sequence Raster 3 Step 2 Step 1 Step Dual
the size of the unit rectangle. It may be affected by the display
Angry 34.05 34.05 34.04 34.01 34.05
resolution and encoding algorithm. In this paper, we carefully
Mole 33.43 33.43 33.41 33.38 33.42
determine the size using a variety of video contents with diverse Animal 33.77 33.77 33.75 33.73 33.77
characteristics through a large number of extensive experiments Akiyo 30.76 30.76 30.75 30.73 30.76
[14]. As shown in Fig. 11, the optimal unit rectangle size is MnD 31.67 31.67 31.66 31.64 31.67
experimentally selected by 160x200 where encoding time of screen
images is minimized, and it is fixed for all our experiments. 16
In this experiment, we mainly focus on region detection 14
Region detection time (ms)

performance, measured by the elapsed time for detection, at a 12

single frame. If modified regions are too large, the overall
10 Raster
performance may be worse rather than VNC only due to the
8 3 Step
increased encoding time by detection. In other words, overhead
6 2 Step
caused by modified region detection is more dominant than its
1 Step
performance gains. Thus, minimizing detection time is greatly 4
Dual
important. 2
Figure 12 shows the average detection time per screen 0
frame, resulting from five detection methods. The proposed angry mole animal akiyo MnD
hierarchical 3 step algorithm saves the detection time by 4 to 5 Sequence

milliseconds without almost PSNR reduction. Also, even with Fig. 13. Total encoding time for a screen image.
‘step 1’ only in the hierarchical 3 step algorithm, we can save
the detection time by approximately 20 milliseconds while the The next experiment evaluates the performance of screen image
received visual quality is still preserved, comparable to the encoding time for MJPEG. For this experiment, we use the same
raster scan order as shown in Table III. It is worth noting that environments as the previous region detection experiment.
the negligible reduction in the PSNR results of Table III are Total time required for encoding a screen image is compared
caused by applying the proposed modified region coding in Fig. 13. We can see from Fig. 13 that our system can provide
H.-Y. Ko et al.: Implementation and Evaluation of Fast Mobile VNC Systems 1217

faster image encoding than ‘VNC’ and ‘VNC with raster’ over VII. CONCLUSION
test sequences. In particular, the proposed ‘VNC with 1 step’ In this paper, we implemented a prototype system for mobile
algorithm produces better performance in terms of encoding VNC, and reported practical performance evaluations. To integrate
time while almost maintaining the same PSNR. video codecs into our VNC system, the existing RFB protocol is
extended, preserving backward compatibility. Also, protocol
TABLE IV
Average ratio of modified blocks per video frame operations are modified to parallel for reducing unnecessary idle
Average ratio of time. In addition to the adoption of video codec we propose a
Sequence modified blocks /total unit rectangles modified region coding to further reduce the encoding time of
Raster 3 Step 2 Step 1 Step Dual screen images. Based on numerous experiments, we found that
Angry 7.71/12 7.71/12 7.70/12 7.69/12 7.71/12 MJPEG is the most suitable for mobile VNC systems in terms of
Mole 5.31/12 5.31/12 5.30/12 5.29/12 5.31/12 both complexity and compression ratio. Besides, the proposed
Animal 7.88/12 7.88/12 7.87/12 7.86/12 7.88/12
modified region coding can further decrease encoding time, and
Akiyo 8.97/12 8.97/12 8.96/12 8.95/12 8.97/12
MnD 9.36/12 9.36/12 9.35/12 9.35/12 9.36/12 consequently increase screen update rate at the client. Various
practical and diverse experimental results demonstrate that the
proposed methods guarantee fast screen image encoding without
19 visual quality degradation. In particular, modified region detection
is significantly effective for gaming video contents with small
17
texture regions.
Screen Uodate Rate

15
VNC
13 REFERENCES
VNC + Raster
11 [1] T. Richardson, Q. Stafford-Fraser, K. Wood, and A. Hopper, “Virtual
VNC + 3 Step
network computing,” IEEE Internet Computing, vol. 2, no. 1, pp. 33–38,
9 VNC + 2 Step Jan./Feb.1998.
VNC + 1 Step [2] P. M. Corcoran, F. Papal, and A. Zoldi, “User interface technologies for
7
home appliances and networks,” IEEE Trans. Consumer Electron., vol.
5 44, no. 3, pp. 679-685, Aug. 1998.
angry mole animal akiyo MnD [3] K. Tsunashima, T. Shida, H. Kawano, T. Sato, and H. Kosaka,
Sequence “Compact programmable network display system for portable
projectors,” IEEE Trans. Consumer Electron., vol. 55, no. 2, pp. 312-
Fig. 14. Screen update rate for game sequences. 315, May 2009.
[4] D. Thommes, Q. Wang, A. Gerlicher, and C. Grecos, “RemoteUI: A
high-performance remote user interface system for mobile consumer
The proposed algorithms depend on the characteristics of
electronics devices,” Proc. of IEEE International Conference on
video sequences. As shown in Table IV, the gaming sequences Consumer Electronics (ICCE 2012), pp. 670-671, Jan. 2012.
relatively have the low ratio of modified blocks due to their little [5] P. M. Corcoran, F. Papal, and A. Zoldi, “User interface technologies for
motion and small texture regions. Meanwhile, the natural video home appliances and networks,” IEEE Trans. Consumer Electron., vol.
44, no. 3, pp. 679-685, Aug. 1998.
sequences such as ‘Akiyo’ and ‘MnD’ typically contain more [6] K. V. Kaplinsky, “VNC tight encoder-data compression for VNC,” Proc.
modified regions, and may lead to less performance gain. of the 7th International Scientific and Practical Conference of Students,
However, the proposed algorithms are still effective for the Post-graduates and Young Scientists (MTT 2001), pp. 155–157, Feb.
2001.
natural video sequences. As confirmed in Fig. 14, the proposed [7] K. J. Tan, “A remote thin client system for real time multimedia
algorithms achieve higher screen update rate than VNC. streaming over VNC,” Proc. of IEEE International Conference on
SUR is a performance metric which is ‘screen update rate’ as Multimedia and Expo (ICME 2010), pp. 992-997, July 2010.
[8] X. Zhang and H. Takahashi, “A hybrid data compression scheme for
measured at the client display. Note that SUR is conceptually improved VNC,” Systemics, Cybernetics and Informatics, Vol. 5, No. 2
equal to frame rate in the video playback. While we measure pp. 1-4, 2007.
and calculate the SUR per frame, we only report on the average [9] H. Shen, “A high-performance remote computing platform,” Proc. of
SUR over hundreds frames. Higher SUR is better. Figure 14 IEEE International Conference on Pervasive Computing and
Communication (PerCom 2009), pp. 1-6, Mar. 2009.
shows the average SUR, resulting from five detection methods [10] S. Rao, H. Vin, and A. Tarafdar, “Comparative evaluation of server-push
and the original VNC. Our system can provide higher SUR than and client-pull architectures for multimedia servers,” Proc. of the 6th
‘original VNC’ and ‘VNC with raster’ over test sequences. As a International Workshop Network and Operating System Support for
Digital Audio and Video (NOSSDAV), pp. 45-48, Apr. 1996.
result, our systems achieve higher screen update rate at the [11] C. Taylor and J. Pasquale, “Improving video performance in VNC under
client due to the reduced encoding complexity. We can see that high latency conditions,” Proc. of the International Symposium on
the proposed algorithms can reduce encoding complexity at the Collaborative Technologies and Systems (CTS 2010), pp.26-35, May 2010.
[12] W. J. Kim, K. Cho, and K. S. Chung, “Stage-based frame-partitioned
server significantly, and achieve higher screen update rate at the parallelization of H.264/AVC decoding,” IEEE Trans. Consumer
client. In addition, applying modified region coding almost Electron., vol. 56, no. 2, pp. 1088-1096, May 2010.
leads to the reduction of compression bit rate because only [13] Sheng Liang, Java Native Interface: Programmer's Guide and
Specification, 1st ed., Prentice Hall, 1999.
modified region is encoded. That is another benefit of the
[14] Jae-Hyeok Lee, Ha-Young Ko, Jong-Ok Kim, “Fast modified region
proposed method even though the aspect of encoding detection for mobile VNC systems,” Proc. of International Conference
complexity is primarily investigated in this paper. on Awareness Science and Technology (iCAST 2012), Aug. 2012.
1218 IEEE Transactions on Consumer Electronics, Vol. 58, No. 4, November 2012

BIOGRAPHIES Jong-Ok Kim (S’05-M’06) received the B.S. and M.S.

degrees in electronic engineering from Korea
Ha-Young Ko received the B.S. degree in Electronic University, Seoul, Korea, in 1994 and 2000,
Engineering from Korea University, Korea, in 2003, She respectively, and the Ph.D. degree in information
is currently pursuing M.S. degree in Electronic networking from Osaka University, Osaka, Japan, in
Engineering at Korea University, Korea. Her research 2006. From 1995 to 1998, he served as an officer in
interests are image signal processing, video compression the Korea Air Force. From 2000 to 2003, he was with
and multimedia communications. SK Telecom R&D Center and Mcubeworks Inc. in
Korea where he was involved in research and
development on mobile multimedia systems. From 2006 to 2009, he was a
researcher in ATR (Advanced Telecommunication Research Institute
Jae-Hyeok Lee received the B.S. degree in Electronic International), Kyoto, Japan. He joined Korea University, Seoul, Korea in
Engineering from Soongsil University, Seoul, Korea, in 2009, and is currently an associate professor. His current research
2011, He is currently pursuing the M.S. degree in interests are in the areas of multimedia communications, video
Electronic Engineering at Korea University, Seoul, compression, and image processing. Dr. Kim was a recipient of Japanese
Korea. His research interests are in the areas of signal Government Scholarship during 2003-2006.
processing, multimedia compression, and
communications.