Performance Comparison and Evaluation of Web Development Technologies in PHP, Python and Node - Js
Performance Comparison and Evaluation of Web Development Technologies in PHP, Python and Node - Js
net/publication/286594024
CITATIONS READS
38 4,478
3 authors, including:
Kai Lei
Peking University
165 PUBLICATIONS 741 CITATIONS
SEE PROFILE
Some of the authors of this publication are also working on these related projects:
ISAO: Knowledge Graph based NLP Information Retrieval and AI based Data Mining. View project
All content following this page was uploaded by Kai Lei on 30 December 2018.
Abstract — Large scale, high concurrency, and vast amount from others in these two aspects. Firstly, we consider from
of data are important trends for the new generation of website. both objective systematic tests (benchmark) and realistic user
Node.js becomes popular and successful to build data-intensive behavior tests (scenario) two sides and use the newest
web applications. To study and compare the performance of commercial testing tool LoadRunner. Secondly, we mainly
Node.js, Python-Web and PHP, we used benchmark tests and concern the performance of supporting concurrent users to
scenario tests. The experimental results yield some valuable meet the demand for IO-intensive real-time websites .
performance data, showing that PHP and Python-Web handle
much less requests than that of Node.js in a certain time. In This paper focuses on the impact on Web performance
conclusion, our results clearly demonstrate that Node.js is quite from three different Web technologies: Node.js, PHP and
lightweight and efficient, which is an idea fit for I/O intensive Python-Web. The security and scalability issues are beyond the
websites among the three, while PHP is only suitable for small scope of the paper. We mainly use the benchmark tests and
and middle scale applications, and Python-Web is developer scenario tests. In addition, one universal method of Web
friendly and good for large web architectures . To the best of our development technique’s evaluation based on the performance
knowledge, this is the first paper to evaluate these Web comparison is proposed in the paper, which can be used to
programming technologies with both objective systematic tests evaluate any new Web technology.
(benchmark) and realistic user behavior tests (scenario),
especially taking Node.js as the main topic to discuss. The main contributions of this paper are listed as follows.
Keywords—Web Development; Performance Evaluation; (1)We consider new web technology Node.js in our
Node.js; Benchmark Test; Scenario Test experiment and analyze the results of it. Then we compare it
with PHP and Python-Web, making a conclusion of which
situation they ought to be used.
I. INTRODUCTION
In the rapid development of Web today, many sites are (2)By means of benchmark tests and scenario tests, we can
faced with new problems, such as the problem multiuser evaluate performance from both objective systematic tests
requests and high concurrency. The dynamic scripting (benchmark) and realistic user behavior tests (scenario). There
language JavaScript has become enormously popular for client is often a dual impact on Web server performance, from the
and is widely used in Web development. Node.js stands for one calculation, and from the number of users. Our experiment has
new technology in JavaScript. Node.js is a platform built on taken each of these effects in account.
Chrome's JavaScript runtime for easily building fast, scalable The rest of this paper is organized as follows. Section 2
network applications [1]. Node.js uses an event-driven, non- discusses related work. Section 3 describes the test bed and
blocking I/O model that makes it lightweight and efficient, configurations in our experiment. Section 4 details our
perfect for data-intensive real-time applications that run across methodology and experimental design of tests. Section 5
distributed devices [1]. Node.js popularity surveys performed presents and analyzes the results of all tests. Section 6 makes a
by official website indicate that the average downloads are conclusion of the paper with a summary of our study and a
over 35,000 since the version 0.10 released in March 2013. future direction.
Corporations are quickly realizing the importance of Node.js
and five major PAAS providers have supported Node.js [2]. II. RELATED WORK
Nowadays, JavaScript has been the first popular language in
GitHub with 133,137 repositories [3]. And talking about There have been lots of studies evaluating and analyzing
evaluation of Web technologies’ performance, many Web server performance. Lance and Martin experimentally
researchers have done the related work. But our work differs evaluated the impact of three different dynamic content
662
IV. TEST METHODOLOGY Value of Fibonacci” and “Select Operation of DB”. “Hello
The experiment evaluated the results from two respects, World” module is a basic module to build a good Web server,
one from the server to do benchmark tests, the other from the then output “hello word” and distinguish the differences of
client to simulate the behavior of users to do scenario tests. In those three technologies. “Calculate Value of Fibonacci”
all the tests, we must follow one-factor-at-a-time experimental module is to calculate some value of Fibonacci and evaluate
design [16] to ensure the accuracy and effectiveness of tests. the performance under compute-intensive tests. “Select
Operation of DB” module is to compare different performance
through querying some value of DB in the IO-intensive
A. Benchmark Test situation. Under all benchmark tests, we keep requests 10000,
1) Benchmark Test Methodology and then we change users from 10 to 1000. TABLE ĉ
According to one-factor-at-a-time experimental design, we summarizes the factors in our experiments.
make three fundamental tests – “Hello World”, “Calculate
2) Benchmark Test Configuration high concurrency. The parameters modified were shown in
In the process of test, we found results of same module are TABLE ċ.
similar. For example, we choose PHP to make three tests under
requests 10000 and users 100. With the number of requests
TABLE III. THE MODIFIED CODE OF LINUX KERNEL PARAMETERS
from 0 to10000, the results of three tests are as shown in Fig.2.
The response time doesn’t have much difference in three tests net.ipv4.conf.default.rp_filter = 1
with the increase of concurrency requests. The average time of net.ipv4.conf.all.rp_filter = 1
net.ipv4.tcp_syncookies = 0
three tests is separate 0.311ms, 0.305ms, and 0.319ms. So we net.ipv6.conf.all.disable_ipv6 = 1
use one test result to evaluate performance in our experiment. net.ipv4.tcp_max_syn_backlog = 819200
In addition, we reboot server in every test to make sure more net.ipv4.tcp_synack_retries = 1
fair in whole experiment. net.ipv4.tcp_max_tw_buckets = 819200
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
B. Scenario Test
1) Scenario Test Methodology
Scenario test aims to simulate realistic user behavior . In
our experiment, we divide scenario tests into two parts, one is
“Login” scenario and the other is “Encryption” scenario. In
Fig. 2. Three tests of PHP under user 100 terms of “Login” scenario, it mainly simulates concurrent users
to login at the same time, and then compares the performance
In origin tests, we found some interrupt when concurrent of three Web technologies in the real IO-intensive scenario. In
requests increased to 200 in Python-Web. Thus we modified the test, we choose 500 users as rendezvous because it appears
the code of Ab and replaced some codes of line 1449 with some errors when users increase to 500. That also means the
codes shown in TABLEĊ. stress is beyond the maximum range Web server can stand. We
then use correct results when users are 500. In the test, we
make statistics of throughput, the average transaction response
TABLE II. THE MODIFIED CODE OF AB
time, and “hits per second” to compare performance of those
bad++; three technologies. Throughput displays the amount of data in
close_connection(c); bytes the Vusers receive from the server at any given second.
return; “Hits per second” displays the number of hits made on the Web
server by Vusers during each second of the load test.
“Encryption” scenario is to simulate a process to encrypt users’
Meanwhile, we modified the Linux Kernel parameters in login password when users login in. It’s mainly to compare
case that the system was regarded as SYN flood attack under performance in the simple real compute-intensive scenario. We
choose the same rendezvous as “Login” scenario.
663
2) Scenario Test Configuration addition, the current users that Node.js can hold are far more
In order to make enough fair in our experiment, we make than PHP, let alone Python-Web. So its performance is much
same configuration as benchmark tests. We reboot server in better than PHP and Python-Web when there are lots of users.
every scenario test.
2) “Calculate Value of Fibonacci” module
FIG.5 and FIG.6 show the Web performance to calculate
V. EXPERIMENTAL the tenth value of Fibonacci when requests are 10000 and
users’ number increases on and on. In terms of Node.js, the
A. Results and Analyses of Benchmark Tests “mean requests per second” is the highest value reaching up to
1) ĀHello World” module 2777.72 times per second and the “mean time per request” is
With growth of users, the performance of three 0.36ms when users are 100. In addition, the “mean requests per
technologies shows a trend of increasing before decreasing second” of Node.js keeps from 2000 to 2800. The peak
when keeping the requests at 10000. As FIG.3 and FIG.4 are requests per second decreases 1.5 times comparing with the
shown, the “mean requests per second” is the highest which same condition of “Hello World” benchmark tests. But to
increases to 3703.5 times per second when the users of Node.js Python-Web, this module has similar results as the last module,
are 100. Meanwhile, the “mean time per request” is the shortest even better results, at the same condition. PHP is also increased
which is 0.27ms. Then the “mean requests per second” slows to peak value when users’ number is up to 100, the “mean
down and maintains a steady state around 2700. As to Python- requests per second” at 3127.98. There is not much difference
Web, the “mean requests per second” keeps stable around 500 between it and “Hello World” module.
and the highest is 559.42. At this moment, the “mean time per
request” is the shortest which is 1.788ms. To PHP, it is also at
a peak when users are 100. The “mean requests per second” is
2977.54 at that time. The “mean time per request” is the
shortest 0.336ms. With users increasing, the “mean requests
per second” decreases to 200 and remains stable.
664
decrease much in performance, especially PHP which “mean Web technologies all don’t adapt to compute-intensive
requests per second” drops from 2000 to 2. Besides PHP, application. However, Node.js performs better among the three
Python-Web reduces from 600 to 3. Node.js also decreases in that test.
much from 2500 to 60. This phenomenon means those three
According to the results at above, we make several tests application, not compute-intensive application, because
with Node.js in order to find performance difference in various compute-intensive applications don’t exploit good advantages
concurrent users as the calculation increases. of Node.js.
FIG.7 and FIG.8 show the results of “Calculate Fibonacci 3) “Select Operation of DB” module
(10/20/30))” as the users grow from 10 to 1000. In order to validate the conclusion to prove Node.js is
adapted to IO-intensive application, we design a “Select
Operation of DB” module. FIG.9 and FIG.10 show the results
of this module.
Fig. 9. Results for “Select Operation of DB” mean requests per second
665
more suitable for IO-intensive application among the three, than Node.js with users increasing. However, Node.js stays
while PHP is applicable to small scale website. steady growth trend because its rate to deal with requests is
higher than PHP with the concurrent users are up. Thus
B. Results and Analyses of Scenario Tests Node.js takes better place when user requests increase.
To validate the results on benchmark tests, we choose two
scenarios as follows.
1) ĀLogin” scenario
We choose peak users at 500 to do tests in “Login”
scenario. We mainly observe “hits per second”, throughput and
average transaction response time as users goes up.
From FIG.11 to FIG.14, the results of “Login” scenario are
shown. The horizontal axis represent the number of concurrent
users, the vertical axis representing hits per second, throughput,
throughput trend, average transaction response time.
FIG.11 shows “hits per second” for different concurrent
users. “Hits per second” measures the number of HTTP Fig. 12. Throughput
requests sent to Web server from virtual users per second in
performance tests. “Hits per second” is larger and the stress to
Web server is larger. It can be seen from FIG.11 that “hits per
second” decreases in large degree when users are up to 150. It
is to say the system can’t hold so many users.
666
In FIG.16, the throughput of Node.js is 4,000,000 byte/s, PHP. Nevertheless, the response time of PHP and Node.js are
going down 1,000,000byte/s contrasting with FIG.12. From the in a good slowly increasing trend with the increasing users.
trend, the throughput of Node.js is keep between 3,000,000
byte/s and 4,000,000 byte/s. All at once, it’s steadily falling as
users increase. PHP is similar to it in “Login” scenario when
users are less than 50 and the throughput of PHP stays 500,000
byte/s now. Then it goes down by a large degree less than
1,000 byte/s. Python is alike with its performance of “Login”
scenario to keep stable, but has a slight decline around 50,000
byte/s. In short, three technologies all aren’t adapted to
compute-intensive application, especially PHP. However,
“Encryption” scenario brings PHP the least effect among the
three. Node.js is more suitable for IO-intensive application
rather than compute-intensive sites.
Fig. 18. Average transaction response time
VI. CONCLUSION
This paper presents a measurement study of three Web
techniques. To the best of our knowledge, this is the first paper
to compare and analyze performance of different Web
development technologies including new technology Node.js
from both objective systematic tests (benchmark) and realistic
user behavior tests (scenario) two aspects. It can get rid of the
deviation of a single kind test and make the results more
referenced and practical.
In short, Node.js performs much better than the traditional
technique PHP in high concurrency situation, no matter in
Fig. 17. Throughput trend benchmark tests or scenario tests. PHP handles small requests
well, but struggles with large requests. Besides, Node.js prefers
The average transaction response time is shown in FIG.18 to be used in the IO-intensive situation, not compute-intensive
and the time of PHP shows very unstable with the increasing sites. Python-Web is also not suitable for the compute-
users. On one hand, it’s due to the effect of “Encryption” intensive website.
scenario. On the other hand, it’s the mode of multi-process in
667
In general, Python-Web has many mature frames to [4] T.Lance, A.Martin and W.Carey, “Performance Comparison of
develop large scale websites, like YouTube and Source Forge. Dynamic Web Technologies”, ACM SIGMETRICS Performance
Evaluation Review, Volume 31 Issue 3, December 2003.
Node.js is an emerging technology and has many advantages in
[5] T.Scott, T.Michiaki, S.Toyotaro, T.Akihiko, and O.Tamiya,
IO-intensive situation, but it’s a little hard for developers who “Performance Comparison of PHP and JSP as Server-Side Scripting
don’t familiar with asynchronous programming. As to PHP, Languages”, Middleware, 2008.
it’s an old technique and popular to be used in small and [6] A.Ranjan, R.Kumar, J.Dhar, “A Comparative Study between Dynamic
middle scale sites. Web Scripting Languages”, Data Engineering and Management, 2012.
[7] J.Hu, S.Mungee, and D.Schmidt, “Techniques for Developing and
In our experiments, we only use the most fundamental Measuring High-Performance Web Servers over ATM Networks",
tests to compare and evaluate the performance of Web Proceedings of IEEE INFOCOM, San Francisco, CA, March/April 1998.
technologies. We just consider from the technologies, not [8] Y.Hu, A.Nanda, and Q.Yang, “Measurement, Analysis, and
including the architecture design. So our future work is focused Performance Improvement of the Apache Web Server", Technical
on the architecture and tries to improve the performance. The Report No. 1097-0001, University of Rhode Island, 1997.
paper mainly concerns the comparison of performance, but [9] E.Cecchet, A.Chanda, S.Elnikety, J.Marguerite, and W.Zwaenepoel,
security and extensibility also requires further examination. “Performance Comparison of Middleware Architectures for Generating
With the popularity of NoSQL, we could bring it into our Dynamic Web Content", Proceedings of 4th Middleware Conference,
Rio de Janeiro, Brazil, June 2003.
future experiments.
[10] U.Ramana, T.Prabhakar, “Some Experiments with the Performance of
LAMP Architecture”, Proceedings of the 2005 Fifth International
ACKNOWLEDGEMENT Conference on Computer and Information Technology, 2005.
[11] S.Warner, J.Worley, “SPECWeb2005 in the Real World: Using Internet
We are grateful to the anonymous reviewers for their Information Server (IIS) and PHP”, 2008 SPEC Benchmark Workshop,
valuable suggestions to improve this paper. This project has 2008.
been financially supported by National Development and [12] P.Neves, N.Paiva, J.Durães, “A comparison between JAVA and PHP”,
Reform Commission Fund of China ([2013]1309), Shenzhen C3S2E '13 Proceedings of the International C* Conference on Computer
Gov Projects (No: JSGG20140516162852628 ˈ Science and Software Engineering, 2013.
JCYJ20130331144541058 and JCYJ20130331144416448). [13] http://www.PHP.net/.
[14] https://www.Python.org/.
[15] R.Jain, “The Art of Computer Systems Performance Analysis:
REFERENCES Techniques for Experimental Design, Measurement, Simulation and
[1] http://Node.js.org/. Modeling”, John Wiley & Sons, Inc., New York, NY, 1991.
[2] http://strongloop.com/developers/Node-js- [16] S.Tilkov, S.Vinoski, “Node.js: Using JavaScript to Build High-
infographic/?utm_source=ourjs.com#3. Performance Network Programs”, IEEE Internet Computing, 2010.
[3] https://github.com/search?l=JavaScript&o=desc&q=stars%3A%3E1&s=
stars&type=Repositories.
668