SCALABILITY

REPORT
December 2019
  
 

Introduction 
 
The following scalability assessment was done for Bakery App Starter 
Flow application. For that purpose we set up a production-like 
scalability testing environment with Amazon EC2 instances. A local 
machine was used to execute Gatling tests against the environment. 
 
The purpose of this document is to show how well Bakery Flow 
scales up and to describe how the testing was done. We also provide 
some recommendations for optimizing the application and setup. 
 
First, we will go through the user journeys, then the test setup. Next, 
basic profiling will be described to show possible configuration 
problems and the most prominent scalability issues. After that, we 
will discuss about the memory usage and general CPU usage of 
Bakery Flow with our user journey. Finally, at the end of the 
document, we will make a summary, some remarks, and a short 
guide on how to continue testing by yourself in the future. 
 
 
 
 
 

​1​ ​Scalability Assessment Report

  
 

User journeys 
 
It is easy to identify at least two separate user journeys: Barista and 
Admin. It is expected that the Barista user journey is at least 10 times 
more common than the Admin user journey. Therefore, we will do 
the scalability test only for the Barista journey. 
 
Barista 
● Log in (view Storefront) 
● Click New Order 
● Fill in customer name and phone number 
● Add an item to the order (first from the combobox) 
● Add another item into order (second from the combobox) 
● Save order 
● Go back to storefront 
 
Admin 
● Log in (view Storefront) 
● Navigate to Dashboard 
● Navigate to Users 
● Navigate to Products 
● Click Strawberry Bun 
● Increase the price by $0.01 
● Click Save. 

​2​ ​Scalability Assessment Report

  
 

Test setup 
 
The application server and database were run on Amazon EC2 (EU 
Frankfurt) m5.2xlarge (8 virtual CPU cores and 32GB RAM) cluster 
consisting of two identical nodes. On the first node, we used 
embedded Spring boot Apache Tomcat/8.5.15 server with Hikari 
connection pool. Tomcat server’s maximum thread count was 
reduced from the default 200 to 100 to reduce thread switching 
overhead. A good guideline for setting the thread count is the 
formula below: 
 
connections = ((core_count * 2) + 1) 
 
(https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing) 
 
The Postgres 9.6 database server was run on the second node of the 
cluster. Hikari connection pool’s size was increased to 17 connections. 
The postgresql.conf file is shown in the appendix. Both the Tomcat 
server and the Postgres server were setup with help of Docker. Used 
Docker files are attached to the appendix. 
 
Gatling was run on a separate 6 core (12 thread) i7 machine. This 
machine was located in Turku, Finland. Our previous tests indicated 
that we were able to run Gatling with at least 10,000 concurrent 
users without significant CPU or network bandwidth limitations. To 

​3​ ​Scalability Assessment Report

be on the safe side, we still recommend using two separate 
machines (possible located in different networks) if going above 
10,000 users. Terminal commands used to run Gatling tests are also 
presented in the appendix. 
 
Tests were run in production mode, and loading of static resources 
such as html and js files were not part of the test (simulating 
separate CDN server).  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

​4​ ​Scalability Assessment Report

  
 

The size of a session 

 
Bakery Flow consumes about 175 MB of memory when idle. The size 
of a session is about 385 kB. The size of a session can be used to 
estimate the memory needed for a production server. For example, if 
you want to serve 2,000 concurrent sessions per server, you should 
reserve at minimum 4.8 GB of memory for it when calculating the 
number of hourly users and with assumption of a 30min session 
timeout, and the average visit time of 6min. 
 
Though, to be on the safe side and enable optimal performance 
(without frequent garbage collection), we recommend reserving at 
least 3.7 GB of memory for every 1,000 concurrent users. In addition, 
you should measure the session size every now and then when 
further developing the web application. The goal is to make sure that 
it will not increase too much or you might need to increase the 
memory reserved for your production servers. 
 
The figure below shows how the memory usage increases 
depending on the number of concurrent users. Please note the 
logarithmic y-axis. The figure is constructed based on the following 
assumptions: 
 
1. Average session size is 385 KB 

​5​ ​Scalability Assessment Report

 2. Session timeout is 30 min 
3. Average visit time is 6 min (after which the session is idle) 
4. Idle memory usage of the application is 175 MB 
5. Recommended memory is 1.5 * minimum memory 
 

 
 
 
 
 
 
 
 
 
 
 

​6​ ​Scalability Assessment Report

  
 

Profiling 
 
The profiling was done in the local development environment using 
Gatling, local Postgres database, and the JProfiler tool. The profiling 
was done with 1,000 concurrent users. Tomcat’s maxThread 
parameter was set to 10, and max heap size was 7 GB. The purpose of 
profiling before actual scalability tests is to verify and possibly fix (at 
least the low-hanging-fruit type issues) the application and the 
environment such that it would be scalable as possible. 
 
The picture below shows basic telemetric of the test when it was run 
for a couple of minutes with constant 1,000 users. Maybe the most 
interesting observations are as follows: 
 
● GC collect activity is modest since the application doesn’t use 
all of it’s 7 GB heap limit. 
● The CPU load is not very high. 
● Maybe most importantly, the time threads spent on the net I/O 
(light blue color) is very high when compared to the time spent 
on normal work (green color). Also there are a lot of idle threads 
(yellow color). 
 

​7​ ​Scalability Assessment Report

  
 
A more detailed view of live threads is shown below. It looks like that 
heavy database queries are the probable reason for the high net I/O. 
 

 
The view of JPA hotspots (below) reveals that over 70% of database 
time is spent fetching previous orders from the database. These 
orders are shown in Grid of Storefront view.  
 

​8​ ​Scalability Assessment Report

  
 
Below is the first general CPU hotspots graph, and the CPU call 
graph opened for the most important parts. 
 

 
 
This indicates clearly that the biggest part of the time (~40 %) is 
spent waiting (I/O) for the Storefront view’s database query to 
complete. After that, the next most significant hotspot is actual CPU 

​9​ ​Scalability Assessment Report

time (approximately ~60%) spent on the same query. One can easily 
identify following remedies for these hotspots: 
 
1. Make the first query (loading the content of Storefront's Grid) 
asynchronous. 
2. Cache (e.g. with Guava Loading Cache) content of the main 
view’s database query. 
3. Reduce the size of the query (i.e. the number of order rows 
fetched in the beginning). 
 
The reasoning behind the first one is that to reduce the synchronous 
I/O wait by freeing business logic to continue while data is fetched 
from the database. The second one is quite obvious, since the 
content of the Storefront view’s Grid is typically the same for every 
user, there is no reason to always fetch it from the database, Finally, 
the third one, at the moment Storefront view’s Grid is loaded with 50 
rows, even though typically only about 10 of those are visible. From 
an implementation point of view, the first one is the most 
complicated, thus we chose the second and third one for the 
optimizations before continuing testing in AWS environment. Below, 
we show how these optimizations can be applied to Bakery. 
 
First, caching the heaviest query. Our analyze above showed that the 
heaviest query is OrderService.findAnyMatchingAfterDueDate. Thus, 
we applied Guava LoadingCache for that query by adding the 
following code snippet to OrderService class: 
 
​ )
LoadingCache<Pair<Pageable, LocalDate>, Page<Order>> ​pageCache ​= CacheBuilder.​newBuilder(
.maximumSize(​1000​)
​
.expireAfterWrite(​5​, TimeUnit.​SECONDS)

​10​ ​Scalability Assessment Report

 .build(​new ​CacheLoader<Pair<Pageable, LocalDate>, Page<Order>>() {
public ​Page<Order> load(Pair<Pageable, LocalDate> keys) {
​return ​orderRepository​.findByDueDateAfter(keys.getValue(), keys.getKey());
}
});

and modified findAnyMatchingAfterDueDate method like below: 

public ​Page<Order> findAnyMatchingAfterDueDate(Optional<String> optionalFilter,

Optional<LocalDate> optionalFilterDate, Pageable pageable) {
​if ​(optionalFilter.isPresent() && !optionalFilter.get().isEmpty()) {
​if ​(optionalFilterDate.isPresent()) {
​return ​orderRepository​.findByCustomerFullNameContainingIgnoreCaseAndDueDateAfter(
optionalFilter.get(), optionalFilterDate.get(), pageable);
} ​else ​{
​return ​orderRepository​.findByCustomerFullNameContainingIgnoreCase(optionalFilter.get(),
pageable);
}
} ​else ​{
​if ​(optionalFilterDate.isPresent()) {
​ ry ​{
t
​return ​pageCache​.get(​new ​Pair<>(pageable, optionalFilterDate.get()));
} ​catch ​(ExecutionException e) {
​return ​orderRepository​.findByDueDateAfter(optionalFilterDate.get(), pageable);
}
} ​else ​{
​return ​orderRepository​.findAll(pageable);
}
}
}

To make this more reliable, one should fully or partially invalidate 

cache after there are known changes, such as new order entities, in 
the database. Another alternative is to keep cache lifetime short 
enough. In the following we decide to keep cache lifetime to 5 
seconds. Thus cache will expire every 5 seconds, even with such low 
lifetime the number of database queries was reduced to less than 2% 
of the original amount. We added similar cache also for two other 

​11​ ​Scalability Assessment Report

most used queries: OrderService.countAnyMatchingAfterDueDate 
and ProductService.find. 
 
The third remedy is very straightforward to apply. Just add the 
following method call to StorefrontView class where the Grid 
containing orders is initialized: 
 
grid​.setPageSize(​25​);

This will reduce the size of the query to half of it since the default 
page size is 50. 
 
To verify the effect of these changes we profiled the application 
again with JProfiler. There is first the same basic telemetric of the 
test when it was run for a couple of minutes with constant 1,000 
users. The proportion of I/O wait time has significantly reduced from 
~40% to <10%. The next figure shows the CPU hotspots after 
optimizations. We can observe that CPU spent to DB operations is 
less than one fifth of the original value. 
 

​12​ ​Scalability Assessment Report

Now the biggest part of time is spent waiting for login to complete. 
Our assumption is that this is mostly spent on general client-server 
communication and sending html, js, and css files to client side 
which could explain the remaining ~10% of I/O wait. One possible 
way to optimize this would be serving static files from a separate 
http server such as Nginx.  
 

 
 
We made these modifications and built a new version. The 
following tests were run on this optimized version of the 
application. 
 
 
 
 
 
 
 
 
 
 
 

​13​ ​Scalability Assessment Report

  
 

General scalability on 

the CPU level 
 
We measured the CPU usage and requests’ response times by 
running the recorded Gatling test (Barista.scala) on the Gatling 
Machine (see the Test setup chapter). Performance was measured by 
running increasing number (500, 1,500 and 2,500) of virtual 
concurrent users with the Gatling tool. The CPU usage was observed 
with Amazon CloudWatch dashboard. 
 
We can observe that 1,500 virtual users produced about 46% of the 
Tomcat server’s CPU usage. Adding 1,000 more users consumed 
about 76% of CPU. As a rough estimate, we suggest to have at least 1 
logical CPU core (for an application server) per every 400 concurrent 
users. Another alternative for adding more CPU cores is of course 
distributing the load by clustering (see Summary). The maximum 
CPU usages are presented in the table below for both the Tomcat 
server and the Postgres server. One can observe that 2,500 users 
consume about two thirds of the available CPU capacity which 
should not yet have a big effect on the response times (see the 
following chapter). It is noteworthy that our optimizations basically 
minimize the CPU consumption of the DB.  
 

​14​ ​Scalability Assessment Report

 Concurrent users  Max Avg (10s) CPU  Required CPU  Max Avg CPU 
Tomcat  cores estimate  Postgres 

500  15%  1.2  <1% 

1,500  46%  3.7  2% 
2,500  76%  6.1  5% 
 
Below, we have estimated how many peak concurrent users you 
should be able to serve with various configurations of AWS one-node 
Tomcat servers before consuming all the available CPU. 
 

Server  CPU cores  RAM (GB)  Peak concurrent users 

m5.large  2  8  800 
m5.xlarge  4  16  1600 
m5.2xlarge  8  32  3200 
m5.4xlarge  16  64  6400 
 
 
 
 
 
 
 
 
 
 
 

​15​ ​Scalability Assessment Report

  
 

User journey 1: Barista 

 
Below is the CPU load graph of the Tomcat server (orange line) for 
500 (red marker), 1,500 (orange marker) and 2,500 (green marker) 
virtual users. The blue line represents the CPU load for the Postgres 
server. The chart’s x-axis represents elapsed time from the beginning 
of the test and the y-axis represents CPU usage. The CPU percentage 
of 100% means that we are fully utilising all available CPU cores. 
 

 
 
In addition to the CPU usage we monitored the network usage of 
Tomcat (blue line: Tomcat network out, orange line: Tomcat in) and 

​16​ ​Scalability Assessment Report

Postgres servers (red line: in, green line: out). The graph is presented 
below. Our previous test with the same server has indicated that it is 
capable to serve several times more network traffic than what is now 
consumed with 2,500 users (~100Mb/s). 
 

 
 
In the figure below there are the response time percentiles over the 
duration of the tests for 500, 1,500 and 2,500 virtual users. With all 
these tested amounts, the response times of the requests were low 
(<500ms) for most of requests (90%). A very small percentage of 
requests (~1%) resulted in increased response times (~700ms) when 
having 1,500 users. With 2,500 users the percentage of slower 
requests (>700ms) was slightly bigger, around 5% of the total 
amount of the requests.  
 

​17​ ​Scalability Assessment Report

  
 
To summarize, we can deduct that the constant maximal concurrent 
user count for our server setup is around 2,500 users since there was 
a small but not yet significant increase in the response times 
compared to the case of 1,500 users. We did also ad-hoc tests with 
3,000 users, but in that case the increase of the response times were 
such big that constant 3,000 users would significantly affect the end 
user experience of the application. 
 
In the table below the data shown above is averaged over the test to 
show expected response times (in milliseconds) for different users. It 
also shows the server’s throughput in requests/second value. For 
 
 

​18​ ​Scalability Assessment Report

example column 95% for row 1,500 means that 95% of all requests are 
taking an average of 109ms to complete. 
 
Users  Min  Avg  50%  75%  95%  99%  Max%  r/s 

500  25  34  27  35  58  111  579  118 

1500  25  40  27  36  109  262  1602  353 

2500  25  49  29  38  129  421  2391  587 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

​19​ ​Scalability Assessment Report

 

 
 

Summary 
 
Having a cluster of two m5.2xlarge nodes for your Bakery App Starter 
Flow application, you should be able to serve constantly about 2,500 
concurrent users and still survive without major problems if the 
concurrent user count occasionally jumps to 3,000 users. Since our 
optimizations reduced the CPU load of the DB server, you could use 
a lower end server for that, e.g. m5.large (2 CPU cores and 8 GB RAM). 
It should be even possible to use the same server used for Bakery for 
DB too. 
 
To translate this to hosting costs, let's assume we would host a 
business system with similar resource use to Bakery for 10,000 users. 
Let’s further assume that the globally distributed workforce uses the 
system so that there are 1,000 - 2,500 concurrent users depending on 
the time of the day. We are taking into account the optimization 
presented and using a single m5.2xlarge AWS reserved instance 
(standard 1-year term) for hosting the application and the DB costs 
$0.2/user/year. 
 
Some additional improvements might be possible to achieve using 
optimised garbage collection, native application server libraries and 

​20​ ​Scalability Assessment Report

by serving static resources from a basic http server (e.g. Nginx or 
Apache2) instead of from an application server such as Tomcat. 
 
You can refer to the table in the beginning of page 11 if you 
considering scaling vertically (i.e. using faster server) the application 
server for bigger amount of concurrent users. On the other hand, we 
recommend clustering your application server (with sticky sessions) 
at least if you are expecting to have constantly more than 5,000 
concurrent users on it. By clustering you should be able to scale out 
Bakery Flow to theoretically any amount of concurrent users. 
Clustering in an early phase also adds capacity for possible rush 
usage even though the expected average load would be lower. 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

​21​ ​Scalability Assessment Report

  
 

Appendix 
 
postgresql.conf generated with https://pgtune.leopard.in.ua/  
 
max_connections = 200
shared_buffers = 7680MB
effective_cache_size = 23040MB
maintenance_work_mem = 1920MB
checkpoint_completion_target = 0.7
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 10485kB
min_wal_size = 1GB
max_wal_size = 2GB
max_worker_processes = 16
max_parallel_workers_per_gather = 4

Dockerfiles 
Tomcat server: 
FROM ​anapsix​/​alpine-java:8_server-jre
ADD ​bakery-app.war .

ARG ​dbip
​ flow_bakery
ARG ​dbname=
​ flow_bakery_user
ARG ​dbuser=
​ flow_bakery_user_pw
ARG ​dbpw=

​ $​dbip
ENV ​dbip=
​ $​dbname
ENV ​dbname=

​22​ ​Scalability Assessment Report

 ​ $​dbuser
ENV ​dbuser=
​ $​dbpw
ENV ​dbpw=

ENTRYPOINT ​java ​-​jar \

​-​Xmx24g ​-​Xms24g \
​-​Dvaadin.productionMode \
​ ​5432​/​$​dbname \
​-​Dspring.datasource.url=jdbc:postgresql:​//​$​dbip: ​
​-​Dspring.datasource.username=$​dbuser ​\
​
​-​Dspring.datasource.password=$​dbpw \
Bakery-app.war

Postgres server: 
FROM ​library​/​postgres:​9.6.9

RUN ​printf ​"max_connections = 200\nshared_buffers = 7680MB\neffective_cache_size =

23040MB\nmaintenance_work_mem = 1920MB\ncheckpoint_completion_target = 0.7\nwal_buffers =
16MB\ndefault_statistics_target = 100\nrandom_page_cost = 1.1\neffective_io_concurrency =
200\nwork_mem = 393kB\nmin_wal_size = 1GB\nmax_wal_size = 2GB\nmax_worker_processes =
16\nmax_parallel_workers_per_gather = 4\n" ​>> /​usr​/​share​/​postgresql​/p
​ ostgresql.conf.sample

ENV ​POSTGRES_USER ​flow_bakery_user

ENV ​POSTGRES_PASSWORD ​flow_bakery_user_pw
​ low_bakery
ENV ​POSTGRES_DB f
EXPOSE ​5432

Gatling commands 
Gatling test were run with Gatling Maven plugin under the project 
with the following parameters, where the IP address (marked as 
xx.xxx.xxx.xx) was the application server’s IP. 
 
500 users: 
mvn -Pscalability gatling:test -Dgatling.sessionCount=500 -Dgatling.sessionStartInterval=140
-Dgatling.sessionRepeats=4 -Dgatling.baseUrl=​http://xx.xxx.xxx.xx:8080  

1,500 users: 
mvn -Pscalability gatling:test -Dgatling.sessionCount=1500 -Dgatling.sessionStartInterval=140
-Dgatling.sessionRepeats=4 -Dgatling.baseUrl=​http://xx.xxx.xxx.xx:8080

​23​ ​Scalability Assessment Report

2,500 users: 
mvn -Pscalability gatling:test -Dgatling.sessionCount=2500 -Dgatling.sessionStartInterval=140
-Dgatling.sessionRepeats=4 -Dgatling.baseUrl=http://xx.xxx.xxx.xx:8080  

​24​ ​Scalability Assessment Report

  
 
 
 
 
 
 
Want to discover the scalability 
potential of your business web app 
built with Vaadin?