General Performance Tuning

Published on 10-01-2010 01:12 AM Number of Views: 863

Below are several suggestions for tuning Wowza Server on your production hardware. If
you would like more specific instructions for configuring Wowza Server on your specific
hardware configuration, send an email with a detailed description of your hardware (# of
CPU/cores, memory, operating system, Java version...) to [email protected]:
1. It is generally best to run the most recent Sun Java JDK (http://java.sun.com).

Note: If you are on a Mac with OS X 10.5.x or greater (Leopard) with 64-bit
hardware you can run the Java 6 VM. To change your default Java VM in OSX,
open the Java preferences application /Applications/Utilities/Java/Java
Preferences and drag the Java SE 6 item to the top of theJava
Applications list. More information here: Changing the default JVM on OSX
Leopard

2. It is best to run a 64-bit operating system with the 64-bit Java VM. The 64-bit
VM enables Java heap sizes greater than 2GB.

Note: Wowza Media Server 2 will run properly on 64-bit Windows platforms.

3. The most recent development builds can be found here:

http://www.wowzamedia.com/devbuild.html

4. It is best to run the -server version of the Java VM. The instructions for
configuring the server version is described in these two forum threads:

Linux and OSX tuning, running the "server" Java VM (tuning)

Windows tuning, running the "server" Java VM (tuning)

5. The default Java heap size is set to 768MB which is great for development but
not large enough for high volume production use. If you are running the 64-bit
version of the Java VM and have 4GB or more of RAM in your machine, we suggest
a Java heap size of between 3000MB to 5000MB. If your are running the 32-bit
version of the Java VM or have less than 3GB of RAM in your machine, we suggest
a Java heap size of 1500MB. Do not attempt to set a heap size greater than
1800MB when using the 32-bit VM. On many Linux platforms the server will start
but will fail with anOutOfMemory error after a short period of time. Memory
settings can be adjusted in the following script files (based on platform):

1. [install-dir]/bin/setenv.sh (Linux, Mac OSX and Solaris)

JAVA_OPTS="-Xmx3000M"

2. [install-dir]/bin/setenv.bat (Windows)

set JAVA_OPTS=-Xmx3000M

3. [install-dir]/bin/WowzaMediaServer-Service.conf (Windows)

wrapper.java.additional.1=-Xmx3000M

2. Garbage collection (GC) tuning in Java is a tricky subject. What works best in
one server setup may not work in another. Through trial and error and customer
feedback we have several suggest approaches:

Garbage Collection Options:

1/4
We strongly recommend No Addtional Settings. The default -server tuning
(see step #4 above) will work quite well for many stream situations. So in most
cases this tuning works great without additional settings.

Only if you are experiencing problems with the Java Heap, such as a build up
of memory usage, you should try implementing one of the following Garbage
Collection options:
a. The Concurrent Collector: The concurrent collector is designed for
applications that prefer shorter garbage collection pauses and that can
afford to share processor resources with the garbage collector while the
application is running. The suggested additional settings are:
Code:
-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC

b. Expiremental: Garbage-First garbage collector (G1): In the Sun Java

6 Update 14 or greater release there is a new garbage collector that is in
the expirmental stage. It is designed for low pause time high throughput
applications such as Wowza Media Server. It is an option to explore. There
is more information below. To enable this garbage collector the addtional
settings are:
Code:
-XX:+UnlockExperimentalVMOptions -XX:+UseG1GC

3.
To change the garbage collection settings, modify the following scripts:
1. [install-dir]/bin/setenv.sh (add at line 10, Linux, Mac OSX and
Solaris). For example:

JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:

+CMSIncrementalMode -XX:+UseParNewGC"
2. [install-dir]/bin/setenv.bat (add at line 8, Windows). For example:

set JAVA_OPTS=%JAVA_OPTS% -XX:+UseConcMarkSweepGC -XX:

+CMSIncrementalMode -XX:+UseParNewGC

3. [install-dir]/bin/WowzaMediaServer-Service.conf (Windows). For

example:

wrapper.java.additional.[n]=-XX:+UseConcMarkSweepGC -XX:
+CMSIncrementalMode -XX:+UseParNewGC

Note: Set [n] to the next available number in the list

of wrapper.java.additional items. So if the last uncommented-
outwrapper.java.additional item is 8 then this item needs to be one
item below it with an index of 9.

There is more detailed information here:

Java SE 6 HotSpot[tm] Virtual Machine Garbage Collection Tuning

Java SE 6 Performance White Paper
The G1 Garbage Collector: Coming in Java 6 Update 14

2. For low latency chat applications, it is best to use smaller socket buffer sizes
(16000 bytes for read and write). The socket buffer sizes are configured
in [install-dir]/conf/VHost.xml:

Code:

2/4
 <ReceiveBufferSize>16000</ReceiveBufferSize>
<SendBufferSize>16000</SendBufferSize>

3. On Linux, increase the maximum number of file descriptors. This will address
the "Too many files open" error message. To do this edit the following two files:

/usr/local/WowzaMediaServer/bin/wms.sh
Code:
from:
#ulimit -n 20000

to:
ulimit -n 20000

/usr/local/WowzaMediaServer/bin/startup.sh
Code:
from:
#ulimit -n 20000

to:
ulimit -n 20000

Note: This may fail on some versions of Linux or if you do not have permission to
change the descriptor limit. If after making this change Wowza Server will not
start, consult the documentation for your Linux distribution to see how to increase
the descriptor limit.

Note: On some versions of Linux there is also a limit in the kernel that may need
to be increased. Consult the documentation for your distribution. You may need to
add the following line to your /etc/sysctl.conf file:
Code:
fs.file-max=20000

4. On Linux, switch to using the Anticipatory elevator (as) elevator algorithm.

It is described here:

Choosing an I/O Scheduler for Red Hat

5. On Linux, mount your disk with the noatime option. This operation will differ
based on your Linux distribution. Here is a basic description of the setting:

Using noatime Attribute

6. To tune your server based on the available CPU resources of your server, use
the following guidelines (this assumes a single virtual host, if you are running
more than one virtual host contact [email protected] for more detailed
instructions).

The [total-core-count] refers to the total number of cpu cores in your server.
For example if you have dual quad core processors (two quad core processors)
the [total-core-count] is:

2x4 = 8

With the number of cores and threads per physical processing growing and
growing we suggest a maximum number of threads for each value below:

3/4
In the configuration file [install-dir]/conf/VHost.xml:
HostPort/ProcessorCount: 2x[total-core-count] (maximum of
24) Note: The HostPort/ProcessorCount field in the Admin HostPort
(/Port "8086") should not be modified.
IdleWorkers/WorkerCount: 2x[total-core-count] (maximum of 24)

NetConnections/ProcessorCount: 2x[total-core-count] (maximum of 24)

RTP/DatagramConfiguration/UnicastIncoming/ProcessorCount: [total-
core-count] (maximum of 12)
RTP/DatagramConfiguration/UnicastOutgoing/ProcessorCount: 2x[total-
core-count] (maximum of 24)

RTP/DatagramConfiguration/MulticastIncoming/ProcessorCount: [total-
core-count] (maximum of 12)
RTP/DatagramConfiguration/MulticastOutgoing/ProcessorCount: [total-
core-count] (maximum of 12)

HandlerThreadPool/PoolSize: (300x[total-core-count])/5 (maximum of

480)
TransportThreadPool/PoolSize: (200x[total-core-count])/5 (maximum of
320)

This assumes you have at least 1GB of memory per core and that in the case you
have 4 or more total cores you are running the 64-bit Java VM using the
suggested memory settings above.

7. If you are not doing low latency streaming and you have a client side buffer of 3
or more seconds (NetStream.bufferTime), you can reduce the CPU load on the
machine and handle more concurrent sessions by editing [install-
dir]/conf/VHost.xml and changing the following two values:

Code:
IdleWorkers/CheckFrequency: 100
Client/IdleFrequency: 500

8. If you are having problems with multiple incoming multicast streams interfering
with each other on Linux you may need to set the Java
propertyjava.net.preferIPv4Stack to true. To do this, edit [install-
dir]/bin/setenv.sh and uncomment the following line (line 13):

Code:
JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"

9. This really isn't tuning but it seems like a good place... When streaming to the
Adobe Flash player, it is important to try and avoid RTMPT(tunneling version of
RTMP) as much as possible. RTMPT uses a polling mechanism which is very chatty
and CPU intensive. We suggest employing a protocol rollover strategy so that only
connections that need RTMPT use RTMPT. Is is described here:

http://www.wowzamedia.com/forums/content.php?50

4/4