Heroku Dev Center H g D C

Get Started Docs Changelog Search

Heroku Architecture › Dynos (app containers) › Dynos and the Dyno Manager

Dynos and the Dyno Manager

D Last updated 09 October 2019

G Table of Contents
Dyno configurations

The Dyno Manager

Dyno types

Scalability

Redundancy

Isolation and security

Networking

CLI commands for dyno management

Dyno sleeping

Startup

Restarting

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Shutdown

Dyno lifecycle latencies

Memory behavior

Connecting to external services

Dynos and requests

All Heroku applications run in a collection of lightweight Linux containers called dynos. This
article describes dyno conventions on the Heroku platform.

For information about dyno pricing, see the Heroku pricing overview.

Dyno configurations
Every dyno belongs to one of the three following configurations:

Web: Web dynos are dynos of the “web” process type that is defined in your Procfile.
Only web dynos receive HTTP traffic from the routers.

Worker: Worker dynos can be of any process type declared in your Procfile, other than
“web”. Worker dynos are typically used for background jobs, queueing systems, and
timed jobs. You can have multiple kinds of worker dynos in your application. For example,
one for urgent jobs and another for long-running jobs. For more information, see Worker
Dynos, Background Jobs and Queueing.

One-off: One-off dynos are temporary dynos that can run detached, or with their
input/output attached to your local terminal. They’re loaded with your latest release.

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 They can be used to handle administrative tasks, such as database migrations and
console sessions. They can also be used to run occasional background work, as with
Heroku Scheduler. For more information, see One-Off Dynos.

Once a web or worker dyno is started, the dyno formation of your app will change (the
number of running dynos of each process type) - and subject to dyno lifecycle, Heroku will
continue to maintain that dyno formation until you change it. One-off dynos, on the other
hand, are only expected to run a short-lived command and then exit, not affecting your dyno
formation.

The Dyno Manager

The dyno manager keeps dynos running automatically; so operating your app is generally
hands-off and maintenance free. The Common Runtime has a single Dyno Manager per
region that is responsible for managing all dynos across all tenants running in a region. The
Private Spaces Runtime has a dedicated Dyno Manager per space. This Dyno Manager only
manages dynos that run within the space

Dyno types
Heroku provides a number of different dyno types each with a set of unique properties and
performance characteristics. Free, Hobby, Standard and Performance dynos are available in
the Common Runtime to all Heroku customers. Private Dynos only run in Private Spaces and
are available in Heroku Enterprise.

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Scalability
To scale horizontally (scale out), add more dynos. For example, adding more web dynos
allows you to handle more concurrent HTTP requests, and therefore higher volumes of
traffic. For more information, see Scaling Your Dyno Formation.

To scale vertically (scale up), use bigger dynos. The maximum amount of RAM available to
your application depends on the dyno type you use. For more information, see Dyno Types
for Common Runtime and Heroku Enterprise for Private Spaces.

Both horizontal and vertical scale are features of the professional dynos, and are not
available to free or hobby dynos.

Redundancy
Applications with multiple running dynos will be more redundant against failure. If some
dynos are lost, the application can continue to process requests while the missing dynos are
replaced. Typically, lost dynos restart promptly, but in the case of a catastrophic failure, it
can take more time. Multiple dynos are also more likely to run on different physical
infrastructure (for example, separate AWS Availability Zones), further increasing
redundancy.

Isolation and security

All dynos are strongly isolated from one another for security purposes. Heroku uses OS
containerization with additional custom hardening to ensure that access is properly
restricted for all customers.

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Free, Hobby and Standard dynos, even though completely isolated, may share an underlying
compute instance. Heroku employs several techniques to ensure fair use of the underlying
resources. However, these dyno types may experience some degree of performance
variability depending on the total load on the underlying instance.

Performance and Private dynos do not share the underlying compute instance with other
dynos. Therefore, these dyno types are not only more powerful but also experience low
variability in performance. In addition to having dedicated compute resources, Private dynos
are furthermore isolated in their own virtual network determined by the Private Space they
are deployed in.

Ephemeral filesystem

Each dyno gets its own ephemeral filesystem, with a fresh copy of the most recently
deployed code. During the dyno’s lifetime its running processes can use the filesystem as a
temporary scratchpad, but no files that are written are visible to processes in any other dyno
and any files written will be discarded the moment the dyno is stopped or restarted. For
example, this occurs any time a dyno is replaced due to application deployment and
approximately once a day as part of normal dyno management.

System clock and time synchronization

Dynos use the Network Time Protocol (NTP) for time synchronization indirectly via kernel
passthrough. The dyno’s underlying host is configured to use time services provided by the
Heroku platform’s Stratum 2 NTP servers. These servers are in turn synchronized to a pool of

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Stratum 1 time servers provided by NIST. Heroku uses the same time servers across its
entire fleet to ensure homogeneous and synchronous time regardless of location.

Networking
Each dyno has its own network interface. The surrounding network configuration depends on
the type of Runtime.

Common Runtime networking

The Common Runtime provides strong isolation by firewalling all dynos off from one another.
The only traffic that can reach a dyno is web requests forwarded from the router to web
processes listening on the port number specified in the $PORT environment variable. Worker
and one-off dynos cannot receive inbound requests.

Individual processes within a dyno can bind to any address or port they want and
communicate among them using e.g. standard TCP. The external networking interface (i.e.:
eth0) for each dyno will be part of a /30 private subnet in the range 172.16.0.0/12, such as
172.16.83.252/30 or 172.30.239.96/30. Processes within one dyno don’t share IPs or
subnets with other dynos, nor can they observe TCP session state of other dynos.

All dyno types in the Common Runtime can make outbound requests to services running
elsewhere on the internet. The originating IP address for these requests cannot be controlled
by the user.

Private Spaces Runtime networking

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 p g

Dynos in a Private Space are all connected via a virtual private network configured as part of
the space. Add-on data services installed in the space are also connected to this network.
Similar to the Common Runtime, web processes can receive web requests by listening on the
port number specified in the $PORT environment variable. In addition, any process in a dyno
can choose to listen on a port number of choice and receive connections from other dynos on
the private network. This is supported for web, worker and one-off processes.

Trusted IP Ranges can be used to control which client IPs are allowed to communicate with
the applications in a Private Space.

Dynos in a Private Space make outbound connections to other internet services via a NAT
gateway which ensures that all connections originate from a set of stable outbound IP
addresses.

CLI commands for dyno management

To view and modify your app’s dyno settings, you can use the Heroku CLI.

Task Example See Also

List the dynos for an app heroku ps Scaling

Start worker dynos. (Look at your Procfile to see the worker process types heroku ps:scale
Scaling
that are defined for your app) worker=2

heroku ps:stop
Stop a particular dyno type * Scaling
worker

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Task Example See Also

heroku ps:stop
Stop a particular dyno * Scaling
worker.2

Dyno
Restart all dynos heroku ps:restart
Manager

heroku ps:restart Dyno

Restart a particular dyno type
web Manager

heroku ps:restart Dyno

Restart a particular dyno
web.1 Manager

heroku ps:scale
Scale horizontally (Add more dynos) Scaling
web=2

heroku ps:scale
Scale horizontally by incrementing the current number of dynos Scaling
web+5

heroku ps:scale
Scale different dyno types horizontally at the same time Scaling
web=1 worker=5

heroku ps:resize Dyno

Scale vertically (Use bigger dynos)
worker=standard-2x Types

Scale horizontally and vertically at the same time. This example scales the heroku ps:scale Dyno
number of web dynos to 3 and resizes them to performance-l web=3:performance-l Types

Get help for the heroku ps command heroku ps --help

One-Off
Launch a one-off dyno that runs bash in a console heroku run bash
Dynos

Launch a one-off dyno that runs the “worker” process type that is present in One-Off
heroku run worker
your application’s Procfile Dynos

heroku logs or
View logs Logging
heroku logs --tail

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 * Running ps:stop on dynos that are part of a scaled process will automatically be restarted.
In Private Spaces, ps:stop will terminate and replace the dedicated instance running the
dyno(s). To permanently stop dynos, scale down the process.

It is also possible to modify some of your app’s dyno settings with the Heroku Dashboard.

Dyno sleeping
Only free dynos will sleep. For more information, see Free Dyno Hours.

Startup

The .profile file

During startup, the container starts a bash shell that runs any code in $HOME/.profile
before executing the dyno’s command. You can put bash code in this file to manipulate the
initial environment, at runtime, for all dyno types in your app.

The .profile script will be sourced after the app’s config vars. To have the config vars take
precedence, use a technique like that shown here with LANG .

# add vendor binaries to the path

export PATH=$PATH:$HOME/vendor/bin

# set a default LANG if it does not exist in the environment

export LANG=${LANG:-en_US.UTF-8}

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 t For most purposes, config vars are more convenient and flexible than .profile. You need
not push new code to edit config vars, whereas .profile is part of your source tree and
must be edited and deployed like any code change.

Local environment variables

The Dyno Manager sets up a number of default environment variables that you can access in
your application.

If the dyno is a web dyno, the $PORT variable will be set. The dyno must bind to this port
number to receive incoming requests.

t The $DYNO variable is experimental and subject to change or removal. Also, $DYNO is
not guaranteed to be unique within an app. For example, during a deploy or restart, the
same dyno identifier could be used for two running dynos. It will be eventually
consistent, however.

The $DYNO variable will be set to the dyno identifier. e.g. web.1 , worker.2 , run.9157 .

Processes

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 After the .profile script is executed, the dyno executes the command associated with the
process type of the dyno. For example, if the dyno is a web dyno, then the command in the
Procfile associated with the web process type will be executed.

Any command that’s executed may spawn additional processes.

Orphan processes within a dyno will be regularly reaped to prevent the accumulation of
zombie/defunct processes.

Process/thread limits
The maximum number of processes/threads that can exist in a dyno at any one time
depends on dyno type:

free , hobby and standard-1x dynos support no more than 256

standard-2x and private-s dynos support no more than 512

performance-m and private-m dynos support no more than 16384

performance-l and private-l dynos support no more than 32768

These limits include all processes and threads, whether they are executing, sleeping or in any
other state. Note that the dyno counts threads and processes towards this limit. For
example, a standard-1x dyno with 255 threads and one process is at the limit, as is a dyno
with 256 processes.

Web dynos

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 A web dyno must bind to its assigned $PORT within 60 seconds of startup. If it doesn’t, it is
terminated by the dyno manager and a R10 Boot Timeout error is logged. Processes can bind
to other ports before and after binding to $PORT .

t Contact support to increase this limit to 120 seconds on a per-application basis. In

general, slow boot times will make it harder to deploy your application and will make
recovery from dyno failures slower, so this should be considered a temporary solution.

Restarting

Automatic dyno restarts

The dyno manager restarts all your app’s dynos whenever you:

create a new release by deploying new code

change your config vars

change your add-ons

run heroku restart

Dynos are also restarted (cycled) at least once per day to help maintain the health of
applications running on Heroku. Any changes to the local filesystem will be deleted. The
cycling happens once every 24 hours (plus up to 216 random minutes, to prevent every dyno

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 for an application from restarting at the same time). Manual restarts ( heroku ps:restart )
and releases (deploys or changing config vars) will reset this 24 hour period. Cycling happens
for all dynos, including one-off dynos, so dynos will run for a maximum of 24 hours + 216
minutes. If you have multiple dynos, they should cycle at different times based on the
random 0 to 216 minutes difference. If you continually make changes to your application
without a 24 hour gap, you won’t see cycling at all. When a dyno cycles, you will see a log
entry similar to this:

2015-08-18T06:20:13+00:00 heroku[web.1]: Cycling

In addition, dynos are restarted as needed for the overall health of the system and your app.
For example, the dyno manager occasionally detects a fault in the underlying hardware and
needs to move your dyno to a new physical location. These things happen transparently and
automatically on a regular basis and are logged to your application logs.

Dynos are also restarted if the command used to start the dyno, exits. The cases when the
command used to start a dyno can exit, are as follows:

Defect in startup code - If your app is missing a critical dependency, or has any other
problem during startup, it will exit immediately with a stack trace.

Transient error on a resource used during startup - If your app accesses a resource
during startup, and that resource is offline, it may exit. For example, if you’re using
Amazon RDS as your database and didn’t create a security group ingress for your
Heroku app, your app will generate an error or time out trying to boot.

Segfault in a binary library - If your app uses a binary library (for example, an XML
parser), and that library crashes, then it may take your entire application with it.
Exception handling can’t trap it, so your process will die.

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Interpreter or compiler bug - The rare case of a bug in an interpreter (Ruby, Python) or in
the results of compilation (Java, Scala) can take down your process.

Dyno crash restart policy

t A dyno “crash” represents any event originating with the process running in the dyno
that causes the dyno to stop. That includes the process exiting with an exit code of 0 (or
any other exit code).

The Common Runtime implements an incremental backoff policy for crashing dynos:

The first time a dyno crashes, it will be restarted immediately.

If the dyno crashes again, it will be subject to a cool-off period before a restart is
attempted.

The first cool-off period is up to 20 minutes, the next one is up to 40 minutes, then up to
60 minutes, up to 180 minutes and finally up to 320 minutes.

After the 320 minute cool-off period is reached, restart attempts happen every 320
minutes.

The cool-off period will be reset when the dyno starts correctly, or when you push a new
release to your app, or when you restart your app (for example, by typing heroku
restart ), or when you scale your dynos to 0 and then scale them back up again.

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 The Private Spaces Runtime does not have a backoff policy. When a dyno crashes it will be
continuously restarted with no cool-off period.

Shutdown

Graceful shutdown with SIGTERM

When the dyno manager restarts a dyno, the dyno manager will request that your processes
shut down gracefully by sending them a SIGTERM signal. This signal is sent to all processes in
the dyno, not just the process type.

t Please note that it is currently possible that processes in a dyno that is being shut down
may receive multiple SIGTERMs

The application processes have 30 seconds to shut down cleanly (ideally, they will do so
more quickly than that). During this time they should stop accepting new requests or jobs
and attempt to finish their current requests, or put jobs back on the queue for other worker
processes to handle. If any processes remain after that time period, the dyno manager will
terminate them forcefully with SIGKILL .

When performing controlled or periodic restarts, new dynos are spun up as soon as
shutdown signals are sent to processes in the old dynos.

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 We can see how this works in practice with a sample worker process. We’ll use Ruby here as
an illustrative language - the mechanism is identical in other languages. Imagine a process
that does nothing but loop and print out a message periodically:

STDOUT.sync = true
puts "Starting up"

trap('TERM') do
puts "Graceful shutdown"
exit
end

loop do
puts "Pretending to do work"
sleep 3
end

If we deploy this (along with the appropriate Gemfile and Procfile ) and heroku ps:scale
worker=1 , we’ll see the process in its loop running on dyno worker.1 :

$ heroku logs
2011-05-31T23:31:16+00:00 heroku[worker.1]: Starting process with command: `bundl
2011-05-31T23:31:17+00:00 heroku[worker.1]: State changed from starting to up
2011-05-31T23:31:17+00:00 app[worker.1]: Starting up
2011-05-31T23:31:17+00:00 app[worker.1]: Pretending to do work
2011-05-31T23:31:20+00:00 app[worker.1]: Pretending to do work
2011-05-31T23:31:23+00:00 app[worker.1]: Pretending to do work

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Restart the dyno, which causes the dyno to receive SIGTERM :

$ heroku restart worker.1

Restarting worker.1 process... done

$ heroku logs
2011-05-31T23:31:26+00:00 app[worker.1]: Pretending to do work
2011-05-31T23:31:28+00:00 heroku[worker.1]: State changed from up to starting
2011-05-31T23:31:29+00:00 heroku[worker.1]: Stopping all processes with SIGTERM
2011-05-31T23:31:29+00:00 app[worker.1]: Graceful shutdown
2011-05-31T23:31:29+00:00 heroku[worker.1]: Process exited

Note that app[worker.1] logged “Graceful shutdown” (as we expect from our code); all the
dyno manager messages log as heroku[worker.1] .

If we modify worker.rb to ignore the TERM signal, like so:

STDOUT.sync = true
puts "Starting up"

trap('TERM') do
puts "Ignoring TERM signal - not a good idea"
end

loop do
puts "Pretending to do work"
sleep 3
end

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Now we see the behavior is changed:

$ heroku restart worker.1

Restarting worker.1 process... done

$ heroku logs
2011-05-31T23:40:57+00:00 heroku[worker.1]: Stopping all processes with SIGTERM
2011-05-31T23:40:57+00:00 app[worker.1]: Ignoring TERM signal - not a good idea
2011-05-31T23:40:58+00:00 app[worker.1]: Pretending to do work
2011-05-31T23:41:01+00:00 app[worker.1]: Pretending to do work
...
2011-05-31T23:41:25+00:00 app[worker.1]: Pretending to do work
2011-05-31T23:41:27+00:00 heroku[worker.1]: Error R12 (Exit timeout) -> Process f
2011-05-31T23:41:27+00:00 heroku[worker.1]: Stopping all processes with SIGKILL
2011-05-31T23:41:28+00:00 heroku[worker.1]: Process exited

Our process ignores SIGTERM and blindly continues on processing. After 30 seconds, the
dyno manager gives up on waiting for the process to shut down gracefully, and kills it with
SIGKILL . It logs Error R12 to indicate that the process is not behaving correctly.

Dyno lifecycle latencies

Dynos in the Common Runtime are optimized for control responsiveness. It only takes a few
seconds to start a one-off dyno process or to scale up a web or worker process.

Dynos in Private Spaces are optimized for robustness and performance. Starting a one-off
dyno or adding one more dyno to an existing web or worker formation can take a few

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 minutes.

Memory behavior
The maximum amount of RAM available to your application depends on the dyno type you
use. The dyno manager will restart your dyno and log an R15 error if the memory usage of a:

free , hobby or standard-1x dyno reaches 1 GB, two times its quota.

standard-2x dyno reaches 2 GB, two times its quota.

performance-m dyno reaches 5 GB, two times its quota.

performance-l dyno reaches 28 GB, two times its quota.

private-s dyno reaches its quota of 1 GB.

private-m dyno reaches its quota of 2.5 GB.

private-l dyno reaches its quota of 14 GB.

Using a dyno type that is too small might cause constant memory swapping, which will
degrade application performance. Application metrics data, including memory usage, is
available via the Metrics tab of the Heroku Dashboard. You can also measure memory with
log-runtime-metrics. Memory usage problems might also be caused by memory leaks in your
app. If you suspect a memory leak, memory profiling tools can be helpful.

r Swap is not available on all dynos in Private Spaces, e.g. Private-M. If you utilize all the
dyno’s memory, it will be restarted.

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 i Using small amounts of swap space and infrequent memory swapping are usually not
problems. Even when your application hasn’t reached its memory limit, it’s common to
see small amounts of memory being swapped to disk as the operating system manages
memory and available disk cache.

Connecting to external services

Applications running on dynos can connect to external services. Heroku can run apps in
multiple regions, so for optimal latency run your services in the same region as the app.

Dynos and requests

A single dyno can serve thousands of requests per second, but performance depends greatly
on the language and framework you use.

A single-threaded, non-concurrent web framework (like Rails 3 in its default configuration)

can process one request at a time. For an app that takes 100ms on average to process each
request, this translates to about 10 requests per second per dyno, which is not optimal.

r Single threaded backends are not recommended for production applications because of
their inefficient handling of concurrent requests. Choose a concurrent backend
whenever developing and running a production service.

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Multi-threaded or event-driven environments like Java, Unicorn, EventMachine, and Node.js
can handle many concurrent requests. Load testing your app is the only realistic way to
determine request throughput.

b Keep reading
B Dynos (app containers)
e Scaling Your Dyno Formation
e Dyno Types
e One-Off Dynos

o Feedback
Log in to submit feedback.

JDyno Runtimes K
Dyno Types

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 INFO R M ATION & SU PPORT

Getting Started
Documentation
Changelog
Compliance Center
Training & Education
Blog
Podcasts
Support Channels
Status

LANGUAG E REFERE NCE

Node.js
Ruby
Java
PHP
Python
Go
Scala
Clojure

OTHE R R ESOURCES

Careers
Elements
Products
Pricing

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Subscribe to our monthly newsletter

Your email address GO

HEROKU IS A COMPANY

 ©2019 Salesforce.com

heroku.com Terms of Service Privacy Cookies

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD