rate_max [FBS] max number of new sessions per second add acl <acl> <pattern>

HAProxy​®​ Troubleshooting Reference Card req_rate [F] HTTP requests per second over last elapsed second add map <map> <key> <value>
2020-03-02 - Use under Creative Commons License (CC BY-SA) req_rate_max [F] max number of HTTP requests per second observed del acl <acl> [ <key> | #<ref> ]
For version 2.2-dev3 req_tot [FB] total number of HTTP requests received del map <map> [ <key> | #<ref> ]
reuse [BS] cumulative number of connection reuses set maxconn frontend <frontend> <value>
rtime [BS] average response time in ms over the 1024 last requests set maxconn server <backend>/<server> <value>
Stats field names (Management Guide §9) rtime_max [BS] the maximum observed response time in ms (0 for TCP) set maxconn global <maxconn>
show stat typed desc scur [LFBS] current sessions set rate-limit connections global <value>
act [BS] number of active servers (backend), server is active (server) sid [LS] server id (unique inside a proxy) set rate-limit http-compression global <value>
addr [LS] address:port or "unix". slim [LFBS] configured session limit set rate-limit sessions global <value>
agent_code [S] numeric code reported by agent if any (unused for now) smax [LFBS] max sessions set rate-limit ssl-sessions global <value>
agent_desc [S] short human-readable description of agent_status src_ilim [S] limit on the number of available idle connections set server <backend>/<server> addr {<ip4>|<ipv6>} [port <port>]
agent_duration [S] time in ms taken to finish last check srv_abrt [BS] number of data transfers aborted by the server set server <backend>/<server> agent [ up | down ]
agent_fall [S] agent's "fall" parameter, normally 1 srv_icur [S] current number of idle connections available for reuse set server <backend>/<server> agent-addr <addr>
agent_health [S] agent's health parameter, between 0 and rise+fall-1 status [LFBS] status (UP/DOWN/NOLB/MAINT/MAINT) set server <backend>/<server> agent-send <value>
agent_rise [S] agent's "rise" parameter, normally 1 stot [LFBS] cumulative number of sessions set server <backend>/<server> health [ up | stopping | down ]
agent_status [S] status of last agent check svname [LFBS] service name set server <backend>/<server> check-port <port>
algo [B] load balancing algorithm throttle [S] current throttle percentage for the active server set server <backend>/</server> state [ready|drain|maint]
bck [BS] number of backup servers (B), server is backup (S) tracked [S] id of proxy/server if tracking is enabled set server <backend>/<server> weight <weight>[%]
bin [LFBS] bytes in ttime [S] avg total session time in ms over the 1024 last requests set server <backend>/<server> fqdn <FQDN>
bout [LFBS] bytes out ttime_max [BS] the maximum observed total session time in ms set weight <backend>/<server> <weight>[%]
btot [BS] total number of times a server was selected type [LFBS] 0=frontend, 1=backend, 2=server, 3=socket/listener set ssl ocsp-response <response|payload>
cache_hits [FB] cumulative number of cache hits weight [BS] total weight (backend), server weight (server) set ssl tls-key <id> <tlskey>
cache_lookups [LB] cumulative number of cache lookups wredis [BS] number of times a request was redispatched to another server
check_code [S] layer5-7 code, if available wretr [BS] number of times a connection to a server was retried clear counters [all]
check_desc [S] short human-readable description of check_status wrew [LFBS] cumulative number of failed header rewriting warnings clear acl <acl>
check_duration [S] time in ms took to finish last health check clear map <map>
check_fall [S] server's "fall" parameter used by checks clear table <table> [[data.<type> <operator> <value>]|[key <key>]]
check_health [S] server's health check value between 0 and rise+fall-1 CLI (Management Guide §9.3) enable|disable agent <backend>/<server>
check_rise [S] server's "rise" parameter used by checks ex: global.stats socket /var/run/haproxy.sock mode 600 level admin enable|disable dynamic-cookie-backend <backend>
check_status [S] status of last health check $ socat /var/run/haproxy.sock readline enable|disable frontend <frontend>
chkdown [BS] number of UP->DOWN transitions $ nc -U /var/run/haproxy.sock enable|disable health <backend>/<frontend>
chkfail [BS] number of failed checks <type> : IPv4, IPv6, integer, string enable|disable server <backend>/<server>
cli_abrt [BS] number of data transfers aborted by the client <operator> : eq, ne, le, ge, lt, gt set dynamic-cookie-key backend <backend> <value>
comp_byp [FB] bytes that bypassed the HTTP compressor <trace-level> : user, proto, state, data, developer set map <map> [ <key> | #<ref> ] <value>
comp_in [FB] number of HTTP response bytes fed to the compressor <trace-criterion> : backend, connection, frontend, listener, set severity-output [none|number|string]
comp_out [FB] number of HTTP response bytes emitted by the compressor nothing, server, session, thread set table <table> key <key> [data.<data_type> <value>]*
comp_rsp [BS] number of HTTP responses that were compressed help shutdown frontend <frontend>
conn_rate [F] number of connections over the last elapsed second prompt shutdown session <id>
conn_rate_max [F] highest known conn_rate quit shutdown sessions server <backend>/<server>
conn_tot [F] cumulative number of connections expert-mode [on|off]
connect [BS] cumulative number of connection establishment attempts operator debug dev <command> [args]*
cookie [BS] server's cookie value or backend's cookie name user set profiling { tasks } { auto | on | off }
ctime [BS] average connect time in ms over the 1024 last requests set timeout_cli <delay> trace
ctime_max [BS] the maximum observed connect time in ms trace 0
dcon [LF] requests denied by "tcp-request connection" rules show acl [<acl>] trace <source> event [ [+|-]!]<name> ]
downtime [BS] total downtime in seconds show backend trace <source> level [<trace-level>]
dreq [LFBS] requests denied because of security concerns show activity trace <source> lock [<trace-criterion>]
dresp [LFBS] responses denied because of security concerns show cli sockets trace <source> { pause | start | stop } [ [+|-]!]event ]
dses [LF] requests denied by "tcp-request session" rules show cache trace <source> sink [<sink>]
econ [BS] errors while trying to connect to a backend server show env [<name>] trace <source> verbosity [<level>]
ereq [LF] request errors show errors [<iid>|<proxy>] [request|response]
eresp [BS] response errors. srv_abrt will be counted here also show events [<sink>] [-w] [-n]
hanafail [S] failed health checks details show fd [<fd>] Master CLI (Management Guide §9.4)
hrsp_other [FBS] http responses with other codes (protocol error) show info [typed|json] [desc] prompt
hrsp_Yxx [FBS] http responses with Yxx code (Y: [1, 2, 3, 4, 5]) show map [<map>] show proc
iid [LFBS] unique proxy id show peers [<peers>] reload
intercepted [FB] cum. number of intercepted requests (monitor, stats) show pools @ID | @!PID switch to process cli
last_agt [BS] last agent check contents or textual error show profiling @ID | @!PID command execute command to @ID or @!PID
last_chk [S] last health check contents or textual error show servers state [<backend>]
lastchg [BS] number of seconds since the last UP<->DOWN transition show sess [<id>]
lastsess [S] seconds since last session assigned to server/backend show stat [{<iid>|<proxy>} <type> <sid] [typed|json] [desc]
mode [LFBS] proxy mode (tcp, http, health, unknown) show schema json
pid [LFBS] process id (0 for first instance, 1 for second, ...) show resolvers [<resolvers>]
pxname [LFBS] proxy name show table [<name> {[data.<type> <operator> <value>]|[key <key>]}]
qcur [BS] current queued requests show threads
qlimit [S] configured maxqueue for the server show tls-keys [<id>|*]
qmax [BS] max value of qcur show trace [<source>]
qtime [BS] average queue time in ms over the 1024 last requests get map <map> <value>
qtime_max [BS] the maximum observed queue time in ms get acl <acl> <value>
rate [FBS] number of sessions per second over last elapsed second get weight <backend>/<server>
rate_lim [F] configured limit on new sessions per second

HAProxy​®​ Troubleshooting Reference Card by Zenetys - 2020-03-12

Timing (Configuration Manual §8.4) HAProxy response code halog Usage
Ta (http) total active time for the request (=TR+Tw+Tc+Tr+Td) Usage: halog [-h|--help] for long help
The error 4xx and 5xx codes above may be customized halog [-q] [-c] [-m <lines>]
Tc time to establish connection to the server 200 access to stats page, and when replying to monitoring requests {-cc|-gt|-pct|-st|-tc|-srv|-u|-uc|-ue|-ua|-ut|-uao|-uto|-uba|-ubt|-ic}
Td data transmission time 400 for an invalid or too large request [-s <skip>] [-e|-E] [-H] [-rt|-RT <time>] [-ad <delay>] [-ac <count>]
Th total time to accept tcp connection and handshakes (ssl) [-v] [-Q|-QS] [-tcn|-TCN <termcode>] [ -hs|-HS [min][:[max]] ]
401 when an authentication is required to perform the action (stats)
Ti (http) idle time before first packet [ -time [min][:[max]] ] < log
403 when a request is forbidden by a "http-request deny" rule
Tq total time to get the client request (=Th+Ti+TR)
TR (http) total time to get the client request after handshakes 408 when the request timeout strikes before the request is complete Input filters (several filters may be combined) :
500 when haproxy encounters an unrecoverable internal error -H only match lines containing HTTP logs (ignore TCP)
Tr (http) server response time -E only match lines without any error (no 5xx status)
Tt total session duration time (=Tq+Tw+Tc+Tr+Td) 502 deny response or empty, invalid or incomplete response by server -e only match lines with errors (status 5xx or negative)
Tw time waiting 503 monitor fail or no server was available to handle the request -rt|-RT <time> only match response times larger|smaller than <time>
504 when the response timeout strikes before the server responds -Q|-QS only match queued requests (any queue|server queue)
-tcn|-TCN <code> only match requests with/without termination code <code>
Session state (Configuration Manual §8.5) 30x redirection, depending on the configured code -hs|-HS <[min][:][max]> only match requests with HTTP status codes within/not
C session unexpectedly aborted by the client within min..max. Any of them may be omitted. Exact
code is checked for if no ':' is specified.
S session unexpectedly aborted or refused by the server Command Line & System (Management Guide §3) -time <[min][:max]> only match requests recorded between timestamps.
P session prematurely aborted by the proxy $ haproxy -f /etc/haproxy.cfg -D \ Any of them may be omitted.
L locally processed and not passed by the server -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) Modifiers
R resource on the proxy exhausted -f CFG define configuration file -v invert the input filtering condition
I internal error -q don't report errors/warnings
-C PATH change directory
-m <lines> limit output to the first <lines> lines
D session killed by proxy, when server goes down -c configuration check -s <skip_n_fields> skip n fields from the beginning of a line (default 5)
U session to backup server killed when others go up -D daemon mode you can also use -n to start from earlier then field 5
K actively killed by admin on proxy -d[x] debug (x as options)
c client timeout -​L NAME local peer name Output filters - only one may be used at a time
s server timeout -N LIMIT default frontend maxconn -c only report the number of lines that would have been printed
-m LIMIT memory limit to LIMIT -pct output connect and response times percentiles
- normal session completion -st output number of requests per HTTP status code
-n LIMIT maxconn limit to LIMIT
R proxy waiting for complete REQUEST from the client -q quiet -cc output number of requests per cookie code (2 chars)
Q proxy waiting in the queue for a slot -tc output number of requests per termination code (2 chars)
-S BIND[,...] master-worker bind options
C proxy waiting for connection to established -srv output statistics per server (time, requests, errors)
-sf <pid>... process to send SIGUSR1 -u* output statistics per URL (time, requests, errors)
H proxy waiting for response headers from server -st <pid>... process to send SIGTERM Additional characters indicate the output sorting key :
D session in DATA phase -V verbose -u : by URL, -uc : request count, -ue : error count
L proxy transmit LAST data to client while server already finished -v[v] version [more verbose] -ua : average response time, -ut : average total time
T tarpit -W master-worker mode -uao, -uto: average times computed on valid ('OK') requests
- normal session after end of data transfer -Ws master-worker with systemd -uba, -ubt: average bytes returned, total bytes returned
-x SOCKET socket migration
CC client aborted before session to server
CD client aborted during data transfer Tricks & Tuning (Management Guide §5-7)
cD client timeout during data transfer $ kill -USR1 <PID> stop listening
CH client aborted while waiting for server response $ kill -USR2 <PID> restart master worker
cH client timeout during POST data from client $ kill -TERM <PID> with “-st”
CQ client aborted during queue $ kill -QUIT <PID> flush pools
CR client aborted before sending full HTTP request $ ss list all system sockets (new)
cR client timeout before sending full HTTP request $ netstat -anp list all system sockets (old)
$ lsof list all open files to find fd
CT client aborted because of tarpit
$ ulimit -n monitor number of file descriptors
LR intercept by proxy cause of redirect or stats ${VAR}​or $
​ VAR​within double quoted strings in config files use environment value
SC the connection was refused by something ● HAPROXY_LOCALPEER
sC connection timeout before been completed ● HAPROXY_CFGFILES
SD connection died with error during DATA ● HAPROXY_MWORKS
sD connection timeout during DATA ● HAPROXY_CLI
SH server aborted before all headers was sent ● HAPROXY_MASTER_CLI
sH server timeout before all headers was sent
sQ session queued too long
PC proxy refused because of limits
PD proxy blocked incorrect caused by invalid chunk
PH proxy blocked response caused by invalid or security reason
PR proxy blocked request caused by invalid or deny
PT proxy blocked request and has tarpit
RC a local resource has been exhausted

HAProxy® is a registered trademark of HAProxy Technologies -

https://www.haproxy.com/trademark-policy/. Thanks to Willy Tarreau,, Thierry Fournier and
many others for their feedback and suggestions.

HAProxy​®​ Troubleshooting Reference Card by Zenetys - 2020-03-12