12. Other sections¶
The sections described below are less commonly used and usually support only a few parameters. There is no implicit relation between any of them. They’re all started using a single keyword. None of them is permitted before a “global” section. The support for some of them might be conditioned by build options (e.g. anything SSL-related).
12.1. Traces¶
For debugging purpose, it is possible to activate traces on an HAProxy’s subsystem. This will dump debug messages about a specific subsystem. It is a very powerful tool to diagnose issues. Traces can be dynamically configured via the CLI. It is also possible to predefined some settings in the configuration file, in dedicated “traces” sections. More details about traces can be found in the management guide. It remains a developer tools used during complex debugging sessions. It is pretty verbose and have a cost, so use it with caution. And because it is a developer tool, there is no warranty about the backward compatibility of this section.
traces
Starts a new traces section. One or multiple “traces” section may be used. All direcitives are evaluated in the declararion order, the last ones overriding previous ones.
trace <source> <args...>
Configures on “trace” subsystem. Each of them can be found in the management manual, and follow the exact same syntax. Any output that the “trace” command would produce will be emitted during the parsing step of the section. Most of the time these will be errors and warnings, but certain incomplete commands might list permissible choices. This command is not meant for regular use, it will generally only be suggested by developers along complex debugging sessions. It is important to keep in mind that depending on the trace level and details, enabling traces can severely degrade the global performance. Please refer to the management manual for the statements syntax.
ring buf1
size 10485760 # 10MB
format timed
backing-file /tmp/h1.traces
ring buf2
size 10485760 # 10MB
format timed
backing-file /tmp/h2.traces
traces
trace h1 sink buf1 level developer verbosity complete start now
trace h2 sink buf1 level developer verbosity complete start now
12.2. Userlists¶
It is possible to control access to frontend/backend/listen sections or to http stats by allowing only authenticated and authorized users. To do this, it is required to create at least one userlist and to define users.
userlist <listname>
Creates new userlist with name <listname>. Many independent userlists can be used to store authentication & authorization data for independent customers.
group <groupname> [users <user>,<user>,(...)]
Adds group <groupname> to the current userlist. It is also possible to attach users to this group by using a comma separated list of names proceeded by “users” keyword.
user <username> [password|insecure-password <password>]
[groups <group>,<group>,(…)] Adds user <username> to the current userlist. Both secure (encrypted) and insecure (unencrypted) passwords can be used. Encrypted passwords are evaluated using the crypt(3) function, so depending on the system’s capabilities, different algorithms are supported. For example, modern Glibc based Linux systems support MD5, SHA-256, SHA-512, and, of course, the classic DES-based method of encrypting passwords.
Attention: Be aware that using encrypted passwords might cause significantly increased CPU usage, depending on the number of requests, and the algorithm used. For any of the hashed variants, the password for each request must be processed through the chosen algorithm, before it can be compared to the value specified in the config file. Most current algorithms are deliberately designed to be expensive to compute to achieve resistance against brute force attacks. They do not simply salt/hash the clear text password once, but thousands of times. This can quickly become a major factor in HAProxy’s overall CPU consumption, and can even lead to application crashes!
To address the high CPU usage of hash functions, one approach is to reduce the number of rounds of the hash function (SHA family algorithms) or decrease the “cost” of the function, if the algorithm supports it.
As a side note, musl (e.g. Alpine Linux) implementations are known to be slower than their glibc counterparts when calculating hashes, so you might want to consider this aspect too.
All passwords are considered normal arguments and are therefore subject to regular section 2.2 Quoting and escaping. Single quoting passwords is therefore recommended.
userlist L1
group G1 users tiger,scott
group G2 users xdb,scott
user tiger password $6$k6y3o.eP$JlKBx9za9667qe4(...)xHSwRv6J.C0/D7cV91
user scott insecure-password 'elgato'
user xdb insecure-password 'hello'
userlist L2
group G1
group G2
user tiger password $6$k6y3o.eP$JlKBx(...)xHSwRv6J.C0/D7cV91 groups G1
user scott insecure-password 'elgato' groups G1,G2
user xdb insecure-password 'hello' groups G2
Please note that both lists are functionally identical.
12.3. Mailers¶
It is possible to send email alerts when the state of servers changes. If configured email alerts are sent to each mailer that is configured in a mailers section. Email is sent to mailers through Lua (see examples/lua/mailers.lua).
mailers <mailersect>
Creates a new mailer list with the name <mailersect>. It is an independent section which is referenced by one or more proxies.
mailer <mailername> <ip>:<port>
Defines a mailer inside a mailers section.
global
# mailers.lua file as provided in the git repository
# adjust path as needed
lua-load examples/lua/mailers.lua
mailers mymailers
mailer smtp1 192.168.0.1:587
mailer smtp2 192.168.0.2:587
backend mybackend
mode tcp
balance roundrobin
email-alert mailers mymailers
email-alert from test1@horms.org
email-alert to test2@horms.org
server srv1 192.168.0.30:80
server srv2 192.168.0.31:80
timeout mail <time>
Defines the time available for a mail/connection to be made and send to the mail-server. If not defined the default value is 10 seconds. To allow for at least two SYN-ACK packets to be send during initial TCP handshake it is advised to keep this value above 4 seconds.
mailers mymailers
timeout mail 20s
mailer smtp1 192.168.0.1:587
12.4. HTTP-errors¶
It is possible to globally declare several groups of HTTP errors, to be imported afterwards in any proxy section. Same group may be referenced at several places and can be fully or partially imported.
http-errors <name>
Create a new http-errors group with the name <name>. It is an independent section that may be referenced by one or more proxies using its name.
errorfile <code> <file>
Associate a file contents to an HTTP error code
Arguments:
<code> is the HTTP status code. Currently, HAProxy is capable of generating codes 200, 400, 401, 403, 404, 405, 407, 408, 410, 425, 429, 500, 501, 502, 503, and 504.
<file> designates a file containing the full HTTP response. It is recommended to follow the common practice of appending “.http” to the filename so that people do not confuse the response with HTML error pages, and to use absolute paths, since files are read before any chroot is performed.
Please referrers to “errorfile” keyword in section 4 for details.
http-errors website-1
errorfile 400 /etc/haproxy/errorfiles/site1/400.http
errorfile 404 /etc/haproxy/errorfiles/site1/404.http
errorfile 408 /dev/null # work around Chrome pre-connect bug
http-errors website-2
errorfile 400 /etc/haproxy/errorfiles/site2/400.http
errorfile 404 /etc/haproxy/errorfiles/site2/404.http
errorfile 408 /dev/null # work around Chrome pre-connect bug
12.5. Rings¶
It is possible to globally declare ring-buffers, to be used as target for log servers or traces.
ring <ringname>
Creates a new ring-buffer with name <ringname>.
backing-file <path>
This replaces the regular memory allocation by a RAM-mapped file to store the ring. This can be useful for collecting traces or logs for post-mortem analysis, without having to attach a slow client to the CLI. Newer contents will automatically replace older ones so that the latest contents are always available. The contents written to the ring will be visible in that file once the process stops (most often they will even be seen very soon after but there is no such guarantee since writes are not synchronous).
When this option is used, the total storage area is reduced by the size of the “struct ring” that starts at the beginning of the area, and that is required to recover the area’s contents. The file will be created with the starting user’s ownership, with mode 0600 and will be of the size configured by the “size” directive. When the directive is parsed (thus even during config checks), any existing non-empty file will first be renamed with the extra suffix “.bak”, and any previously existing file with suffix “.bak” will be removed. This ensures that instant reload or restart of the process will not wipe precious debugging information, and will leave time for an admin to spot this new “.bak” file and to archive it if needed. As such, after a crash
the file designated by <path> will contain the freshest information, and if
the service is restarted, the “<path>.bak” file will have it instead. This means that the total storage capacity required will be double of the ring size. Failures to rotate the file are silently ignored, so placing the file into a directory without write permissions will be sufficient to avoid the backup file if not desired.
WARNING: there are stability and security implications in using this feature. First, backing the ring to a slow device (e.g. physical hard drive) may cause perceptible slowdowns during accesses, and possibly even panics if too many threads compete for accesses. Second, an external process modifying the area could cause the haproxy process to crash or to overwrite some of its own memory with traces. Third, if the file system fills up before the ring, writes to the ring may cause the process to crash.
The information present in this ring are structured and are NOT directly readable using a text editor (even though most of it looks barely readable). The output of this file is only intended for developers.
description <text>
The description is an optional description string of the ring. It will appear on CLI. By default, <name> is reused to fill this field.
format <format>
Format used to store events into the ring buffer.
Arguments:
<format> is the log format used when generating syslog messages. It may be one of the following :
iso A message containing only the ISO date, followed by the text. The PID, process name and system name are omitted. This is designed to be used with a local log server.
local Analog to rfc3164 syslog message format except that hostname field is stripped. This is the default. Note: option “log-send-hostname” switches the default to rfc3164.
raw A message containing only the text. The level, PID, date, time, process name and system name are omitted. This is designed to be used in containers or during development, where the severity only depends on the file descriptor used (stdout/stderr). This is the default.
rfc3164 The RFC3164 syslog message format. (https://tools.ietf.org/html/rfc3164)
rfc5424 The RFC5424 syslog message format. (https://tools.ietf.org/html/rfc5424)
short A message containing only a level between angle brackets such as ‘<3>’, followed by the text. The PID, date, time, process name and system name are omitted. This is designed to be used with a local log server. This format is compatible with what the systemd logger consumes.
priority A message containing only a level plus syslog facility between angle brackets such as ‘<63>’, followed by the text. The PID, date, time, process name and system name are omitted. This is designed to be used with a local log server.
timed A message containing only a level between angle brackets such as ‘<3>’, followed by ISO date and by the text. The PID, process name and system name are omitted. This is designed to be used with a local log server.
maxlen <length>
The maximum length of an event message stored into the ring, including formatted header. If an event message is longer than <length>, it will be truncated to this length.
server <name> <address> [param*]
Used to configure a syslog tcp server to forward messages from ring buffer. This supports for all “server” parameters found in 5.2 paragraph. Some of these parameters are irrelevant for “ring” sections. Important point: there is little reason to add more than one server to a ring, because all servers will receive the exact same copy of the ring contents, and as such the ring will progress at the speed of the slowest server. If one server does not respond, it will prevent old messages from being purged and may block new messages from being inserted into the ring. The proper way to send messages to multiple servers is to use one distinct ring per log server, not to attach multiple servers to the same ring. Note that specific server directive “log-proto” is used to set the protocol used to send messages.
size <size>
This is the optional size in bytes for the ring-buffer. Default value is set to BUFSIZE.
timeout connect <timeout>
Set the maximum time to wait for a connection attempt to a server to succeed.
Arguments:
<timeout> is the timeout value specified in milliseconds by default, but can be in any other unit if the number is suffixed by the unit, as explained at the top of this document.
timeout server <timeout>
Set the maximum time for pending data staying into output buffer.
Arguments:
<timeout> is the timeout value specified in milliseconds by default, but can be in any other unit if the number is suffixed by the unit, as explained at the top of this document.
global
log ring@myring local7
ring myring
description "My local buffer"
format rfc5424
maxlen 1200
size 32764
timeout connect 5s
timeout server 10s
server mysyslogsrv 127.0.0.1:6514 log-proto octet-count
12.6. Log forwarding¶
It is possible to declare one or multiple log forwarding section, HAProxy will forward all received log messages to a log servers list.
log-forward <name>
Creates a new log forwarder proxy identified as <name>.
backlog <conns>
Give hints to the system about the approximate listen backlog desired size on connections accept.
bind <addr> [param*]
Used to configure a stream log listener to receive messages to forward. This supports the “bind” parameters found in 5.1 paragraph including those about ssl but some statements such as “alpn” may be irrelevant for syslog protocol over TCP. Those listeners support both “Octet Counting” and “Non-Transparent-Framing” modes as defined in rfc-6587.
dgram-bind <addr> [param*]
Used to configure a datagram log listener to receive messages to forward. Addresses must be in IPv4 or IPv6 form,followed by a port. This supports for some of the “bind” parameters found in 5.1 paragraph among which “interface”, “namespace” or “transparent”, the other ones being silently ignored as irrelevant for UDP/syslog case.
log globallog <target> [len <length>] [format <format>] [sample <ranges>:<sample_size>]<facility> [<level> [<minlevel>]] Used to configure target log servers. See more details on proxies documentation. If no format specified, HAProxy tries to keep the incoming log format. Configured facility is ignored, except if incoming message does not present a facility but one is mandatory on the outgoing format. If there is no timestamp available in the input format, but the field exists in output format, HAProxy will use the local date.
global
log stderr format iso local7
ring myring
description "My local buffer"
format rfc5424
maxlen 1200
size 32764
timeout connect 5s
timeout server 10s
# syslog tcp server
server mysyslogsrv 127.0.0.1:514 log-proto octet-count
log-forward sylog-loadb
dgram-bind 127.0.0.1:1514
bind 127.0.0.1:1514
# all messages on stderr
log global
# all messages on local tcp syslog server
log ring@myring local0
# load balance messages on 4 udp syslog servers
log 127.0.0.1:10001 sample 1:4 local0
log 127.0.0.1:10002 sample 2:4 local0
log 127.0.0.1:10003 sample 3:4 local0
log 127.0.0.1:10004 sample 4:4 local0
maxconn <conns>
Fix the maximum number of concurrent connections on a log forwarder. 10 is the default.
timeout client <timeout>
Set the maximum inactivity time on the client side.
option assume-rfc6587-ntf
Directs HAProxy to treat incoming TCP log streams always as using non-transparent framing. This option simplifies the framing logic and ensures consistent handling of messages, particularly useful when dealing with improperly formed starting characters.
option dont-parse-log
Enables HAProxy to relay syslog messages without attempting to parse and restructure them, useful for forwarding messages that may not conform to traditional formats. This option should be used with the format raw setting on destination log targets to ensure the original message content is preserved.
option host { replace | fill | keep | append }
Set the host strategy that should be used on the log-forward section regarding syslog hostname field for outbound rfc3164 or rfc5424 messages.
replace If input message already contains a value for the hostname field, we replace it by the source IP address from the sender. If input message doesn’t contain a value for the hostname field (ie: ‘-’ as input rfc5424 message or non compliant rfc3164 or rfc5424 message), we use the source IP address from the sender as hostname field.
fill If input message already contains a value for the hostname field, we keep it. If input message doesn’t contain a value for the hostname field (ie: ‘-’ as input rfc5424 message or non compliant rfc3164 or rfc5424 message), we use the source IP address from the sender as hostname field. (This is the default)
keep If input message already contains a value for the hostname field, we keep it. If input message doesn’t contain a value for the hostname field, we set it to ‘localhost’ (rfc3164) or ‘-’ (rfc5424).
append If input message already contains a value for the hostname field, we append a comma followed by the IP address from the sender. If input message doesn’t contain a value for the hostname field, we use the source IP address from the sender.
For all options above, if the source IP address from the sender is not available (ie: UNIX/ABNS socket), then the resulting strategy is “keep”.
Note that this option is only relevant for rfc3164 or rfc5424 destination log format. Else setting the option will have no visible effect.
12.7. Certificate Storage¶
HAProxy uses an internal storage mechanism to load and store certificates used in the configuration. This storage can be configured by using a “crt-store” section. It allows to configure certificate definitions and which files should be loaded in it. A certificate definition must be written before it is used elsewhere in the configuration.
crt-store [<name>]
The “crt-store” takes an optional name in argument. If a name is specified, every certificate of this store must be referenced using “@<name>/<crt>” or “@<name>/<alias>”.
Files in the certificate storage can also be updated dynamically with the CLI. See “set ssl cert” in the section 9.3 of the management guide.
The following keywords are supported in the “crt-store” section :
crt-basekey-baseload
crt-base <dir>
Assigns a default directory to fetch SSL certificates from when a relative path is used with “crt” directives. Absolute locations specified prevail and ignore “crt-base”. When used in a crt-store, the crt-base of the global section is ignored.
key-base <dir>
Assigns a default directory to fetch SSL private keys from when a relative path is used with “key” directives. Absolute locations specified prevail and ignore “key-base”. When used in a crt-store, the key-base of the global section is ignored.
load [crt <filename>] [param*]
Load SSL files in the certificate storage. For the parameter list, see section “12.7.1. Load options”
crt-store
load crt "site1.crt" key "site1.key" ocsp "site1.ocsp" alias "site1"
load crt "site2.crt" key "site2.key"
frontend in2
bind *:443 ssl crt "@/site1" crt "site2.crt"
crt-store web
crt-base /etc/ssl/certs/
key-base /etc/ssl/private/
load crt "site3.crt" alias "site3"
load crt "site4.crt" key "site4.key"
frontend in2
bind *:443 ssl crt "@/site1" crt "site2.crt" crt "@web/site3" crt "@web/site4.crt"
12.7.1. Load options¶
Load SSL files in the certificate storage. The load keyword can take multiple parameters which are listed below. These keywords are also usable in a crt-list.
crt <filename>
This argument is mandatory, it loads a PEM which must contain the public certificate but could also contain the intermediate certificates and the private key. If no private key is provided in this file, a key can be provided with the “key” keyword.
acme <string>
This option allows to configure the ACME protocol for a given certificate. This is an experimental feature which needs the “expose-experimental-directives” keyword in the global section.
When using the “acme” keyword in a crt-store, it is possible to start without an existing certificate on the disk. Instead, a temporary key pair will be used until the ACME certificate is generated. This behavior is exclusives to crt-stores, neither a crt-list line nor an ssl-f-use line can achieve the same without declaring a crt-store first.
See also
Section 12.8 (“ACME”) and “domains” in this section.
alias <string>
Optional argument. Allow to name the certificate with an alias, so it can be referenced with it in the configuration. An alias must be prefixed with ‘@/’ when called elsewhere in the configuration.
domains <string>
Configure the list of domains that will be used for ACME certificates. The first domain of the list is used as the CN. Domains are separated by commas in the list.
See also
Section 12.8 (“ACME”) and “acme” in this section.
load crt "example.com.pem" acme LE domains "bar.example.com,foo.example.com"
ips <string>
Configure the list of IP addresses that will be included as IP SANs in the ACME certificate. IP addresses are separated by commas in the list.
Generating a certificate with IPs might require the use of the “shortlived” profile.
See also
Section 12.8 (“ACME”), “acme” and “domains” in this section.
load crt "server.pem" acme LE ips "192.0.2.1,2001:db8::1"
key <filename>
This argument is optional. Load a private key in PEM format. If a private key was already defined in “crt”, it will overwrite it.
ocsp <filename>
This argument is optional, it loads an OCSP response in DER format. It can be updated with the CLI.
issuer <filename>
This argument is optional. Load the OCSP issuer in PEM format. In order to identify which certificate an OCSP Response applies to, the issuer’s certificate is necessary. If the issuer’s certificate is not found in the “crt” file, it could be loaded from a file with this argument.
sctl <filename>
This argument is optional. Support for Certificate Transparency (RFC6962) TLS extension is enabled. The file must contain a valid Signed Certificate Timestamp List, as described in RFC. File is parsed to check basic syntax, but no signatures are verified.
ocsp-update [ off | on ]
Enable automatic OCSP response update when set to ‘on’, disable it otherwise. Its value defaults to ‘off’. To enable the OCSP auto update on a bind line, you can use this option in a crt-store or you can use the global option “tune.ocsp-update.mode”. If a given certificate is used in multiple crt-lists with different values of the ‘ocsp-update’ set, an error will be raised. Likewise, if a certificate inherits from the global option on a bind line and has an incompatible explicit ‘ocsp-update’ option set in a crt-list, the same error will be raised.
Here is an example configuration enabling it with a crt-list:
haproxy.cfg: frontend fe bind :443 ssl crt-list haproxy.list
haproxy.list: server_cert.pem [ocsp-update on] foo.bar
Here is an example configuration enabling it with a crt-store:
haproxy.cfg:
crt-store load crt foobar.pem ocsp-update on
frontend fe bind :443 ssl crt foobar.pem
When the option is set to ‘on’, we will try to get an ocsp response whenever an ocsp uri is found in the frontend’s certificate. The only limitation of this mode is that the certificate’s issuer will have to be known in order for the OCSP certid to be built. Each OCSP response will be updated at least once an hour, and even more frequently if a given OCSP response has an expire date earlier than this one hour limit. A minimum update interval of 5 minutes will still exist in order to avoid updating too often responses that have a really short expire time or even no ‘Next Update’ at all. Because of this hard limit, please note that when auto update is set to ‘on’, any OCSP response loaded during init will not be updated until at least 5 minutes, even if its expire time ends before now+5m. This should not be too much of a hassle since an OCSP response must be valid when it gets loaded during init (its expire time must be in the future) so it is unlikely that this response expires in such a short time after init. On the other hand, if a certificate has an OCSP uri specified and no OCSP response, setting this option to ‘on’ for the given certificate will ensure that the OCSP response gets fetched automatically right after init. The default minimum and maximum delays (5 minutes and 1 hour respectively) can be configured by the “ocsp-update.maxdelay” and “ocsp-update.mindelay” global options.
Whenever an OCSP response is updated by the auto update task or following a call to the “update ssl ocsp-response” CLI command, a dedicated log line is emitted. It follows a dedicated format that contains the following header “<OCSP-UPDATE>” and is followed by specific OCSP-related information:
the path of the corresponding frontend certificate
a numerical update statusa textual update statusthe number of update failures for the given response
the number of update successes for the givan response
See “show ssl ocsp-updates” CLI command for a full list of error codes and error messages. This line is emitted regardless of the success or failure of the concerned OCSP response update. The OCSP request/response is sent and received through an http_client instance that has the dontlog-normal option set and that uses the regular HTTP log format in case of error (unreachable OCSP responder for instance). If such an error occurs, another log line that contains HTTP-related information will then be emitted alongside the “regular” OCSP one (which will likely have “HTTP error” as text status). But if a purely HTTP error happens (unreachable OCSP responder for instance), an extra log line that follows the regular HTTP log-format will be emitted. Here are two examples of such log lines, with a successful OCSP update log line first and then an example of an HTTP error with the two different lines (lines were spit and the URL was shortened for readability): <133>Mar 6 11:16:53 haproxy[14872]: <OCSP-UPDATE> /path_to_cert/foo.pem 1 \ “Update successful” 0 1
<133>Mar 6 11:18:55 haproxy[14872]: <OCSP-UPDATE> /path_to_cert/bar.pem 2 \ “HTTP error” 1 0 <133>Mar 6 11:18:55 haproxy[14872]: -:- [06/Mar/2023:11:18:52.200] \ <OCSP-UPDATE> -/- 2/0/-1/-1/3009 503 217 - - SC– 0/0/0/0/3 0/0 {} \ “GET http://127.0.0.1:12345/MEMwQT HTTP/1.1”
Troubleshooting: A common error that can happen with Let’s Encrypt certificates is if the DNS resolution provides an IPv6 address and your system does not have a valid outgoing IPv6 route. In such a case, you can either create the appropriate route or set the “httpclient.resolvers.prefer ipv4” option in the global section. In case of “OCSP response check failure” error, you might want to check that the issuer certificate that you provided is valid. A more precise error message might also be displayed between parenthesis after the “generic” error message. It can happen for “OCSP response check failure” or “Error during insertion” errors.
jwt [ off | on ]
Allow for this certificate to be used for JWT validation or decryption via the “jwt_verify_cert”, “jwt_decrypt_cert” or “jwt_decrypt” converters when set to ‘on’. Its value defaults to ‘off’.
When set to ‘on’ for a given certificate, the CLI command “del ssl cert” will not work. In order to be deleted, a certificate must not be used, either for SSL handshakes or JWT validation.
This option can be changed during runtime via the “add ssl jwt” and “del ssl jwt” CLI commands. See also “show ssl jwt” CLI command.
generate-dummy [ off | on ]
Allow the generation of a private key and its self-signed certificate at parsing time when set to ‘on’. This may be useful if one does not have a certificate at disposal during testing phase for instance. In this case, “keytype”, “bits” and “curves” may be used to customize the private key. When not used, the default value is ‘off’. (also see “keytype”, “bits” and “curves”).
keytype [ RSA | ECDSA ]
Allow the selection of the private key type used to generate at parsing time a self-signed certificate. This is the case if “generate-dummy” is set to ‘on’ for this certificate. When not used, the default is ‘RSA’. (also see “generate-dummy”).
bits <number>
Configure the number of bits to generate an RSA self-signed certificate when “generate-dummy” is set to ‘on’ for this self-signed certificate and “keytype” is set to ‘RSA’. When not used, the default is 2048. (also see “generate-dummy”).
curves <string>
Configure the curves when “generate-dummy” is set to ‘on’ and “keytype” is set to ‘ECDSA” for this self-signed certificate. The default is ‘P-384’.
12.8. ACME¶
acme <name>
The ACME protocol can be configured using the “acme” section. The section takes a “<name>” argument, which is used to link a certificate to the section.
The ACME section allows to configure HAProxy as an ACMEv2 client. This feature is experimental meaning that “expose-experimental-directives” must be in the global section so this can be used.
A guide is available on the HAProxy wiki https://github.com/haproxy/wiki/wiki/ACME:–native-haproxy
Current limitations:
The feature is limited to the http-01, dns-01 or dns-persist-01 challenges for now. http-01 is completely handled by HAProxy, but dns-01 and dns-persist-01 needs either the dataplaneAPI, a lua script using event_hdl or another 3rd party tool to talk to a DNS provider API. dns-persist-01 only needs the TXT entry to be set once, so it could be set manually without a tool.
It is possible to start without an existing certificate on the disk. To do so, the certificate must configured in a crt-store. When using the “acme” keyword in a crt-store, a temporary key pair will be used until the ACME certificate is generated.
The current HAProxy architecture is a non-blocking model, access to the disk is not supposed to be done after the configuration is loaded, because it could block the event loop, blocking the traffic on the same thread. Meaning that the certificates and keys generated from HAProxy will need to be dumped from outside HAProxy using “dump ssl cert” on the stats socket. It’s possible to automate the dump of the certificates by using the dataplaneAPI or the haproxy-dump-certs script provided in the admin/cli/ directory.
The ACME scheduler starts at HAProxy startup, it will loop over the certificates and start an ACME renewal task when the notAfter task is past curtime + (notAfter - notBefore) / 12, or 7 days if notBefore is not defined. The scheduler will then sleep and wakeup after 12 hours. It is possible to start manually a renewal task with “acme renew’.
See also
acme status” in the management guide.
The following keywords are usable in the ACME section:
account-key <filename>
Configure the path to the account key. The key need to be generated before launching HAProxy. If no account keyword is used, the acme section will try to load a filename using the section name “<name>.account.key”. If the file doesn’t exist, HAProxy will generate one, using the parameters from the acme section.
You can also generate manually an RSA private key with openssl:
openssl genrsa -out account.key 2048
Or an ecdsa one:
openssl ecparam -name secp384r1 -genkey -noout -out account.key
acme-vars <string>
Pass arbitrary variables to the external DNS provisioning tool (e.g. the dataplaneAPI) via the “dpapi” sink. The semantics are tool-specific; refer to your DNS provisioning tool’s documentation.
This keyword is only meaningful when the challenge type is “dns-01” or “dns-persist-01”.
See also
“challenge”, “provider-name”
bits <number>
Configure the number of bits to generate an RSA certificate. Default to 2048. Setting a too high value can trigger a warning if your machine is not powerful enough. (This can be configured with “warn-blocked-traffic-after” but blocking the traffic too long could trigger the watchdog.)
challenge <string>
Takes a challenge type as parameter, this must be http-01, dns-01 or dns-persist-01. When not used the default is http-01.
dns-persist-01 implements draft-ietf-acme-dns-persist. Unlike dns-01, it uses a static TXT record at “_validation-persist.<domain>” that is set once and never changes between renewals. The record must contain the account URI and an optional policy. This challenge type does not require write access to the DNS provider API on each renewal.
challenge-ready <value>[,<value>]*
Configure the conditions that must be met before notifying the ACME server that a dns-01 challenge is ready to be validated. Accepted values are:
cli - wait for an operator to signal readiness via the CLI command “acme challenge_ready <crt> domain <domain>” on the master CLI or the stats socket. This allows an external DNS provisioning tool to confirm that the TXT record has been set before HAProxy proceeds. It is also possible to signal the “cli” readiness using the ACME.challenge_ready() lua function.
dns - perform a DNS pre-check by resolving the TXT record for “_acme-challenge.<domain>” using the configured “default” resolvers section, not the authoritative name servers. The challenge is not submitted until the TXT record matches the expected token. Results may therefore be affected by DNS caching at the resolver level. The delay between resolution attempts is controlled by “dns-delay”. This option is independent of the CLI command, so no human intervention is required.
For dns-01, the TXT record at “_acme-challenge.<domain>” is resolved and must match the expected token. For dns-persist-01, the TXT record at “_validation-persist.<domain>” is resolved and only its presence is checked.
delay - apply an initial wait of “dns-delay” before proceeding. Without “dns”, the challenge is submitted after the delay expires. When combined with “dns”, the initial wait is applied before starting the DNS pre-checks.
none - no readiness condition; the challenge is submitted to the ACME server immediately without waiting for any external confirmation. This option cannot be combined with others.
Multiple values can be combined with a comma. When several conditions are specified, HAProxy processes them in the following order: first it waits for the CLI confirmation (“cli”), then applies the initial delay (“delay”), then performs the DNS pre-checks (“dns”).
This option is only compatible with the dns-01 and dns-persist-01 challenge types.
When “challenge” is set to “dns-01” and this option is not configured, the default is “cli”.
When “challenge” is set to “dns-persist-01” and this option is not configured, the default is “dns,delay”.
When “challenge” is set to “dns-persist-01”, an initial opportunistic DNS check is always performed before the challenge-ready conditions are evaluated. Since the “_validation-persist.<domain>” TXT record is set once and never changes between renewals, HAProxy checks at renewal time whether the record is already present. If the check succeeds for all domains, the challenge is submitted immediately without going through the challenge-ready steps (cli, delay, dns). If the check fails, HAProxy falls back to the normal challenge-ready flow.
# Wait for CLI confirmation, then verify DNS propagation
challenge-ready cli,dns
contact <string>
The contact email that will be associated to the account key in the CA.
curves <string>
When using the ECDSA keytype, configure the curves. The default is P-384.
directory <string>
This keyword configures the directory URL for the CA used by this acme section. This keyword is mandatory as there is no default URL.
directory https://acme-staging-v02.api.letsencrypt.org/directory
dns-delay <time>
Configure the delay used by “challenge-ready” conditions “delay” and “dns”. The value is a time expressed in HAProxy time format (e.g. “5m”, “300s”). Default is 30 seconds.
Its role depends on the “challenge-ready” conditions in use:
delay - the challenge is submitted after this delay expires, without any DNS pre-check.
dns - the delay between two consecutive DNS resolution attempts. The first probe fires immediately without any initial wait.
dns+delay - the initial wait before the first DNS resolution attempt, and the delay between subsequent retries.
Note that the resolution goes through the configured “default” resolvers section, not the authoritative name servers. Results may therefore still be affected by DNS caching at the resolver level.
dns-timeout <time>
When “challenge-ready” includes “dns”, configure the maximum time allowed to successfully resolve the TXT record before aborting the challenge. The value is a time expressed in HAProxy time format (e.g. “10m”, “600s”). Default is 600 seconds.
The timer starts from the moment the first DNS resolution attempt is triggered (after the initial “dns-delay”). If the next resolution attempt would be triggered after the timeout has elapsed, the challenge is aborted with an error. This prevents an infinite retry loop when DNS propagation fails.
See also
“dns-delay”
keytype <string>
Configure the type of key that will be generated. Value can be either “RSA” or “ECDSA”. You can also configure the “curves” for ECDSA and the number of “bits” for RSA. By default EC384 keys are generated.
map <map>
Configure the map which will be used to store token (key) and thumbprint (value), which is useful to reply to a challenge when there are multiple account used. The acme task will add entries before validating the challenge and will remove the entries at the end of the task.
profile <string>
Request a specific certificate profile from the CA by including a “profile” field in the newOrder request. This implements draft-ietf-acme-profiles.
Profile names are CA-specific short identifiers (e.g. “classic”, “shortlived”). When set, the profile name is sent as-is in the newOrder JSON payload. The CA is free to ignore the request or return an error if the profile is not supported. When not set, no profile field is included and the CA uses its default issuance policy.
See https://letsencrypt.org/docs/profiles/ for Let’s Encrypt profiles.
# Request short-lived certificates
profile shortlived
provider-name <string>
Set the DNS provider name passed to the external DNS provisioning tool (e.g. the dataplaneAPI) via the “dpapi” sink. The accepted values are tool-specific; refer to your DNS provisioning tool’s documentation.
This keyword is only meaningful when the challenge type is “dns-01” or “dns-persist-01”.
See also
“challenge”, “acme-vars”
reuse-key { on | off }
If set to “on”, HAProxy won’t generate a new private key and will keep the previous one. Rotating private keys is recommended, when enabling this option it is recommended to regenerate manually the keys regularly.
This option might be useful when using RSA keys bigger than 2048 that can take time to generate and might slow down one thread doing so.
Using the same key can be useful when using the cache of your ACME server, it can help to retrieve a valid certificate corresponding to the current key.
The default setting is “off”.
global
expose-experimental-directives
httpclient.resolvers.prefer ipv4
frontend in
bind *:80
bind *:443 ssl
http-request return status 200 content-type text/plain lf-string "%[path,field(-1,/)].%[path,field(-1,/),map(virt@acme)]\n" if { path_beg '/.well-known/acme-challenge/' }
ssl-f-use crt "foo.example.com.pem.rsa" acme LE1 domains "foo.example.com.pem,bar.example.com"
ssl-f-use crt "foo.example.com.pem.ecdsa" acme LE2 domains "foo.example.com.pem,bar.example.com"
acme LE1
directory https://acme-staging-v02.api.letsencrypt.org/directory
account-key /etc/haproxy/letsencrypt.account.key
contact john.doe@example.com
challenge http-01
keytype RSA
bits 2048
map virt@acme
acme LE2
directory https://acme-staging-v02.api.letsencrypt.org/directory
account-key /etc/haproxy/letsencrypt.account.key
contact john.doe@example.com
challenge http-01
keytype ECDSA
curves P-384
map virt@acme
eab-key-id <filename>
Configure the path to the EAB key id file. The credential is provided by the CA and must be placed at the specified path before starting HAProxy. It is used during account creation only.
The file must contain a plain ASCII string.
EAB credentials are only required during the initial ACME account creation and can be removed afterwards, either from the config or by emptying the files. An empty file is silently ignored. Whitespace is not ignored, except for the trailing newline.
See also
“eab-mac-key”, “eab-mac-alg”
eab-mac-key <filename>
Configure the path to the EAB MAC key file. The credential is provided by the CA and must be placed at the specified path before starting HAProxy. It is used during account creation only.
The file must contain a base64url encoded MAC key.
EAB credentials are only required during the initial ACME account creation and can be removed afterwards, either from the config or by emptying the files. An empty file is silently ignored. Whitespace is not ignored, except for the trailing newline.
See also
“eab-key-id”, “eab-mac-alg”
eab-mac-alg { HS256 | HS384 | HS512 }
Configure MAC algorithm used for EAB signing. Default is HS256. EAB MAC key must be large enough to support specified MAC algorithm. Not all CAs support algorithms other than HS256.
See also
“eab-key-id”, “eab-mac-key”
12.9. Healthchecks¶
It is possible to globally declare several health-checks that could be used by servers across all the configuration, overriding the local proxy configuration.
healthcheck <name>
Created a new healthcheck with name <name>. This name must be unique. It should be used on server line to reference a specific health-check section.
type <type>
Defines the health-check type. This parameter is mandatory. Following types of health-check are supported:
tcp-checkhttpchkssl-hello-chksmtpchkpgsql-checkredis-checkmysql-checkldap-checkspop-check
Each type uses the same parameters, if any, than the corresponding proxy’s option. For instance, the method, the uri… may be speficied for the “httpchk” type:
healthcheck my-http-check
type httpchk GET /health HTTP/1.1 %[srv_name]
See also
“option tcp-check”, “option httpchk”, “option ssl-hello-chk”,
“option smtpchk”, “option mysql-check”, “option pgsql-check”, “option redis-check”, “option ldap-check and “option spop-check”
http-check comment <string>http-check connect [default] [port <expr>] [addr <ip>] [send-proxy][via-socks4] [ssl] [sni <sni>] [alpn <alpn>] [linger] [proto <name>] [comment <msg>]
http-check disable-on-404http-check expect [min-recv <int>] [comment <msg>][ok-status <st>] [error-status <st>] [tout-status <st>] [on-success <fmt>] [on-error <fmt>] [status-code <expr>] [!] <match> <pattern>
http-check send [meth <method>] [{ uri <uri> | uri-lf <fmt> }>] [ver <version>]
[comment <msg>]
http-check send-statehttp-check set-var(<var-name>[,<cond>...]) <expr>http-check set-var-fmt(<var-name>[,<cond>...]) <fmt>http-check unset-var(<var-name>)Add a specific http-check rule for a “httpchk” healthcheck. The same syntax than the corresponding proxy’s directives is used. See the corresponding proxy documentation for details.
tcp-check comment <string>tcp-check connect [default] [port <expr>] [addr <ip>] [send-proxy] [via-socks4][ssl] [sni <sni>] [alpn <alpn>] [linger] [proto <name>] [comment <msg>]
tcp-check expect [min-recv <int>] [comment <msg>]
[ok-status <st>] [error-status <st>] [tout-status <st>] [on-success <fmt>] [on-error <fmt>] [status-code <expr>] [!] <match> <pattern>
tcp-check send <data> [comment <msg>]tcp-check send-lf <fmt> [comment <msg>]tcp-check send-binary <hexstring> [comment <msg>]tcp-check send-binary-lf <hexfmt> [comment <msg>]tcp-check set-var(<var-name>[,<cond>...]) <expr>tcp-check set-var-fmt(<var-name>[,<cond>...]) <fmt>tcp-check unset-var(<var-name>)Add a specific tcp-check rule for a “tcp-check” healthcheck. The same syntax than the corresponding proxy’s directives is used. See the corresponding proxy documentation for details.
12.10. HTTP hardening¶
HTTP is a very tolerant protocol. Over the years, its specifications have accumulated many optional features, alternate syntaxes and compatibility exceptions, and HAProxy, like most implementations, accepts the vast majority of them in order to remain interoperable with the widest possible range of clients and servers. This tolerance comes at a cost: some of these constructs are almost never met in normal traffic, and when they are, they are more often associated with attack attempts (request smuggling, cache poisoning, evasion of filtering rules, exploitation of fragile applications) than with legitimate use. Others are perfectly valid but may cause information to leak, such as internal headers or details accidentally sent to the other side. While HAProxy tends to be robust against these variations, passing valid certain but unusual constructs can needlessly expose potentially fragile internal implementations.
Not every deployment needs to accept everything. A frontend exposed to the internet usually has good reasons to be stricter than one serving internal users, and a backend protecting a fragile application may only want to receive a narrow subset of what the protocol permits. HTTP hardening profiles make it possible to decide, for each exposed side, which of these uncommon or risky constructs are rejected, removed or adjusted, and which ones are simply let through as usual.
A profile is declared in an “http-hardening” section and applied to a “bind” and/or “server” line using the “http-hardening” keyword. Each control in a profile comes with a default, that is usually off, but may be implicit (for certain sections, see below), or explicit (inherited from another existing section). Since the same profile may be used on either side, controls are expressed relative to the traffic received or sent by that side, and never in terms of requests or responses.
http-hardening <name> [from <name2>]
Declares a new hardening profile <name>. A profile can only be declared once. The name “auto” is reserved. When “from” is used, the profile starts with the settings of the already declared profile <name2>, otherwise it starts with the defaults. A profile must be declared before being referenced by a “bind” or “server” line or by “from”, like for “defaults” sections.
The profiles “untrusted” and “trusted” always exist even if never declared. They may be declared once to change their settings, in which case they start from their defaults. They are supported in “from” to create new profiles. The reserved name “auto” may not be defined nor used in “from”.
Enables or disables the hardening check designated by <keyword> in the
profile being declared. Keywords are processed in order, so a later line
overrides an earlier one. The keyword "all" designates all options at once,
which allows to preset or clear everything before adjusting some of them.
Each option applies to requests, responses, or both. The list of known
keywords is presented below, and may be listed from the command line using
"haproxy -dKcfg".
Example :
http-hardening untrusted
all on
http-hardening trusted
all off
frontend www
bind :80 http-hardening untrusted
backend app
server app1 192.168.0.1:80 http-hardening trusted
All configurable checks try to follow the same naming scheme:
- reject-xxxx: this applies to inbound traffic (request, response or both),
and rejects the whole request or response if a specific condition is
matched. An error is returned.
- delete-xxxx: this applies to inbound traffic (request, response or both),
after the location where "reject" checks are applied, so that "reject"
checks always have precedence over "delete" ones. Such rules indicate that
the offending element (generally a header field) will be deleted very early
before the HTTP message enters the core HTTP processing. It will then not
be seen by processing rules. These actions may be used to fix up protocol
glitches without rejecting traffic.
- adjust-xxxx: this applies to inbound traffic (request, response or both),
after the location where "delete" checks are applied, so that "delete"
checks always have precedence over "adjust" ones. Such rules indicate that
the offending element will be adjusted or reformatted very early before the
HTTP message enters the core HTTP processing. It will then not be seen in
its original form by processing rules. These actions may be used to fix up
protocol glitches without rejecting traffic.
- block-outgoing-xxxx: this applies to outbound traffic (request, response,
or both). It will refuse to emit a message (request or response) that
violates the designated control. This is generally used to avoid emitting
possibly ambiguous protocol elements to a client or a server. An error will
be produced.
- delete-outgoing-xxxx: this applies to outbound traffic (request, response
or both), after the location where "block-outgoing" checks are applied, so
that "block-outgoing" checks always have precedence over "delete-outgoing"
ones. Such rules indicate that the offending element (generally a header
field) will be deleted before the HTTP message is emitted so that nothing
ambiguous is emitted to a client or server. These actions may be used to
gracefully adapt to sensitive components, and to clean up private
information.
- adjust-outgoing-xxxx: this applies to outbound traffic (request, response
or both), after the location where "delete-outgoing" checks are applied, so
that "delete-outgoing" checks always have precedence over "adjust-outgoing"
ones. Such rules indicate that the offending element will be adjusted or
reformatted before the HTTP message is emitted so that nothing ambiguous is
emitted to a client or server. These actions may be used to gracefully
adapt to with sensitive components, and to clean up private information.
Supported http-hardening checks and descriptions:
- all
This keyword covers all known keywords together. It is essentially used
to enable or disable all checks at once.
- all-reject
This keyword covers all keywords below whose name starts with "reject".
It may be used on the CLI or in the config to quickly disable any rule
that rejects traffic in case of issues.
- all-delete
This keyword covers all keywords below whose name starts with "delete".
It may be used on the CLI or in the config to quickly disable any rule
that deletes elements in case of issues.
- delete-all-trailer-fields (inbound, request and response)
Deletes from requests and responses any trailer field, regardless of
its content or syntax. This is done after syntactic checks though. The
vast majority of public facing applications do not use trailers because
they are not reliable (intermediaries are allowed to drop or fold them)
and their presence on public networks is generally associated with
attack attempts. It is generally safe to drop them by default on public
facing listeners, and even on servers meant to deliver public contents,
in order to avoid any information leakage. Note that gRPC services use
them, in responses only.
- delete-underscore-in-header-name (inbound, request and response)
Deletes from requests and responses any header field whose name
contains an underscore character ('_' or \x5C). While this character is
perfectly permitted, some application frameworks which create automatic
variables already turn the dash ('-') into an underscore in variable
names, leading to an ambiguity with the underscore which can result in
the same variable name. This could be abused to bypass some
access-control policy for example. No standard HTTP header uses this,
so it is recommended to enable on untrusted listeners. Note that this
applies only after "reject-underscore-in-header-name" was accepted.
- delete-underscore-in-trailer-name (inbound, request and response)
Deletes from requests and responses any trailer field whose name
contains an underscore character ('_' or \x5C). While this character is
perfectly permitted, some application frameworks which create automatic
variables already turn the dash ('-') into an underscore in variable
names, leading to an ambiguity with the underscore which can result in
the same variable name. No standard HTTP trailer uses this, so it is
recommended to enable on untrusted listeners. Note that this applies
only after "reject-underscore-in-trailer-name" was accepted.
- delete-unregistered-header-fields (inbound, request and response)
Deletes from requests and responses any header field whose name starts
with 'x-' (upper or lower case). These header fields are not registered
with IANA and are not used across the internet by standard components.
They may appear in untrusted requests in attacks trying to exploit some
backend framework weaknesses however, as well as in old browsers that
were trying to work around known server bugs (e.G. "X-Pad"). While it
would be risky to block such requests, it is safe and even recommended
to delete such headers from inbound internet requests. It is preferable
not to do it on responses however when applications uses such header
fields to report information to HAProxy.
- delete-unregistered-trailer-fields (inbound, request and response)
Deletes from requests and responses any trailer field whose name starts
with 'x-' (upper or lower case). These trailer fields are not
registered with IANA and are not used across the internet by standard
components. They may appear in untrusted requests in attacks trying to
exploit some backend framework weaknesses however, so it is generally
safe and even recommended to delete such trailers from inbound internet
requests.
- reject-content-length-leading-zero (inbound request and response)
Rejects requests and responses having a Content-Length header field
whose value contains a leading zero before a digit. Some client and
server applications tend to parse this as an octal value giving a
different body size (e.g. "0300" = 192). This can be used in attacks
trying to desynchronize a client or server from the rest of the
communication, and for example for cache poisionning attacks. These may
seldom be seen on legacy applications but not on standard components
found on the internet, so it is generally safe to reject them on public
facing listeners. It is enabled by default on untrusted connections.
- reject-content-length-with-te (inbound request and response)
Rejects requests and responses having both a Content-Length header
field and a Transfer-Encoding header field. While these are permitted
by HTTP/1.1 (Transfer-Encoding wins), it can cause problems when the
message first passed through an HTTP/1.0 gateway that does not
recognize Transfer-Encoding, and can desynchronize this gateway from
the rest of the communication. These are still seldom seen in legacy
client or server applications, sometimes in relation with compression
and/or caching. While HAProxy properly normalizes them, it is generally
safe to reject them on public facing listeners. It is enabled by
default on untrusted connections.
- reject-ctl-in-header-trailer (inbound request and response)
Rejects requests and responses which contain control characters in
their header or trailer field values (codes 0-8,10-31,127). These ones
are forbidden but have long experienced a certain tolerance among
implementations, which has encouraged some rare applications to use
them as record delimiters for example. No standard component produces
them on the public internet, except attacks. By default, HAProxy only
blocks the strictly required ones (0,10,13) and lets other ones pass,
which can make certain backend applications trigger connection errors.
As such, it is both safe and recommended to block them on public facing
listeners. See also the has_ctl() converter used for detecting them.
It is enabled by default on untrusted connections.
- reject-duplicate-host-header (inbound request)
Rejects requests containing more than one "Host" header field, even if
valid and identical. This is recommended on untrusted listeners. Note that
even if not rejected, duplicate host header fields are folded into a
single one.
- reject-header-trailer-folding (inbound request and response)
Rejects requests and responses which contain folded header fields or
trailer fields. Folding consists in having a single field span multiple
lines. This practice is obsoleted by modern HTTP standards, absolutely
not seen anymore in valid messages over the internet but might still be
present in a few antique internal services dealing with large cookies.
On untrusted networks, they are always associated with attacks trying
to inject headers into non-compliant components, and should be rejected
as a sign of break-in attempt. Note that HAProxy does not have problems
with them and systematically unfolds these messages before forwarding
them. It is enabled by default on untrusted connections.
- reject-lowercase-method (inbound request)
Rejects requests whose method contains a lower-case character. No
standard HTTP method uses lower-case characters and these ones are
essentially used in desynchronization attack scenarios, by trying to
exploit different case matching schemes along the chain. This is
recommended on untrusted listeners. It is enabled by default on
untrusted connections.
- reject-underscore-in-header-name (inbound, request and response)
Rejects requests and responses in which one header field name contains
an underscore character ('_' or \x5C). This character is perfectly
permitted, but some application frameworks which create automatic
variables tend to already turn the dash ('-') into an underscore in
variable names, leading to an ambiguity with the underscore which can
result in the same variable name. This could be abused to bypass some
access-control policy for example. No standard HTTP header uses this,
so it is safe to enable on untrusted listeners. See also the
"delete-underscore-in-header-name" variant.
- reject-underscore-in-trailer-name (inbound, request and response)
Rejects requests and responses in which one tailer field name contains
an underscore character ('_' or \x5C). This character is perfectly
permitted, but some application frameworks which create automatic
variables tend to already turn the dash ('-') into an underscore in
variable names, leading to an ambiguity with the underscore which can
result in the same variable name. No standard HTTP trailer uses this,
so it is safe to enable on untrusted listeners. See also the
"delete-underscore-in-trailer-name" variant.
/*
Local variables:
fill-column: 79
End:
*/