blog に戻る

2021年09月09日 Sumo Logic

All you need to know about HAProxy log format

HAProxy is one of the fastest and most widely-used load balancing solutions available today. If you’re already using HAProxy, or if you’re considering using HAProxy in your environment, then this is a great place to start. On this page, we discuss HAProxy logging and why logging is such a vital component of the load balancer implementation. We then take a deep dive into the logging offered by HAProxy. Finally, you’ll read about working with the HAProxy logging format and how you can configure the logging to suit your needs better.

Why Load Balancer Logging is Critical

When connecting your application to the outside world, the load balancer is an essential component. In its simplest form, the load balancer is responsible for analyzing all inbound requests, determining their validity, and then passing them to the appropriate destination. The load balancer does more than just route traffic; it ensures that requests are dispersed evenly between all healthy instances of your application, preventing requests from reaching unhealthy instances and ensuring high availability.

As the critical component that handles all requests, the load balancer often becomes the focal point of investigations relative to connectivity issues. Timely and accurate logs are an essential part of any investigation. HAProxy logs can provide unique insights into all aspects of failed and successful requests, including performance and routing information.

HAProxy Logging 101

HAProxy provides extremely detailed logging with millisecond precision to provide actionable data about your infrastructure at a macro and a micro-level. Let’s look at some of the information you can glean from an HAProxy log and how each data point helps provide comprehensive observability for your application.

Traffic Metrics

Understanding the volume and size of traffic moving through your application can provide essential guidance about required resources downstream, and critical insights about how, why, and when failures might occur.

Request and Response Details

If you’re investigating error conditions that affect targeted users or regions, inspecting the headers, payloads, and status codes of requests and responses can help you isolate the exact cause of a problem and provide detailed information about the behaviors that clients may experience.

Routing Decisions

HAProxy logs also include information about each request’s decisions, such as the identity of which downstream target to which the request was routed, and filtering applied to requests and information about persistence between requests from the same client.

Error Tracing

Finally, the logs contain information that engineers can use to identify where request life cycle failures occur. The logs can also include information relating to active sessions and their termination status.

Default and Pre-Configured Logging Formats

HAProxy derives the logging format from settings within the HAProxy configuration file. You can use the default logging format by excluding the option directive from the configuration, or setting one of two pre-configured formats.

  • For TCP or Layer 4 operation mode, the directive to include is option tcplog.

  • For HTTP or Layer 7 operation mode, the configuration directive is option httplog.

We’ll break down an example of each log format below.

These example logs are from an instance of HAProxy version 2.4.2, installed on an AWS instance running AWS Linux 4.14 and configured to manage traffic between two HTTP servers within the same subnet. The logs were captured using RSyslog and stored locally on the host server.

Default Log Format (No option configured)

Jul 12 06:32:30 localhost haproxy[2679]: Connect from 67.171.183.156:50871 to 172.31.30.201:80 (http_front/TCP)



Jul 12 06:32:30

Log Time Stamp

localhost

Hostname or IP address of HAProxy host

haproxy[2679]:

Process ID for the HAProxy process

Connect from 67.171.183.156:50871 to 172.31.30.201:80

Connect from

Source IP:Source Port

To

Destination IP:Destination Port

(http_front/TCP)

Front-end name / Front-end mode


TCP / Layer 4 Log Format (option tcplog)

Jul 12 06:24:02 localhost haproxy[2590]: 67.171.183.156:54500 [12/Jul/2021:06:23:21.058] http_front http_back/webserver1 1/0/40996 383 -- 2/2/1/0/0 0/0



Jul 12 06:24:02

Log Time Stamp

localhost

Hostname or IP address of HAProxy host

haproxy[2590]:

Process ID for the HAProxy process

67.171.183.156:54500

Source IP:Source Port

[12/Jul/2021:06:23:21.058]

Timestamp request accepted, with millisecond accuracy

http_front

Front-end name

http_back/webserver1

Target request was routed to

1/0/40996

Time waiting in queues (ms) /

Time to establish the connection to destination server (ms) /

Total time from request receipt until last close (ms)

383

Bytes read

--

Termination state preceded by --

2/2/1/0/0

Active connections /

Front-end connections /

Back-end connections /

Server connections /

Retries

0/0

Server queue /

Back-end queue


HTTP / Layer 7 (option httplog)

Jul 12 05:54:55 localhost haproxy[1060]: 67.171.183.156:64978 [12/Jul/2021:05:54:55.077] http_front http_back/webserver1 0/0/0/1/1 200 288 - - ---- 2/2/0/0/0 0/0 "GET / HTTP/1.1"



Jul 12 05:54:55

Log Time Stamp

localhost

Hostname or IP address of HAProxy host

haproxy[1060]:

Process ID for the HAProxy process

67.171.183.156:64978

Source IP:Source Port

[12/Jul/2021:05:54:55.077]

Timestamp request accepted, with millisecond accuracy

http_front

Front-end name

http_back/webserver1

Target request was routed to

0/0/0/1/1

Time waiting for full request from client (ms) /

Time waiting in queues (ms) /

Time to establish connection to destination server (ms) /

Time for destination server to send response (ms) /

Total time request active in HAProxy (ms)

200

HTTP Response Code

288

Bytes read

- -

Optional values:

  • Captured request cookie

  • Captured response cooking

----

Termination state preceded by --

2/2/0/0/0

Active connections /

Front-end connections /

Back-end connections /

Server connections /

Retries

0/0

Server queue size /

Back-end queue size

"GET / HTTP/1.1"

HTTP Request


Customizing the HAProxy Log Format

While the default and pre-configured log formats available in HAProxy can provide substantial insights, you might have noticed an absence of metrics and log elements related to activities supporting SSL encryption, headers, and the payload itself. While these elements can significantly increase the amount of data in the log, they can also help resolve problems.

If you want to use a customized log format, you can replace the option direction with a log-format directive, followed by a string indicating the contents and format of the desired log messages. If we were to replicate the HTTP log format described above, the directive would be as follows:

log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"


A complete reference of available options and their correct usage is listed in the HAProxy Documentation. Some of the available options that you may want to include to support advanced troubleshooting include:

  • Connection handshake time (%Th)

  • SSL ciphers and versions (%sslc / %sslv)

  • Request headers (%hr or %hrl for CLF formatting)

  • Response headers (%hs or %hsl for CLF formatting)

  • The full HTTP request (%r)

Managing Log Volume at Scale

HAProxy is one of the most performant load-balancing solutions currently available, and as such, can reasonably handle thousands of requests each second. While HAProxy can easily create a log entry for each request, you might find yourself in a position where you need to explore creative options to support the storage of these logs. Identifying alternate options is critical if your team prototyped an HAProxy installation that directed the logs to disk storage.


In addition to the volume of log data, you also need to search the logs for anomalies, or even better, automate anomaly detection to alert your team when problems occur. Partnering with a log management solution partner such as Sumo Logic will help resolve both of these problems and provide your teams with fast and reliable tools to support monitoring efforts and analysis when issues arise. You can learn more about integrating HAProxy with Sumo Logic here and sign up for a free trial to try it out for yourself.

Complete visibility for DevSecOps

Reduce downtime and move from reactive to proactive monitoring.

Sumo Logic cloud-native SaaS analytics

Build, run, and secure modern applications and cloud infrastructures.

Start free trial

Sumo Logic

More posts by Sumo Logic.

これを読んだ人も楽しんでいます