blog/_posts/2014-05-15-key-value-logs-with-nginx.md

permalink	title	published_date	layout	data
/{{ year }}/{{ month }}/{{ day }}/key-value-logs-with-nginx	key=value logs with nginx	2014-04-15 18:24:00 +0200	post.liquid	route blog

In Six Ways to Make Your Production Logs More Useful @roidrage talked about how to make your logs much more useful. One of the proposed solutions was to use a format that is more structured and thus easier to read: key=value logging. This way it's easy to parse for a human eye, but a machine will have no problems either. And of course your standard shell tools (with the combination of some basic regular expressions) will still work to find what you want in your logs.

This blog and all of my other sites run on nginx and with nginx it is very easy to use a custom log format. Just define a new log_format, give it a meaningful name and the layout you want. For example I use the following now:

log_format  keyvalue  'time=$time_iso8601 ip=$remote_addr user=$remote_user req="$request" '
                  'status=$status bytes_sent=$body_bytes_sent req_time=$request_time ref="$http_referer" '
                  'ua="$http_user_agent" forwarded="$http_x_forwarded_for"';

Put this in your http {} block and wherever you define a new access log file append keyvalue (or the name you have chosen) like this:

access_log  /var/log/nginx/access.log  keyvalue;

And voilà:

$ tail -1 /var/log/nginx/access.log
time=2014-04-15T18:20:54+02:00 ip=127.0.0.1 user=- req="GET / HTTP/1.1" status=301 bytes_sent=184 req_time=0.000 ref="-" ua="curl/7.36.0" forwarded="-"

If you want to know which variables are available, read the nginx docs.

Update: @rhoml pointed me to a full overview of available variables.