Conditional Logging

gmadarm · 2014-06-29T04:58:28+00:00

Couple of things for anyone ready to play with rrdtool and lua:1. Get lua 5.1. Lua 5.2 will not work with RRDtool since some calls have been replaced from ma...

Conditional Logging

gmadarm

Hi,

I want to conditionally log the entire HTTP request/response (including request headers, POST body, response headers and response body) to a separate log file. An example condition for logging is if a given keyword is found anywhere within the request headers or body. I could use ngx.log() for this but would need to modify

ngx_log.h to bypass the hard-coded 2048 byte limitation. Is there a better way to implement this?

thanks,
- G

agentzh

Hello!

On Sat, Jun 28, 2014 at 1:58 PM, Gordon Madarm wrote:
> I want to conditionally log the entire HTTP request/response (including
> request headers, POST body, response headers and response body) to a
> separate log file. An example condition for logging is if a given keyword is
> found anywhere within the request headers or body. I could use ngx.log() for
> this but would need to modify ngx_log.h to bypass the hard-coded 2048 byte
> limitation. Is there a better way to implement this?
>

File-based logging is blocking due to blocking normal file syscalls
and nginx's error logging is especially expensive in that it disables
buffering completely and should only be used for true errors and
debugging. Better log to sockets, for example,

    https://github.com/cloudflare/lua-resty-logger-socket#name

Regards,
-agentzh

rpaprocki

Would we also be able to use the socket functionality found within thestandard Lua nginx module(http://wiki.nginx.org/HttpLuaModule#tcpsock:connect)? We would like tobe able to log data including variables set during other phases,specifically the access phase. Is the Lua module UDP socket sendfunctionality synchronous but non-blocking?


On 6/28/2014 17:43, Yichun Zhang (agentzh) wrote:

Hello!

On Sat, Jun 28, 2014 at 1:58 PM, Gordon Madarm wrote:

I want to conditionally log the entire HTTP request/response (including
request headers, POST body, response headers and response body) to a
separate log file. An example condition for logging is if a given keyword is
found anywhere within the request headers or body. I could use ngx.log() for
this but would need to modify ngx_log.h to bypass the hard-coded 2048 byte
limitation. Is there a better way to implement this?

File-based logging is blocking due to blocking normal file syscalls
and nginx's error logging is especially expensive in that it disables
buffering completely and should only be used for true errors and
debugging. Better log to sockets, for example,

     https://github.com/cloudflare/lua-resty-logger-socket#name

Regards,
-agentzh

agentzh

Hello!

On Sat, Jun 28, 2014 at 7:37 PM, Robert Paprocki wrote:
> Would we also be able to use the socket functionality found within the
> standard Lua nginx module
> (http://wiki.nginx.org/HttpLuaModule#tcpsock:connect)? We would like to be
> able to log data including variables set during other phases, specifically
> the access phase. Is the Lua module UDP socket send functionality
> synchronous but non-blocking?
>

The lua-resty-logger-socket library does use the ngx.socket.tcp API
internally and can be used in phases other than the log phase.

Regards,
-agentzh

peter_booth

Gordon,

Some questions:

1. What fraction of requests do you estimate need "full logging" ?

2. Are your web servers fronted with a hardware load balancer like an F 5 LTM or similar ?

3. Why do you need this?

I figure you have a good reason for this, but I don't know what it is.

Peter

Sent from my iPhone

On Jun 28, 2014, at 4:58 PM, Gordon Madarm <gma...@gmail.com> wrote:

Hi,

I want to conditionally log the entire HTTP request/response (including request headers, POST body, response headers and response body) to a separate log file. An example condition for logging is if a given keyword is found anywhere within the request headers or body. I could use ngx.log() for this but would need to modify ngx_log.h to bypass the hard-coded 2048 byte limitation. Is there a better way to implement this? thanks, - G

.