request logging design patterns

jenders · 2016-03-17T09:27:34+00:00

Well that was straight forward :)Thanks!On Monday, March 14, 2016 at 9:52:20 PM UTC, rpaprocki wrote:Rewrite phase handlers run before access phase handler (...

request logging design patterns

jenders

Hi everybody,

I'm working on a project to gather log data from several sources that (only) offer streaming logs via json posts. My goal is to ingest the log, normalize the field data and then write it out to durable storage. This seemed like a natural fit for OpenResty but I'm having some issues that hopefully you can help me solve.

My first implementation parses the post body of each request using content_by_lua_file and then sets global nginx variables that are referred to within a log_format directive and logged locally to disk via the logging phase. This works quite well if there is only one message per request but often, a provider will batch multiple messages within a single HTTP request for efficiency. To continue with the current design, I seemingly need to invoke multiple logging phases for a single request. This doesn't seem very elegant or possible under the current API.

It suspect it's not very efficient but could this be done via subrequests or is there a better way to go about this? Maybe using Jiale's and Yichun's https://github.com/cloudflare/lua-resty-logger-socket and sending formatted log messages to syslog-ng?

Thanks for your help!

agentzh

Hello!

On Wed, Mar 16, 2016 at 6:27 PM, 'Josh Enders' via openresty-en wrote:
>
> It suspect it's not very efficient but could this be done via subrequests or
> is there a better way to go about this? Maybe using Jiale's and Yichun's
> https://github.com/cloudflare/lua-resty-logger-socket and sending formatted
> log messages to syslog-ng?
>

Yes. Or you can create a lua-resty-logger-http library in a similar way.

Regards,
-agentzh