Best practices for logging to external services

peter · 2013-12-04T04:08:03+00:00

It may use it in places, but I think most of the per connection/request stuff is all pool based. One way to find out is compile with jemalloc and compare per...

Best practices for logging to external services

peter

Hello,

In a log_by_lua directive, I’m creating per-request log records that I’m serializing and storing in a shared dictionary. A self-perpetuating ngx.timer periodically flushes entries from the dictionary and sends them to RabbitMQ. It’s a lot like what was discussed in this thread, but it looks like that conversation predated ngx.timer:

https://groups.google.com/d/msg/openresty-en/U7kwHEJsXd4/m6xcrvyFreUJ

I have a couple of questions:

Is this still the best/most generally accepted way of logging directly to an external service from OpenResty?
Are there any guidelines I should follow around the frequency of the ngx.timer calls? I’m currently running it every 5 seconds, but I just want to be sure I’m staying far away from some of the problems described in the documentation around the accumulation of timers.

Thank you,

Peter

agentzh

Hello!

On Tue, Dec 3, 2013 at 12:08 PM, Peter Schwartz wrote:
> In a log_by_lua directive, I’m creating per-request log records that I’m
> serializing and storing in a shared dictionary. A self-perpetuating
> ngx.timer periodically flushes entries from the dictionary and sends them to
> RabbitMQ.

Yes, using buffers and timers is the recommended approach. See also
the lua-resty-logger-socket library:

  https://github.com/cloudflare/lua-resty-logger-socket

There are still a few important unresolved tickets for a few bad
corner cases in this library though:

  https://github.com/cloudflare/lua-resty-logger-socket/issues

My colleague Jiale Zhi will work on these issues very soon.

Best regards,
-agentzh

peter

Great, thanks for your prompt reply. Good to know I'm on the right track for now. I'll definitely keep an eye on lua-resty-logger-socket, and am hopeful that the cosocket API will one day be accessible in the logging phase :-)

On Tuesday, December 3, 2013 3:20:01 PM UTC-5, agentzh wrote:

Hello!

On Tue, Dec 3, 2013 at 12:08 PM, Peter Schwartz wrote:
> In a log_by_lua directive, I’m creating per-request log records that I’m
> serializing and storing in a shared dictionary. A self-perpetuating
> ngx.timer periodically flushes entries from the dictionary and sends them to
> RabbitMQ.

Yes, using buffers and timers is the recommended approach. See also
the lua-resty-logger-socket library:

https://github.com/cloudflare/lua-resty-logger-socket

There are still a few important unresolved tickets for a few bad
corner cases in this library though:

https://github.com/cloudflare/lua-resty-logger-socket/issues

My colleague Jiale Zhi will work on these issues very soon.

Best regards,
-agentzh