How to overcome nginx output filter limitations when using body_filter_by_lua?

gauravphoenix · 2014-11-18T04:17:52+00:00

Thanks you!I wasn't aware of hard coded length restriction. After putting string.length in lua script, I think I am getting expected length. Thank...

How to overcome nginx output filter limitations when using body_filter_by_lua?

gauravphoenix

I need to put a string in a third party system (mongodb/redis etc) when using body_filter_by_lua. As per the documentation,

Note that the following API functions are currently disabled within this context due to the limitations in NGINX output filter's current implementation:

Output API functions (e.g., ngx.say and ngx.send_headers)
Control API functions (e.g., ngx.exit and ngx.exec)
Subrequest API functions (e.g., ngx.location.capture and ngx.location.capture_multi)
Cosocket API functions (e.g., ngx.socket.tcp and ngx.req.socket).

I am not able to use following code

local red = redis:new()

when using body_filter_by_lua. I think it is due to the fact that redis library must be using ngx.socket or any of the functions which are disabled as per documentation.

What is the recommended way to overcome this limitation?

agentzh

Hello!

On Mon, Nov 17, 2014 at 12:17 PM, Gaurav Kumar wrote:
> I need to put a string in a third party system (mongodb/redis etc) when
> using body_filter_by_lua. As per the documentation,
>
[...]
> when using body_filter_by_lua. I think it is due to the fact that redis
> library must be using ngx.socket or any of the functions which are disabled
> as per documentation.
>

The work-around is also officially documented :) To quote,

https://github.com/openresty/lua-nginx-module#cocockets-not-available-everywhere

"There exists a work-around, however, when the original context does
not need to wait for the cosocket results. That is, creating a 0-delay
timer via the ngx.timer.at API and do the cosocket results in the
timer handler, which runs asynchronously as to the original context
creating the timer."

BTW, for the best performance, you'd better buffer the messages on the
Lua land a little bit so that you can take advantage of the redis
pipelining feature to reduce the (expensive) redis roundtrips and
socket-related system calls. Or in other words, use the "batch
processing" model. The lua-resty-logger-socket library, for example,
uses this buffering model as well:
https://github.com/cloudflare/lua-resty-logger-socket

Best regards,
-agentzh

gauravphoenix

Thank you! I didn't refer to the GitHub documentation. Perhaps wiki documentation can refer to github documentation also ?

Anyways, if I understand correctly, I can use lua-resty-logger-socket library under body_filter_by_lua and log string using lua-resty-logger-socket?

On Mon, Nov 17, 2014 at 12:32 PM, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

Hello!

On Mon, Nov 17, 2014 at 12:17 PM, Gaurav Kumar wrote:
> I need to put a string in a third party system (mongodb/redis etc) when
> using body_filter_by_lua. As per the documentation,
>
[...]
> when using body_filter_by_lua. I think it is due to the fact that redis
> library must be using ngx.socket or any of the functions which are disabled
> as per documentation.
>

The work-around is also officially documented :) To quote,

https://github.com/openresty/lua-nginx-module#cocockets-not-available-everywhere

"There exists a work-around, however, when the original context does
not need to wait for the cosocket results. That is, creating a 0-delay
timer via the ngx.timer.at API and do the cosocket results in the
timer handler, which runs asynchronously as to the original context
creating the timer."

BTW, for the best performance, you'd better buffer the messages on the
Lua land a little bit so that you can take advantage of the redis
pipelining feature to reduce the (expensive) redis roundtrips and
socket-related system calls. Or in other words, use the "batch
processing" model. The lua-resty-logger-socket library, for example,
uses this buffering model as well:
https://github.com/cloudflare/lua-resty-logger-socket

Best regards,
-agentzh

.

agentzh

Hello!

On Mon, Nov 17, 2014 at 12:47 PM, Gaurav Kumar wrote:
> Thank you! I didn't refer to the GitHub documentation. Perhaps wiki
> documentation can refer to github documentation also ?
>

The wiki and github versions are almost always synchronized. If not,
please let me know :)

> Anyways, if I understand correctly, I can use lua-resty-logger-socket
> library under body_filter_by_lua and log string using
> lua-resty-logger-socket?
>

Yes, you could though you don't have to. It's all up to you :)

Regards,
-agentzh

gauravphoenix

Nevermind, I think I answered my own questions. Since Cocockets are not available, lua-resty-logger-socket won't work either. So I need to experiment with ngx.timer.at API.

On Mon, Nov 17, 2014 at 12:47 PM, Gaurav Kumar <gaurav...@gmail.com> wrote:

Thank you! I didn't refer to the GitHub documentation. Perhaps wiki documentation can refer to github documentation also ?

Anyways, if I understand correctly, I can use lua-resty-logger-socket library under body_filter_by_lua and log string using lua-resty-logger-socket?

On Mon, Nov 17, 2014 at 12:32 PM, Yichun Zhang (agentzh) <age...@gmail.com> wrote:
Hello!

On Mon, Nov 17, 2014 at 12:17 PM, Gaurav Kumar wrote:
> I need to put a string in a third party system (mongodb/redis etc) when
> using body_filter_by_lua. As per the documentation,
>
[...]
> when using body_filter_by_lua. I think it is due to the fact that redis
> library must be using ngx.socket or any of the functions which are disabled
> as per documentation.
>

The work-around is also officially documented :) To quote,

https://github.com/openresty/lua-nginx-module#cocockets-not-available-everywhere

"There exists a work-around, however, when the original context does
not need to wait for the cosocket results. That is, creating a 0-delay
timer via the ngx.timer.at API and do the cosocket results in the
timer handler, which runs asynchronously as to the original context
creating the timer."

BTW, for the best performance, you'd better buffer the messages on the
Lua land a little bit so that you can take advantage of the redis
pipelining feature to reduce the (expensive) redis roundtrips and
socket-related system calls. Or in other words, use the "batch
processing" model. The lua-resty-logger-socket library, for example,
uses this buffering model as well:
https://github.com/cloudflare/lua-resty-logger-socket

Best regards,
-agentzh

.

agentzh

Hello!

On Mon, Nov 17, 2014 at 12:50 PM, Gaurav Kumar wrote:
> Nevermind, I think I answered my own questions. Since Cocockets are not
> available,  lua-resty-logger-socket won't work either. So I need to
> experiment with ngx.timer.at API.
>

IIRC, lua-resty-logger-socket already handles the timer part
automatically for you. It should work out of the box. If not, let me
know :)

FWIW, we use this library directly in log_by_lua at CloudFlare :)

Regards,
-agentzh