body_filter with Redis/Mysql?

gabor.lengyel · 2013-07-02T02:45:06+00:00

Hello! Sorry for the delay! I missed your last email somehow. On Thu, Jun 20, 2013 at 3:21 PM, mfeldheim wrote: > so the next step will be to get the wh...

body_filter with Redis/Mysql?

gabor.lengyel

Hi,

As far as I understand, Redis, Mysql, etc. cannot be used in body_filter_by_lua. How do I get around this, if in a reverse proxy setup I need to modify the content served by the upstream server, before passing it on to the client, and the modification needs data from a database?

Sorry if this is obvious, I'm new to Openresty / Nginx.

Thanks!
Gabor

agentzh

Hello!

On Mon, Jul 1, 2013 at 7:46 AM,  gabor.lengyel wrote:
> As far as I understand, Redis, Mysql, etc. cannot be used in
> body_filter_by_lua. How do I get around this, if in a reverse proxy setup I
> need to modify the content served by the upstream server, before passing it
> on to the client, and the modification needs data from a database?
>

Instead of using a body filter, you can just use ngx.location.capture
to do a subrequest to access your location configured by ngx_proxy and
modify the whole response body in Lua before sending it out. See

    http://wiki.nginx.org/HttpLuaModule#ngx.location.capture

Alternatively, you can cache your modification rules stored in Redis
or MySQL in either your nginx's Lua variables or shared memory
dictionaries so that you do not have to do I/O in your body filters.
In this setting, you can also use ngx.timer.at to periodically
synchronize your local rule cache with your backend storage. See

    http://wiki.nginx.org/HttpLuaModule#ngx.timer.at

BTW, please join the mailing list before posting, otherwise your posts
require manual moderation.

Thanks!
-agentzh

gabor.lengyel

Hi!

On Monday, July 1, 2013 8:49:42 PM UTC+2, agentzh wrote:

Hello!

On Mon, Jul 1, 2013 at 7:46 AM, gabor.lengyel wrote:
> As far as I understand, Redis, Mysql, etc. cannot be used in
> body_filter_by_lua. How do I get around this, if in a reverse proxy setup I
> need to modify the content served by the upstream server, before passing it
> on to the client, and the modification needs data from a database?
>

Instead of using a body filter, you can just use ngx.location.capture
to do a subrequest to access your location configured by ngx_proxy and
modify the whole response body in Lua before sending it out. See

http://wiki.nginx.org/HttpLuaModule#ngx.location.capture

Alternatively, you can cache your modification rules stored in Redis
or MySQL in either your nginx's Lua variables or shared memory
dictionaries so that you do not have to do I/O in your body filters.
In this setting, you can also use ngx.timer.at to periodically
synchronize your local rule cache with your backend storage. See

http://wiki.nginx.org/HttpLuaModule#ngx.timer.at

Awsome! I will fiddle with this and see what I can come up with, both options seem feasible.
Thank you for the very quick and helpful answer!

Gabor

kroptgye

Alternative sounds great, but how much data i can store in «Lua variables or shared memory
dictionaries»
Thanks
On Monday, July 1, 2013 10:49:42 PM UTC+4, agentzh wrote:

Hello!

On Mon, Jul 1, 2013 at 7:46 AM, gabor.lengyel wrote:
> As far as I understand, Redis, Mysql, etc. cannot be used in
> body_filter_by_lua. How do I get around this, if in a reverse proxy setup I
> need to modify the content served by the upstream server, before passing it
> on to the client, and the modification needs data from a database?
>

Instead of using a body filter, you can just use ngx.location.capture
to do a subrequest to access your location configured by ngx_proxy and
modify the whole response body in Lua before sending it out. See

http://wiki.nginx.org/HttpLuaModule#ngx.location.capture

Alternatively, you can cache your modification rules stored in Redis
or MySQL in either your nginx's Lua variables or shared memory
dictionaries so that you do not have to do I/O in your body filters.
In this setting, you can also use ngx.timer.at to periodically
synchronize your local rule cache with your backend storage. See

http://wiki.nginx.org/HttpLuaModule#ngx.timer.at

BTW, please join the mailing list before posting, otherwise your posts
require manual moderation.

Thanks!
-agentzh

agentzh

Hello!

On Mon, Jul 8, 2013 at 5:00 PM, Viktor malezink wrote:
> Alternative sounds great, but how much data i can store in «Lua variables or
> shared memory
> dictionaries»

For the former, it depends on the GC limit of your Lua VM (if you're
using LuaJIT 2.0 on x86_64, there is a 1 GB limit). For the latter, it
depends on the actual size you configure via the lua_shared_dict
directive. And of course, your physical memory size is always the hard
limit too.

Regards,
-agentzh

kroptgye

So that mean i can store lot of data(hash table as example) in dict and sync with ngx.timer.

Thanks for answer.

On Tuesday, July 9, 2013 4:10:30 AM UTC+4, agentzh wrote:

Hello!

On Mon, Jul 8, 2013 at 5:00 PM, Viktor malezink wrote:
> Alternative sounds great, but how much data i can store in «Lua variables or
> shared memory
> dictionaries»

For the former, it depends on the GC limit of your Lua VM (if you're
using LuaJIT 2.0 on x86_64, there is a 1 GB limit). For the latter, it
depends on the actual size you configure via the lua_shared_dict
directive. And of course, your physical memory size is always the hard
limit too.

Regards,
-agentzh