ngx_lua: Reading POST by chunks possible?

dererk · 2013-10-05T08:11:28+00:00

Hello! On Thu, Oct 10, 2013 at 12:31 PM, Rohit Yadav wrote: > I found was that the shared Lua dictionary lives within the virtual > memory and shared...

ngx_lua: Reading POST by chunks possible?

dererk

Hi there!


Is there any way to read the data sent at a POST by chunks on ngx_lua,
instead of using req.get_body_data or req.get_post_args to get it in a
single shot?

The idea behind it is to have the availability to analyse the data being
sent before having the full transfer completed at the client side,
allowing, for example, to abort the transfer or crazy scenarios like those.

By taking a quick look at ngx_http_lua_req_body.c at
ngx_http_lua_ngx_req_read_body call, it seems that nginx is indeed
offering the capability to read the buffer by chunks:

    ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
            "lua start to read buffered request body");

    rc = ngx_http_read_client_request_body(r,
ngx_http_lua_req_body_post_read);


I know that it might appear that this scenario is a little bit
complicated, which it is :-), but it seems that the idea behind
performing some how buffered actions does work as suggested at the
ngx.req.init_body (with different effects of course):

    ngx.req.init_body(128 * 1024)  -- buffer is 128KB
    for chunk in next_data_chunk() do
        ngx.req.append_body(chunk) -- each chunk can be 4KB
    end
    ngx.req.finish_body()


Any Ideas? :-)


Thanks in advance and have a great weekend!


Cheers,

Dererk

-- 
BOFH excuse #399:
We are a 100% Microsoft Shop.

agentzh

Hello!

On Fri, Oct 4, 2013 at 5:11 PM, Dererk wrote:
> Is there any way to read the data sent at a POST by chunks on ngx_lua,
> instead of using req.get_body_data or req.get_post_args to get it in a
> single shot?
>

Use the ngx.req.socket API to read the request body data by chunks in Lua:

    http://wiki.nginx.org/HttpLuaModule#ngx.req.socket

And the lua-resty-upload library makes use of this API:

    https://github.com/agentzh/lua-resty-upload

> The idea behind it is to have the availability to analyse the data being
> sent before having the full transfer completed at the client side,
> allowing, for example, to abort the transfer or crazy scenarios like those.
>

Yes, this is what the ngx.req.socket API was originally designed for :)

> By taking a quick look at ngx_http_lua_req_body.c at
> ngx_http_lua_ngx_req_read_body call, it seems that nginx is indeed
> offering the capability to read the buffer by chunks:
>
[...]
>
>     rc = ngx_http_read_client_request_body(r,
> ngx_http_lua_req_body_post_read);
>

No, this standard Nginx request reader API does not support streaming
processing at all. The ngx_lua module implements an alternative
request body reader via ngx.req.socket :)

It may be worth mentioning that the Nginx team is going to implement
an input filter mechanism in future versions of the Nginx core, but
that is another story :)

> but it seems that the idea behind
> performing some how buffered actions does work as suggested at the
> ngx.req.init_body (with different effects of course):
>
>     ngx.req.init_body(128 * 1024)  -- buffer is 128KB
>     for chunk in next_data_chunk() do
>         ngx.req.append_body(chunk) -- each chunk can be 4KB
>     end
>     ngx.req.finish_body()
>

Well, this set of API is usually used with the ngx.req.socket API when
you still need to forward the request body to some other Nginx modules
and ngx_proxy and ngx_fastcgi eventually.

Regards,
-agentzh