Does the ngx lua module support chunked requests (i.e. HTTP POST)

rvsw · 2014-01-30T22:01:04+00:00

Hello,Thanks Brian for your comments and suggestion on this. I will check the possibility to use redis as replacement of core proxy cache. I want to cache la...

Does the ngx lua module support chunked requests (i.e. HTTP POST)

rvsw

I need to modify HTTP POST. I can do this by using lua code in a rewrite_by_lua directive like this :

my_request_body = ngx.req.get_body_data()

my_request_body_modified = modify_with_some_func(my_request_body)

ngx.req.set_body_data(my_request_body_modified)

If the request body is large and put into a file than I can use similar functions that work with disk files

All these works ok when the request body is not chunked

It is not clear from the documentation what will happen if the request body chunked. Will the entire request body chunks be buffered and delivered OR will there be separate chunks. I see the following in documentation for ngx.req.socket.

If any request body data has been pre-read into the Nginx core request header buffer, the resulting cosocket object will take care of this to avoid potential data loss resulting from such pre-reading. Chunked request bodies are not yet supported in this API.

Does this pertain to ALL the API in LUA module

agentzh

Hello!

On Thu, Jan 30, 2014 at 6:01 AM, rvsw wrote:
> All these works ok when the request body is not chunked
> It is not clear from the documentation what will happen if the request body
> chunked.

For chunked request bodies, the nginx core's request body reader (used
by ngx.req.read_body() and lua_need_request_body) can handle it
transparently. So it should work out of the box for you. If it is not,
please report it as a bug.

> Will the entire request body chunks  be buffered and delivered OR
> will there be separate chunks. I see the following in documentation for
> ngx.req.socket.
[...]
>
> Does this pertain to ALL the API in LUA module
>

No. This limitation only applies to ngx.req.socket which implements
its own request body reader (which has nothing to do with nginx's
builtin reader).

Regards,
-agentzh

agentzh

Hello!

On Thu, Jan 30, 2014 at 3:22 PM, Yichun Zhang (agentzh) wrote:
>> Will the entire request body chunks  be buffered and delivered OR
>> will there be separate chunks.

Forgot to mention that the nginx builtin request reader always fully
buffer the request body even when it is using the chunked encoding. So
from the caller's perspective, there is no difference with normal
non-chunked request bodies.

Regards,
-agentzh

jenders

Will large request body chunks block the worker processes? Is there a way to read the chunks as they come in?

On Thursday, January 30, 2014 at 3:24:38 PM UTC-8, agentzh wrote:

Hello!

On Thu, Jan 30, 2014 at 3:22 PM, Yichun Zhang (agentzh) wrote:
>> Will the entire request body chunks be buffered and delivered OR
>> will there be separate chunks.

Forgot to mention that the nginx builtin request reader always fully
buffer the request body even when it is using the chunked encoding. So
from the caller's perspective, there is no difference with normal
non-chunked request bodies.

Regards,
-agentzh

agentzh

Hello!

On Mon, Mar 28, 2016 at 11:35 PM, 'Josh Enders' via openresty-en wrote:
> Will large request body chunks block the worker processes? Is there a way to
> read the chunks as they come in?
>

If the nginx core buffers the large request body data into disk (as
temp files), then it may block the nginx processes for a little bit.
But it usually is not an issue if your disk is not painfully slow.

Using the raw downstream cosocket can read chunked request body data
in a streaming fashion, there is a tested example in the mailing list
archive and the ngx_lua's official test suite.

Regards,
-agentzh

ganzhi

You can turn off request buffering in nginx 1.7.11+ now. Check the Nginx document here: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_request_buffering

On Mon, 28 Mar 2016 at 23:35 'Josh Enders' via openresty-en <openresty-en@googlegroups.com> wrote:

Will large request body chunks block the worker processes? Is there a way to read the chunks as they come in?

On Thursday, January 30, 2014 at 3:24:38 PM UTC-8, agentzh wrote:
Hello!

On Thu, Jan 30, 2014 at 3:22 PM, Yichun Zhang (agentzh) wrote:
>> Will the entire request body chunks be buffered and delivered OR
>> will there be separate chunks.

Forgot to mention that the nginx builtin request reader always fully
buffer the request body even when it is using the chunked encoding. So
from the caller's perspective, there is no difference with normal
non-chunked request bodies.

Regards,
-agentzh

.

jenders

On Wednesday, April 6, 2016 at 9:04:19 AM UTC-7, agentzh wrote:

Hello!

On Mon, Mar 28, 2016 at 11:35 PM, 'Josh Enders' via openresty-en wrote:
> Will large request body chunks block the worker processes? Is there a way to
> read the chunks as they come in?
>

If the nginx core buffers the large request body data into disk (as
temp files), then it may block the nginx processes for a little bit.
But it usually is not an issue if your disk is not painfully slow.

Using the raw downstream cosocket can read chunked request body data
in a streaming fashion, there is a tested example in the mailing list
archive and the ngx_lua's official test suite.

Regards,
-agentzh

Excellent, thanks! Here it is for anyone else looking for it: https://github.com/openresty/lua-nginx-module/blob/75cc29ea64a87bc5cd447525893fda76b8d664b4/t/116-raw-req-socket.t#L789-L856 I'll give that a shot. Any chance there will be a higher-level api for this in the future?