ngx_lua POST request body reading issues

kiprasm · 2015-05-15T00:54:43+00:00

Hello! On Wed, Jun 3, 2015 at 12:51 AM, waziers wrote: > However, closing the file to create a valid zip file is more problematic: I > can detect the...

ngx_lua POST request body reading issues

kiprasm

Hey,

i'm having issues trying to proxy POST requests with ngx_lua.

I'm using https://github.com/pintsized/lua-resty-http to run the actual requests to upstream.

The problem is with request body reading.

The default (and afaik most efficient) way to do this is to use the lua-resty-http function. But this doesn't work when under an HTTPS SPDY connection, because it uses to read the request body (which doesn't work under SPDY).

So in that case we can fall back to using , but now we face an issue when POST data size exceeds , and the response data is now buffered into temporary files and this method returns .

So then you can then call to get the temporary file name, which it does return. However, trying to read the file gives an error like this:

error while opening file: /usr/local/openresty/nginx/client_body_temp/0000000100: No such file or directory

Which makes no sense, considering that did return this file path, which means it is actually used. It probably gets deleted before you can try to access it.

One workaround is to set to "on" or "clean" in nginx config, in which case you can actually read that file and forward the request body stored in it. But i fear this may have performance implications, since afaik all POST requests bodies (even small ones) will now be buffered to disk.

Is anyone aware of these issues, and if so - do you know of any stable, recommended way of dealing with POST request bodies?

kiprasm

Turns out the issue (not being able to open the temporary request body file) was caused by calling after . When is not called - everything works well.

Seems like a bug in ngx_lua or perhaps some info missing from documentation?

nginx version: openresty/1.7.4.1

On Thursday, May 14, 2015 at 7:54:42 PM UTC+3, Kipras Mancevičius wrote:

Hey,

i'm having issues trying to proxy POST requests with ngx_lua.
I'm using https://github.com/pintsized/lua-resty-http to run the actual requests to upstream.

The problem is with request body reading.
The default (and afaik most efficient) way to do this is to use the lua-resty-http get_client_body_reader() function. But this doesn't work when under an HTTPS SPDY connection, because it uses to read the request body (which doesn't work under SPDY).

So in that case we can fall back to using , but now we face an issue when POST data size exceeds , and the response data is now buffered into temporary files and this method returns .

So then you can then call to get the temporary file name, which it does return. However, trying to read the file gives an error like this:

error while opening file: /usr/local/openresty/nginx/client_body_temp/0000000100: No such file or directory

Which makes no sense, considering that did return this file path, which means it is actually used. It probably gets deleted before you can try to access it.

One workaround is to set to "on" or "clean" in nginx config, in which case you can actually read that file and forward the request body stored in it. But i fear this may have performance implications, since afaik all POST requests bodies (even small ones) will now be buffered to disk.

Is anyone aware of these issues, and if so - do you know of any stable, recommended way of dealing with POST request bodies?

kiprasm

ok, so i get this error when calling for POST requests with a large body size (> 16kb):

requesty body in temp file not supported

So i guess, this is probably expected behavior for the current ngx_lua, and in this case (when request body is in a temp file) you have to parse POST arguments yourself in Lua, right?

Suggestion: adding some quick notice about this in the docs (for the ) would help, i think :)

On Friday, May 15, 2015 at 12:49:54 PM UTC+3, Kipras Mancevičius wrote:

Turns out the issue (not being able to open the temporary request body file) was caused by calling after . When is not called - everything works well.

Seems like a bug in ngx_lua or perhaps some info missing from documentation?

nginx version: openresty/1.7.4.1

On Thursday, May 14, 2015 at 7:54:42 PM UTC+3, Kipras Mancevičius wrote:
Hey,

i'm having issues trying to proxy POST requests with ngx_lua.
I'm using https://github.com/pintsized/lua-resty-http to run the actual requests to upstream.

The problem is with request body reading.
The default (and afaik most efficient) way to do this is to use the lua-resty-http get_client_body_reader() function. But this doesn't work when under an HTTPS SPDY connection, because it uses to read the request body (which doesn't work under SPDY).

So in that case we can fall back to using , but now we face an issue when POST data size exceeds , and the response data is now buffered into temporary files and this method returns .

So then you can then call to get the temporary file name, which it does return. However, trying to read the file gives an error like this:

error while opening file: /usr/local/openresty/nginx/client_body_temp/0000000100: No such file or directory

Which makes no sense, considering that did return this file path, which means it is actually used. It probably gets deleted before you can try to access it.

One workaround is to set to "on" or "clean" in nginx config, in which case you can actually read that file and forward the request body stored in it. But i fear this may have performance implications, since afaik all POST requests bodies (even small ones) will now be buffered to disk.

Is anyone aware of these issues, and if so - do you know of any stable, recommended way of dealing with POST request bodies?

hamish.forbes

You could try proxying the https SPDY connections to a non-SPDY endpoint (e.g. an alternate port on localhost) and then use lua-resty-http and get_client_body_reader() to stream the request to your upstream from there.

Our architecture has a vanilla Nginx proxy terminating SPDY/TLS in front of any systems running Lua anyway so we haven't had this problem.

That approach still buffers the whole request on the SPDY/Nginx proxy before its streamed through to your resty-http and upstream though.

Does anyone know if http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_request_buffering works with SPDY? As that would resolve that problem.

Alternatively I think you should be able to write a wrapper that uses ngx.req.get_body_file() and reads the buffered file (in chunks) and passes them through to lua-resty-http.

As long as your function returns data in chunks until its done, then returns nil it should just drop in.

Still have to let Nginx buffer the request to disk first though.

On Thursday, 14 May 2015 17:54:42 UTC+1, Kipras Mancevičius wrote:

Hey,

i'm having issues trying to proxy POST requests with ngx_lua.
I'm using https://github.com/pintsized/lua-resty-http to run the actual requests to upstream.

The problem is with request body reading.
The default (and afaik most efficient) way to do this is to use the lua-resty-http get_client_body_reader() function. But this doesn't work when under an HTTPS SPDY connection, because it uses to read the request body (which doesn't work under SPDY).

So in that case we can fall back to using , but now we face an issue when POST data size exceeds , and the response data is now buffered into temporary files and this method returns .

So then you can then call to get the temporary file name, which it does return. However, trying to read the file gives an error like this:

error while opening file: /usr/local/openresty/nginx/client_body_temp/0000000100: No such file or directory

Which makes no sense, considering that did return this file path, which means it is actually used. It probably gets deleted before you can try to access it.

One workaround is to set to "on" or "clean" in nginx config, in which case you can actually read that file and forward the request body stored in it. But i fear this may have performance implications, since afaik all POST requests bodies (even small ones) will now be buffered to disk.

Is anyone aware of these issues, and if so - do you know of any stable, recommended way of dealing with POST request bodies?

kiprasm

Alternatively I think you should be able to write a wrapper that uses ngx.req.get_body_file() and reads the buffered file (in chunks) and passes them through to lua-resty-http.

Yes, that's what i did. Works just fine in simple tests but i still have yet to stress test it.

On Thursday, 14 May 2015 17:54:42 UTC+1, Kipras Mancevičius wrote:
Hey,

i'm having issues trying to proxy POST requests with ngx_lua.
I'm using https://github.com/pintsized/lua-resty-http to run the actual requests to upstream.

The problem is with request body reading.
The default (and afaik most efficient) way to do this is to use the lua-resty-http get_client_body_reader() function. But this doesn't work when under an HTTPS SPDY connection, because it uses to read the request body (which doesn't work under SPDY).

So in that case we can fall back to using , but now we face an issue when POST data size exceeds , and the response data is now buffered into temporary files and this method returns .

So then you can then call to get the temporary file name, which it does return. However, trying to read the file gives an error like this:

error while opening file: /usr/local/openresty/nginx/client_body_temp/0000000100: No such file or directory

Which makes no sense, considering that did return this file path, which means it is actually used. It probably gets deleted before you can try to access it.

One workaround is to set to "on" or "clean" in nginx config, in which case you can actually read that file and forward the request body stored in it. But i fear this may have performance implications, since afaik all POST requests bodies (even small ones) will now be buffered to disk.

Is anyone aware of these issues, and if so - do you know of any stable, recommended way of dealing with POST request bodies?