chunked request bodies not supported yet - lua-resty-upload

hook1988 · 2015-04-04T03:59:41+00:00

Hello! On Thu, Jul 30, 2015 at 7:23 AM, Gavin Rolleston wrote: > We sometimes use empty rewrite_by_lua directives in the location {} level to > preven...

chunked request bodies not supported yet - lua-resty-upload

hook1988

Hi,

I am experiencing issues when trying to upload a file with chunked request bodies.

I already tried to debug this issue, and it doesn't seem to be the client which causes issues. So it has to do something with the nginx core itself or lua.

Here's what I'm sending to the server:

POST /ul/8R1Jg_IST4w HTTP/1.1
Host: EXAMPLE.COM
User-Agent: upload test
Accept: * /*; q=0.01
Content-Type: multipart/form-data; boundary=---------------------------16470503901521535624504986122
Connection: close
Transfer-Encoding: chunked
Pragma: no-cache
Cache-Control: no-cache

b2
-----------------------------16470503901521535624504986122
Content-Disposition: form-data; name="file1"; filename="test.dat"
Content-Type: application/x-ns-proxy-autoconfig


1f4
Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et e
1f4
a rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feu
bb
giat nulla facilisis at vero eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue duis dolore te feugait nulla facilisi. Lorem ipsum dolor sit amet,

40

-----------------------------16470503901521535624504986122--

0

And this is the response:

HTTP/1.1 500 Internal Server Error
Server: openresty
Date: Fri, 03 Apr 2015 19:48:31 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: close

4f
{"status":500,"msg":"upload failed: chunked request bodies not supported yet"}

0

The relevant lua code for resty upload (nothing special):

local upload = require "resty.upload"
local chunk_size = 16384 -- should be set to >4096 for real-world settings
require "functions"

-- some more code which has nothign to to with uploading

local form, err = upload:new(chunk_size)
if not form then
    displayError("upload failed: " .. err, 500) -- displayError is defined in functions.lua and adds error to nginx log and outputs in json format
end

Is there anything I am doing wrong here, or are chunked POST's just not supported by nginx/openresty?

Thanks for your help!

hook1988

Sorry, forgot the most important thing:
I'm using nginx version: openresty/1.7.10.1

Am Freitag, 3. April 2015 21:59:40 UTC+2 schrieb Michael Hock:

Hi,

I am experiencing issues when trying to upload a file with chunked request bodies.

I already tried to debug this issue, and it doesn't seem to be the client which causes issues. So it has to do something with the nginx core itself or lua.

Here's what I'm sending to the server:

POST /ul/8R1Jg_IST4w HTTP/1.1 Host: EXAMPLE.COM User-Agent: upload test Accept: * /*; q=0.01 Content-Type: multipart/form-data; boundary=---------------------------16470503901521535624504986122 Connection: close Transfer-Encoding: chunked Pragma: no-cache Cache-Control: no-cache b2 -----------------------------16470503901521535624504986122 Content-Disposition: form-data; name="file1"; filename="test.dat" Content-Type: application/x-ns-proxy-autoconfig 1f4 Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et e 1f4 a rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feu bb giat nulla facilisis at vero eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue duis dolore te feugait nulla facilisi. Lorem ipsum dolor sit amet, 40 -----------------------------16470503901521535624504986122-- 0

And this is the response:

HTTP/1.1 500 Internal Server Error Server: openresty Date: Fri, 03 Apr 2015 19:48:31 GMT Content-Type: application/json Transfer-Encoding: chunked Connection: close 4f {"status":500,"msg":"upload failed: chunked request bodies not supported yet"} 0

The relevant lua code for resty upload (nothing special):

local upload = require "resty.upload" local chunk_size = 16384 -- should be set to >4096 for real-world settings require "functions" -- some more code which has nothign to to with uploading local form, err = upload:new(chunk_size) if not form then displayError("upload failed: " .. err, 500) -- displayError is defined in functions.lua and adds error to nginx log and outputs in json format end

Is there anything I am doing wrong here, or are chunked POST's just not supported by nginx/openresty?

Thanks for your help!

agentzh

Hello!

On Fri, Apr 3, 2015 at 12:59 PM, Michael Hock wrote:
> I am experiencing issues when trying to upload a file with chunked request
> bodies.
>

Yeah, the ngx.req.socket() API function [1] used by lua-resty-upload
does not support chunked encoding yet. Adding the support has been on
my TODO list. Patches welcome as always :)

Best regards,
-agentzh

[1] https://github.com/openresty/lua-nginx-module#ngxreqsocket

hook1988

Hello agentzh,

Thank you for your very fast response! :)

Unfortunately, I'm not into the C programming that much, so I doubt I would be of any help here ... :(

Is there any other possibility you can think of to send data to lua-resty-upload without knowing the acutal size in advance?

Thanks, best regards and happy easter holidays!

Michael

Am Samstag, 4. April 2015 03:22:16 UTC+2 schrieb agentzh:

Hello!

On Fri, Apr 3, 2015 at 12:59 PM, Michael Hock wrote:
> I am experiencing issues when trying to upload a file with chunked request
> bodies.
>

Yeah, the ngx.req.socket() API function [1] used by lua-resty-upload
does not support chunked encoding yet. Adding the support has been on
my TODO list. Patches welcome as always :)

Best regards,
-agentzh

[1] https://github.com/openresty/lua-nginx-module#ngxreqsocket

agentzh

Hello!

On Sat, Apr 4, 2015 at 4:27 AM, Michael wrote:
> Is there any other possibility you can think of to send data to
> lua-resty-upload without knowing the acutal size in advance?
>

There does exist a work-around for reading the chunked request body
data in a streaming fashion, via using the "raw" request socket (i.e.,
ngx.req.socket(true)), as demonstrated by the following (declarative)
test case in ngx_lua' existing test suite:

https://github.com/openresty/lua-nginx-module/blob/75cc29ea64a87bc5cd447525893fda76b8d664b4/t/116-raw-req-socket.t#L784

But it would require some more work to make it work with
lua-resty-upload. Good luck with this (dirty) route :)

The most beautiful way to achieve this, however, as I've said in my
previous email, is to make the non-raw request socket API support
chunked encoding transparently, such that existing Lua libraries like
lua-resty-upload will just work with chunked request bodies out of the
box.

Regards,
-agentzh

hook1988

Hello agentz!

I had a look on the work-around with lua, but it seems too much work needs to be done to get it working with the lua-resty-upload module. Also, I like the solution with a lua-nginx-module modification better ...
Therefore I tried to implement it directly in nginx.
I looked at the function static int ngx_http_lua_req_socket(lua_State *L) in ngx_http_lua_socket_tcp.c, because the error with chunked request bodies gets thrown there.
I know I can check if the request is a chunked request by checking r->headers_in.chunked.
I tried to use the function ngx_http_parse_chunked(ngx_http_request_t *r, ngx_buf_t *b, ngx_http_chunked_t *ctx); when it's a chunked request in the bottom of the /* request body reader */ part, but I never managed to get any data to lua.
The function seemed to be the key for me, because it's also used in some changes where nginx added support for chunked request bodies:
https://github.com/nginx/nginx/commit/a79aeb7ca33d55f3bed89062b1bdfe24ee9a46cb

But to be honest, I have not much of an idea on which place modifications are to be done, so that lua gets the un-chunked-encoded data ...
Would it be possible to tell roughly in which parts changes need to be done?

Thank you!

Best Regards,
Michael

Am Sonntag, 5. April 2015 19:56:15 UTC+2 schrieb agentzh:

Hello!

On Sat, Apr 4, 2015 at 4:27 AM, Michael wrote:
> Is there any other possibility you can think of to send data to
> lua-resty-upload without knowing the acutal size in advance?
>

There does exist a work-around for reading the chunked request body
data in a streaming fashion, via using the "raw" request socket (i.e.,
ngx.req.socket(true)), as demonstrated by the following (declarative)
test case in ngx_lua' existing test suite:

https://github.com/openresty/lua-nginx-module/blob/75cc29ea64a87bc5cd447525893fda76b8d664b4/t/116-raw-req-socket.t#L784

But it would require some more work to make it work with
lua-resty-upload. Good luck with this (dirty) route :)

The most beautiful way to achieve this, however, as I've said in my
previous email, is to make the non-raw request socket API support
chunked encoding transparently, such that existing Lua libraries like
lua-resty-upload will just work with chunked request bodies out of the
box.

Regards,
-agentzh