Http 2.0 upload / ngx.req.socket

rwesolowski · 2016-02-17T06:13:24+00:00

在 2016年2月13日星期六 UTC+8下午6:45:05，harveyfa...@gmail.com写道：I am trying...

Http 2.0 upload / ngx.req.socket

rwesolowski

Hi,

Is it possible to handle file uploading with HTTP2 module enabled and openresty? We get "http v2 not supported yet" error while we try to read from ngx.req.socket.

Is it alternative way or patching openresty core is needed ?:)

ggroups

Simply create a dedicated server section for the upload which speaks http 1.x and is listening on localhost or unix socket and proxy to it. For example i have this configuration running successfully for h2 uploads:

server {
    listen *.80;
    server_name domain.com;
    root /some/path/;
    
    ...
    
    location = /api/uploading {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_http_version 1.1;
        proxy_request_buffering off;
        proxy_pass http://unix:/run/nginx/uploading.sock;
    }
    
    ...
    
}

server {
    listen unix:/run/nginx/uploading.sock;
    server_name domain.com;
    root /some/path/;

    location = /api/uploading {
        content_by_lua_file /lua/file/using/resty/upload.lua;
    }
}

One really important setting is to turn off request buffering using:

proxy_request_buffering off;

Otherwise the full upload will be buffered in memory or disk, which is bad. (Then you could just get rid of resty.upload anyways)

On Tuesday, February 16, 2016 at 11:13:24 PM UTC+1, r--w wrote:

Hi,
Is it possible to handle file uploading with HTTP2 module enabled and openresty? We get "http v2 not supported yet" error while we try to read from ngx.req.socket.
Is it alternative way or patching openresty core is needed ?:)

ggroups

Actually with h2, the request body is always fully buffered, regardless of the proxy_request_buffering setting, so please ignore my post.

On Wednesday, February 17, 2016 at 12:19:48 AM UTC+1, bjoe2k4 wrote:

Simply create a dedicated server section for the upload which speaks http 1.x and is listening on localhost or unix socket and proxy to it. For example i have this configuration running successfully for h2 uploads:

server { listen *.80; server_name domain.com; root /some/path/; ... location = /api/uploading { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_http_version 1.1; proxy_request_buffering off; proxy_pass http://unix:/run/nginx/uploading.sock; } ... } server { listen unix:/run/nginx/uploading.sock; server_name domain.com; root /some/path/; location = /api/uploading { content_by_lua_file /lua/file/using/resty/upload.lua; } }

One really important setting is to turn off request buffering using: proxy_request_buffering off;Otherwise the full upload will be buffered in memory or disk, which is bad. (Then you could just get rid of resty.upload anyways)

On Tuesday, February 16, 2016 at 11:13:24 PM UTC+1, r--w wrote:
Hi,
Is it possible to handle file uploading with HTTP2 module enabled and openresty? We get "http v2 not supported yet" error while we try to read from ngx.req.socket.
Is it alternative way or patching openresty core is needed ?:)

rwesolowski

HI,

Thank you for your answer, unfortunately it's not possible for us to have a special location for uploads/POSTs

We have a proxy service and we need to inspect POST queries as they come.

W dniu środa, 17 lutego 2016 08:23:54 UTC+1 użytkownik bjoe2k4 napisał:

Actually with h2, the request body is always fully buffered, regardless of the proxy_request_buffering setting, so please ignore my post.

On Wednesday, February 17, 2016 at 12:19:48 AM UTC+1, bjoe2k4 wrote:
Simply create a dedicated server section for the upload which speaks http 1.x and is listening on localhost or unix socket and proxy to it. For example i have this configuration running successfully for h2 uploads:

server { listen *.80; server_name domain.com; root /some/path/; ... location = /api/uploading { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_http_version 1.1; proxy_request_buffering off; proxy_pass http://unix:/run/nginx/uploading.sock; } ... } server { listen unix:/run/nginx/uploading.sock; server_name domain.com; root /some/path/; location = /api/uploading { content_by_lua_file /lua/file/using/resty/upload.lua; } }

One really important setting is to turn off request buffering using: proxy_request_buffering off;Otherwise the full upload will be buffered in memory or disk, which is bad. (Then you could just get rid of resty.upload anyways)

On Tuesday, February 16, 2016 at 11:13:24 PM UTC+1, r--w wrote:
Hi,
Is it possible to handle file uploading with HTTP2 module enabled and openresty? We get "http v2 not supported yet" error while we try to read from ngx.req.socket.
Is it alternative way or patching openresty core is needed ?:)

agentzh

Hello!

On Wed, Feb 17, 2016 at 2:13 PM, r--w wrote:
> Thank you for your answer, unfortunately it's not possible for us to have a
> special location for uploads/POSTs
> We have a proxy service and we need to inspect POST queries as they come.
>

If you really need streaming reading request bodies, then you
shouldn't use the HTTP 2.0 protocol in the first place. HTTP 2.0
inherits the in-connection multiplexing feature of SPDY, which makes
such "pulling paradigm" very hard to implement, if not impossible.

Regards,
-agentzh

rwesolowski

Hi,

Thank you, so currenty the only way to inspect POSTs is to put another nginx proxy with HTTP 2.0 support in front of WAF and use HTTP 1.1 between them?

W dniu czwartek, 18 lutego 2016 01:44:57 UTC+1 użytkownik agentzh napisał:

Hello!

On Wed, Feb 17, 2016 at 2:13 PM, r--w wrote:
> Thank you for your answer, unfortunately it's not possible for us to have a
> special location for uploads/POSTs
> We have a proxy service and we need to inspect POST queries as they come.
>

If you really need streaming reading request bodies, then you
shouldn't use the HTTP 2.0 protocol in the first place. HTTP 2.0
inherits the in-connection multiplexing feature of SPDY, which makes
such "pulling paradigm" very hard to implement, if not impossible.

Regards,
-agentzh

agentzh

Hello!

On Fri, Feb 19, 2016 at 5:09 AM, r--w wrote:
> Thank you, so currenty the only way to inspect POSTs is to put another nginx
> proxy with HTTP 2.0 support in front of WAF and use HTTP 1.1 between them?
>

I doubt its value since your frontend nginx doing HTTP 2.0 still has
to buffer all the request body locally anyway. Why not just inspect
the body via ngx.req.get_body_data() and ngx.req.get_body_file() if
you will always have to buffer the body with HTTP 2.0?

Regards,
-agentzh

ggroups

A patch that enables unbuffered uploads for h2 has just been submitted to the nginx-devel mailinglist. So there is hope :-)

On Friday, February 19, 2016 at 8:22:16 PM UTC+1, agentzh wrote:

Hello!

On Fri, Feb 19, 2016 at 5:09 AM, r--w wrote:
> Thank you, so currenty the only way to inspect POSTs is to put another nginx
> proxy with HTTP 2.0 support in front of WAF and use HTTP 1.1 between them?
>

I doubt its value since your frontend nginx doing HTTP 2.0 still has
to buffer all the request body locally anyway. Why not just inspect
the body via ngx.req.get_body_data() and ngx.req.get_body_file() if
you will always have to buffer the body with HTTP 2.0?

Regards,
-agentzh