subreques 'capture' through reverse proxy truncated at 38395 bytes?

dholth · 2014-11-15T05:36:25+00:00

+1 for ffi based cosocket in lua-resty-core On Tuesday, November 18, 2014 3:44:57 AM UTC+8, agentzh wrote:Hello! On Sat, Nov 15, 2014 at 4:24 PM, Jerome Laf...

subreques 'capture' through reverse proxy truncated at 38395 bytes?

dholth

All,

Do I need to change a proxy buffer or some other setting to be able to successfully receive more data from .capture()?

I'm using nginx 1.6.2 on Ubuntu with this script:

https://github.com/dholth/pouchdb-bulk-get/blob/master/nginx/dbproxy.lua

and this nginx config:

https://github.com/dholth/pouchdb-bulk-get/blob/master/nginx/dbproxy.conf#L43

which does a bunch of subrequests to effect many GET requests in a single request. The lua script is supposed to stream out each subrequest body as it is received.

However I started noticing that I would get invalid json for larger requests. This is because many of the .capture() subrequests are truncated. NGINX complains about "connection reset by peer" but the reverse-proxied CouchDB peer does not complain.

The maximum line length is 38395 bytes...

Thanks,

Daniel

agentzh

Hello!

On Fri, Nov 14, 2014 at 1:36 PM, Daniel Holth wrote:
> and this nginx config:
>
> https://github.com/dholth/pouchdb-bulk-get/blob/master/nginx/dbproxy.conf#L43
>

One obvious thing is "proxy_buffering off;". Please do not turn off
proxy buffering when using ngx.location.capture().

This is a known issue.

> The lua script is supposed to stream out each subrequest body as it
> is received.
>

The ngx.location.capture* API does not support streaming by design. It
always buffers the whole response, no matter what.

It's recommended to use cosocket-based lua-resty-* libraries that
support streaming output.

Regards,
-agentzh