Content Hangs Until Keepalive_Timeout Expires

bellasys · 2015-11-26T07:15:43+00:00

Hello! On Wed, Nov 25, 2015 at 4:01 AM, Brent Gerig <brent...@gmail.com> wrote: > Hi, how long is it usually before openresty adopts new nginx vers...

Content Hangs Until Keepalive_Timeout Expires

bellasys

ENV: Debian 8, OpenResty 1.9.3.2, PostgreSQL 9.4, LuaRocks, Pgmoon module.

basic structure:

http {

keepalive_timeout 5;

server{

listen 80;

server_name www.mydomain.net mydomain.net;

location /file {

rewrite_by_lua_block {

local pgmoon = require("pgmoon");

local pg = pgmoon.new({

host = "127.0.0.1",

port = "5432",

database = "mypgdb",

user = "mypgdb",

password = "password"

});

-- do stuff ...

local res = pg:query("select stuff");

local derive = ... -- processing res

-- do more stuff

local res2 = pg:query("select more stuff");

local derive2 = ... -- processing res2

pg:disconnect();

ngx.ctx.derive = derive;

ngx.ctx.derive2 = derive2;

}

header_filter_by_lua_block {

local derive = ngx.ctx.derive;

ngx.header["Set-Cookie"] = "derive=" .. derive .. "; Domain=" .. ngx.var.host .. ";Path=/; Expires=" .. ngx.cookie_time(ngx.time() + 31436000);

}

body_filter_by_lua_block {

local derive2 = ngx.ctx.derive2;

ngx.arg[1] = derive2;

ngx.arg[2] = "./";

}

} # end location

}#end server

}#end http

This used to filter body with my modifications immediately ( 250ms) and send output, including 1 initial call to psql using ngx_postgres module... that included processing the response using basic Lua code inside the blocks.

Now that I've developed a solution that uses multiple calls to db, plus many other mods, my results are constantly held up by the keepalive_timeout. Without explicit keepalive_timeout a remote domain waits (the default) 75 seconds + ~500ms more (code execution time) to receive the body data. Otherwise it matches 500ms + the keepalive time I specify. I want it to be 15, but have set it to 5 (and to 0, to 30 and 60) to confirm my findings. After 50 tests, it always works the same, related to the keepalive_timeout.

Because I've added so much code, and I'm kinda new to OpenResty and Lua (although more familiar with Nginx), I don't know which way to begin troubleshooting.

Since I just now found a post on Nginx forum talking about "Content-Encoding: gzip" header, ( https://forum.nginx.org/read.php?2,170139,209671#msg-209671 ) and the zlib.output_compression being set by PHP.. While I don't use PHP , I am utilizing my own zlib compression via FFI bindings with LuaJIT in my code... so I will begin troubleshooting there.

Interesting because I was using these bindings before adding any Postgres calls, and this did not have such effect. Still, I think that condition sounds like a good fit.

SOOO my questions, based on imagination + ignorance are:

1. Is there some latency issue with data binding when is involved? What about when is passed between Lua blocks?

2. Has someone else solved a similar problem re: content hangs until keepalive_timeout is reached, and can perhaps add general advice here? I know I am not providing a lot of code, but what if, for example, I could manually trigger the end of body conditions.

So I'm just looking for what might be obvious and fundamental- I would imagine many people who are new have encountered similar issues during development.

I will return to post as I'm going to attempt a solution now... in the meantime any help would be appreciated.

agentzh

Hello!

On Thu, Nov 26, 2015 at 7:15 AM, Max Bellasys wrote:
>              header_filter_by_lua_block {
>                local derive = ngx.ctx.derive;
>                ngx.header["Set-Cookie"] = "derive=" .. derive .. "; Domain="
> .. ngx.var.host .. ";Path=/; Expires=" .. ngx.cookie_time(ngx.time() +
> 31436000);
>              }
>              body_filter_by_lua_block {
>                 local derive2 = ngx.ctx.derive2;
>                 ngx.arg[1] = derive2;
>                 ngx.arg[2] = "./";
>             }

Since you're modifying the response body in the body filter, you'd
better clear the response header Content-Length in your header filter
to avoid confusing the client (like waiting for more data that does
not exist at all).

>
> Because I've added so much code, and I'm kinda new to OpenResty and Lua
> (although more familiar with Nginx), I don't know which way to begin
> troubleshooting.
>

Usually, looking at the nginx debugging logs can be a good start:

    http://nginx.org/en/docs/debugging_log.html

Also, looking at the wire with tools like wireshark or tcpdump can be helpful.

> Since I just now found a post on Nginx forum talking about
> "Content-Encoding: gzip" header, (
> https://forum.nginx.org/read.php?2,170139,209671#msg-209671 )  and the
> zlib.output_compression being set by PHP.. While I don't use PHP , I am
> utilizing my own zlib compression via FFI bindings with LuaJIT in my code...
> so I will begin troubleshooting there.
>

Be careful when writing your own FFI bindings since the FFI
programming is as unsafe as C programming. There's no hand-holding
there.

> Interesting because I was using these bindings before adding any Postgres
> calls, and this did not have such effect. Still, I think that condition
> sounds like a good fit.
>

You can try removing stuff while keeping the issue present. Debugging
a minimal setting is much easier.

> 1. Is there some latency issue with data binding when ngx.ctx.foo is
> involved? What about when ngx.ctx.foo is passed between Lua blocks?
>

Impossible. ngx.ctx accesses do not involve any I/O nor any syscalls
or context switches at all.

> 2. Has someone else solved a similar problem re: content hangs until
> keepalive_timeout is reached, and can perhaps add general advice here?

Looks like the client is waiting for something that does not exist due
to incorrect Content-Length header values or etc (see above).

> I
> know I am not providing a lot of code, but what if, for example,  I could
> manually trigger the end of body conditions.
>

If you want us to debug the issues for you, then you'd better try
providing a minimal and isolated example that we can reproduce the
problem on our side. (Often, people find mistakes in their own code
during this minimization process.)

Best regards,
-agentzh

bellasys

Yes this confirms the direction I'm headed re: Content-Length.

No I don't really want help debugging- I learn through the work.

I would have to be pretty desperate for that ;)

Thanks very much for the response. You answered my critical questions and gave me studied advice on how to proceed.

-- bellasys

On Wednesday, November 25, 2015 at 7:57:34 PM UTC-8, agentzh wrote:

Hello!

On Thu, Nov 26, 2015 at 7:15 AM, Max Bellasys wrote:
> header_filter_by_lua_block {
> local derive = ngx.ctx.derive;
> ngx.header["Set-Cookie"] = "derive=" .. derive .. "; Domain="
> .. ngx.var.host .. ";Path=/; Expires=" .. ngx.cookie_time(ngx.time() +
> 31436000);
> }
> body_filter_by_lua_block {
> local derive2 = ngx.ctx.derive2;
> ngx.arg[1] = derive2;
> ngx.arg[2] = "./";
> }

Since you're modifying the response body in the body filter, you'd
better clear the response header Content-Length in your header filter
to avoid confusing the client (like waiting for more data that does
not exist at all).

>
> Because I've added so much code, and I'm kinda new to OpenResty and Lua
> (although more familiar with Nginx), I don't know which way to begin
> troubleshooting.
>

Usually, looking at the nginx debugging logs can be a good start:

http://nginx.org/en/docs/debugging_log.html

Also, looking at the wire with tools like wireshark or tcpdump can be helpful.

> Since I just now found a post on Nginx forum talking about
> "Content-Encoding: gzip" header, (
> https://forum.nginx.org/read.php?2,170139,209671#msg-209671 ) and the
> zlib.output_compression being set by PHP.. While I don't use PHP , I am
> utilizing my own zlib compression via FFI bindings with LuaJIT in my code...
> so I will begin troubleshooting there.
>

Be careful when writing your own FFI bindings since the FFI
programming is as unsafe as C programming. There's no hand-holding
there.

> Interesting because I was using these bindings before adding any Postgres
> calls, and this did not have such effect. Still, I think that condition
> sounds like a good fit.
>

You can try removing stuff while keeping the issue present. Debugging
a minimal setting is much easier.

> 1. Is there some latency issue with data binding when ngx.ctx.foo is
> involved? What about when ngx.ctx.foo is passed between Lua blocks?
>

Impossible. ngx.ctx accesses do not involve any I/O nor any syscalls
or context switches at all.

> 2. Has someone else solved a similar problem re: content hangs until
> keepalive_timeout is reached, and can perhaps add general advice here?

Looks like the client is waiting for something that does not exist due
to incorrect Content-Length header values or etc (see above).

> I
> know I am not providing a lot of code, but what if, for example, I could
> manually trigger the end of body conditions.
>

If you want us to debug the issues for you, then you'd better try
providing a minimal and isolated example that we can reproduce the
problem on our side. (Often, people find mistakes in their own code
during this minimization process.)

Best regards,
-agentzh

bellasys

FYI- setting in header_filter_by_lua immediately resolved my problem.

I am typically thorough when I am learning, and would have tested many configurations of Content-Encoding, plus activated gzip encoding in Nginx (turned off at this time for dev) to learn about the interactions. However, just getting it to work right now was important, so thanks again.

-- bellasys