many external requests cause all sockets to be used in TIME_WAIT and proper functioning to stop

james_357f · 2014-10-03T00:13:42+00:00

Hello! On Tue, Dec 30, 2014 at 5:47 AM, Tor Hveem wrote: > Maybe nginx lua could be extended to re-use file stat and open > functions from Nginx? Yea...

many external requests cause all sockets to be used in TIME_WAIT and proper functioning to stop

james_357f

here is a simplified example as we have many separate files normally:

worker_processes 8;
error_log logs/error.log;
worker_rlimit_nofile 262144;
events {
use epoll;
multi_accept on;
worker_connections 16384;
}
http {
tcp_nodelay on;
merge_slashes off;
resolver 10.0.0.2;
proxy_http_version 1.1;
proxy_set_header Connection "";
send_timeout 1s;
resolver_timeout 1s;
client_body_timeout 50ms;
client_header_timeout 50ms;
proxy_connect_timeout 50ms;
proxy_send_timeout 100ms;
proxy_read_timeout 200ms;
keepalive_requests 1024;
keepalive_timeout 1s;

upstream external_server {
server api.thing.com:80;
keepalive 256;
}

server {

listen 80;

location /thing/proxy {
internal;
proxy_connect_timeout 50ms;
proxy_send_timeout 50ms;
proxy_read_timeout 200ms;

proxy_pass_request_headers off;
proxy_set_header Host $http_host;
proxy_set_header Content-Type application/json;

proxy_pass http://external_server/method;
}

location /call/ {
error_log logs/error.log crit;
lua_need_request_body on;
keepalive_timeout 0;
content_by_lua '
local ok, resp = pcall(
ngx.location.capture,
"/thing/proxy",
{method = ngx.HTTP_POST,
body = {},
args = {}}
)
';
}
}
}

Now we call the /call/ location many times eg 600/second and all of our machines open sockets fill up and go into the TIME_WAIT state and then the rest of the machines responsibilities start failing.

We've tried turning on and off the lingering_close directive and tried setting tcp_fin_timeout and tcp_tw_recycle and tcp_tw_reuse to various settings. Nothing seems to help the machine be able to handle the load.

Any help would be appreciated

agentzh

Hello!

On Thu, Oct 2, 2014 at 9:13 AM, James Marlowe <ja...@lumate.com> wrote:
> Now we call the /call/ location many times eg 600/second and all of our
> machines open sockets fill up and go into the TIME_WAIT state and then the
> rest of the machines responsibilities start failing.
>

Seems like your proxy module's HTTP 1.1 keepalive and connection
pooling do not work. Your upstream connections get established and
closed too frequently, exhausting the
Ephemeral ports available in your system (at maximum just 60K+ ports
for a single remote peer).

Maybe your backend HTTP server does not support or enable HTTP 1.1
keepalive? I suggest you investigate why connection pooling does not
work in your ngx_proxy setup. Enabling the nginx debug logging offline
might give you some clues:

http://nginx.org/en/docs/debugging_log.html

Using common system tools like netstat can quckly check whether the
keepalive connections work or not.

> We've tried turning on and off the lingering_close directive and tried
> setting tcp_fin_timeout and tcp_tw_recycle and tcp_tw_reuse to various
> settings. Nothing seems to help the machine be able to handle the load.
>

These configurations are not really relevant here.

Regards,
-agentzh

vincent.mc.li

On Thursday, October 2, 2014 12:40:13 PM UTC-7, agentzh wrote:

Hello!

On Thu, Oct 2, 2014 at 9:13 AM, James Marlowe <ja...@lumate.com> wrote:
> Now we call the /call/ location many times eg 600/second and all of our
> machines open sockets fill up and go into the TIME_WAIT state and then the
> rest of the machines responsibilities start failing.
>

Seems like your proxy module's HTTP 1.1 keepalive and connection
pooling do not work. Your upstream connections get established and
closed too frequently, exhausting the
Ephemeral ports available in your system (at maximum just 60K+ ports
for a single remote peer).

sorry to hijack this thread, but I thought it is somehow relevant.

it is common problem when system under connection load and there is only one source ip and ~2^16 port limitation
for server side connection in F5 BIGIP too, so there is feature in BIGIP to have pool of source ips to use
for server side connection and each source ip get ~2^16 port, that is a way to mitigate the port exhaustion issue

reason I mention above here is I am wondering if nginx could have similar feature to use pool of source ips for upstream
connection, I looked there is a proxy_bind http://wiki.nginx.org/HttpProxyModule#proxy_bind, but that can only bind to one source ip
is it possible to extend it to bind to more than one source ip? or develop a upstream directive "source_ip" like upstream directive "server" to have pool of "source_ip" for
upstream connections?

I am newbie to nginx, but I could help developing this feature if it is not too complicate and with some expert guidance

Regards,

Vincent

agentzh

Hello!

On Sun, Oct 5, 2014 at 9:23 AM,  vincent.mc.li wrote:
> it is common problem when system under connection load and there is only one
> source ip and ~2^16 port limitation
> for server side connection in F5 BIGIP too, so there is feature in BIGIP to
> have pool of source ips to use
> for server side connection and each source ip get ~2^16 port, that is a way
> to mitigate the port exhaustion issue
>

Yes.

>  reason I mention above here is I am wondering if nginx could have similar
> feature to use pool of source ips for upstream
> connection, I looked there is a proxy_bind
> http://wiki.nginx.org/HttpProxyModule#proxy_bind, but that can only bind to
> one source ip
> is it possible to extend it to bind to more than one source ip? or develop a
> upstream directive "source_ip" like upstream directive "server" to have pool
> of "source_ip" for
> upstream connections?
>

Yes, we could implement a bind() method for cosockets but there's a
caveat at least for Linux systems (not sure about other OSes,
comments?): the bind() and connect() call sequence will not give us
more ephemeral ports when only binding to multiple local IP addresses
(that is, without specifying a local port number to bind to at the
same time). See the following post for details:

http://aleccolocco.blogspot.com/2008/11/ephemeral-ports-problem-and-solution.html

So for such (broken) systems which do not assign ephemeral ports
automatically when an "incomplete" bind() is used before connect(), we
need to do the port bookkeeping ourselves on the userland. Alas.
Though the Lua programmer can implement this on the Lua land, it can
be tricky for the user Lua code to actually know when an upstream
(cosocket) connection is closed and its port can get reused. So it's
better to be implemented in the cosocket facility itself.

> I am newbie to nginx, but I could help developing this feature if it is not
> too complicate and  with some expert guidance
>

The bind() method implementation itself should be easy. Just steal
proxy_bind's stuff. But to really solve the ephemeral port range
problem with bind() everywhere (including Linux), we definitely need
more work to do :) And yeah, patches welcome, as always :)

Best regards,
-agentzh

vincent.mc.li

On Monday, October 6, 2014 10:15:16 PM UTC-7, agentzh wrote:

Hello!

On Sun, Oct 5, 2014 at 9:23 AM, vincent.mc.li wrote:
> it is common problem when system under connection load and there is only one
> source ip and ~2^16 port limitation
> for server side connection in F5 BIGIP too, so there is feature in BIGIP to
> have pool of source ips to use
> for server side connection and each source ip get ~2^16 port, that is a way
> to mitigate the port exhaustion issue
>

Yes.

> reason I mention above here is I am wondering if nginx could have similar
> feature to use pool of source ips for upstream
> connection, I looked there is a proxy_bind
> http://wiki.nginx.org/HttpProxyModule#proxy_bind, but that can only bind to
> one source ip
> is it possible to extend it to bind to more than one source ip? or develop a
> upstream directive "source_ip" like upstream directive "server" to have pool
> of "source_ip" for
> upstream connections?
>

Yes, we could implement a bind() method for cosockets but there's a
caveat at least for Linux systems (not sure about other OSes,
comments?): the bind() and connect() call sequence will not give us
more ephemeral ports when only binding to multiple local IP addresses
(that is, without specifying a local port number to bind to at the
same time). See the following post for details:

http://aleccolocco.blogspot.com/2008/11/ephemeral-ports-problem-and-solution.html

I have found that link before while trouble shooting similar issue port exhaustion issue, but didn't
fully understand, you made it clear to me. thanks

So for such (broken) systems which do not assign ephemeral ports
automatically when an "incomplete" bind() is used before connect(), we
need to do the port bookkeeping ourselves on the userland. Alas.
Though the Lua programmer can implement this on the Lua land, it can
be tricky for the user Lua code to actually know when an upstream
(cosocket) connection is closed and its port can get reused. So it's
better to be implemented in the cosocket facility itself.

nice to hear cosocket could achieve the goal

> I am newbie to nginx, but I could help developing this feature if it is not
> too complicate and with some expert guidance
>

The bind() method implementation itself should be easy. Just steal
proxy_bind's stuff. But to really solve the ephemeral port range
problem with bind() everywhere (including Linux), we definitely need
more work to do :) And yeah, patches welcome, as always :)

I don't think I understand Nginx or Lua enough to contribute, I will keep learning :)

Best regards,
-agentzh