hard time understanding and using lint_conn.

matthieu.labour

I would greatly appreciate some help on using limit_conn. Thank you in advance for your time.

Specifically, I do not understand why Experience#01 and Experience#02 as shown below do NOT produce the same number of connections on SERVER.

Please note the values for keepalive, rate and limit_conn.

We would like that Experience#01 and Experience#02 achieve the same number of connections on SERVER.

Experience#01

PROXY01 => SERVER

Experience#02

PROXY01 AND PROXY02 => SERVER

Experience#1

==========

PROXY01 nginx.conf

[...]
server {
listen 8002;
proxy_intercept_errors on;

location / {
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_pass http://locallayer;
}
error_page 302 400 403 404 408 500 502 503 504 = /empty;
location /empty {
return 204;
}
}

upstream locallayer {
server 127.0.0.1:7999;
keepalive 32;
}

upstream dynamic {
least_conn;
server 192.168.10.131:9990;
keepalive 32;
}

limit_conn_zone $binary_remote_addr zone=peripconn:100m;
limit_req_zone $binary_remote_addr zone=peripreq:1000m rate=6000r/s;

server {
listen 7999;
server_name local.com;
proxy_intercept_errors on;
location / {
limit_conn peripconn 32;
limit_req zone=peripreq burst=100 nodelay;
limit_conn_status 503;
limit_req_status 503;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_connect_timeout 10ms;
proxy_send_timeout 10ms;
proxy_read_timeout 60ms;
proxy_pass http://dynamic;
}
error_page 302 400 403 404 408 500 502 503 504 = /empty;
location /empty {
return 204;
}
}
[...]

I am seeing the following traffic on SERVER

ubuntu@ip-192-168-10-131:~$ while true;do ~/count_req.sh;done
QPS: 6024
Active connections: 39
QPS: 6032
Active connections: 55
QPS: 6036
Active connections: 53
QPS: 6036

Experience#2

==========

PROXY01 and PROXY02 have identical nginx.conf configurations. Please note that we simply divided the limit, number of connection and keepalive by 2.

[...]
server {
listen 8002;
proxy_intercept_errors on;

location / {
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_pass http://locallayer;
}
error_page 302 400 403 404 408 500 502 503 504 = /empty;
location /empty {
return 204;
}
}

upstream locallayer {
server 127.0.0.1:7999;
keepalive 16;
}

upstream dynamic {
least_conn;
server 192.168.10.131:9990;
keepalive 16;
}

limit_conn_zone $binary_remote_addr zone=peripconn:100m;
limit_req_zone $binary_remote_addr zone=peripreq:1000m rate=3000r/s;

server {
listen 7999;
server_name local.com;
proxy_intercept_errors on;
location / {
limit_conn peripconn 16;
limit_req zone=peripreq burst=100 nodelay;
limit_conn_status 503;
limit_req_status 503;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_connect_timeout 10ms;
proxy_send_timeout 10ms;
proxy_read_timeout 60ms;
proxy_pass http://dynamic;
}
error_page 302 400 403 404 408 500 502 503 504 = /empty;
location /empty {
return 204;
}
}
[...]

I am seeing the following traffic on SERVER

ubuntu@ip-192-168-10-131:~$ while true;do ~/count_req.sh;done
QPS: 6056
Active connections: 111
QPS: 6074
Active connections: 105
QPS: 6062
Active connections: 112 <=================== I would expect the number of connections to be the same number of connections seem for Experiment#1

agentzh

Hello!

On Fri, Jun 17, 2016 at 1:32 PM, Matthieu Labour wrote:
> I am seeing the following traffic on SERVER
>
> ubuntu@ip-192-168-10-131:~$ while true;do ~/count_req.sh;done
> QPS:                    6056
> Active connections:     111
> QPS:                    6074
> Active connections:     105
> QPS:                    6062
> Active connections:     112   <=================== I would expect the number
> of connections to be the same number of connections seem for Experiment#1
>

Are you using the real traffic or just initiating fake traffic from a
fixed IP address? If it's the former, then it could be much more
complicated to reason about the effect of division (because the
clients may not fall ideally evenly upon the two servers and can lead
to varying results). Better try the latter instead.

Regards,
-agentzh