Number of active connections

simon.eskildsen · 2015-01-22T23:57:32+00:00

Hello! On Wed, Jan 21, 2015 at 4:28 PM, Lord Nynex wrote: > Any hope for https://github.com/openresty/lua-resty-redis/pull/49 ? > This pull request i...

Number of active connections

simon.eskildsen

Hello!

We're starting to make more and more use of ngx-lua, but one of my worries is exceeding `worker_connections` under load. I've been trying to monitor the number of active connections (client, upstream, ngx lua tcp connections) that, as far as I understand, all count towards this limit. I've been trying to grab `cycle->free_connection_n` with stap, but it just gives me a static number even under load. (See https://raw.githubusercontent.com/Sirupsen/system-tools/nginx-cycle/nginx/cycle.stp for the script). The `ngx-active-req` from the resty stap tools only seems to count client connections, and not all connections—I haven't been able to successfully tweak it to count all connections.

Would love some help!

brian

Are you using http://nginx.org/en/docs/http/ngx_http_stub_status_module.html ? You can look through it's code to see how it counts connections.

FWIW, I've ran instances that had 100k+ connections per nginx worker with no issues in production doing "real work" in Lua. Depends on your workload, etc.

On Thu, Jan 22, 2015 at 10:57 AM, Simon Eskildsen <simon.e...@shopify.com> wrote:

Hello!

We're starting to make more and more use of ngx-lua, but one of my worries is exceeding `worker_connections` under load. I've been trying to monitor the number of active connections (client, upstream, ngx lua tcp connections) that, as far as I understand, all count towards this limit. I've been trying to grab `cycle->free_connection_n` with stap, but it just gives me a static number even under load. (See https://raw.githubusercontent.com/Sirupsen/system-tools/nginx-cycle/nginx/cycle.stp for the script). The `ngx-active-req` from the resty stap tools only seems to count client connections, and not all connections—I haven't been able to successfully tweak it to count all connections.

Would love some help!

.

simon.eskildsen

I did a test with the status module where I opened 100 connections from Lua and slept for 10s. I saw the connections with `lsof`, but no number on the status page climbed above 100. It seems to not count the Lua connections?

On Thu, Jan 22, 2015 at 2:00 PM, Brian Akins <br...@akins.org> wrote:

Are you using http://nginx.org/en/docs/http/ngx_http_stub_status_module.html ? You can look through it's code to see how it counts connections.

FWIW, I've ran instances that had 100k+ connections per nginx worker with no issues in production doing "real work" in Lua. Depends on your workload, etc.

On Thu, Jan 22, 2015 at 10:57 AM, Simon Eskildsen <simon.e...@shopify.com> wrote:
Hello!

We're starting to make more and more use of ngx-lua, but one of my worries is exceeding `worker_connections` under load. I've been trying to monitor the number of active connections (client, upstream, ngx lua tcp connections) that, as far as I understand, all count towards this limit. I've been trying to grab `cycle->free_connection_n` with stap, but it just gives me a static number even under load. (See https://raw.githubusercontent.com/Sirupsen/system-tools/nginx-cycle/nginx/cycle.stp for the script). The `ngx-active-req` from the resty stap tools only seems to count client connections, and not all connections—I haven't been able to successfully tweak it to count all connections.

Would love some help!

.

simon.eskildsen

Hm, with the test described in the previous email, monitoring `free_connection_n` (which I was monitoring in the first place) is actually what I want; it's climbing above 100. It looks like the test I used the first time wasn't right. So I've got the numbers I needed now!

The `stub_status_module` page only shows 1 additional connections for such a request. I checked the source, and it indeed only shows a count of HTTP connections. I'm assuming other people are monitoring `free_connection_n` to make sure `ngx.timer` and TCP connections from Nginx lua are counted as well?

Cheers

On Thu, Jan 22, 2015 at 2:06 PM, Simon Hørup Eskildsen <simon.e...@shopify.com> wrote:

I did a test with the status module where I opened 100 connections from Lua and slept for 10s. I saw the connections with `lsof`, but no number on the status page climbed above 100. It seems to not count the Lua connections?

On Thu, Jan 22, 2015 at 2:00 PM, Brian Akins <br...@akins.org> wrote:
Are you using http://nginx.org/en/docs/http/ngx_http_stub_status_module.html ? You can look through it's code to see how it counts connections.

FWIW, I've ran instances that had 100k+ connections per nginx worker with no issues in production doing "real work" in Lua. Depends on your workload, etc.

On Thu, Jan 22, 2015 at 10:57 AM, Simon Eskildsen <simon.e...@shopify.com> wrote:
Hello!

We're starting to make more and more use of ngx-lua, but one of my worries is exceeding `worker_connections` under load. I've been trying to monitor the number of active connections (client, upstream, ngx lua tcp connections) that, as far as I understand, all count towards this limit. I've been trying to grab `cycle->free_connection_n` with stap, but it just gives me a static number even under load. (See https://raw.githubusercontent.com/Sirupsen/system-tools/nginx-cycle/nginx/cycle.stp for the script). The `ngx-active-req` from the resty stap tools only seems to count client connections, and not all connections—I haven't been able to successfully tweak it to count all connections.

Would love some help!

.

agentzh

Hello!

On Thu, Jan 22, 2015 at 7:57 AM, Simon Eskildsen wrote:
> I've been trying
> to grab `cycle->free_connection_n` with stap, but it just gives me a static
> number even under load. (See
> https://raw.githubusercontent.com/Sirupsen/system-tools/nginx-cycle/nginx/cycle.stp
> for the script).

The direction is correct but your systemtap script has bugs itself.
For example, ngx_cycle itself is a pointer, addressing it with "&"
again turn it into something else. Also, ngx_cycle->files is an
instance of ngx_list_t, whose ->size field is not what you want (it's
just the size of each element in the queue, not the size of the
queue).

I've just tweaked your script and come up with this:

    https://github.com/openresty/stapxx/blob/master/samples/ngx-count-conns.sxx

I've added it as a sample tool for my stap++ wrapper for systemtap.
Hopefully you don't mind :)

Also added some docs for it:

    https://github.com/openresty/stapxx#ngx-count-conns

One sample output in a production box:

    $ ngx-count-conns.sxx -x 32743
    Start tracing 32743 (/opt/nginx/sbin/nginx)...

    ====== CONNECTIONS ======
    Max connections: 32768
    Free connections: 27674
    Used connections: 5094

    ====== FILES ======
    Max files: 131072
    Open files: 2
    Free files: 131070

Regarding your questions, yes, ngx_lua's cosocket connections and
timers are also counted in the "Used connections" numbers shown above.

Regards,
-agentzh

simon.eskildsen

You rock, thanks Yichun!

I've written another script which may be useful to other people in the same situation as me (want to monitor connection/timer ceilings, critical to not loose data) to measure # of timers in ngx-lua: https://github.com/openresty/stapxx/pull/10

On Thu, Jan 22, 2015 at 4:50 PM, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

Hello!

On Thu, Jan 22, 2015 at 7:57 AM, Simon Eskildsen wrote:
> I've been trying
> to grab `cycle->free_connection_n` with stap, but it just gives me a static
> number even under load. (See
> https://raw.githubusercontent.com/Sirupsen/system-tools/nginx-cycle/nginx/cycle.stp
> for the script).

The direction is correct but your systemtap script has bugs itself.
For example, ngx_cycle itself is a pointer, addressing it with "&"
again turn it into something else. Also, ngx_cycle->files is an
instance of ngx_list_t, whose ->size field is not what you want (it's
just the size of each element in the queue, not the size of the
queue).

I've just tweaked your script and come up with this:

https://github.com/openresty/stapxx/blob/master/samples/ngx-count-conns.sxx

I've added it as a sample tool for my stap++ wrapper for systemtap.
Hopefully you don't mind :)

Also added some docs for it:

https://github.com/openresty/stapxx#ngx-count-conns

One sample output in a production box:

$ ngx-count-conns.sxx -x 32743
Start tracing 32743 (/opt/nginx/sbin/nginx)...

====== CONNECTIONS ======
Max connections: 32768
Free connections: 27674
Used connections: 5094

====== FILES ======
Max files: 131072
Open files: 2
Free files: 131070

Regarding your questions, yes, ngx_lua's cosocket connections and
timers are also counted in the "Used connections" numbers shown above.

Regards,
-agentzh

.

simon.eskildsen

Another question for this thread. I just ran ngx-count-conns (https://github.com/openresty/stapxx/blob/master/samples/ngx-count-conns.sxx) in production and get:

====== CONNECTIONS ======

Max connections: 20480

Free connections: 20419

Used connections: 61

====== FILES ======

Max files: 200000

Open normal files: 30

Then I `curl` to the `ngx_http_stub_status_module` (http://nginx.org/en/docs/http/ngx_http_stub_status_module.html) and get something like:

Active connections: 8036

server accepts handled requests

351374622 351374622 553813639

Reading: 1 Writing: 89 Waiting: 7930

Why are they so different? I confirmed that Lua TCP sockets indeed make it to "Used connections" for the stap script, but I have a hard time figuring out what each number here means now.

On Thu, Jan 22, 2015 at 5:24 PM, Simon Hørup Eskildsen <simon.e...@shopify.com> wrote:

You rock, thanks Yichun!

I've written another script which may be useful to other people in the same situation as me (want to monitor connection/timer ceilings, critical to not loose data) to measure # of timers in ngx-lua: https://github.com/openresty/stapxx/pull/10

On Thu, Jan 22, 2015 at 4:50 PM, Yichun Zhang (agentzh) <age...@gmail.com> wrote:
Hello!

On Thu, Jan 22, 2015 at 7:57 AM, Simon Eskildsen wrote:
> I've been trying
> to grab `cycle->free_connection_n` with stap, but it just gives me a static
> number even under load. (See
> https://raw.githubusercontent.com/Sirupsen/system-tools/nginx-cycle/nginx/cycle.stp
> for the script).

The direction is correct but your systemtap script has bugs itself.
For example, ngx_cycle itself is a pointer, addressing it with "&"
again turn it into something else. Also, ngx_cycle->files is an
instance of ngx_list_t, whose ->size field is not what you want (it's
just the size of each element in the queue, not the size of the
queue).

I've just tweaked your script and come up with this:

https://github.com/openresty/stapxx/blob/master/samples/ngx-count-conns.sxx

I've added it as a sample tool for my stap++ wrapper for systemtap.
Hopefully you don't mind :)

Also added some docs for it:

https://github.com/openresty/stapxx#ngx-count-conns

One sample output in a production box:

$ ngx-count-conns.sxx -x 32743
Start tracing 32743 (/opt/nginx/sbin/nginx)...

====== CONNECTIONS ======
Max connections: 32768
Free connections: 27674
Used connections: 5094

====== FILES ======
Max files: 131072
Open files: 2
Free files: 131070

Regarding your questions, yes, ngx_lua's cosocket connections and
timers are also counted in the "Used connections" numbers shown above.

Regards,
-agentzh

.

agentzh

Hello!

On Thu, Jan 22, 2015 at 3:16 PM, Simon Hørup Eskildsen wrote:
> Then I `curl` to the `ngx_http_stub_status_module`
> (http://nginx.org/en/docs/http/ngx_http_stub_status_module.html) and get
> something like:
>
> Active connections: 8036
> server accepts handled requests
>  351374622 351374622 553813639
> Reading: 1 Writing: 89 Waiting: 7930
>
> Why are they so different?

Maybe some of your worker processes crashed abnormally since the
startup? Better enable core dump for your nginx worker processes.

Also please keep in mind that the stub_status metrics are per-server
(including all the workers) while the stap script is per-worker.

Regards,
-agentzh

alhakeem

Hello,
Can anyone share the connections per second and the maximum concurrent connections 
ever tested on Openresty, with details of the reference hardware ?
Cheers,
Abdul Hakeem

-----Original Message-----
From: openresty-en@googlegroups.com [mailto:openresty-en@googlegroups.com] On 
Behalf Of Yichun Zhang (agentzh)
Sent: Thursday, January 22, 2015 11:25 PM
To: openresty-en
Subject: Re: [openresty-en] Number of active connections

Hello!

On Thu, Jan 22, 2015 at 3:16 PM, Simon Hï¿½rup Eskildsen wrote:
> Then I `curl` to the `ngx_http_stub_status_module`
> (http://nginx.org/en/docs/http/ngx_http_stub_status_module.html) and get
> something like:
>
> Active connections: 8036
> server accepts handled requests
>  351374622 351374622 553813639
> Reading: 1 Writing: 89 Waiting: 7930
>
> Why are they so different?

Maybe some of your worker processes crashed abnormally since the
startup? Better enable core dump for your nginx worker processes.

Also please keep in mind that the stub_status metrics are per-server
(including all the workers) while the stap script is per-worker.

Regards,
-agentzh.