connection pool shared at server level instead of worker level ?

flavien.lebarbe · 2016-05-24T00:48:14+00:00

Hello! On Mon, May 23, 2016 at 7:51 AM, AL wrote: > Every few hours, I run into the pcre exception -27 > (PCRE_ERROR_JIT_STACKLIMIT) > > This...

connection pool shared at server level instead of worker level ?

flavien.lebarbe

Hi,

I start a thread with ngx.thread.spawn()
Inside that thread, I query a remote server using client:request {
        url = "">        method = method,
        keepalive = 1000,
    }

When the query terminates, the connection goes to the pool. Fine with me.
Then, if I issue another request using the same code snippet right after (ie in the same thread), the connection is reused (as long as the timeout of 1000ms is not expired). Great.

Now, when the thread terminates, it looks that the connection pool is emptied : the connection is closed (I can see a "closed keepalive connection" in the logs and tcpdump shows it's at the nginx initiative, not the remote). I read in the docs that : "Note that the cosocket connection pool is per Nginx worker process rather than per Nginx server instance". The observed behaviour is the one intended.

I'd like to share this connection pool across workers/threads. Is that possible ?

I can mitigate the cost of tcp connections adding a local proxy in between but I'd like to avoid that.

Thanks,
Flavien.

agentzh

Hello!

On Mon, May 23, 2016 at 9:48 AM, Flavien L wrote:
> I'd like to share this connection pool across workers/threads. Is that
> possible ?
>

Nope. That would be too expensive to pass connections across the
process boundary.

> I can mitigate the cost of tcp connections adding a local proxy in between
> but I'd like to avoid that.
>

We usually do not configure as many nginx worker processes as with
Apache httpd, so the per-worker pool model shouldn't be too much a
problem in practice.

Regards,
-agentzh

flavien.lebarbe_ba6b

On Fri, May 27, 2016 at 11:49 PM, Yichun Zhang (agentzh) <age...@gmail.com>
wrote:

> Hello!
>
> On Mon, May 23, 2016 at 9:48 AM, Flavien L wrote:
> > I'd like to share this connection pool across workers/threads. Is that
> > possible ?
> >
>
> Nope. That would be too expensive to pass connections across the
> process boundary.
>

Depends on what you compare it to ! ;o) (see below).

> > I can mitigate the cost of tcp connections adding a local proxy in
> between
> > but I'd like to avoid that.
>

My LUA script is some sort of smart proxy for HTTP requests : Depending on
configuration and request type/content, it rewrites some parts, adds
headers, removes some, sends requests to different machines, analyzes the
results, etc.

The problem I have is that establishing connections to remote hosts is
costly as they are in a different DC. Therefore, I really do need to use
keepalive between the host that runs my smart proxy and the remote hosts.
For each request that is received, a lua context is created and the
cosockets that I create in the script are closed when the script exits.

luis.gasca

Maybe I am not understanding your problem, but why do you close the cosockets ? You can return the connection to the pool with tcpsock:setkeepalive() call instead of the close() call.

Note also that you can have several pools in case you need, for example, per DC pools:

tcpsock:connect(host, port, options_table?)

An optional Lua table can be specified as the last argument to this method to specify various connect options:
pool specify a custom name for the connection pool being used. If omitted, then the connection pool name will be generated from the string template "<host>:<port>" or "<unix-socket-path>".

Regards,
Luis

El 02/06/2016 a las 11:52, Flavien Lebarbe escribió:

On Fri, May 27, 2016 at 11:49 PM, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

Hello!

On Mon, May 23, 2016 at 9:48 AM, Flavien L wrote:
> I'd like to share this connection pool across workers/threads. Is that
> possible ?
>

Nope. That would be too expensive to pass connections across the
process boundary.

Depends on what you compare it to ! ;o) (see below).

> I can mitigate the cost of tcp connections adding a local proxy in between
> but I'd like to avoid that.

My LUA script is some sort of smart proxy for HTTP requests : Depending on configuration and request type/content, it rewrites some parts, adds headers, removes some, sends requests to different machines, analyzes the results, etc.

The problem I have is that establishing connections to remote hosts is costly as they are in a different DC. Therefore, I really do need to use keepalive between the host that runs my smart proxy and the remote hosts.
For each request that is received, a lua context is created and the cosockets that I create in the script are closed when the script exits. From an isolation point of view, this is totally reasonable and I respect that. The thing is : the socket is closed, and I'll have to open a new one for the next request. It works from a functional point of view, but I'm not using keep alive. So from a performance point of view (latencywise), it's not satisfactory.

What I ended up doing is use a proxypass in front of the remote hosts : The cosockets talk to it (without keepalive, as above), and the proxypass talks to the remote hosts. The cosockets are closed when the request is finished. The proxypass part takes care of the sockets to the remote hosts, it does not close them : it uses keepalive. The cost of opening a new socket is then small as they are local.

I tried to use a location.capture() to talk to the proxypass. It works. However, I sometimes need to discard the content that comes back from the remote. In that situation, I cannot close the connection that I don't care about anymore. The worker is blocked until the remote answers.

Flavien.

.