UDP I/O

zxcvbn4038 · 2016-01-05T05:55:32+00:00

Hello! On Mon, Jan 4, 2016 at 9:45 AM, Yichun Zhang (agentzh) wrote: > cd ~/git/opsboy/misc/logs > ./dispatcher -r -t 270 -a 'linux x86_64' -f...

UDP I/O

zxcvbn4038

I've seen several examples where content_by_lua is used to send/receive a request via UDP with socket.udp()/setpeername()/send()/recv(). I believe that this would create one UDP socket per request and destroy it when the request terminates.

Is there any way to instead create one shared UDP socket per nginx worker, and use some sort of unique identifier in the UDP payload to route data received back to the original request? Basically to avoid the overhead of creating/destroying a socket with every request.

Or is it actually more efficient to just create a socket every time then to do everything necessary to multiplex?

lordnynex

Hello,

UDP is stateless so there is no concept of pooling or persistence. There
isn't really any socket overhead because the system will never receive an
ACK packet in return. There is no performance gain to be had from what
you're describing because there is no socket to pool or 'hold open'. At the
OS level this is not possible. Unfortunately, most socket libraries create
functions that give this appearance for consistency with the TCP
implementation and it can be misleading.

Here is an example illustrating this on linux systems

echo "some data" > /dev/udp/192.168.1.10/2008

Where the IP is the target IP and the last number is the UDP port number.

agentzh

Hello!

On Mon, Jan 4, 2016 at 2:31 PM, Lord Nynex wrote:
> UDP is stateless so there is no concept of pooling or persistence.

True, UDP is connection-less. But still we can save the overhead of
the socket(), close(), and bind() syscalls by recycling the socket
objects with pools. I won't call it a "connection pool", but just a
"socket pool" :)

I've been thinking about adding the setkeepalive() method for the UDP
cosocket objects too but I haven't got the time to do that myself.
Volunteers welcome! Patches welcome! :)

Best regards,
-agentzh

zxcvbn4038

Ok, so the way I would envision this working then is I'd create an array to hold the socket_udp objects sometime in the initialization stage, then in the request stages I'd try to pop a socket_udp from the array, create a new one if the array is empty, include some magic number in my UDP payload so I can be certain that anything I receive is for the current request and not a previous request, then at the end of the request I push the socket_udp object back into the table for the next request. Thats a nice middle-of-the-road setup I hadn't considered before.

However, with that setup I'd still be concerned that because nginx is so crazy good at what it does, it would be practical to open more then > 64K sockets unless I had each worker bind outgoing UDP to a seperate IP address, so I'd still like to see if there is some way I could multiplex requests/responses to a backend though a single UDP socket per worker. Within the lua framework, is there some way I could send out the nginx request context (an integer) then have the backend echo that back and somehow use that to direct the response received to the right request context?

On Monday, January 4, 2016 at 4:55:32 PM UTC-5, CJ Ess wrote:

I've seen several examples where content_by_lua is used to send/receive a request via UDP with socket.udp()/setpeername()/send()/recv(). I believe that this would create one UDP socket per request and destroy it when the request terminates.

Is there any way to instead create one shared UDP socket per nginx worker, and use some sort of unique identifier in the UDP payload to route data received back to the original request? Basically to avoid the overhead of creating/destroying a socket with every request.

Or is it actually more efficient to just create a socket every time then to do everything necessary to multiplex?

agentzh

Hello!

On Tue, Jan 5, 2016 at 8:35 AM, CJ Ess wrote:
> Ok, so the way I would envision this working then is I'd create an array to
> hold the socket_udp objects sometime in the initialization stage, then in
> the request stages I'd try to pop a socket_udp from the array, create a new
> one if the array is empty, include some magic number in my UDP payload so I
> can be certain that anything I receive is for the current request and not a
> previous request, then at the end of the request I push the socket_udp
> object back into the table for the next request. Thats a nice
> middle-of-the-road setup I hadn't considered before.
>
> However, with that setup I'd still be concerned that because nginx is so
> crazy good at what it does, it would be practical to open more then > 64K
> sockets unless I had each worker bind outgoing UDP to a seperate IP address,
> so I'd still like to see if there is some way I could multiplex
> requests/responses to a backend though a single UDP socket per worker.
> Within the lua framework, is there some way I could send out the nginx
> request context (an integer) then have the backend echo that back and
> somehow use that to direct the response received to the right request
> context?

Well, you could do that NOW with init_worker_by_lua* + ngx.timer.at +
ngx.socket.udp + ngx.semaphore. Just keep a live UDP cosocket object
in a background light thread initiated by the timer in
init_worker_by_lua and use this UDP cosocket to multiplex all the
requests of your NGINX's downstream request handlers. And use
ngx.semaphore to synchronize the light threads. This should be very
efficient. You need at least the following version of OpenResty
though:

https://openresty.org/download/ngx_openresty-1.9.7.2rc0.tar.gz

Good luck :)

Best regards,
-agentzh