[ANN] ngx_lua_ipc : async interprocess communication for openresty workers

leo · 2017-03-04T04:27:13+00:00

Sounds about right. Getting the mutex operations out of the hot path will make a noticable difference.Having a read-write lock implementation for shared dict...

[ANN] ngx_lua_ipc : async interprocess communication for openresty workers

leo

Interprocess communication for lua_nginx_module and Openresty. Send
named alerts with string data between Nginx worker processes.
Asynchronous, nonblocking, non-locking.

Very simple to use:

local ipc = require "ngx.ipc"

ipc.send(destination_worker_pid, ipc_alert_name, data_string)

ipc.receive(ipc_alert_name, function(data)
  --ipc receiver function for all alerts with string name ipc_alert_name
  print("received alert from " .. ipc.sender)
  ipc.reply("you said " .. data)
end)


https://github.com/slact/ngx_lua_ipc

aapo.talvensaari

On Friday, 3 March 2017 22:27:15 UTC+2, Leo Ponomarev wrote:

Interprocess communication for lua_nginx_module and Openresty. Send
named alerts with string data between Nginx worker processes.
Asynchronous, nonblocking, non-locking.

Thank you Leo, this is really nice and valuable!

Regards

Aapo

erik.l.nelson

Leo Ponomarev wrote on Friday, March 03, 2017 3:27 PM
> Subject: [openresty-en] [ANN] ngx_lua_ipc : async interprocess
> communication for openresty workers
> 

This looks great!  How does the broadcast functionality keep track of the worker pids?  Does the list update when a worker process is restarted by the master?

Erik

----------------------------------------------------------------------
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer.   If you are not the intended recipient, please delete this message.

rpaprocki

On Fri, Mar 3, 2017 at 12:27 PM, Leo Ponomarev <l...@nchan.io> wrote:

Interprocess communication for lua_nginx_module and Openresty. Send
named alerts with string data between Nginx worker processes.
Asynchronous, nonblocking, non-locking.

Very simple to use:

local ipc = require "ngx.ipc"

ipc.send(destination_worker_pid, ipc_alert_name, data_string)

ipc.receive(ipc_alert_name, function(data)
--ipc receiver function for all alerts with string name ipc_alert_name
print("received alert from " .. ipc.sender)
ipc.reply("you said " .. data)
end)

https://github.com/slact/ngx_lua_ipc

Way cool! Have you done any stress testing? How does this hold up in high-concurrency environments?

leo

The workers are tracked in a small slab of shared memory. The PIDs (and
their associated nginx process slots) are stored at worker initialization.

So yes, the list is updated when individual workers are killed or when
nginx is SIGHUP-reloaded.

'Nelson, Erik - 2' via openresty-en:
> 
> Leo Ponomarev wrote on Friday, March 03, 2017 3:27 PM
>> Subject: [openresty-en] [ANN] ngx_lua_ipc : async interprocess
>> communication for openresty workers
>>
> 
> This looks great!  How does the broadcast functionality keep track of the worker pids?  Does the list update when a worker process is restarted by the master?
> 
> Erik
> 
> ----------------------------------------------------------------------
> This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer.   If you are not the intended recipient, please delete this message.
>

leo

I haven't fully stress-tested this implementation yet. I will say that
in the Nchan module, which is where I originally developed the IPC
system, it could handle >200K IPC alerts per second easy.

I'd say this implementation is an order of magnitude slower, as it's
detuned to be much more general-purpose, and there's also bit of
overhead in the way incoming alerts are delivered into a valid Lua thread.

I'll do some benchmarks next week and I'll be sure to post them.

Robert Paprocki:
> On Fri, Mar 3, 2017 at 12:27 PM, Leo Ponomarev <l...@nchan.io> wrote:
> 
>> Interprocess communication for lua_nginx_module and Openresty. Send
>> named alerts with string data between Nginx worker processes.
>> Asynchronous, nonblocking, non-locking.
>>
>> Very simple to use:
>>
>> local ipc = require "ngx.ipc"
>>
>> ipc.send(destination_worker_pid, ipc_alert_name, data_string)
>>
>> ipc.receive(ipc_alert_name, function(data)
>>   --ipc receiver function for all alerts with string name ipc_alert_name
>>   print("received alert from " .. ipc.sender)
>>   ipc.reply("you said " .. data)
>> end)
>>
>>
>> https://github.com/slact/ngx_lua_ipc
> 
> 
> Way cool! Have you done any stress testing? How does this hold up in
> high-concurrency environments?
>

aapo.talvensaari

On Friday, 3 March 2017 23:20:33 UTC+2, Leo Ponomarev wrote:

I haven't fully stress-tested this implementation yet. I will say that
in the Nchan module, which is where I originally developed the IPC
system, it could handle >200K IPC alerts per second easy.

Of course this would open possibilities for high performance applications, but this has use in other cases as well that don't require performance:

- worker level cache invalidation

- configuration reloading

- websocket broadcasting

Maybe it could be considered as something that could at some point be included in OpenResty core.

leo

Did a few benchmarks, the results are really good for high concurrency:

Test case: Broadcasting from 1 worker, followed by many K rounds of
back-and-forth communication between the initial broadcaster and all the
other workers. Done on an elderly i5-2450M 2.50GHz (2 core, 4 thread)
CPU. Nginx compiled with -O3.

Time to do 50K round-trips between broadcaster and all workers (100K ipc
alerts total):

  4 workers: 18.32 sec
 10 workers: 6.15 sec
 20 workers: 3.00 sec
 30 workers: 1.92 sec
 40 workers: 1.49 sec
 50 workers: 1.22 sec
 60 workers: 1.05 sec
 70 workers: 0.94 sec
100 workers: 0.83 sec
150 workers: 0.81 sec

So at crazy concurrency, it can do 120K alerts/sec on this machine.

Robert Paprocki:
> On Fri, Mar 3, 2017 at 12:27 PM, Leo Ponomarev <l...@nchan.io> wrote:
> 
>> Interprocess communication for lua_nginx_module and Openresty. Send
>> named alerts with string data between Nginx worker processes.
>> Asynchronous, nonblocking, non-locking.
>>
>> Very simple to use:
>>
>> local ipc = require "ngx.ipc"
>>
>> ipc.send(destination_worker_pid, ipc_alert_name, data_string)
>>
>> ipc.receive(ipc_alert_name, function(data)
>>   --ipc receiver function for all alerts with string name ipc_alert_name
>>   print("received alert from " .. ipc.sender)
>>   ipc.reply("you said " .. data)
>> end)
>>
>>
>> https://github.com/slact/ngx_lua_ipc
> 
> 
> Way cool! Have you done any stress testing? How does this hold up in
> high-concurrency environments?
>

jaksprats

This is very useful. I did an IPC module for nginx workers last year but have not been maintaining it very well.

Can you tell us more about how you implemented it. How does IPC work under the hood (TCP,UDP,SM)? I had to make some changes to the nginx core for mine to work, you didnt, now I am very interested in the internals.

Nice work :)

On Saturday, March 4, 2017 at 9:16:28 PM UTC-8, Leo Ponomarev wrote:

Did a few benchmarks, the results are really good for high concurrency:

Test case: Broadcasting from 1 worker, followed by many K rounds of
back-and-forth communication between the initial broadcaster and all the
other workers. Done on an elderly i5-2450M 2.50GHz (2 core, 4 thread)
CPU. Nginx compiled with -O3.

Time to do 50K round-trips between broadcaster and all workers (100K ipc
alerts total):

4 workers: 18.32 sec
10 workers: 6.15 sec
20 workers: 3.00 sec
30 workers: 1.92 sec
40 workers: 1.49 sec
50 workers: 1.22 sec
60 workers: 1.05 sec
70 workers: 0.94 sec
100 workers: 0.83 sec
150 workers: 0.81 sec

So at crazy concurrency, it can do 120K alerts/sec on this machine.

Robert Paprocki:
> On Fri, Mar 3, 2017 at 12:27 PM, Leo Ponomarev <l...@nchan.io> wrote:
>
>> Interprocess communication for lua_nginx_module and Openresty. Send
>> named alerts with string data between Nginx worker processes.
>> Asynchronous, nonblocking, non-locking.
>>
>> Very simple to use:
>>
>> local ipc = require "ngx.ipc"
>>
>> ipc.send(destination_worker_pid, ipc_alert_name, data_string)
>>
>> ipc.receive(ipc_alert_name, function(data)
>> --ipc receiver function for all alerts with string name ipc_alert_name
>> print("received alert from " .. ipc.sender)
>> ipc.reply("you said " .. data)
>> end)
>>
>>
>> https://github.com/slact/ngx_lua_ipc
>
>
> Way cool! Have you done any stress testing? How does this hold up in
> high-concurrency environments?
>

leo

Here's how it works:

Let's say we have n workers. At master process initialization
(postconfig), n (unix) sockets are created with the same logic that
initializes worker slots [1]. Then, when each worker starts, it adds a
listen event on the socket in its worker slot to the nginx event tree
[2], and stores its process id associated with the worker slot in shared
memory [3].

Then, to send an ipc alert to a target worker by process id, look up its
slot in shared memory, then send data to the associated socket [4]. Add
some send [5] and receive [6] buffering, and voila.

Importing the data into a valid Lua thread was a little tricky. When a
user calls ipc.receive() in the config, the module starts a (very long)
timer with a unique timeout using ngx.timer.at() [7]. This timer is then
yanked off the timer tree (by finding it using the unique timeout) [8],
and registered with the lua ipc module. Then, when an alert arrives [9],
this timer event is deleted from the timer tree, and re-inserted with
timeout 0 [10]. Once the incoming events are processed, the timer
re-inserts itself [11] with a unique timeout again, and the process repeats.

[1] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L49-L115
[2] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L229-L269
[3]
https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L385-L394
[4]
https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L239-L254
[5] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L186-L227
[6] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L271-L477
[7]
https://github.com/slact/ngx_lua_ipc/blob/master/src/lua/register_receive_handler.lua
[8]
https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L194-L237
[9]
https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L269-L327
[10]
https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L313-L317
[11]
https://github.com/slact/ngx_lua_ipc/blob/master/src/lua/register_receive_handler.lua#L19-L20




Russell Sullivan:
> 
> This is very useful. I did an IPC module for nginx workers last year but 
> have not been maintaining it very well.
> 
> Can you tell us more about how you implemented it. How does IPC work under 
> the hood (TCP,UDP,SM)? I had to make some changes to the nginx core for 
> mine to work, you didnt, now I am very interested in the internals.
> 
> Nice work :)
> 
> On Saturday, March 4, 2017 at 9:16:28 PM UTC-8, Leo Ponomarev wrote:
>>
>> Did a few benchmarks, the results are really good for high concurrency: 
>>
>> Test case: Broadcasting from 1 worker, followed by many K rounds of 
>> back-and-forth communication between the initial broadcaster and all the 
>> other workers. Done on an elderly i5-2450M 2.50GHz (2 core, 4 thread) 
>> CPU. Nginx compiled with -O3. 
>>
>> Time to do 50K round-trips between broadcaster and all workers (100K ipc 
>> alerts total): 
>>
>>   4 workers: 18.32 sec 
>>  10 workers: 6.15 sec 
>>  20 workers: 3.00 sec 
>>  30 workers: 1.92 sec 
>>  40 workers: 1.49 sec 
>>  50 workers: 1.22 sec 
>>  60 workers: 1.05 sec 
>>  70 workers: 0.94 sec 
>> 100 workers: 0.83 sec 
>> 150 workers: 0.81 sec 
>>
>> So at crazy concurrency, it can do 120K alerts/sec on this machine. 
>>
>> Robert Paprocki: 
>>> On Fri, Mar 3, 2017 at 12:27 PM, Leo Ponomarev <l...@nchan.io 
>> <javascript:>> wrote: 
>>>
>>>> Interprocess communication for lua_nginx_module and Openresty. Send 
>>>> named alerts with string data between Nginx worker processes. 
>>>> Asynchronous, nonblocking, non-locking. 
>>>>
>>>> Very simple to use: 
>>>>
>>>> local ipc = require "ngx.ipc" 
>>>>
>>>> ipc.send(destination_worker_pid, ipc_alert_name, data_string) 
>>>>
>>>> ipc.receive(ipc_alert_name, function(data) 
>>>>   --ipc receiver function for all alerts with string name 
>> ipc_alert_name 
>>>>   print("received alert from " .. ipc.sender) 
>>>>   ipc.reply("you said " .. data) 
>>>> end) 
>>>>
>>>>
>>>> https://github.com/slact/ngx_lua_ipc 
>>>
>>>
>>> Way cool! Have you done any stress testing? How does this hold up in 
>>> high-concurrency environments? 
>>>
>>
>>

jaksprats

Hi Leo,

You got the architecture right: non-blocking Unix Domain Sockets, worker to socket mapping in SharedMemory, and the receive-coroutine is correct too.

My implementation did the first two, but the receive logic was done by hacking the nginx core and using nginx's internals, it was a hack.

I did not know there was an easy way to add a listening socket to nginx, but it looks like you found one :) I did a lot of ugly stuff, only took 2 days, and it shows :)

The reason I need this is I shard requests to sticky workers. The simplest analogy is all requests for users [A-H] go to worker0, users [I-P] go to worker1, users[Q-Z] go to worker3. I do it for data linearizability using a LMDB backend, LMDB runs in SharedMemory but I do multiple key updates that need to be serialized, so I pushed that requirement into the request receive layer.

Your implementation looks cleaner than mine, so I will probably end up using it if you maintain it.

Are there any gotchas? It doesn't look like it, the architecture is done correctly.

Again, very nice work :)

- Russ

BTW: nice writeup with the numbered links, helps understand the code and implementation

On Monday, March 6, 2017 at 6:47:49 PM UTC-8, Leo Ponomarev wrote:

Here's how it works:

Let's say we have n workers. At master process initialization
(postconfig), n (unix) sockets are created with the same logic that
initializes worker slots [1]. Then, when each worker starts, it adds a
listen event on the socket in its worker slot to the nginx event tree
[2], and stores its process id associated with the worker slot in shared
memory [3].

Then, to send an ipc alert to a target worker by process id, look up its
slot in shared memory, then send data to the associated socket [4]. Add
some send [5] and receive [6] buffering, and voila.

Importing the data into a valid Lua thread was a little tricky. When a
user calls ipc.receive() in the config, the module starts a (very long)
timer with a unique timeout using ngx.timer.at() [7]. This timer is then
yanked off the timer tree (by finding it using the unique timeout) [8],
and registered with the lua ipc module. Then, when an alert arrives [9],
this timer event is deleted from the timer tree, and re-inserted with
timeout 0 [10]. Once the incoming events are processed, the timer
re-inserts itself [11] with a unique timeout again, and the process repeats.

[1] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L49-L115
[2] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L229-L269
[3]
https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L385-L394
[4]
https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L239-L254
[5] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L186-L227
[6] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L271-L477
[7]
https://github.com/slact/ngx_lua_ipc/blob/master/src/lua/register_receive_handler.lua
[8]
https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L194-L237
[9]
https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L269-L327
[10]
https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L313-L317
[11]
https://github.com/slact/ngx_lua_ipc/blob/master/src/lua/register_receive_handler.lua#L19-L20

Russell Sullivan:
>
> This is very useful. I did an IPC module for nginx workers last year but
> have not been maintaining it very well.
>
> Can you tell us more about how you implemented it. How does IPC work under
> the hood (TCP,UDP,SM)? I had to make some changes to the nginx core for
> mine to work, you didnt, now I am very interested in the internals.
>
> Nice work :)
>
> On Saturday, March 4, 2017 at 9:16:28 PM UTC-8, Leo Ponomarev wrote:
>>
>> Did a few benchmarks, the results are really good for high concurrency:
>>
>> Test case: Broadcasting from 1 worker, followed by many K rounds of
>> back-and-forth communication between the initial broadcaster and all the
>> other workers. Done on an elderly i5-2450M 2.50GHz (2 core, 4 thread)
>> CPU. Nginx compiled with -O3.
>>
>> Time to do 50K round-trips between broadcaster and all workers (100K ipc
>> alerts total):
>>
>> 4 workers: 18.32 sec
>> 10 workers: 6.15 sec
>> 20 workers: 3.00 sec
>> 30 workers: 1.92 sec
>> 40 workers: 1.49 sec
>> 50 workers: 1.22 sec
>> 60 workers: 1.05 sec
>> 70 workers: 0.94 sec
>> 100 workers: 0.83 sec
>> 150 workers: 0.81 sec
>>
>> So at crazy concurrency, it can do 120K alerts/sec on this machine.
>>
>> Robert Paprocki:
>>> On Fri, Mar 3, 2017 at 12:27 PM, Leo Ponomarev <l...@nchan.io
>> <_javascript_:>> wrote:
>>>
>>>> Interprocess communication for lua_nginx_module and Openresty. Send
>>>> named alerts with string data between Nginx worker processes.
>>>> Asynchronous, nonblocking, non-locking.
>>>>
>>>> Very simple to use:
>>>>
>>>> local ipc = require "ngx.ipc"
>>>>
>>>> ipc.send(destination_worker_pid, ipc_alert_name, data_string)
>>>>
>>>> ipc.receive(ipc_alert_name, function(data)
>>>> --ipc receiver function for all alerts with string name
>> ipc_alert_name
>>>> print("received alert from " .. ipc.sender)
>>>> ipc.reply("you said " .. data)
>>>> end)
>>>>
>>>>
>>>> https://github.com/slact/ngx_lua_ipc
>>>
>>>
>>> Way cool! Have you done any stress testing? How does this hold up in
>>> high-concurrency environments?
>>>
>>
>>

leo

I've been using a variant of this IPC system since about 2009 (as part
of the nginx http push module, now Nchan), so it's rather well
time-tested already. As I recall, it took about a week of digging
through the Nginx guts to figure out a proper IPC mechanism. (I used the
master->worker signal forwarding system as a template to build from.)

One small correction - the sockets are not in shared memory, but in
copy-on-write memory inherited from the master process. worker PID and
slot info is what's in shared memory. (This made reloading logic a bit
simpler)

There are 3 minor catches so far: the alert name is limited to 64K, the
data to 4T, and the alert handler runs in an ngx.timer context. That's
about all.

Russell -- incidentally, I'd be interested to see your code for your IPC
system.

Russell Sullivan:
> Hi Leo,
> 
> You got the architecture right: non-blocking Unix Domain Sockets, worker to 
> socket mapping in SharedMemory, and the receive-coroutine is correct too.
> 
> My implementation did the first two, but the receive logic was done by 
> hacking the nginx core and using nginx's internals, it was a hack.
> 
> I did not know there was an easy way to add a listening socket to nginx, 
> but it looks like you found one :) I did a lot of ugly stuff, only took 2 
> days, and it shows :)
> 
> The reason I need this is I shard requests to sticky workers. The simplest 
> analogy is all requests for users [A-H] go to worker0, users [I-P] go to 
> worker1, users[Q-Z] go to worker3. I do it for data linearizability using a 
> LMDB backend, LMDB runs in SharedMemory but I do multiple key updates that 
> need to be serialized, so I pushed that requirement into the request 
> receive layer.
> 
> Your implementation looks cleaner than mine, so I will probably end up 
> using it if you maintain it.
> 
> Are there any gotchas? It doesn't look like it, the architecture is done 
> correctly.
> 
> Again, very nice work :)
> 
> - Russ
> 
> BTW: nice writeup with the numbered links, helps understand the code and 
> implementation
> 
> 
> On Monday, March 6, 2017 at 6:47:49 PM UTC-8, Leo Ponomarev wrote:
>>
>> Here's how it works: 
>>
>> Let's say we have n workers. At master process initialization 
>> (postconfig), n (unix) sockets are created with the same logic that 
>> initializes worker slots [1]. Then, when each worker starts, it adds a 
>> listen event on the socket in its worker slot to the nginx event tree 
>> [2], and stores its process id associated with the worker slot in shared 
>> memory [3]. 
>>
>> Then, to send an ipc alert to a target worker by process id, look up its 
>> slot in shared memory, then send data to the associated socket [4]. Add 
>> some send [5] and receive [6] buffering, and voila. 
>>
>> Importing the data into a valid Lua thread was a little tricky. When a 
>> user calls ipc.receive() in the config, the module starts a (very long) 
>> timer with a unique timeout using ngx.timer.at() [7]. This timer is then 
>> yanked off the timer tree (by finding it using the unique timeout) [8], 
>> and registered with the lua ipc module. Then, when an alert arrives [9], 
>> this timer event is deleted from the timer tree, and re-inserted with 
>> timeout 0 [10]. Once the incoming events are processed, the timer 
>> re-inserts itself [11] with a unique timeout again, and the process 
>> repeats. 
>>
>> [1] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L49-L115 
>> [2] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L229-L269 
>> [3] 
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L385-L394 
>> [4] 
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L239-L254 
>> [5] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L186-L227 
>> [6] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L271-L477 
>> [7] 
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/lua/register_receive_handler.lua 
>> [8] 
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L194-L237 
>> [9] 
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L269-L327 
>> [10] 
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L313-L317 
>> [11] 
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/lua/register_receive_handler.lua#L19-L20 
>>
>>
>>
>>
>> Russell Sullivan: 
>>>
>>> This is very useful. I did an IPC module for nginx workers last year but 
>>> have not been maintaining it very well. 
>>>
>>> Can you tell us more about how you implemented it. How does IPC work 
>> under 
>>> the hood (TCP,UDP,SM)? I had to make some changes to the nginx core for 
>>> mine to work, you didnt, now I am very interested in the internals. 
>>>
>>> Nice work :) 
>>>
>>> On Saturday, March 4, 2017 at 9:16:28 PM UTC-8, Leo Ponomarev wrote: 
>>>>
>>>> Did a few benchmarks, the results are really good for high concurrency: 
>>>>
>>>> Test case: Broadcasting from 1 worker, followed by many K rounds of 
>>>> back-and-forth communication between the initial broadcaster and all 
>> the 
>>>> other workers. Done on an elderly i5-2450M 2.50GHz (2 core, 4 thread) 
>>>> CPU. Nginx compiled with -O3. 
>>>>
>>>> Time to do 50K round-trips between broadcaster and all workers (100K 
>> ipc 
>>>> alerts total): 
>>>>
>>>>   4 workers: 18.32 sec 
>>>>  10 workers: 6.15 sec 
>>>>  20 workers: 3.00 sec 
>>>>  30 workers: 1.92 sec 
>>>>  40 workers: 1.49 sec 
>>>>  50 workers: 1.22 sec 
>>>>  60 workers: 1.05 sec 
>>>>  70 workers: 0.94 sec 
>>>> 100 workers: 0.83 sec 
>>>> 150 workers: 0.81 sec 
>>>>
>>>> So at crazy concurrency, it can do 120K alerts/sec on this machine. 
>>>>
>>>> Robert Paprocki: 
>>>>> On Fri, Mar 3, 2017 at 12:27 PM, Leo Ponomarev <l...@nchan.io 
>>>> <javascript:>> wrote: 
>>>>>
>>>>>> Interprocess communication for lua_nginx_module and Openresty. Send 
>>>>>> named alerts with string data between Nginx worker processes. 
>>>>>> Asynchronous, nonblocking, non-locking. 
>>>>>>
>>>>>> Very simple to use: 
>>>>>>
>>>>>> local ipc = require "ngx.ipc" 
>>>>>>
>>>>>> ipc.send(destination_worker_pid, ipc_alert_name, data_string) 
>>>>>>
>>>>>> ipc.receive(ipc_alert_name, function(data) 
>>>>>>   --ipc receiver function for all alerts with string name 
>>>> ipc_alert_name 
>>>>>>   print("received alert from " .. ipc.sender) 
>>>>>>   ipc.reply("you said " .. data) 
>>>>>> end) 
>>>>>>
>>>>>>
>>>>>> https://github.com/slact/ngx_lua_ipc 
>>>>>
>>>>>
>>>>> Way cool! Have you done any stress testing? How does this hold up in 
>>>>> high-concurrency environments? 
>>>>>
>>>>
>>>>
>>
>>

mat999

Wow this looks so much nicer than our IPC.

The only thing I think this is missing that we implemented is guarantee of delivery (acknowledgements to dequeue) but we can wrap that I think.

On Wednesday, March 8, 2017 at 9:06:09 AM UTC+11, Leo Ponomarev wrote:

I've been using a variant of this IPC system since about 2009 (as part
of the nginx http push module, now Nchan), so it's rather well
time-tested already. As I recall, it took about a week of digging
through the Nginx guts to figure out a proper IPC mechanism. (I used the
master->worker signal forwarding system as a template to build from.)

One small correction - the sockets are not in shared memory, but in
copy-on-write memory inherited from the master process. worker PID and
slot info is what's in shared memory. (This made reloading logic a bit
simpler)

There are 3 minor catches so far: the alert name is limited to 64K, the
data to 4T, and the alert handler runs in an ngx.timer context. That's
about all.

Russell -- incidentally, I'd be interested to see your code for your IPC
system.

Russell Sullivan:
> Hi Leo,
>
> You got the architecture right: non-blocking Unix Domain Sockets, worker to
> socket mapping in SharedMemory, and the receive-coroutine is correct too.
>
> My implementation did the first two, but the receive logic was done by
> hacking the nginx core and using nginx's internals, it was a hack.
>
> I did not know there was an easy way to add a listening socket to nginx,
> but it looks like you found one :) I did a lot of ugly stuff, only took 2
> days, and it shows :)
>
> The reason I need this is I shard requests to sticky workers. The simplest
> analogy is all requests for users [A-H] go to worker0, users [I-P] go to
> worker1, users[Q-Z] go to worker3. I do it for data linearizability using a
> LMDB backend, LMDB runs in SharedMemory but I do multiple key updates that
> need to be serialized, so I pushed that requirement into the request
> receive layer.
>
> Your implementation looks cleaner than mine, so I will probably end up
> using it if you maintain it.
>
> Are there any gotchas? It doesn't look like it, the architecture is done
> correctly.
>
> Again, very nice work :)
>
> - Russ
>
> BTW: nice writeup with the numbered links, helps understand the code and
> implementation
>
>
> On Monday, March 6, 2017 at 6:47:49 PM UTC-8, Leo Ponomarev wrote:
>>
>> Here's how it works:
>>
>> Let's say we have n workers. At master process initialization
>> (postconfig), n (unix) sockets are created with the same logic that
>> initializes worker slots [1]. Then, when each worker starts, it adds a
>> listen event on the socket in its worker slot to the nginx event tree
>> [2], and stores its process id associated with the worker slot in shared
>> memory [3].
>>
>> Then, to send an ipc alert to a target worker by process id, look up its
>> slot in shared memory, then send data to the associated socket [4]. Add
>> some send [5] and receive [6] buffering, and voila.
>>
>> Importing the data into a valid Lua thread was a little tricky. When a
>> user calls ipc.receive() in the config, the module starts a (very long)
>> timer with a unique timeout using ngx.timer.at() [7]. This timer is then
>> yanked off the timer tree (by finding it using the unique timeout) [8],
>> and registered with the lua ipc module. Then, when an alert arrives [9],
>> this timer event is deleted from the timer tree, and re-inserted with
>> timeout 0 [10]. Once the incoming events are processed, the timer
>> re-inserts itself [11] with a unique timeout again, and the process
>> repeats.
>>
>> [1] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L49-L115
>> [2] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L229-L269
>> [3]
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L385-L394
>> [4]
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L239-L254
>> [5] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L186-L227
>> [6] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L271-L477
>> [7]
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/lua/register_receive_handler.lua
>> [8]
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L194-L237
>> [9]
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L269-L327
>> [10]
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L313-L317
>> [11]
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/lua/register_receive_handler.lua#L19-L20
>>
>>
>>
>>
>> Russell Sullivan:
>>>
>>> This is very useful. I did an IPC module for nginx workers last year but
>>> have not been maintaining it very well.
>>>
>>> Can you tell us more about how you implemented it. How does IPC work
>> under
>>> the hood (TCP,UDP,SM)? I had to make some changes to the nginx core for
>>> mine to work, you didnt, now I am very interested in the internals.
>>>
>>> Nice work :)
>>>
>>> On Saturday, March 4, 2017 at 9:16:28 PM UTC-8, Leo Ponomarev wrote:
>>>>
>>>> Did a few benchmarks, the results are really good for high concurrency:
>>>>
>>>> Test case: Broadcasting from 1 worker, followed by many K rounds of
>>>> back-and-forth communication between the initial broadcaster and all
>> the
>>>> other workers. Done on an elderly i5-2450M 2.50GHz (2 core, 4 thread)
>>>> CPU. Nginx compiled with -O3.
>>>>
>>>> Time to do 50K round-trips between broadcaster and all workers (100K
>> ipc
>>>> alerts total):
>>>>
>>>> 4 workers: 18.32 sec
>>>> 10 workers: 6.15 sec
>>>> 20 workers: 3.00 sec
>>>> 30 workers: 1.92 sec
>>>> 40 workers: 1.49 sec
>>>> 50 workers: 1.22 sec
>>>> 60 workers: 1.05 sec
>>>> 70 workers: 0.94 sec
>>>> 100 workers: 0.83 sec
>>>> 150 workers: 0.81 sec
>>>>
>>>> So at crazy concurrency, it can do 120K alerts/sec on this machine.
>>>>
>>>> Robert Paprocki:
>>>>> On Fri, Mar 3, 2017 at 12:27 PM, Leo Ponomarev <l...@nchan.io
>>>> <_javascript_:>> wrote:
>>>>>
>>>>>> Interprocess communication for lua_nginx_module and Openresty. Send
>>>>>> named alerts with string data between Nginx worker processes.
>>>>>> Asynchronous, nonblocking, non-locking.
>>>>>>
>>>>>> Very simple to use:
>>>>>>
>>>>>> local ipc = require "ngx.ipc"
>>>>>>
>>>>>> ipc.send(destination_worker_pid, ipc_alert_name, data_string)
>>>>>>
>>>>>> ipc.receive(ipc_alert_name, function(data)
>>>>>> --ipc receiver function for all alerts with string name
>>>> ipc_alert_name
>>>>>> print("received alert from " .. ipc.sender)
>>>>>> ipc.reply("you said " .. data)
>>>>>> end)
>>>>>>
>>>>>>
>>>>>> https://github.com/slact/ngx_lua_ipc
>>>>>
>>>>>
>>>>> Way cool! Have you done any stress testing? How does this hold up in
>>>>> high-concurrency environments?
>>>>>
>>>>
>>>>
>>
>>

mat999

Of additional note, how is worker termination (shutdown / reload) handled?

What happens to queued messages (particularly broadcast messages)?

On Wednesday, March 8, 2017 at 9:06:09 AM UTC+11, Leo Ponomarev wrote:

I've been using a variant of this IPC system since about 2009 (as part
of the nginx http push module, now Nchan), so it's rather well
time-tested already. As I recall, it took about a week of digging
through the Nginx guts to figure out a proper IPC mechanism. (I used the
master->worker signal forwarding system as a template to build from.)

One small correction - the sockets are not in shared memory, but in
copy-on-write memory inherited from the master process. worker PID and
slot info is what's in shared memory. (This made reloading logic a bit
simpler)

There are 3 minor catches so far: the alert name is limited to 64K, the
data to 4T, and the alert handler runs in an ngx.timer context. That's
about all.

Russell -- incidentally, I'd be interested to see your code for your IPC
system.

Russell Sullivan:
> Hi Leo,
>
> You got the architecture right: non-blocking Unix Domain Sockets, worker to
> socket mapping in SharedMemory, and the receive-coroutine is correct too.
>
> My implementation did the first two, but the receive logic was done by
> hacking the nginx core and using nginx's internals, it was a hack.
>
> I did not know there was an easy way to add a listening socket to nginx,
> but it looks like you found one :) I did a lot of ugly stuff, only took 2
> days, and it shows :)
>
> The reason I need this is I shard requests to sticky workers. The simplest
> analogy is all requests for users [A-H] go to worker0, users [I-P] go to
> worker1, users[Q-Z] go to worker3. I do it for data linearizability using a
> LMDB backend, LMDB runs in SharedMemory but I do multiple key updates that
> need to be serialized, so I pushed that requirement into the request
> receive layer.
>
> Your implementation looks cleaner than mine, so I will probably end up
> using it if you maintain it.
>
> Are there any gotchas? It doesn't look like it, the architecture is done
> correctly.
>
> Again, very nice work :)
>
> - Russ
>
> BTW: nice writeup with the numbered links, helps understand the code and
> implementation
>
>
> On Monday, March 6, 2017 at 6:47:49 PM UTC-8, Leo Ponomarev wrote:
>>
>> Here's how it works:
>>
>> Let's say we have n workers. At master process initialization
>> (postconfig), n (unix) sockets are created with the same logic that
>> initializes worker slots [1]. Then, when each worker starts, it adds a
>> listen event on the socket in its worker slot to the nginx event tree
>> [2], and stores its process id associated with the worker slot in shared
>> memory [3].
>>
>> Then, to send an ipc alert to a target worker by process id, look up its
>> slot in shared memory, then send data to the associated socket [4]. Add
>> some send [5] and receive [6] buffering, and voila.
>>
>> Importing the data into a valid Lua thread was a little tricky. When a
>> user calls ipc.receive() in the config, the module starts a (very long)
>> timer with a unique timeout using ngx.timer.at() [7]. This timer is then
>> yanked off the timer tree (by finding it using the unique timeout) [8],
>> and registered with the lua ipc module. Then, when an alert arrives [9],
>> this timer event is deleted from the timer tree, and re-inserted with
>> timeout 0 [10]. Once the incoming events are processed, the timer
>> re-inserts itself [11] with a unique timeout again, and the process
>> repeats.
>>
>> [1] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L49-L115
>> [2] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L229-L269
>> [3]
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L385-L394
>> [4]
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L239-L254
>> [5] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L186-L227
>> [6] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L271-L477
>> [7]
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/lua/register_receive_handler.lua
>> [8]
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L194-L237
>> [9]
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L269-L327
>> [10]
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L313-L317
>> [11]
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/lua/register_receive_handler.lua#L19-L20
>>
>>
>>
>>
>> Russell Sullivan:
>>>
>>> This is very useful. I did an IPC module for nginx workers last year but
>>> have not been maintaining it very well.
>>>
>>> Can you tell us more about how you implemented it. How does IPC work
>> under
>>> the hood (TCP,UDP,SM)? I had to make some changes to the nginx core for
>>> mine to work, you didnt, now I am very interested in the internals.
>>>
>>> Nice work :)
>>>
>>> On Saturday, March 4, 2017 at 9:16:28 PM UTC-8, Leo Ponomarev wrote:
>>>>
>>>> Did a few benchmarks, the results are really good for high concurrency:
>>>>
>>>> Test case: Broadcasting from 1 worker, followed by many K rounds of
>>>> back-and-forth communication between the initial broadcaster and all
>> the
>>>> other workers. Done on an elderly i5-2450M 2.50GHz (2 core, 4 thread)
>>>> CPU. Nginx compiled with -O3.
>>>>
>>>> Time to do 50K round-trips between broadcaster and all workers (100K
>> ipc
>>>> alerts total):
>>>>
>>>> 4 workers: 18.32 sec
>>>> 10 workers: 6.15 sec
>>>> 20 workers: 3.00 sec
>>>> 30 workers: 1.92 sec
>>>> 40 workers: 1.49 sec
>>>> 50 workers: 1.22 sec
>>>> 60 workers: 1.05 sec
>>>> 70 workers: 0.94 sec
>>>> 100 workers: 0.83 sec
>>>> 150 workers: 0.81 sec
>>>>
>>>> So at crazy concurrency, it can do 120K alerts/sec on this machine.
>>>>
>>>> Robert Paprocki:
>>>>> On Fri, Mar 3, 2017 at 12:27 PM, Leo Ponomarev <l...@nchan.io
>>>> <_javascript_:>> wrote:
>>>>>
>>>>>> Interprocess communication for lua_nginx_module and Openresty. Send
>>>>>> named alerts with string data between Nginx worker processes.
>>>>>> Asynchronous, nonblocking, non-locking.
>>>>>>
>>>>>> Very simple to use:
>>>>>>
>>>>>> local ipc = require "ngx.ipc"
>>>>>>
>>>>>> ipc.send(destination_worker_pid, ipc_alert_name, data_string)
>>>>>>
>>>>>> ipc.receive(ipc_alert_name, function(data)
>>>>>> --ipc receiver function for all alerts with string name
>>>> ipc_alert_name
>>>>>> print("received alert from " .. ipc.sender)
>>>>>> ipc.reply("you said " .. data)
>>>>>> end)
>>>>>>
>>>>>>
>>>>>> https://github.com/slact/ngx_lua_ipc
>>>>>
>>>>>
>>>>> Way cool! Have you done any stress testing? How does this hold up in
>>>>> high-concurrency environments?
>>>>>
>>>>
>>>>
>>
>>

bogdan

Very interesting project, thank you

I would like to ask if this might be a feasable way to replace thePUB/SUB events queues from Redis with IPC messages. What do you think ofthis, regarding performance and stability?We are using PUB/SUB events from Redis with websockets to trigger data"push" on websocket channels on various type of events (incoming dataevents, to be more specific).

Any suggestions and ideas are welcome.

Thank you

Bogdan Irimia

aapo.talvensaari

On Wed, Mar 8, 2017 at 10:58 AM, Bogdan Irimia <bo...@digitair.ro> wrote:

Very interesting project, thank you

I would like to ask if this might be a feasable way to replace the PUB/SUB events queues from Redis with IPC messages. What do you think of this, regarding performance and stability?
We are using PUB/SUB events from Redis with websockets to trigger data "push" on websocket channels on various type of events (incoming data events, to be more specific).
Any suggestions and ideas are welcome.

I think it could. Redis on the other hand has advantage of supporting web farms. This is for single Nginx workers IPC, AFAIK.

jaksprats

Hi Leo,

Here is the post where I talked about my solution: https://groups.google.com/d/msg/openresty-en/feJePTfPC0M/XMLIT22tMgAJ

And here is the github link: https://github.com/JakSprats/lua-nginx-module

Most of the code is in here: https://github.com/JakSprats/lua-nginx-module/blob/master/src/ngx_http_lua_unique_socket_per_worker.c

Here is a test running the code: https://github.com/JakSprats/lua-nginx-module/tree/master/TEST

This last link gives a basic idea of how I set it all up. My solution needed a separate server in nginx's config that was later overridden to be unique per worker.

It was a hack, but it worked for my purposes.

Not sure you can learn anything from it :)

- Russ

On Tuesday, March 7, 2017 at 2:06:09 PM UTC-8, Leo Ponomarev wrote:

I've been using a variant of this IPC system since about 2009 (as part
of the nginx http push module, now Nchan), so it's rather well
time-tested already. As I recall, it took about a week of digging
through the Nginx guts to figure out a proper IPC mechanism. (I used the
master->worker signal forwarding system as a template to build from.)

One small correction - the sockets are not in shared memory, but in
copy-on-write memory inherited from the master process. worker PID and
slot info is what's in shared memory. (This made reloading logic a bit
simpler)

There are 3 minor catches so far: the alert name is limited to 64K, the
data to 4T, and the alert handler runs in an ngx.timer context. That's
about all.

Russell -- incidentally, I'd be interested to see your code for your IPC
system.

Russell Sullivan:
> Hi Leo,
>
> You got the architecture right: non-blocking Unix Domain Sockets, worker to
> socket mapping in SharedMemory, and the receive-coroutine is correct too.
>
> My implementation did the first two, but the receive logic was done by
> hacking the nginx core and using nginx's internals, it was a hack.
>
> I did not know there was an easy way to add a listening socket to nginx,
> but it looks like you found one :) I did a lot of ugly stuff, only took 2
> days, and it shows :)
>
> The reason I need this is I shard requests to sticky workers. The simplest
> analogy is all requests for users [A-H] go to worker0, users [I-P] go to
> worker1, users[Q-Z] go to worker3. I do it for data linearizability using a
> LMDB backend, LMDB runs in SharedMemory but I do multiple key updates that
> need to be serialized, so I pushed that requirement into the request
> receive layer.
>
> Your implementation looks cleaner than mine, so I will probably end up
> using it if you maintain it.
>
> Are there any gotchas? It doesn't look like it, the architecture is done
> correctly.
>
> Again, very nice work :)
>
> - Russ
>
> BTW: nice writeup with the numbered links, helps understand the code and
> implementation
>
>
> On Monday, March 6, 2017 at 6:47:49 PM UTC-8, Leo Ponomarev wrote:
>>
>> Here's how it works:
>>
>> Let's say we have n workers. At master process initialization
>> (postconfig), n (unix) sockets are created with the same logic that
>> initializes worker slots [1]. Then, when each worker starts, it adds a
>> listen event on the socket in its worker slot to the nginx event tree
>> [2], and stores its process id associated with the worker slot in shared
>> memory [3].
>>
>> Then, to send an ipc alert to a target worker by process id, look up its
>> slot in shared memory, then send data to the associated socket [4]. Add
>> some send [5] and receive [6] buffering, and voila.
>>
>> Importing the data into a valid Lua thread was a little tricky. When a
>> user calls ipc.receive() in the config, the module starts a (very long)
>> timer with a unique timeout using ngx.timer.at() [7]. This timer is then
>> yanked off the timer tree (by finding it using the unique timeout) [8],
>> and registered with the lua ipc module. Then, when an alert arrives [9],
>> this timer event is deleted from the timer tree, and re-inserted with
>> timeout 0 [10]. Once the incoming events are processed, the timer
>> re-inserts itself [11] with a unique timeout again, and the process
>> repeats.
>>
>> [1] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L49-L115
>> [2] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L229-L269
>> [3]
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L385-L394
>> [4]
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L239-L254
>> [5] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L186-L227
>> [6] https://github.com/slact/ngx_lua_ipc/blob/master/src/ipc.c#L271-L477
>> [7]
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/lua/register_receive_handler.lua
>> [8]
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L194-L237
>> [9]
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L269-L327
>> [10]
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/ngx_lua_ipc.c#L313-L317
>> [11]
>>
>> https://github.com/slact/ngx_lua_ipc/blob/master/src/lua/register_receive_handler.lua#L19-L20
>>
>>
>>
>>
>> Russell Sullivan:
>>>
>>> This is very useful. I did an IPC module for nginx workers last year but
>>> have not been maintaining it very well.
>>>
>>> Can you tell us more about how you implemented it. How does IPC work
>> under
>>> the hood (TCP,UDP,SM)? I had to make some changes to the nginx core for
>>> mine to work, you didnt, now I am very interested in the internals.
>>>
>>> Nice work :)
>>>
>>> On Saturday, March 4, 2017 at 9:16:28 PM UTC-8, Leo Ponomarev wrote:
>>>>
>>>> Did a few benchmarks, the results are really good for high concurrency:
>>>>
>>>> Test case: Broadcasting from 1 worker, followed by many K rounds of
>>>> back-and-forth communication between the initial broadcaster and all
>> the
>>>> other workers. Done on an elderly i5-2450M 2.50GHz (2 core, 4 thread)
>>>> CPU. Nginx compiled with -O3.
>>>>
>>>> Time to do 50K round-trips between broadcaster and all workers (100K
>> ipc
>>>> alerts total):
>>>>
>>>> 4 workers: 18.32 sec
>>>> 10 workers: 6.15 sec
>>>> 20 workers: 3.00 sec
>>>> 30 workers: 1.92 sec
>>>> 40 workers: 1.49 sec
>>>> 50 workers: 1.22 sec
>>>> 60 workers: 1.05 sec
>>>> 70 workers: 0.94 sec
>>>> 100 workers: 0.83 sec
>>>> 150 workers: 0.81 sec
>>>>
>>>> So at crazy concurrency, it can do 120K alerts/sec on this machine.
>>>>
>>>> Robert Paprocki:
>>>>> On Fri, Mar 3, 2017 at 12:27 PM, Leo Ponomarev <l...@nchan.io
>>>> <_javascript_:>> wrote:
>>>>>
>>>>>> Interprocess communication for lua_nginx_module and Openresty. Send
>>>>>> named alerts with string data between Nginx worker processes.
>>>>>> Asynchronous, nonblocking, non-locking.
>>>>>>
>>>>>> Very simple to use:
>>>>>>
>>>>>> local ipc = require "ngx.ipc"
>>>>>>
>>>>>> ipc.send(destination_worker_pid, ipc_alert_name, data_string)
>>>>>>
>>>>>> ipc.receive(ipc_alert_name, function(data)
>>>>>> --ipc receiver function for all alerts with string name
>>>> ipc_alert_name
>>>>>> print("received alert from " .. ipc.sender)
>>>>>> ipc.reply("you said " .. data)
>>>>>> end)
>>>>>>
>>>>>>
>>>>>> https://github.com/slact/ngx_lua_ipc
>>>>>
>>>>>
>>>>> Way cool! Have you done any stress testing? How does this hold up in
>>>>> high-concurrency environments?
>>>>>
>>>>
>>>>
>>
>>

erik.l.nelson

I was hoping to use bungle’s lua-resty-session to add CSRF protection to a rails app. My configuration is pretty simple, like

location / {

proxy_buffering off;

proxy_pass $scheme://127.0.0.1:8080;

}

How would I use lua-resty-session in this case? Is it even possible, or is there a simpler way to add csrf protection? I poked about on the net and didn’t find much.

Thank you

Erik

This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.