Updating ngx.shared.dict from continuous loop

james_357f · 2014-05-28T03:04:05+00:00

Hello! On Tue, May 27, 2014 at 11:41 AM, arthurx wrote: > I am experimenting in modularizing some of my codes and have a problem about > variable scop...

Updating ngx.shared.dict from continuous loop

james_357f

I would like to be able to populate a shared dict from messages outside of an nginx request

It would be really cool to be able to update based on messages from a queue like zmq, potentially like https://github.com/imatix/zguide/blob/master/examples/Lua/psenvsub.lua. It could be accomplished via an intermediate program loop that received messages and then made an HTTP request to a url set to put the body into the dict, but I would rather do it more natively.

Is there a place where I could put a continuous loop that wouldn't be blocking for nginx and be able to update the shared dict? I could do it in a language other than lua (probably C) if that would make it possible.

brian

You could use a timer. I've had Lua in a timer subscribe to redis pub sub , for example.

AFAIK, you can't use zmq inside nginx because of the way event loops and threads are done - I could be wrong.

james_357f

Does that capture messages that were published in between timer runs?

brian

It depends on how you code your app. In my case, the job kicked off by the timer never exited - it was an infinite loop, basically.

james_357f

Oh, cool. That was basically my question I guess. I figured having an infinite loop inside an nginx process would make bad things happen.

brian

It yields on every network call that may block. Don't do a "real" infinite loop - ie, just doing math or something - or it will block the event loop.

Sent from my iPhone

On May 27, 2014, at 6:13 PM, James Marlowe <ja...@lumate.com> wrote:

Oh, cool. That was basically my question I guess. I figured having an infinite loop inside an nginx process would make bad things happen.

.

agentzh

Hello!

On Tue, May 27, 2014 at 3:05 PM, Brian Akins wrote:
> It depends on how you code your app. In my case, the job kicked off by the
> timer never exited - it was an infinite loop, basically.
>

Well, it is better to use a recurring timer than a true infinite loop
doing nonblocking I/O because currently we allocate some small memory
blocks (including the cosocket API) in the "request memory pool" (in
case of timers, they are "request pools" for fake requests), so you
may see some kind of memory leaks using the latter approach. Nginx
assumes that requests are usually short-lived so the request memory
pool design makes sense for true requests. I'd eventually prevent
allocating in the request memory pool in ngx_lua's Lua API wherever
possible but I'm not there yet.

Regards,
-agentzh

brian

I guess I've always been lucky ;). The actual use case was a bit more complicated than I described as the loop wasn't quite infinite, so to speak.

Sent from my iPhone

> On May 30, 2014, at 12:48 AM, "Yichun Zhang (agentzh)" <age...@gmail.com> wrote:
> 
> Hello!
> 
>> On Tue, May 27, 2014 at 3:05 PM, Brian Akins wrote:
>> It depends on how you code your app. In my case, the job kicked off by the
>> timer never exited - it was an infinite loop, basically.
> 
> Well, it is better to use a recurring timer than a true infinite loop
> doing nonblocking I/O because currently we allocate some small memory
> blocks (including the cosocket API) in the "request memory pool" (in
> case of timers, they are "request pools" for fake requests), so you
> may see some kind of memory leaks using the latter approach. Nginx
> assumes that requests are usually short-lived so the request memory
> pool design makes sense for true requests. I'd eventually prevent
> allocating in the request memory pool in ngx_lua's Lua API wherever
> possible but I'm not there yet.
> 
> Regards,
> -agentzh.