On Tuesday, 3 November 2015 16:28:56 UTC+2, bogdan wrote:
Ok, but how should I do that?
1. Say you have 4 workers that to counting of 8 different calculations or parameters
2. Each worker will need to process 2 parameters
Now on init_by_lua compile a list of parameters in SHM.
Then in init_worker_by_lua each worker will read 2 different parameters from that list. When this gets merger you can lpop in init_worker_by_lua, but lets not think about that now, let's think how you can do this now:
https://github.com/openresty/lua-nginx-module/pull/586
How to do this now:
One thing I'm not sure is that are the workers started serially or in parallel. Lets just say (I'm not sure if it is correct, that they are started in parallel so they run init_worker_by_lua concurrently, if not then this is even easier).
1. On init worker by lua start looping that list of parameters
2. Check if parameter is already processed and if it is, then skip it, if not
3. Lock against parameter name with 0 ms timeout so that only one worker will take that
4. Mark the parameter as processed
5. Unlock parameter
6. Break the loop when ceil(parameters / workers count) parameters have been taken by this worker
Now each worker has their own set of parameters. Just in case Nginx worker crashes
store the names in SHM with worker pid.
Now start a recurring timer for each parameter.
If crash happens, then before 1. You need to check if there are previously registered
parameters for this worker in SHM. If there is, then use those. And skip 2 - 6.
Now if workers are started serially (this can be easily tested), this is easy, just pop out
ceil(parameters / workers count) from that SHM list built on init_by_lua. If the list is empty
all the work has already been taken care of. Aka if you have 4 workers and 2 parameters,
then only 2 workers will have something to do with.
And that's about it. Now you have Nginx running as an event loop for these timers and the
timers are divided between workers.
