openresty, redis: application configuration files question

sparvu · 2014-09-04T23:21:20+00:00

Hello! On Thu, Sep 4, 2014 at 11:35 PM, muteblinddeaf wrote: > How secure is this installation by default? It's good. > Is there any suggestions fo...

openresty, redis: application configuration files question

sparvu

Hi,

Im trying to understand how can I read some json files and map some needed settings into
Redis, during application start. For example, our system uses the following application configurations files:

 users.json (keeps some user configuration information)
 objects.json (keeps information about monitoring objects

and during our application boot, meaning OpenResty is up and running we want to
read some information from these files and cache them in Redis that all the other coming
requests will find all needed settings, prepared. Sort of reading app config files
phase.

Now, can anyone comment if this approach is ok or how can we handle this
part, since Redis has no idea about filesystem or files:

 - during first request, read the files and extract the needed information in Lua
   OpenResty and map that to Redis

 - have a Redis dump (RDB), which can be imported into Redis everytime we start Redis ?
   this means we could process everything on the redis side ?

 - anything else ?

Any pointers or ideas ?

Thanks a lot,

-- 
Stefan Parvu <sp...@systemdatarecorder.org>

hamish.forbes

Why does it need to be in Redis?

It seems like the easiest option would be to read the config files during init_by_lua and save it into a module level variable that gets shared across all workers.

If it *has* to be in Redis for some reason then you could use ngx.timer.at in init_worker_by_lua to read the files and save them into Redis immediately after startup.

On Thursday, 4 September 2014 16:21:21 UTC+1, Stefan Parvu wrote:

Hi,

Im trying to understand how can I read some json files and map some needed settings into
Redis, during application start. For example, our system uses the following application configurations files:

users.json (keeps some user configuration information)
objects.json (keeps information about monitoring objects

and during our application boot, meaning OpenResty is up and running we want to
read some information from these files and cache them in Redis that all the other coming
requests will find all needed settings, prepared. Sort of reading app config files
phase.

Now, can anyone comment if this approach is ok or how can we handle this
part, since Redis has no idea about filesystem or files:

- during first request, read the files and extract the needed information in Lua
OpenResty and map that to Redis

- have a Redis dump (RDB), which can be imported into Redis everytime we start Redis ?
this means we could process everything on the redis side ?

- anything else ?

Any pointers or ideas ?

Thanks a lot,

--
Stefan Parvu <spa...@systemdatarecorder.org>

sparvu

On Thu, 4 Sep 2014 11:28:31 -0700 (PDT)
Hamish Forbes <hamish...@gmail.com> wrote:

> Why does it need to be in Redis?
> It seems like the easiest option would be to read the config files during 
> init_by_lua and save it into a module level variable that gets shared 
> across all workers.

We populate some data structures in Redis and store user information within. 
Initial settings I was thinking can be mapped similar way in Redis. I need to check
if we really need to map the settings within Redis or not.

I see, if we go via OpenResty we could use init_by_lua + variable. For Redis we could
use ngx.timer.at ... 

We need to experiment with these. 

Thanks a lot,

-- 
Stefan Parvu <sp...@systemdatarecorder.org>

bogdan

Thank you

Me and Stefan are working together to this project.
If we would need to have the data in redis, why should we use "init_worker_by_lua+timer", and not directly load the data in redis during "init_by_lua"? Are there any restrictions regarding the code that can be run in "init_by_lua"?
Also, "init_worker_by_lua" runs for each worker, so we would need a semaphore to load the data in redis only once. Correct?

Thank you

Bogdan

Hamish Forbes

Thursday, September 04, 2014 9:28 PM

Why does it need to be in Redis?
It seems like the easiest option would be to read the config files during init_by_lua and save it into a module level variable that gets shared across all workers.

If it *has* to be in Redis for some reason then you could use ngx.timer.at in init_worker_by_lua to read the files and save them into Redis immediately after startup.

On Thursday, 4 September 2014 16:21:21 UTC+1, Stefan Parvu wrote:

- during first request, read the files and extract the needed information in Lua
OpenResty and map that to Redis

- have a Redis dump (RDB), which can be imported into Redis everytime we start Redis ?
this means we could process everything on the redis side ?

- anything else ?

Any pointers or ideas ?

Thanks a lot,

hamish.forbes

Yep you can't use tcp sockets during init_by_lua or init_worker_by_lua and you can't use ngx.timer.at in init_by_lua.

Take a look at the 'context' sections in the docs:

https://github.com/openresty/lua-nginx-module#ngxsockettcp

https://github.com/openresty/lua-nginx-module#ngxtimerat

So instead, you trigger a timer with 0 delay in init_worker, within that timer callback you can use tcp connections (e.g. talk to redis).

Depending on how long it takes to read the config file, process it and write to Redis you will have a short window where requests could be served before you've finished writing to Redis.

Yes you'll need to lock, using a shared dictionary, to avoid multiple workers writing into Redis.

Have a look at this: https://gist.github.com/hamishforbes/1bdc679893eabbbab3bd

On Friday, 5 September 2014 09:44:33 UTC+1, bogdan wrote:

Thank you

Me and Stefan are working together to this project.
If we would need to have the data in redis, why should we use "init_worker_by_lua+timer", and not directly load the data in redis during "init_by_lua"? Are there any restrictions regarding the code that can be run in "init_by_lua"?
Also, "init_worker_by_lua" runs for each worker, so we would need a semaphore to load the data in redis only once. Correct?

Thank you

Bogdan

Hamish Forbes

Thursday, September 04, 2014 9:28 PM

Why does it need to be in Redis?
It seems like the easiest option would be to read the config files during init_by_lua and save it into a module level variable that gets shared across all workers.

If it *has* to be in Redis for some reason then you could use ngx.timer.at in init_worker_by_lua to read the files and save them into Redis immediately after startup.

On Thursday, 4 September 2014 16:21:21 UTC+1, Stefan Parvu wrote:

- during first request, read the files and extract the needed information in Lua
OpenResty and map that to Redis

- have a Redis dump (RDB), which can be imported into Redis everytime we start Redis ?
this means we could process everything on the redis side ?

- anything else ?

Any pointers or ideas ?

Thanks a lot,

bogdan

Yep, everything is clear now.
Thank you very much.

Bogdan

Hamish Forbes wrote:

Yep you can't use tcp sockets during init_by_lua or init_worker_by_lua
and you can't use ngx.timer.at in init_by_lua.
Take a look at the 'context' sections in the docs:
https://github.com/openresty/lua-nginx-module#ngxsockettcp
https://github.com/openresty/lua-nginx-module#ngxtimerat

So instead, you trigger a timer with 0 delay in init_worker, within
that timer callback you can use tcp connections (e.g. talk to redis).
Depending on how long it takes to read the config file, process it and
write to Redis you will have a short window where requests could be
served before you've finished writing to Redis.

Yes you'll need to lock, using a shared dictionary, to avoid multiple
workers writing into Redi s.
Have a look at this:
https://gist.github.com/hamishforbes/1bdc679893eabbbab3bd

On Friday, 5 September 2014 09:44:33 UTC+1, bogdan wrote:

    Thank you

    Me and Stefan are working together to this project.
    If we would need to have the data in redis, why should we use
    "init_worker_by_lua+timer", and not directly load the data in
    redis during "init_by_lua"? Are there any restrictions regarding
    the code that can be run in "init_by_lua"?
    Also, "init_worker_by_lua" runs for each worker, so we would need
    a semaphore to load the data in redis only once. Correct?

    Thank you

    Bogdan

    Hamish Forbes <_javascript_:>
    Thursday, September 04, 2014 9:28 PM
    Why does it need to be in Redis?
    It seems like the easiest option would be to read the config
    files during init_by_lua and save it into a module level variable
    that gets shared across all workers.

    If it *has* to be in Redis for some reason then you could use
    ngx.timer.at <http://ngx.timer.at> in init_worker_by_lua to read
    the files and save them into Redis immediately after startup.

    On Thursday, 4 September 2014 16:21:21 UTC+1, Stefan Parvu wrote:
    --
   .