Using nginx variables to store large data instead of memcached

alexl · 2016-05-03T22:29:17+00:00

Hi,On Fri, Mar 17, 2017 at 9:26 PM, chintans <chintansharrma@gmail.com> wrote:Hi Robert,I tried ngx.ctx for sharing value between phases, unfortunately...

Using nginx variables to store large data instead of memcached

alexl

Hey!

So, a weird question - if I store a rather large static string(~17kb json) in an nginx variable, will it be as fast as retreiving it from local memcached?

I will have one JSON per .conf file, but will probably eventually have thousands of .conf files, so a lot of JSONs to keep in memcached.

My reasons are - I would be required to dynamically generate a package of *.conf scripts, one script per web-site, and update them if they change. I will soon begin writing a service to sync those. At the same time I need to keep jsons updated in memcached on the same machine, to access them from lua.

This would probably be the same service, and so I thought, why not include the JSONs in the .conf files? Maybe save them to a text file and somehow include them in .conf, so it would get loaded once, upon nginx start?

Anyone done something like that?

luis.gasca

Take a look at confd[1]. Maybe it can fit your use-case. I am using it with consul. Your json variables are consul http api calls, with a shcache[2] to speed things up.

Hope it helps.

[1] https://github.com/kelseyhightower/confd
[2] https://github.com/mtourne/ngx.shcache/

El 03/05/2016 a las 16:29, al...@chameleonx.com escribió:

Hey!
So, a weird question - if I store a rather large static string(~17kb json) in an nginx variable, will it be as fast as retreiving it from local memcached?

I will have one JSON per .conf file, but will probably eventually have thousands of .conf files, so a lot of JSONs to keep in memcached.

My reasons are - I would be required to dynamically generate a package of *.conf scripts, one script per web-site, and update them if they change. I will soon begin writing a service to sync those. At the same time I need to keep jsons updated in memcached on the same machine, to access them from lua.

This would probably be the same service, and so I thought, why not include the JSONs in the .conf files? Maybe save them to a text file and somehow include them in .conf, so it would get loaded once, upon nginx start?

Anyone done something like that?
.

rpaprocki

You may also want to look at storing these as lua module members (see https://github.com/openresty/lua-nginx-module#data-sharing-within-an-nginx-worker), depending on how youre using your data - JSON strings translate nicely to lua tables :). This might be a bit easier to work with, and if you initialize your data as part of an init_by_lua handler you will enjoy COW memory benefits.

On Tue, May 3, 2016 at 7:29 AM, Luis Gasca <luis...@interactive3g.com> wrote:

Take a look at confd[1]. Maybe it can fit your use-case. I am using it with consul. Your json variables are consul http api calls, with a shcache[2] to speed things up.

Hope it helps.

[1] https://github.com/kelseyhightower/confd
[2] https://github.com/mtourne/ngx.shcache/

El 03/05/2016 a las 16:29, al...@chameleonx.com escribió:

Hey!
So, a weird question - if I store a rather large static string(~17kb json) in an nginx variable, will it be as fast as retreiving it from local memcached?

I will have one JSON per .conf file, but will probably eventually have thousands of .conf files, so a lot of JSONs to keep in memcached.

My reasons are - I would be required to dynamically generate a package of *.conf scripts, one script per web-site, and update them if they change. I will soon begin writing a service to sync those. At the same time I need to keep jsons updated in memcached on the same machine, to access them from lua.

This would probably be the same service, and so I thought, why not include the JSONs in the .conf files? Maybe save them to a text file and somehow include them in .conf, so it would get loaded once, upon nginx start?

Anyone done something like that?
.