Non-blocking options for static permissions

marcos · 2015-01-12T15:28:59+00:00

Am 07.01.15 um 21:19 schrieb Yichun Zhang (agentzh): Hi, > Several obvious issues: thanks for your patience and pointing me in the right directions. I'...

Non-blocking options for static permissions

marcos

I'd like to create a static website that also has authorisation for certain content.

Authentication can be managed with federated login (E.g. login with Facebook/Google/GitHub etc.), and using the openresty Lua plugin to verify the token.

However, there's two pieces I'm still considering the options for:

1) Permissions. Just like the static content, I would like to generate static permissions somehow. My first thought was an SQLite DB, but Googling for that lead me to this forum. It seems doing that would adversely affect Nginx performance. I'm thus wondering about options that, preferably, don't require another 'moving piece' so to speak.

2) Also related to perms, but rather than check every resource for perms (only to find it's 'public'), I'd like to do it for any arbitrary URL/file. One way I thought was to use a special extension /index.html.secure, and use a regex on *.secure. Before 404ing, on /index.html, Nginx could pass on to the perms module to check if index.html.secure was available. Another would be to use UNIX file perms somehow. Does anyone have advice on what would be the most performant?

marcos

Just to add, the post I read on this was here: https://groups.google.com/forum/#!topic/openresty-en/lUH10YE_6Z8

Someone's attempted it here with a native module: https://github.com/kunal/sqlite-http-basic-auth-nginx-module

agentzh

Hello!

On Sun, Jan 11, 2015 at 11:28 PM, Marcos Scriven wrote:
> 1) Permissions. Just like the static content, I would like to generate
> static permissions somehow.

I'm not sure what do you exactly mean by "static permissions". If they
are truly static and not subject to change often, then you can just
generate a JSON file and load it via lua-cjson (or some else Lua JSON
library) in your Lua code (like loading in the context init_by_lua).

If these permission rules are subject to change often, then you can
just run a redis server (probably with persistence support enabled)
and use lua-resty-redis to talk to it nonblockingly.

> 2) Also related to perms, but rather than check every resource for perms
> (only to find it's 'public'), I'd like to do it for any arbitrary URL/file.
> One way I thought was to use a special extension /index.html.secure, and use
> a regex on *.secure. Before 404ing, on /index.html, Nginx could pass on to
> the perms module to check if index.html.secure was available.

It's much more efficient (and also much easier) to do the checking
upon every request in access_by_lua, for example. Please keep in mind
that file I/O syscalls are usually *much* more expensive compared to
pure memory computations in your Lua code.

Regards,
-agentzh

marcos

On Tuesday, January 13, 2015 at 6:43:25 AM UTC+11, agentzh wrote:

Hello!

Hi - and thanks for the quick and friendly response!

I'm not sure what do you exactly mean by "static permissions". If they
are truly static and not subject to change often, then you can just
generate a JSON file and load it via lua-cjson (or some else Lua JSON
library) in your Lua code (like loading in the context init_by_lua).

Well, by 'static permissions', I meant something you could just copy to a file, or check in to GitHub for instance, that would be generated at the same time as the content. Hence the SQLite idea, as that can be represented as a file.

If these permission rules are subject to change often, then you can
just run a redis server (probably with persistence support enabled)
and use lua-resty-redis to talk to it nonblockingly.

The idea is that the permissions *could* change, just like the 'static html', but not very often. Reading from a JSON file could work, so long as I could get it to read an updated JSON file every so often, without having to restart or HUP the Nginx process. I'd rather not rely on having a Redis setup - the idea is you can just checkout the latest version of the content, and serve it with Nginx. No app servers, no databases, just Nginx, Lua plugin, and the right config. Certainly you could the setup with Redis DBs, page view counts and so forth, but I'd rather that it worked flawlessly when any of those extra pieces were down or failing.

It's much more efficient (and also much easier) to do the checking
upon every request in access_by_lua, for example. Please keep in mind
that file I/O syscalls are usually *much* more expensive compared to
pure memory computations in your Lua code.

I note Lua can access shared Nginx mem via a dict. Would it be possible to have a very lightweight process that polled a JSON file (or whatever), and updated that in memory dict? Basically I want a way to update a file, and for that to update what 'access_by_lua' decided, in a simple, non-blocking way. Perhaps just a simple bash script fastcgi, that span up to re-read the JSON file every so often?

agentzh

Hello!

On Mon, Jan 12, 2015 at 1:07 PM, Marcos Scriven wrote:
> The idea is that the permissions *could* change, just like the 'static
> html', but not very often. Reading from a JSON file could work, so long as I
> could get it to read an updated JSON file every so often, without having to
> restart or HUP the Nginx process.

Easy: just create a re-occurring timer in one of your nginx workers to
check if the JSON file on disk has changed. A more efficient approach
is presented below.

> I note Lua can access shared Nginx mem via a dict. Would it be possible to
> have a very lightweight process that polled a JSON file (or whatever), and
> updated that in memory dict?

Yes, you can just expose an internal endpoint in your nginx by a
special "location = /update", for example, to trigger data updates in
the shdict. Any worker process serving this HTTP request can populate
the new data into the shm zone visible to all the workers.

No fastcgi backend needed, and the fastcgi thing is much slower.

Regards,
-agentzh

marcos

On Tuesday, January 13, 2015 at 8:53:31 AM UTC+11, agentzh wrote:

Yes, you can just expose an internal endpoint in your nginx by a
special "location = /update", for example, to trigger data updates in
the shdict. Any worker process serving this HTTP request can populate
the new data into the shm zone visible to all the workers.

Thanks - I went Googling for Nginx Lua timers, and found an answer by you on Stackoverflow: http://stackoverflow.com/questions/19035286/how-to-create-asynchronous-cron-like-scheduler-inside-nginx

Will try out this appeoach