nginx cache in front of a location served by 'content_by_lua'

grvsaxena419 · 2017-06-07T09:19:27+00:00

What about updating your `lua_package_path` to:lua_package_path "/usr/local/share/lua/5.1/?.lua;/usr/local/openresty/lualib/?.lua;/usr/local/openresty/n...

nginx cache in front of a location served by 'content_by_lua'

grvsaxena419

Hi,

I am doing some calculation in a location that generates content using 'content_by_lua'. I would like this calculation to be cached for a period of time. I have tried couple things but none of those work. I would appreciate any insight you have to make this work.

proxy_cache_path  /tmp/cache levels=1:2 keys_zone=my_cache:8m max_size=30m inactive=2m;

server {
  listen 9001;

  location /test {
    proxy_cache my_cache;  
    proxy_cache_valid 200 2m;
    proxy_cache_min_uses 1;
    proxy_cache_key "fooo";
    proxy_ignore_headers Expires Cache-Control;


    content_by_lua '
      local res = ngx.location.capture("/main")
      ngx.status = res.status
      ngx.print(res.body)
      ';


    more_set_headers "X-Cache-Status: $upstream_cache_status";
  }

  location /main {
    proxy_cache my_cache;  # this line is crucial or the cache will be disabled!
    proxy_cache_valid 200 2m;
    proxy_cache_min_uses 1;
    proxy_cache_key "roooo";
    proxy_ignore_headers Expires Cache-Control;

    content_by_lua '
      local res = ngx.location.capture("/index.html")
      ngx.status = res.status
      for key, value in pairs(res.header) do
        ngx.log (ngx.NOTICE, "copying header key:"..key.." value="..value)
        ngx.header[key] = value
      end
      ngx.say("time :", ngx.now())
      ngx.say("cache-status: ", res.header["X-Cache-Status"])
      ngx.print(res.body)
      ';
    expires 2m;
  }

  location /index.html {
    return 200 "hoooofooookoooo";
  }
}

Both /main and /test are not doing any caching and for every request I can see the updated time. I am not sure what I am doing wrong.

rpaprocki

Hi,

On Tue, Jun 6, 2017 at 6:19 PM, Gaurav Saxena <grvsa...@gmail.com> wrote:

Hi,

proxy_cache_path  /tmp/cache levels=1:2 keys_zone=my_cache:8m max_size=30m inactive=2m;

server {
  listen 9001;

  location /test {
    proxy_cache my_cache;  
    proxy_cache_valid 200 2m;
    proxy_cache_min_uses 1;
    proxy_cache_key "fooo";
    proxy_ignore_headers Expires Cache-Control;


    content_by_lua '
      local res = ngx.location.capture("/main")
      ngx.status = res.status
      ngx.print(res.body)
      ';


    more_set_headers "X-Cache-Status: $upstream_cache_status";
  }

  location /main {
    proxy_cache my_cache;  # this line is crucial or the cache will be disabled!
    proxy_cache_valid 200 2m;
    proxy_cache_min_uses 1;
    proxy_cache_key "roooo";
    proxy_ignore_headers Expires Cache-Control;

    content_by_lua '
      local res = ngx.location.capture("/index.html")
      ngx.status = res.status
      for key, value in pairs(res.header) do
        ngx.log (ngx.NOTICE, "copying header key:"..key.." value="..value)
        ngx.header[key] = value
      end
      ngx.say("time :", ngx.now())
      ngx.say("cache-status: ", res.header["X-Cache-Status"])
      ngx.print(res.body)
      ';
    expires 2m;
  }

  location /index.html {
    return 200 "hoooofooookoooo";
  }
}

Both /main and /test are not doing any caching and for every request I can see the updated time. I am not sure what I am doing wrong.

proxy_cache implies that the location is being served by proxy_pass. Content generated by a handler other than proxy_pass (e.g. content_by_lua) is not cached by proxy pass, and therefore needs to be cached manually via some other mechanism.

.

grvsaxena419

Thanks a lot for the quick reply.

Do you know of any examples for custom caching in lua?

On Tuesday, June 6, 2017 at 6:20:53 PM UTC-7, rpaprocki wrote:

Hi,

On Tue, Jun 6, 2017 at 6:19 PM, Gaurav Saxena <grvsax...@gmail.com> wrote:

Hi,

proxy_cache_path  /tmp/cache levels=1:2 keys_zone=my_cache:8m max_size=30m inactive=2m;

server {
  listen 9001;

  location /test {
    proxy_cache my_cache;  
    proxy_cache_valid 200 2m;
    proxy_cache_min_uses 1;
    proxy_cache_key "fooo";
    proxy_ignore_headers Expires Cache-Control;


    content_by_lua '
      local res = ngx.location.capture("/main")
      ngx.status = res.status
      ngx.print(res.body)
      ';


    more_set_headers "X-Cache-Status: $upstream_cache_status";
  }

  location /main {
    proxy_cache my_cache;  # this line is crucial or the cache will be disabled!
    proxy_cache_valid 200 2m;
    proxy_cache_min_uses 1;
    proxy_cache_key "roooo";
    proxy_ignore_headers Expires Cache-Control;

    content_by_lua '
      local res = ngx.location.capture("/index.html")
      ngx.status = res.status
      for key, value in pairs(res.header) do
        ngx.log (ngx.NOTICE, "copying header key:"..key.." value="..value)
        ngx.header[key] = value
      end
      ngx.say("time :", ngx.now())
      ngx.say("cache-status: ", res.header["X-Cache-Status"])
      ngx.print(res.body)
      ';
    expires 2m;
  }

  location /index.html {
    return 200 "hoooofooookoooo";
  }
}

Both /main and /test are not doing any caching and for every request I can see the updated time. I am not sure what I am doing wrong.

.

rpaprocki

Hey Gaurav,

On Tue, Jun 6, 2017 at 9:23 PM, Gaurav Saxena <grvsa...@gmail.com> wrote:

Thanks a lot for the quick reply.
Do you know of any examples for custom caching in lua?

I don't think there are any boilerplate examples of caching manually generated content, but it shouldn't be too hard to build your own, as long as you know how to key your data. Have a look at a few community libs as reference examples:

https://github.com/openresty/lua-nginx-module#ngxshareddict

https://github.com/openresty/lua-resty-lrucache

https://github.com/hamishforbes/lua-resty-tlc
https://github.com/thibaultcha/lua-resty-mlcache

Basically you just need to figure out how you want to cache that data (e.g. the cache key), then store that data in the appropriate medium (per-worker LRU, shared dict, remote network server, etc), and generate your content when the cache lookup misses. Good luck with it!

grvsaxena419

Thanks a lot, Robert. Appreciate the detailed pointers :).

Thanks and Regards ,
Gaurav

On Wed, Jun 7, 2017 at 11:33 AM, Robert Paprocki <rpap...@fearnothingproductions.net> wrote:

Hey Gaurav,

On Tue, Jun 6, 2017 at 9:23 PM, Gaurav Saxena <grvsa...@gmail.com> wrote:
Thanks a lot for the quick reply.
Do you know of any examples for custom caching in lua?

I don't think there are any boilerplate examples of caching manually generated content, but it shouldn't be too hard to build your own, as long as you know how to key your data. Have a look at a few community libs as reference examples:

https://github.com/openresty/lua-nginx-module#ngxshareddict
https://github.com/openresty/lua-resty-lrucache
https://github.com/hamishforbes/lua-resty-tlc
https://github.com/thibaultcha/lua-resty-mlcache

Basically you just need to figure out how you want to cache that data (e.g. the cache key), then store that data in the appropriate medium (per-worker LRU, shared dict, remote network server, etc), and generate your content when the cache lookup misses. Good luck with it!

zchao1995

Let's suppose that you have two location A and B, in the location A, you can set some directives about proxy_cache and proxy_pass to location B, the location B is where you can use content_by_lua generating contents.

Frankly, i don't think this is a good solution. :(

On Wednesday, June 7, 2017 at 9:19:27 AM UTC+8, Gaurav Saxena wrote:

Hi,

proxy_cache_path  /tmp/cache levels=1:2 keys_zone=my_cache:8m max_size=30m inactive=2m;

server {
  listen 9001;

  location /test {
    proxy_cache my_cache;  
    proxy_cache_valid 200 2m;
    proxy_cache_min_uses 1;
    proxy_cache_key "fooo";
    proxy_ignore_headers Expires Cache-Control;


    content_by_lua '
      local res = ngx.location.capture("/main")
      ngx.status = res.status
      ngx.print(res.body)
      ';


    more_set_headers "X-Cache-Status: $upstream_cache_status";
  }

  location /main {
    proxy_cache my_cache;  # this line is crucial or the cache will be disabled!
    proxy_cache_valid 200 2m;
    proxy_cache_min_uses 1;
    proxy_cache_key "roooo";
    proxy_ignore_headers Expires Cache-Control;

    content_by_lua '
      local res = ngx.location.capture("/index.html")
      ngx.status = res.status
      for key, value in pairs(res.header) do
        ngx.log (ngx.NOTICE, "copying header key:"..key.." value="..value)
        ngx.header[key] = value
      end
      ngx.say("time :", ngx.now())
      ngx.say("cache-status: ", res.header["X-Cache-Status"])
      ngx.print(res.body)
      ';
    expires 2m;
  }

  location /index.html {
    return 200 "hoooofooookoooo";
  }
}

Both /main and /test are not doing any caching and for every request I can see the updated time. I am not sure what I am doing wrong.

grvsaxena419

Thanks. Is it possible to do a proxy_pass to a different location? I tried proxy_pass to a different location and that didn't work.

On Wednesday, June 7, 2017 at 11:11:14 PM UTC-7, tokers wrote:

Hi

Let's suppose that you have two location A and B, in the location A, you can set some directives about proxy_cache and proxy_pass to location B, the location B is where you can use content_by_lua generating contents.
Frankly, i don't think this is a good solution. :(
On Wednesday, June 7, 2017 at 9:19:27 AM UTC+8, Gaurav Saxena wrote:
Hi,

I am doing some calculation in a location that generates content using 'content_by_lua'. I would like this calculation to be cached for a period of time. I have tried couple things but none of those work. I would appreciate any insight you have to make this work.
proxy_cache_path  /tmp/cache levels=1:2 keys_zone=my_cache:8m max_size=30m inactive=2m;
server {
  listen 9001;

  location /test {
    proxy_cache my_cache;  
    proxy_cache_valid 200 2m;
    proxy_cache_min_uses 1;
    proxy_cache_key "fooo";
    proxy_ignore_headers Expires Cache-Control;
    content_by_lua '
      local res = ngx.location.capture("/main")
      ngx.status = res.status
      ngx.print(res.body)
      ';
    more_set_headers "X-Cache-Status: $upstream_cache_status";
  }

  location /main {
    proxy_cache my_cache;  # this line is crucial or the cache will be disabled!
    proxy_cache_valid 200 2m;
    proxy_cache_min_uses 1;
    proxy_cache_key "roooo";
    proxy_ignore_headers Expires Cache-Control;

    content_by_lua '
      local res = ngx.location.capture("/index.html")
      ngx.status = res.status
      for key, value in pairs(res.header) do
        ngx.log (ngx.NOTICE, "copying header key:"..key.." value="..value)
        ngx.header[key] = value
      end
      ngx.say("time :", ngx.now())
      ngx.say("cache-status: ", res.header["X-Cache-Status"])
      ngx.print(res.body)
      ';
    expires 2m;
  }

  location /index.html {
    return 200 "hoooofooookoooo";
  }
}
Both /main and /test are not doing any caching and for every request I can see the updated time. I am not sure what I am doing wrong. 