Using ngx_srcache for responses exceeding 1MB (Was Re: Thank You Your for Your Program)

agentzh · 2012-03-21T17:32:34+00:00

Hi agentzh, sorry for delay. Today I released ngx_http_redis-0.3.6. <ChangeLog> *) Feature: redis_gzip_flag. Usefull if you are prefer to store...

Using ngx_srcache for responses exceeding 1MB (Was Re: Thank You Your for Your Program)

agentzh

On Mon, Mar 19, 2012 at 3:34 PM, chaeplin <chaeplin@gmail.com> wrote:
> Anyhow the site has over 1MB files. So I check how to do that.
> I have studied to join srcache and Redis2 module.
>
> 1) === use $request_body as Redis vlaue
>         : result : value truncated
>         : error log : srcache_fetch: cache sent invalid status line while
> sending to client
>         : need more debug.
>

I'll work out a working ngx_srcache + ngx_redis/ngx_redis2 example for
you to try in the next few days or so :)

> 2) ===  Inclease memcached object limit to over 1MB
>        - someone say there will be performance degration.
>

I haven't tried this myself and I am not sure about the statistical
distribution of your page sizes though.

> Which one is best way for over 1MB cache ? just use disk?
>

It depends on concrete application characteristics. So the best way to
decide is to test them on your side.

>
> FYI: Currentely I have used plain cookie to skip srcache yet. and
> use header_filter_by_lua to skip.
> !-------------------------------------------------
> header_filter_by_lua '
>         if (ngx.header["X-Authed-Site"] == "ONOFFMIX") then
> ngx.var.skip = 1
>         end
> ';
>

That's fine :)

P.S. I've cc'd the openresty mailing group such that the whole team
(as well as others) can see our conversation here :)

Best,
-agentzh

agentzh

On Wed, Mar 21, 2012 at 5:32 PM, agentzh <age...@gmail.com> wrote:
>
> I'll work out a working ngx_srcache + ngx_redis/ngx_redis2 example for
> you to try in the next few days or so :)
>

Here is a working example that has been tested carefully on my side:

location /foo {
default_type text/css;

set $key $uri;
set_escape_uri $escaped_key $key;

srcache_fetch GET /redis $key;
srcache_store POST /redis2 key=$escaped_key&exptime=120;

echo_duplicate 2097152 a;
}

location = /redis {
internal;

set $redis_key $args;
redis_pass 127.0.0.1:6379;
}

location = /redis2 {
internal;

set_unescape_uri $exptime $arg_exptime;
set_unescape_uri $key $arg_key;

redis2_query set $key $echo_request_body;
redis2_query expire $key $exptime;
redis2_pass 127.0.0.1:6379;
}

This example uses ngx_echo's echo_duplicate directive to emit 2MB's data (2097152 a's).

This example makes use of the $echo_request_body variable provided by the ngx_echo module. Note that you need the latest version of ngx_echo, v0.38rc2:

https://github.com/agentzh/echo-nginx-module/tags

Earlier versions of ngx_echo will not work reliably.

Also, you need both the ngx_redis and ngx_redis2 modules here:

http://wiki.nginx.org/HttpRedisModule
http://wiki.nginx.org/HttpRedis2Module

Sergey A. Osokin's ngx_redis module's current latest version, 0.3.5, has a bug that could cause problems. A patch is provided here:

https://groups.google.com/group/openresty/browse_thread/thread/b5ddf1b24d3c9677

Finally, the Nginx core also has a bug that could prevent ngx_redis2's pipelining support from working properly in certain extreme conditions. And the following patch fixes this:

http://mailman.nginx.org/pipermail/nginx-devel/2012-March/002040.html

Note that, if you're using the ngx_openresty 1.0.11.27 bundle or later, then you only need to prepare Sergey A. Osokin's ngx_redis module and the patch mentioned above, because other patches and components are already applied in the bundle properly but the ngx_redis module is not included in the bundle yet.

If you have any other issues, please let us know :)

Enjoy!
-agentzh