using gzip with srcache-nginx-module + memc-nginx-module

jim.robinson · 2015-04-23T14:19:42+00:00

Hello! On Fri, May 8, 2015 at 2:57 AM, Aapo Talvensaari wrote: > So the cache gets also updates for going on request if there is a new > request comin...

using gzip with srcache-nginx-module + memc-nginx-module

jim.robinson

Hi folks,

The documentation for srcache-nginx-module outlines the command srcache_ignore_content_encoding, and explains how, by default, srcache will skip storing a response if the Content-Encoding header is present. That section ends with this warning:

It's recommended to always disable gzip/deflate compression on your backend server by specifying the following line in your nginx.conf file:

proxy_set_header Accept-Encoding "";

I'd like to understand why this advice is being given. I'm interested in using gzip to store the compressed version of responses in memcached. I was thinking that, since the implementation stores and returns http headers, that it'd make sense to set Accept-Encoding when talking to the origin and whatever was compressed by the origin would stay compressed when added to memcached. I thought the nginx modules for gunzip could then handle uncompressing the data if necessary.

Given the warning discussed above, I'm thinking I'm missing something important in that logic, and was hoping someone could explain the problem?

Here's my test configuration, it seemed to be doing what I wanted (data fetched from the origin that was compressed stayed compressed in memcache, and I could get it compressed or uncompressed as needed when I queried nginx using Accept-Encoding:gzip or not).

worker_processes 1;

error_log logs/error.log;

events {

worker_connections 1024;

}

http {

include mime.types;

default_type application/octet-stream;

log_format main '$remote_addr - $remote_user [$time_local] "$request" '

access_log logs/access.log main;

gzip on;

gzip_comp_level 9;

gzip_proxied any;

gzip_types application/atom+xml application/rdf+xml application/x-_javascript_ application/xhtml+xml application/xml text/css text/_javascript_ text/plain text/xml;

gunzip on;

upstream memcached_pool_01 {

server 127.0.0.1:11211;

keepalive 10;

}

server {

listen 80;

server_name localhost;

charset utf-8;

access_log logs/host.access.log main;

location /memc {

internal;

memc_connect_timeout 100ms;

memc_send_timeout 100ms;

memc_read_timeout 100ms;

memc_ignore_client_abort on;

set $memc_key $query_string;

memc_pass memcached_pool_01;

}

location / {

set $key $uri$args;

srcache_fetch GET /memc $key;

srcache_store PUT /memc $key;

srcache_store_statuses 200 301 302;

srcache_ignore_content_encoding on;

proxy_set_header Accept-Encoding "gzip;q=1.0, identity; q=0.5, *;q=0";

proxy_pass http://origin.example.org:80;

}

agentzh

Hello!

On Thu, Apr 23, 2015 at 2:19 PM, Jim Robinson wrote:
> I'd like to understand why this advice is being given.  I'm interested in
> using gzip to store the compressed version of responses in memcached.

This is because one must be very careful about clients supporting gzip
compression *and* clients not supporting gzip compression.

If you only store the gzip'd contents in the cache, then when a client
not supporting gzip accesses your server, it will panic.

One solution is to automatically decompress the compressed data for
such weak clients by means of something like the ngx_gunzip module:

http://nginx.org/en/docs/http/ngx_http_gunzip_module.html

Another solution is to store both the compressed version and
decompressed version in the cache by differentiating the cache keys.

Well, just my 2 cents.

Regards,
-agentzh

jimr

Great, thank you for confirming that adding the gunzip module would address the concerns raised in the documentation.

On Thu, May 7, 2015 at 5:16 AM Yichun Zhang (agentzh) <age...@gmail.com> wrote:

Hello!

On Thu, Apr 23, 2015 at 2:19 PM, Jim Robinson wrote:
> I'd like to understand why this advice is being given. I'm interested in
> using gzip to store the compressed version of responses in memcached.

This is because one must be very careful about clients supporting gzip
compression *and* clients not supporting gzip compression.

If you only store the gzip'd contents in the cache, then when a client
not supporting gzip accesses your server, it will panic.

One solution is to automatically decompress the compressed data for
such weak clients by means of something like the ngx_gunzip module:

http://nginx.org/en/docs/http/ngx_http_gunzip_module.html

Another solution is to store both the compressed version and
decompressed version in the cache by differentiating the cache keys.

Well, just my 2 cents.

Regards,
-agentzh.

agentzh

Hello!

On Thu, May 7, 2015 at 9:38 PM, James A. Robinson wrote:
> Great, thank you for confirming that adding the gunzip module would address
> the concerns raised in the documentation.
>

Well, I've never tried it out myself, you still need to test it to be sure :)

Good luck!

Best regards,
-agentzh

jim.robinson

Yes, per my original post it seems to work just fine.

On Thursday, May 7, 2015 at 6:42:13 AM UTC-7, agentzh wrote:

Hello!

On Thu, May 7, 2015 at 9:38 PM, James A. Robinson wrote:
> Great, thank you for confirming that adding the gunzip module would address
> the concerns raised in the documentation.
>

Well, I've never tried it out myself, you still need to test it to be sure :)

Good luck!

Best regards,
-agentzh