best practice to cache only valid POST responses, f.i. checked by reg ex

dasschwarzesystem · 2015-11-21T19:52:28+00:00

for anyone interested in how I solved this, I am attaching a first draft of a module that does both leader election and group membership using ngx.shared.DIC...

best practice to cache only valid POST responses, f.i. checked by reg ex

dasschwarzesystem

Hello!

I have a fine working NGINX setup to cache a web service with proxy_cache_methods set to POST.

The proxy_cache_key is set to "$request_uri|$request_body", because the responses are identical for equal requests about the same day.

So the cache is flushed every midnight.

The problem is, that I only want to cache "valid" responses. It's necessary to check the web service response before caching.

Rarely the web service responses are not valid because of special messages inside the response.

It is sufficient to detect these responses with simple regular expressions.

And these responses shouldn't be cached. If the client repeats the request, the response of the web service mostly is valid again within seconds.

But with my actual setup the not valid responses are served by the cache for the whole day.

Is it possible to solve this use case with Lua and what's the best practice?

I tried with body_filter_by_lua to detect not valid responses. That works fine, but the not valid response is already cached before the filter.

Thanks in advance,

Falk Berger

info

Apart from a hack into proxy_cache maybe you could store the invalid entry into shared memory, run a timer which checks shared memory for invalid entry markers and call each to invalidate the (proxy)cached one. Invalid cached responses should then have a lifetime of a few seconds.
Or just call proxycache again to expire(purge) the value.

perusio

I would proxy the request so that in the "origin" the response body is parsed and set an entry in a shared dict. Like a flag. Then use it to skip the cache by stting proxy_cache_bypass and proxy_no_cache to non 0.

Just store the responses that shouldn't be cached in the shared dict.

Just an idea.

Grüße,
Antonio

<i...@ecsystems.nl> schrieb am Sa., 21. Nov. 2015 21:37:

Apart from a hack into proxy_cache maybe you could store the invalid entry into shared memory, run a timer which checks shared memory for invalid entry markers and call each to invalidate the (proxy)cached one. Invalid cached responses should then have a lifetime of a few seconds.
Or just call proxycache again to expire(purge) the value.

.

dasschwarzesystem

Hey .. thanks for the hint.
But where do I set the entry into the shared dict? Could you give an example please.

Regards

Falk

Am Samstag, 21. November 2015 21:55:38 UTC+1 schrieb perusio:

I would proxy the request so that in the "origin" the response body is parsed and set an entry in a shared dict. Like a flag. Then use it to skip the cache by stting proxy_cache_bypass and proxy_no_cache to non 0.

Just store the responses that shouldn't be cached in the shared dict.

Just an idea.

Grüße,
Antonio

perusio

On Sun, Nov 22, 2015 at 5:53 PM Falk Berger <dasschwa...@gmail.com> wrote:

Hey .. thanks for the hint.
But where do I set the entry into the shared dict? Could you give an example please.

I just had a better idea that bypasses shared dict. Just use a proxy that does nothing more than issue a request to your backend using the resty-http library.

Then based on the response set a header that then is used in the proxy_cache_bypass, proxy_no_cache directives.

In your proxy location.

proxy_cache_bypass $upstream_http_x_no_cache;

proxy_no_cache $upstream_http_x_no_cache;

proxy_pass http://my_proxy_upstream;

In the proxy upstream you do request to your backend:

-- Create a new connection handler. local http_connection = http:new()
-- Request the translation from the dictionary service. local res, err = http_connection:request_uri('http://example.com', { method = 'GET'})

-- If the connection failed exit and log it.
if not res then
ngx.log(ngx.ERR, format('Failed request: %s', err))
return
end

-- Now evaluate the response body (res.body). If the pattern for being
-- non-cacheable is there set the header.

res.headers['X-No-Cache'] = 'Y'

Check the library at https://github.com/pintsized/lua-resty-http

I haven't tested this myself. Just an idea.

Grüße,

Antonio

Regards
Falk

Am Samstag, 21. November 2015 21:55:38 UTC+1 schrieb perusio:
I would proxy the request so that in the "origin" the response body is parsed and set an entry in a shared dict. Like a flag. Then use it to skip the cache by stting proxy_cache_bypass and proxy_no_cache to non 0.

Just store the responses that shouldn't be cached in the shared dict.

Just an idea.

Grüße,
Antonio

.

peter_booth

I guess that your root problem is that your API isn’t well behaved -its returning a 200 with error visible in the content .

I would try something like the following to set a 404 or better code.

The nginx tutorial sections on directive execution order, combined with a debug built instance of nginx,

are essential for debugging stuff like this.

http://openresty.org/download/agentzh-nginx-tutorials-en.html

location = /my_api {
content_by_lua '
res = ngx.location.capture(“/api");

if (string.match(res, “ERR”)) then

return ngx.exit(ngx.HTTP_NOT_FOUND)

end

ngx.print(res.body)';
}

On 22 Nov 2015, at 7:43 PM, António P. P. Almeida <per...@gmail.com> wrote:

On Sun, Nov 22, 2015 at 5:53 PM Falk Berger <dasschwa...@gmail.com> wrote:
Hey .. thanks for the hint.
But where do I set the entry into the shared dict? Could you give an example please.

I just had a better idea that bypasses shared dict. Just use a proxy that does nothing more than issue a request to your backend using the resty-http library.
Then based on the response set a header that then is used in the proxy_cache_bypass, proxy_no_cache directives.

In your proxy location.

proxy_cache_bypass $upstream_http_x_no_cache;
proxy_no_cache $upstream_http_x_no_cache;
proxy_pass http://my_proxy_upstream;

In the proxy upstream you do request to your backend:

-- Create a new connection handler. local http_connection = http:new()
-- Request the translation from the dictionary service. local res, err = http_connection:request_uri('http://example.com', { method = 'GET'})
-- If the connection failed exit and log it.
if not res then
   ngx.log(ngx.ERR, format('Failed request: %s', err))
   return
end

-- Now evaluate the response body (res.body). If the pattern for being
-- non-cacheable is there set the header.

res.headers['X-No-Cache'] = 'Y'

Check the library at https://github.com/pintsized/lua-resty-http

I haven't tested this myself. Just an idea.

Grüße,
Antonio

Regards
Falk

Am Samstag, 21. November 2015 21:55:38 UTC+1 schrieb perusio:
I would proxy the request so that in the "origin" the response body is parsed and set an entry in a shared dict. Like a flag. Then use it to skip the cache by stting proxy_cache_bypass and proxy_no_cache to non 0.
Just store the responses that shouldn't be cached in the shared dict.
Just an idea.
Grüße,
Antonio

--
.