Remove cookies from upstream response before proxy_cache

lagged · 2019-09-23T15:34:01+00:00

In my own C module I run a pre-access filter that uses `ngx_http_read_client_request_body_handler` to load the request body on the request struct. &#xA...

Remove cookies from upstream response before proxy_cache

lagged

Hello,

I'm having a hard time caching requests such as the following due to the "Set-Cookie" entries returned by the upstream application:

~$ curl -I https://[redacted]/[redacted]/amp
HTTP/1.1 200 OK
Date: Mon, 23 Sep 2019 07:28:09 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
X-Pingback: https://[redacted]/xmlrpc.php
Set-Cookie: pvc_visits[0]=1569310089b5472; expires=Tue, 24-Sep-2019 07:28:09 GMT; Max-Age=86400; path=/; secure; HttpOnly
Set-Cookie: cookielawinfo-checkbox-necessary=yes; expires=Mon, 23-Sep-2019 08:28:09 GMT; Max-Age=3600; path=/
Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: max-age=3600
Expires: Mon, 23 Sep 2019 08:28:09 GMT
Vary: Accept-Encoding
Referrer-Policy: no-referrer-when-downgrade
Alternate-Protocol: 443:npn-http/2
X-Request-ID: iptrysbtcqgffnvy
X-Cache-Status: MISS-0

Would it be possible to strip cookies using a regex from the backend response using Lua, before the request reaches proxy_cache, in order to force it to cache? Unfortunately, I have no control over the backend application, and I would like to only selectively remove cookies versus all cookies to avoid content issues. Any help would be greatly appreciated

lagged

Does anyone know if this is possible using Lua? If so, does anyone have an example or suggestions?

On Mon, Sep 23, 2019 at 2:33 AM Andrei <la...@gmail.com> wrote:

Hello,

I'm having a hard time caching requests such as the following due to the "Set-Cookie" entries returned by the upstream application:

~$ curl -I https://[redacted]/[redacted]/amp
HTTP/1.1 200 OK
Date: Mon, 23 Sep 2019 07:28:09 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
X-Pingback: https://[redacted]/xmlrpc.php
Set-Cookie: pvc_visits[0]=1569310089b5472; expires=Tue, 24-Sep-2019 07:28:09 GMT; Max-Age=86400; path=/; secure; HttpOnly
Set-Cookie: cookielawinfo-checkbox-necessary=yes; expires=Mon, 23-Sep-2019 08:28:09 GMT; Max-Age=3600; path=/
Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: max-age=3600
Expires: Mon, 23 Sep 2019 08:28:09 GMT
Vary: Accept-Encoding
Referrer-Policy: no-referrer-when-downgrade
Alternate-Protocol: 443:npn-http/2
X-Request-ID: iptrysbtcqgffnvy
X-Cache-Status: MISS-0

Would it be possible to strip cookies using a regex from the backend response using Lua, before the request reaches proxy_cache, in order to force it to cache? Unfortunately, I have no control over the backend application, and I would like to only selectively remove cookies versus all cookies to avoid content issues. Any help would be greatly appreciated

lagged

I'm having a really hard time with GDPR cookies breaking cache. Does anyone have a suggested solution? In varnish for example, vcl can be used to selectively strip cookies from the request before it checks for cache. How can this be done in Lua?

On Wed, Sep 25, 2019, 10:31 Andrei <la...@gmail.com> wrote:

Does anyone know if this is possible using Lua? If so, does anyone have an example or suggestions?

On Mon, Sep 23, 2019 at 2:33 AM Andrei <la...@gmail.com> wrote:
Hello,

I'm having a hard time caching requests such as the following due to the "Set-Cookie" entries returned by the upstream application:

~$ curl -I https://[redacted]/[redacted]/amp
HTTP/1.1 200 OK
Date: Mon, 23 Sep 2019 07:28:09 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
X-Pingback: https://[redacted]/xmlrpc.php
Set-Cookie: pvc_visits[0]=1569310089b5472; expires=Tue, 24-Sep-2019 07:28:09 GMT; Max-Age=86400; path=/; secure; HttpOnly
Set-Cookie: cookielawinfo-checkbox-necessary=yes; expires=Mon, 23-Sep-2019 08:28:09 GMT; Max-Age=3600; path=/
Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: max-age=3600
Expires: Mon, 23 Sep 2019 08:28:09 GMT
Vary: Accept-Encoding
Referrer-Policy: no-referrer-when-downgrade
Alternate-Protocol: 443:npn-http/2
X-Request-ID: iptrysbtcqgffnvy
X-Cache-Status: MISS-0

Would it be possible to strip cookies using a regex from the backend response using Lua, before the request reaches proxy_cache, in order to force it to cache? Unfortunately, I have no control over the backend application, and I would like to only selectively remove cookies versus all cookies to avoid content issues. Any help would be greatly appreciated

lagged

An example of what I used to do in Varnish for Google Analytics for example, was to save the cookie to a custom header, strip it from the incoming request before cache processing, then add it back to the client returned data and strip the custom header. That way the cookie would remain intact, cache would be used, and content differences wouldn't appear as it was only for specific cookies I knew wouldn't make a difference in content. I'm basically trying to do the same in OpenResty but I'm not having any luck. Please help :)

On Fri, Oct 4, 2019 at 1:55 AM Andrei <la...@gmail.com> wrote:

I'm having a really hard time with GDPR cookies breaking cache. Does anyone have a suggested solution? In varnish for example, vcl can be used to selectively strip cookies from the request before it checks for cache. How can this be done in Lua?

On Wed, Sep 25, 2019, 10:31 Andrei <la...@gmail.com> wrote:
Does anyone know if this is possible using Lua? If so, does anyone have an example or suggestions?

On Mon, Sep 23, 2019 at 2:33 AM Andrei <la...@gmail.com> wrote:
Hello,

I'm having a hard time caching requests such as the following due to the "Set-Cookie" entries returned by the upstream application:

~$ curl -I https://[redacted]/[redacted]/amp
HTTP/1.1 200 OK
Date: Mon, 23 Sep 2019 07:28:09 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
X-Pingback: https://[redacted]/xmlrpc.php
Set-Cookie: pvc_visits[0]=1569310089b5472; expires=Tue, 24-Sep-2019 07:28:09 GMT; Max-Age=86400; path=/; secure; HttpOnly
Set-Cookie: cookielawinfo-checkbox-necessary=yes; expires=Mon, 23-Sep-2019 08:28:09 GMT; Max-Age=3600; path=/
Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: max-age=3600
Expires: Mon, 23 Sep 2019 08:28:09 GMT
Vary: Accept-Encoding
Referrer-Policy: no-referrer-when-downgrade
Alternate-Protocol: 443:npn-http/2
X-Request-ID: iptrysbtcqgffnvy
X-Cache-Status: MISS-0

Would it be possible to strip cookies using a regex from the backend response using Lua, before the request reaches proxy_cache, in order to force it to cache? Unfortunately, I have no control over the backend application, and I would like to only selectively remove cookies versus all cookies to avoid content issues. Any help would be greatly appreciated

piotrprz

One thing you can try is to split your proxy into two layers:

1. Client-facing cache, listening on public port. Uses layer 2 as upstream. Converts custom header to cookie.

2. Proxy layer, listening on local socket. Fetches data from your real upstream and moves your cookie to custom header.

In other terms: insert a proxy that will transform your response before it's cached, and use this proxy as your upstream.

W dniu piątek, 4 października 2019 08:58:38 UTC+2 użytkownik Andrei napisał:

An example of what I used to do in Varnish for Google Analytics for example, was to save the cookie to a custom header, strip it from the incoming request before cache processing, then add it back to the client returned data and strip the custom header. That way the cookie would remain intact, cache would be used, and content differences wouldn't appear as it was only for specific cookies I knew wouldn't make a difference in content. I'm basically trying to do the same in OpenResty but I'm not having any luck. Please help :)

On Fri, Oct 4, 2019 at 1:55 AM Andrei <lag...@gmail.com> wrote:
I'm having a really hard time with GDPR cookies breaking cache. Does anyone have a suggested solution? In varnish for example, vcl can be used to selectively strip cookies from the request before it checks for cache. How can this be done in Lua?

On Wed, Sep 25, 2019, 10:31 Andrei <lag...@gmail.com> wrote:
Does anyone know if this is possible using Lua? If so, does anyone have an example or suggestions?

On Mon, Sep 23, 2019 at 2:33 AM Andrei <lag...@gmail.com> wrote:
Hello,

I'm having a hard time caching requests such as the following due to the "Set-Cookie" entries returned by the upstream application:

~$ curl -I https://[redacted]/[redacted]/amp
HTTP/1.1 200 OK
Date: Mon, 23 Sep 2019 07:28:09 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
X-Pingback: https://[redacted]/xmlrpc.php
Set-Cookie: pvc_visits[0]=1569310089b5472; expires=Tue, 24-Sep-2019 07:28:09 GMT; Max-Age=86400; path=/; secure; HttpOnly
Set-Cookie: cookielawinfo-checkbox-necessary=yes; expires=Mon, 23-Sep-2019 08:28:09 GMT; Max-Age=3600; path=/
Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: max-age=3600
Expires: Mon, 23 Sep 2019 08:28:09 GMT
Vary: Accept-Encoding
Referrer-Policy: no-referrer-when-downgrade
Alternate-Protocol: 443:npn-http/2
X-Request-ID: iptrysbtcqgffnvy
X-Cache-Status: MISS-0

Would it be possible to strip cookies using a regex from the backend response using Lua, before the request reaches proxy_cache, in order to force it to cache? Unfortunately, I have no control over the backend application, and I would like to only selectively remove cookies versus all cookies to avoid content issues. Any help would be greatly appreciated

lagged

Hello,

Thanks for the input, that seems to be a common suggestion. However, it should be possible to selectively strip cookie data in Lua before it checks / passes proxy_cache.

On Wed, Oct 16, 2019 at 1:34 PM Piotr Przybylski <pio...@gmail.com> wrote:

One thing you can try is to split your proxy into two layers:
1. Client-facing cache, listening on public port. Uses layer 2 as upstream. Converts custom header to cookie.
2. Proxy layer, listening on local socket. Fetches data from your real upstream and moves your cookie to custom header.

In other terms: insert a proxy that will transform your response before it's cached, and use this proxy as your upstream.

W dniu piątek, 4 października 2019 08:58:38 UTC+2 użytkownik Andrei napisał:
An example of what I used to do in Varnish for Google Analytics for example, was to save the cookie to a custom header, strip it from the incoming request before cache processing, then add it back to the client returned data and strip the custom header. That way the cookie would remain intact, cache would be used, and content differences wouldn't appear as it was only for specific cookies I knew wouldn't make a difference in content. I'm basically trying to do the same in OpenResty but I'm not having any luck. Please help :)

On Fri, Oct 4, 2019 at 1:55 AM Andrei <lag...@gmail.com> wrote:
I'm having a really hard time with GDPR cookies breaking cache. Does anyone have a suggested solution? In varnish for example, vcl can be used to selectively strip cookies from the request before it checks for cache. How can this be done in Lua?

On Wed, Sep 25, 2019, 10:31 Andrei <lag...@gmail.com> wrote:
Does anyone know if this is possible using Lua? If so, does anyone have an example or suggestions?

On Mon, Sep 23, 2019 at 2:33 AM Andrei <lag...@gmail.com> wrote:
Hello,

I'm having a hard time caching requests such as the following due to the "Set-Cookie" entries returned by the upstream application:

~$ curl -I https://[redacted]/[redacted]/amp
HTTP/1.1 200 OK
Date: Mon, 23 Sep 2019 07:28:09 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
X-Pingback: https://[redacted]/xmlrpc.php
Set-Cookie: pvc_visits[0]=1569310089b5472; expires=Tue, 24-Sep-2019 07:28:09 GMT; Max-Age=86400; path=/; secure; HttpOnly
Set-Cookie: cookielawinfo-checkbox-necessary=yes; expires=Mon, 23-Sep-2019 08:28:09 GMT; Max-Age=3600; path=/
Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: max-age=3600
Expires: Mon, 23 Sep 2019 08:28:09 GMT
Vary: Accept-Encoding
Referrer-Policy: no-referrer-when-downgrade
Alternate-Protocol: 443:npn-http/2
X-Request-ID: iptrysbtcqgffnvy
X-Cache-Status: MISS-0

Would it be possible to strip cookies using a regex from the backend response using Lua, before the request reaches proxy_cache, in order to force it to cache? Unfortunately, I have no control over the backend application, and I would like to only selectively remove cookies versus all cookies to avoid content issues. Any help would be greatly appreciated