is php fastcgi as body rewrite possible

matt.meisberger · 2016-06-08T08:56:01+00:00

Hi Matthieu, I found this module https://github.com/GuyCheung/falcon-ngx_metric, maybe it will help you On Wednesday, 8 June 2016 00:09:50 UTC...

is php fastcgi as body rewrite possible

matt.meisberger

I'm trying to proxy_pass and use an already developed large php library to modify the body for the end client. I can't seem to find a good way to do it yet.

Is there a way to first do a proxy pass and then write to stdout and then hand off to fastcgi pass and read stdin?

Any others ideas would be appreciated also.

Thanks

matt.meisberger

I think I have a path forward on this but if someone has a better idea feel free to suggest it.

I essentially have 3 locations in my conf file to support this. The initial location will just spawn a thread and hit the desired location with a unique key. proxy_cache_lock and proxy_cache are set to on with very low cache time. So that location block will ignore the results of the thread it just gets the request started. I then proxy_pass into the location which has the fastcgi. I then curl back into that location with the key for the http request which should lock me until it finishes downloading the html.

If interested here is the proof of concept code that I will be rewriting that seems to work.

location ~* /preflight/(.*)/(.*) {
    resolver 8.8.8.8 valid=60s;

    proxy_cache html;
    proxy_cache_lock on;
    proxy_cache_valid 404 1s;
    proxy_cache_valid any 5s;
    proxy_cache_key $1;


    # needed since nginx will otherwise auto put the regex on proxy_pass
    set $thing "";

    proxy_pass http://bing.com/$thing;
}

location ~* /postflight/(.*)/(.*) {

    fastcgi_param HTTP_FETCH_KEY        $1; # php should pass this back through to preflight
    fastcgi_param URL_ID                $2;

    fastcgi_param SCRIPT_FILENAME /vagrant/src/web/lua-proxy.php;
    fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
    include ../conf/fastcgi_params;
    fastcgi_buffers 32 32k;
    fastcgi_buffer_size 64k;
}

location  /proof-of-concept {

    # just start priming the pumps in the background
    rewrite_by_lua_block {
        function send()
            http = require "resty.http"
            local httpc = http.new()
            httpc:request_uri("https://127.0.0.1/preflight/1/abcwebsiteurlid", {
                method = "GET",
                ssl_verify = false
            })
        end

        ngx.thread.spawn(send)
    }

    # now that the pumps are priming, get request to fpm
    proxy_pass https://127.0.0.1/postflight/1/abcwebsiteurlid;
}

On Tuesday, June 7, 2016 at 7:56:01 PM UTC-5, Matt Meisberger wrote:

I'm trying to proxy_pass and use an already developed large php library to modify the body for the end client. I can't seem to find a good way to do it yet.

Is there a way to first do a proxy pass and then write to stdout and then hand off to fastcgi pass and read stdin?

Any others ideas would be appreciated also.

Thanks

matt.meisberger

After posting I realized I probably can just move the fastcgi stuff and replace the proxy pass in the proof-of-concept block

On Wednesday, June 8, 2016 at 6:08:41 AM UTC-5, Matt Meisberger wrote:

I think I have a path forward on this but if someone has a better idea feel free to suggest it.

I essentially have 3 locations in my conf file to support this. The initial location will just spawn a thread and hit the desired location with a unique key. proxy_cache_lock and proxy_cache are set to on with very low cache time. So that location block will ignore the results of the thread it just gets the request started. I then proxy_pass into the location which has the fastcgi. I then curl back into that location with the key for the http request which should lock me until it finishes downloading the html.

If interested here is the proof of concept code that I will be rewriting that seems to work.
location ~* /preflight/(.*)/(.*) {
    resolver 8.8.8.8 valid=60s;

    proxy_cache html;
    proxy_cache_lock on;
    proxy_cache_valid 404 1s;
    proxy_cache_valid any 5s;
    proxy_cache_key $1;
    # needed since nginx will otherwise auto put the regex on proxy_pass
    set $thing "";

    proxy_pass http://bing.com/$thing;
}

location ~* /postflight/(.*)/(.*) {

    fastcgi_param HTTP_FETCH_KEY        $1; # php should pass this back through to preflight
    fastcgi_param URL_ID                $2;

    fastcgi_param SCRIPT_FILENAME /vagrant/src/web/lua-proxy.php;
    fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
    include ../conf/fastcgi_params;
    fastcgi_buffers 32 32k;
    fastcgi_buffer_size 64k;
}

location  /proof-of-concept {
    # just start priming the pumps in the background
    rewrite_by_lua_block {
        function send()
            http = require "resty.http"
            local httpc = http.new()
            httpc:request_uri("https://127.0.0.1/preflight/1/abcwebsiteurlid", {
                method = "GET",
                ssl_verify = false
            })
        end

        ngx.thread.spawn(send)
    }

    # now that the pumps are priming, get request to fpm
    proxy_pass https://127.0.0.1/postflight/1/abcwebsiteurlid;
}
On Tuesday, June 7, 2016 at 7:56:01 PM UTC-5, Matt Meisberger wrote:
I'm trying to proxy_pass and use an already developed large php library to modify the body for the end client. I can't seem to find a good way to do it yet.

Is there a way to first do a proxy pass and then write to stdout and then hand off to fastcgi pass and read stdin?

Any others ideas would be appreciated also.

Thanks

jonathan

On Tuesday, June 7, 2016 at 8:56:01 PM UTC-4, Matt Meisberger wrote:

Is there a way to first do a proxy pass and then write to stdout and then hand off to fastcgi pass and read stdin?

have you looked at the lua-rest-http driver? I think that can do what you want.

https://github.com/pintsized/lua-resty-http

matt.meisberger

I do use that but how would I pass the data to my php process? Would I write to stdout and read the stdin from php? haven't had much luck passing the html into php.

Also, I'm a little worried about loading it all into memory.

On Wed, Jun 8, 2016 at 8:30 PM <jon...@findmeon.com> wrote:

On Tuesday, June 7, 2016 at 8:56:01 PM UTC-4, Matt Meisberger wrote:
Is there a way to first do a proxy pass and then write to stdout and then hand off to fastcgi pass and read stdin?

have you looked at the lua-rest-http driver? I think that can do what you want.

https://github.com/pintsized/lua-resty-http

.

jonathan

I was just thinking of very simply grabbing the result from one and POSTing it into the other

agentzh

Hello!

On Tue, Jun 7, 2016 at 5:56 PM, Matt Meisberger wrote:
> I'm trying to proxy_pass and use an already developed large php library to
> modify the body for the end client. I can't seem to find a good way to do it
> yet.
>
> Is there a way to first do a proxy pass and then write to stdout and then
> hand off to fastcgi pass and read stdin?
>
> Any others ideas would be appreciated also.
>

Your PHP/FastCGI part can easily become the bottleneck of your whole
nginx server. Better do that in-place in nginx.

Regards,
-agentzh

matt.meisberger

Hi, I think I've got a path forward. I'm using http in a thread to fire off a request to a proxy locked pass thru then I drop that and give control to php which then makes a curl request to the same unique url that is passed as a fast cgi param.

On Wednesday, June 22, 2016 at 12:45:45 AM UTC-5, agentzh wrote:

Hello!

On Tue, Jun 7, 2016 at 5:56 PM, Matt Meisberger wrote:
> I'm trying to proxy_pass and use an already developed large php library to
> modify the body for the end client. I can't seem to find a good way to do it
> yet.
>
> Is there a way to first do a proxy pass and then write to stdout and then
> hand off to fastcgi pass and read stdin?
>
> Any others ideas would be appreciated also.
>

Your PHP/FastCGI part can easily become the bottleneck of your whole
nginx server. Better do that in-place in nginx.

Regards,
-agentzh