Making http requests in ssl_certificate_by_lua_file context

moseleymark · 2016-06-04T05:48:05+00:00

mex. Could you share some samples how you achieved backend connections limit? At now I can't understand how lua-resty-limit-traffic library can hel...

Making http requests in ssl_certificate_by_lua_file context

moseleymark

I've been testing the new RC (1.9.15.1rc1) and have been running into an odd issue. I'm on Ubuntu 12.04 and use resty for a number of things in production including what's breaking in the RC. I've been testing on 1.9.15 and 1.11.1. I'm hoping for some guidance on whether this is a bug in the RC, and/or where to go next with debugging.

In the RC version, if I try to make an HTTP request in ssl_certificate_by_lua_file, the request just dies. The same code works fine in a production 1.9.9 instance. I assume it's something weird with my setup, but I'm at a loss as to what to look at next.

I've tried a couple of different resty.http modules (and to reiterate, this was testing code that already is in production on 1.9.9 and works fine doing HTTP reqs in the ssl_certificate_by_lua_file context). I've added ngx.log statements to two resty.http modules I've tried and in both cases they die in the resty C code (specifically when they call connect on ngx.socket.tcp). I've not seen a single logged error in all my tests. The worker isn't segfaulting either, it's still around. strace'ing gives no indication of what it's choking on.

The connection is actually made but then immediately shut down by nginx (and not by the remote server, lots of tcpdump'ing to confirm this). There's two cases: in one, the nginx box immediately sends a RST when the remote server SYN-ACKs (presumably the nginx box closed it really quickly); in the other, the nginx box will ACK the SYN-ACK, and then send a FIN.

I've been using the super stripped-down script below to test (as well as going through the standard nginx process of pulling out all the 3rd party modules that I could). If I use this script in ssl_certificate_by_lua_file, it never reaches 'GOT HERE 2'. If I do NOT use it in ssl_certificate_by_lua_file and rather use this script in content_by_lua_file, it DOES reach 'GOT HERE 2'.

<begin>

ngx.log(ngx.ERR, "GOT HERE" )

local url = "" href="http://www.perl.com">http://www.perl.com"

local http = require "resty.http"

local hc = http.new()

local ok = hc:request_uri( url)

ngx.log(ngx.ERR, "GOT HERE 2" )

return

</end>

agentzh

Hello!

On Fri, Jun 3, 2016 at 2:48 PM, Mark Moseley wrote:
> I've been testing the new RC (1.9.15.1rc1) and have been running into an odd
> issue. I'm on Ubuntu 12.04 and use resty for a number of things in
> production including what's breaking in the RC. I've been testing on 1.9.15
> and 1.11.1.

OpenResty bundles with its own NGINX core. Are you swapping the nginx
core with your own? Don't do that since you'll need to apply
OpenResty's patches to the nginx core yourself.

Swapping out the nginx core may introduce various compatibility
issues. For example, the OpenResty components have never officially
been tested against the nginx 1.11.1 core yet. You are at your own
risk.

> <begin>
> ngx.log(ngx.ERR, "GOT HERE" )
>
> local url = "http://www.perl.com";
> local http = require "resty.http"
> local hc = http.new()
> local ok = hc:request_uri( url)
>
> ngx.log(ngx.ERR, "GOT HERE 2" )

I've tried your example on my side with OpenResty 1.9.15.1 RC2 with
the following configuration:

    lua_package_path "/home/agentzh/git/lua-resty-http/lib/?.lua;;";

    server {
        listen 443 ssl;
        ssl_certificate /home/agentzh/git/lua-nginx-module/t/cert/test.crt;
        ssl_certificate_key /home/agentzh/git/lua-nginx-module/t/cert/test.key;

        ssl_certificate_by_lua_block {
            ngx.log(ngx.ERR, "GOT HERE" )

            local url = "http://www.perl.com";
            local http = require "resty.http"
            local hc = http.new()
            local ok = hc:request_uri( url)

            ngx.log(ngx.ERR, "GOT HERE 2" )
        }
    }

And then use the following command to test this server locally:

    curl https://127.0.0.1:443/ -k

And I'm getting the expected error.log entries in my error.log file:

    2016/06/03 15:04:19 [error] 93335#0: [lua]
ssl_certificate_by_lua:2: GOT HERE, context: ssl_certificate_by_lua*,
client: 127.0.0.1, server: 0.0.0.0:443
    2016/06/03 15:04:19 [error] 93335#0: [lua]
ssl_certificate_by_lua:9: GOT HERE 2, context:
ssl_certificate_by_lua*, client: 127.0.0.1, server: 0.0.0.0:443

The symptom you described should be that you're using your own nginx
core which lacks the crucial patch for the nginx core required to get
ssl_certificate_by_lua* work. See

    https://github.com/openresty/lua-nginx-module#ssl_certificate_by_lua_block

To quote,

"This directive currently requires the following NGINX core patch to
work correctly:

http://mailman.nginx.org/pipermail/nginx-devel/2016-January/007748.html

The bundled version of the NGINX core in OpenResty 1.9.7.2 (or above)
already has this patch applied."

Best regards,
-agentzh

moseleymark

On Fri, Jun 3, 2016 at 3:06 PM, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

Hello!

On Fri, Jun 3, 2016 at 2:48 PM, Mark Moseley wrote:
> I've been testing the new RC (1.9.15.1rc1) and have been running into an odd
> issue. I'm on Ubuntu 12.04 and use resty for a number of things in
> production including what's breaking in the RC. I've been testing on 1.9.15
> and 1.11.1.

OpenResty bundles with its own NGINX core. Are you swapping the nginx
core with your own? Don't do that since you'll need to apply
OpenResty's patches to the nginx core yourself.

Swapping out the nginx core may introduce various compatibility
issues. For example, the OpenResty components have never officially
been tested against the nginx 1.11.1 core yet. You are at your own
risk.

> <begin>
> ngx.log(ngx.ERR, "GOT HERE" )
>
> local url = "" href="http://www.perl.com" rel="noreferrer" target="_blank">http://www.perl.com"
> local http = require "resty.http"
> local hc = http.new()
> local ok = hc:request_uri( url)
>
> ngx.log(ngx.ERR, "GOT HERE 2" )

I've tried your example on my side with OpenResty 1.9.15.1 RC2 with
the following configuration:

lua_package_path "/home/agentzh/git/lua-resty-http/lib/?.lua;;";

server {
listen 443 ssl;
ssl_certificate /home/agentzh/git/lua-nginx-module/t/cert/test.crt;
ssl_certificate_key /home/agentzh/git/lua-nginx-module/t/cert/test.key;

ssl_certificate_by_lua_block {
ngx.log(ngx.ERR, "GOT HERE" )

local url = "" href="http://www.perl.com" rel="noreferrer" target="_blank">http://www.perl.com"
local http = require "resty.http"
local hc = http.new()
local ok = hc:request_uri( url)

ngx.log(ngx.ERR, "GOT HERE 2" )
}
}

And then use the following command to test this server locally:

curl https://127.0.0.1:443/ -k

And I'm getting the expected error.log entries in my error.log file:

2016/06/03 15:04:19 [error] 93335#0: [lua]
ssl_certificate_by_lua:2: GOT HERE, context: ssl_certificate_by_lua*,
client: 127.0.0.1, server: 0.0.0.0:443
2016/06/03 15:04:19 [error] 93335#0: [lua]
ssl_certificate_by_lua:9: GOT HERE 2, context:
ssl_certificate_by_lua*, client: 127.0.0.1, server: 0.0.0.0:443

The symptom you described should be that you're using your own nginx
core which lacks the crucial patch for the nginx core required to get
ssl_certificate_by_lua* work. See

https://github.com/openresty/lua-nginx-module#ssl_certificate_by_lua_block

To quote,

"This directive currently requires the following NGINX core patch to
work correctly:

http://mailman.nginx.org/pipermail/nginx-devel/2016-January/007748.html

The bundled version of the NGINX core in OpenResty 1.9.7.2 (or above)
already has this patch applied."

That is, of course, exactly the issue :)

It's been a while since I rolled the production package and I completely forgot about the patch. Is that slated to be incorporated into the nginx core?

Works perfectly now on 1.11.1 (haven't tested 1.9.15 yet). I figured it was me doing something stupid. Thanks!