Lua SSL LetsEncrypt prevent restarting nginx

c0nw0nks · 2017-03-20T10:30:02+00:00

I've had this happen when the upstream sends gzip-encoded data. Subrequests using ngx.location.capture pass on the client headers, so...

Lua SSL LetsEncrypt prevent restarting nginx

c0nw0nks

So I am curious if there is a way with Lua I won't have to restart my Nginx when the SSL cert/key/chain currently being used for the server in the defined directory gets updated / modified by the letsencrypt program (scheduled task).

server {
listen 80; #IPv4 Unsecured
listen [::]:80; #IPv6 Unsecured

listen 443 ssl http2; #IPv4 HTTPS PORT
listen [::]:443 ssl http2; #IPv6 HTTPS PORT

server_name networkflare.com www.networkflare.com; #domain name(s)

#Certificate Paths
ssl_certificate /filepath/cert/site/certificate.pem;
ssl_certificate_key /filepath/cert/site/key.key;

#SSL Settings
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_session_tickets off;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:ECDH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!eNULL:!MD5:!DSS:!EXP:!ADH:!LOW:!MEDIUM;

Be much more useful if Lua could make it so we don't need our nginx restarted every time letsencrypt updates a server locations certs, Every time nginx restarts it wipes our caches etc. Also who ever the unlucky person who tries to visit in the seconds / milliseconds that an nginx restarts takes place obviously can't access the site.

rpaprocki

Hi,

Sent from my iPhone

On Mar 19, 2017, at 19:30, c0nw0nks via openresty-en <openresty-en@googlegroups.com> wrote:

So I am curious if there is a way with Lua I won't have to restart my Nginx when the SSL cert/key/chain currently being used for the server in the defined directory gets updated / modified by the letsencrypt program (scheduled task).
server {
listen 80; #IPv4 Unsecured
listen [::]:80; #IPv6 Unsecured

listen 443 ssl http2; #IPv4 HTTPS PORT
listen [::]:443 ssl http2; #IPv6 HTTPS PORT

server_name networkflare.com www.networkflare.com; #domain name(s)

#Certificate Paths
ssl_certificate /filepath/cert/site/certificate.pem;
ssl_certificate_key /filepath/cert/site/key.key;

#SSL Settings
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_session_tickets off;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:ECDH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!eNULL:!MD5:!DSS:!EXP:!ADH:!LOW:!MEDIUM;
Be much more useful if Lua could make it so we don't need our nginx restarted every time letsencrypt updates a server locations certs, Every time nginx restarts it wipes our caches etc.

Have a look at the balancer by Lua directive, which lets you set TLS cert info on the fly. https://github.com/GUI/lua-resty-auto-ssl is a pretty slick turnkey project.

Also who ever the unlucky person who tries to visit in the seconds / milliseconds that an nginx restarts takes place obviously can't access the site.

This is what graceful reloads are for. A reload via HUP signal is a zero downtime operation. See http://nginx.org/en/docs/control.html.This also obviates your above concerns :p

c0nw0nks

Hey there,

Thanks for the info and links.

I see they use this

ssl_certificate_by_lua_block {

I take it with that Lua directive I can just define ssl_cert_path_to_certificate and ssl_key_path_to_key to use that would load any dynamicly changing certificate and key from the directory i specify on the fly in a non blocking manner does anyone have a example ?

I already have a existing process that communicates with letsencrypt and gets the certificates all i need is the ssl certs dynamicly loading that github from what I see also wants to connect and obtain certificates from letsencrypt.

On Monday, 20 March 2017 03:16:46 UTC, rpaprocki wrote:

Hi,

Sent from my iPhone
On Mar 19, 2017, at 19:30, c0nw0nks via openresty-en <openre...@googlegroups.com> wrote:
So I am curious if there is a way with Lua I won't have to restart my Nginx when the SSL cert/key/chain currently being used for the server in the defined directory gets updated / modified by the letsencrypt program (scheduled task).
server {
listen 80; #IPv4 Unsecured
listen [::]:80; #IPv6 Unsecured

listen 443 ssl http2; #IPv4 HTTPS PORT
listen [::]:443 ssl http2; #IPv6 HTTPS PORT

server_name networkflare.com www.networkflare.com; #domain name(s)

#Certificate Paths
ssl_certificate /filepath/cert/site/certificate.pem;
ssl_certificate_key /filepath/cert/site/key.key;

#SSL Settings
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_session_tickets off;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:ECDH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!eNULL:!MD5:!DSS:!EXP:!ADH:!LOW:!MEDIUM;
Be much more useful if Lua could make it so we don't need our nginx restarted every time letsencrypt updates a server locations certs, Every time nginx restarts it wipes our caches etc.
Have a look at the balancer by Lua directive, which lets you set TLS cert info on the fly. https://github.com/GUI/lua-resty-auto-ssl is a pretty slick turnkey project.

Also who ever the unlucky person who tries to visit in the seconds / milliseconds that an nginx restarts takes place obviously can't access the site.

This is what graceful reloads are for. A reload via HUP signal is a zero downtime operation. See http://nginx.org/en/docs/control.html.This also obviates your above concerns :p

rpaprocki

Hi,

On Mon, Mar 20, 2017 at 5:22 AM, c0nw0nks via openresty-en <openresty-en@googlegroups.com> wrote:

Hey there,

Thanks for the info and links.

I see they use this
ssl_certificate_by_lua_block {
I take it with that Lua directive I can just define ssl_cert_path_to_certificate and ssl_key_path_to_key to use that would load any dynamicly changing certificate and key from the directory i specify on the fly in a non blocking manner does anyone have a example ?

Yes, look at the documentation for this directive :) https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/balancer.md

c0nw0nks

So in order to keep it reloading/refreshing grabbing the latest generated SSL cert and KEY this is what I have found so far.

ssl_certificate_by_lua_block {
    local ssl = require "ngx.ssl"
    ssl.clear_certs()

    local ssl_cert_and_key_directory = "/etc/httpsacme-v01.api.letsencrypt.org/"

    -- load cert from SNI
    local f = assert(io.open(ssl_cert_and_key_directory .. ssl.server_name() .. ".der"))
    local cert_data = f:read("*a")
    f:close()
    -- set cert
    local ok, err = ssl.set_der_cert(cert_data)
    if not ok then
        ngx.log(ngx.ERR, "failed to set DER cert: ", err)
        return
    end

    -- load key from SNI
    local f = assert(io.open(ssl_cert_and_key_directory .. ssl.server_name() .. ".key"))
    local pkey_data = f:read("*a")
    f:close()
    -- set key
    local ok, err = ssl.set_der_priv_key(pkey_data)
    if not ok then
        ngx.log(ngx.ERR, "failed to set DER private key: ", err)
        return
    end
}

These are the certificate and key files i have to choose from but i notice the Lua syntax says DER and these files that was generated by my letsencrypt are all PEM format.

https://cloud.githubusercontent.com/assets/12204587/24085357/84e08b68-0cf2-11e7-860d-a2f5fb48e3bf.png

The chain file is PEM the only DER file is this one "networkflare.com-crt"

On Monday, 20 March 2017 16:03:01 UTC, rpaprocki wrote:

Hi,
On Mon, Mar 20, 2017 at 5:22 AM, c0nw0nks via openresty-en <openre...@googlegroups.com> wrote:
Hey there,

Thanks for the info and links.

I see they use this
ssl_certificate_by_lua_block {
I take it with that Lua directive I can just define ssl_cert_path_to_certificate and ssl_key_path_to_key to use that would load any dynamicly changing certificate and key from the directory i specify on the fly in a non blocking manner does anyone have a example ?
Yes, look at the documentation for this directive :) https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/balancer.md

rpaprocki

On Mon, Mar 20, 2017 at 3:06 PM, <c0n...@googlemail.com> wrote:

So in order to keep it reloading/refreshing grabbing the latest generated SSL cert and KEY this is what I have found so far.

ssl_certificate_by_lua_block {
    local ssl = require "ngx.ssl"
    ssl.clear_certs()

    local ssl_cert_and_key_directory = "/etc/httpsacme-v01.api.letsencrypt.org/"

    -- load cert from SNI
    local f = assert(io.open(ssl_cert_and_key_directory .. ssl.server_name() .. ".der"))
    local cert_data = f:read("*a")
    f:close()
    -- set cert
    local ok, err = ssl.set_der_cert(cert_data)
    if not ok then
        ngx.log(ngx.ERR, "failed to set DER cert: ", err)
        return
    end

    -- load key from SNI
    local f = assert(io.open(ssl_cert_and_key_directory .. ssl.server_name() .. ".key"))
    local pkey_data = f:read("*a")
    f:close()
    -- set key
    local ok, err = ssl.set_der_priv_key(pkey_data)
    if not ok then
        ngx.log(ngx.ERR, "failed to set DER private key: ", err)
        return
    end
}

I certainly wouldn't use io.open to constantly read off disk. Cache this data in memory and invalidate it as needed.

These are the certificate and key files i have to choose from but i notice the Lua syntax says DER and these files that was generated by my letsencrypt are all PEM format.

https://cloud.githubusercontent.com/assets/12204587/24085357/84e08b68-0cf2-11e7-860d-a2f5fb48e3bf.png

The chain file is PEM the only DER file is this one "networkflare.com-crt"

There are ssl_by_lua routines to handle PEM data:

https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/ssl.md#cert_pem_to_der
https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/ssl.md#priv_key_pem_to_der

https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/ssl.md#parse_pem_cert
https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/ssl.md#parse_pem_priv_key

c0nw0nks

Thank's :) Il post my full complete code here when i get it working.

Do you know what the ssl.server_name() var will return every time even if i use subdomains because my certificates start with "networkflare.com" as i showed here https://cloud.githubusercontent.com/assets/12204587/24085357/84e08b68-0cf2-11e7-860d-a2f5fb48e3bf.png but I don't know how it will react to subdomains like www. or media1. sub1. z1. etc etc

I am also looking at a way to cache the certificate here https://github.com/openresty/lua-nginx-module#ngxshareddict

On Monday, 20 March 2017 22:15:32 UTC, rpaprocki wrote:

On Mon, Mar 20, 2017 at 3:06 PM, <c0nw...@googlemail.com> wrote:
So in order to keep it reloading/refreshing grabbing the latest generated SSL cert and KEY this is what I have found so far.

ssl_certificate_by_lua_block {
    local ssl = require "ngx.ssl"
    ssl.clear_certs()

    local ssl_cert_and_key_directory = "/etc/httpsacme-v01.api.letsencrypt.org/"

    -- load cert from SNI
    local f = assert(io.open(ssl_cert_and_key_directory .. ssl.server_name() .. ".der"))
    local cert_data = f:read("*a")
    f:close()
    -- set cert
    local ok, err = ssl.set_der_cert(cert_data)
    if not ok then
        ngx.log(ngx.ERR, "failed to set DER cert: ", err)
        return
    end

    -- load key from SNI
    local f = assert(io.open(ssl_cert_and_key_directory .. ssl.server_name() .. ".key"))
    local pkey_data = f:read("*a")
    f:close()
    -- set key
    local ok, err = ssl.set_der_priv_key(pkey_data)
    if not ok then
        ngx.log(ngx.ERR, "failed to set DER private key: ", err)
        return
    end
}

I certainly wouldn't use io.open to constantly read off disk. Cache this data in memory and invalidate it as needed.

These are the certificate and key files i have to choose from but i notice the Lua syntax says DER and these files that was generated by my letsencrypt are all PEM format.

https://cloud.githubusercontent.com/assets/12204587/24085357/84e08b68-0cf2-11e7-860d-a2f5fb48e3bf.png

The chain file is PEM the only DER file is this one "networkflare.com-crt"

There are ssl_by_lua routines to handle PEM data:

https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/ssl.md#cert_pem_to_der
https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/ssl.md#priv_key_pem_to_der

https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/ssl.md#parse_pem_cert
https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/ssl.md#parse_pem_priv_key

jonathan

I open sourced our solution that *mostly* handles this a few months ago:

https://github.com/aptise/peter_sslers-lua-resty

peter_sslers-lua-resty is an openresty package that does a 4 tiered caching fallback for ssl certificate data:

* per worker native objects

* shared dict PEM data

* direct redis lookup for PEM data

* upstream http lookup for json document with PEM data

it would not be hard to adapt this to your needs, replacing the redis/upstream lookups with local files.

zxcvbn4038

Reading from the disk will block the thread until the IO completes, and querying the cert from an external resource every request adds latency, its best to cache the cert locally in memory and only reload it periodically or on demand (i.e. setup an internal URL you can hit which will cause OpenResty to re-read the cert and store it in memory). Use a shared dictionary - otherwise each thread has its own copy and you have to update them individually.

On Monday, March 20, 2017 at 6:15:32 PM UTC-4, rpaprocki wrote:

On Mon, Mar 20, 2017 at 3:06 PM, <c0nw...@googlemail.com> wrote:
So in order to keep it reloading/refreshing grabbing the latest generated SSL cert and KEY this is what I have found so far.

ssl_certificate_by_lua_block {
    local ssl = require "ngx.ssl"
    ssl.clear_certs()

    local ssl_cert_and_key_directory = "/etc/httpsacme-v01.api.letsencrypt.org/"

    -- load cert from SNI
    local f = assert(io.open(ssl_cert_and_key_directory .. ssl.server_name() .. ".der"))
    local cert_data = f:read("*a")
    f:close()
    -- set cert
    local ok, err = ssl.set_der_cert(cert_data)
    if not ok then
        ngx.log(ngx.ERR, "failed to set DER cert: ", err)
        return
    end

    -- load key from SNI
    local f = assert(io.open(ssl_cert_and_key_directory .. ssl.server_name() .. ".key"))
    local pkey_data = f:read("*a")
    f:close()
    -- set key
    local ok, err = ssl.set_der_priv_key(pkey_data)
    if not ok then
        ngx.log(ngx.ERR, "failed to set DER private key: ", err)
        return
    end
}

I certainly wouldn't use io.open to constantly read off disk. Cache this data in memory and invalidate it as needed.

These are the certificate and key files i have to choose from but i notice the Lua syntax says DER and these files that was generated by my letsencrypt are all PEM format.

https://cloud.githubusercontent.com/assets/12204587/24085357/84e08b68-0cf2-11e7-860d-a2f5fb48e3bf.png

The chain file is PEM the only DER file is this one "networkflare.com-crt"

There are ssl_by_lua routines to handle PEM data:

https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/ssl.md#cert_pem_to_der
https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/ssl.md#priv_key_pem_to_der

https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/ssl.md#parse_pem_cert
https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/ssl.md#parse_pem_priv_key

zxcvbn4038

One other thought I had is that your clearing the certificates too early - once you clear the certificates you are past the point of no return. You will need to supply a valid key and certificate in their place or you'll probably segv shortly after you return from the handler. So do all of your loading and error checking first, and then clear the current certs right before you call set_der_priv_key and set_der_cert. Those functions don't do any validation btw, as I found out when someone sent me a malformed cert and a mismatched key one day, they just shove a blobs into memory.

On Monday, March 20, 2017 at 6:06:43 PM UTC-4, c0nw...@googlemail.com wrote:

So in order to keep it reloading/refreshing grabbing the latest generated SSL cert and KEY this is what I have found so far.

ssl_certificate_by_lua_block {
    local ssl = require "ngx.ssl"
    ssl.clear_certs()

    local ssl_cert_and_key_directory = "/etc/httpsacme-v01.api.letsencrypt.org/"

    -- load cert from SNI
    local f = assert(io.open(ssl_cert_and_key_directory .. ssl.server_name() .. ".der"))
    local cert_data = f:read("*a")
    f:close()
    -- set cert
    local ok, err = ssl.set_der_cert(cert_data)
    if not ok then
        ngx.log(ngx.ERR, "failed to set DER cert: ", err)
        return
    end

    -- load key from SNI
    local f = assert(io.open(ssl_cert_and_key_directory .. ssl.server_name() .. ".key"))
    local pkey_data = f:read("*a")
    f:close()
    -- set key
    local ok, err = ssl.set_der_priv_key(pkey_data)
    if not ok then
        ngx.log(ngx.ERR, "failed to set DER private key: ", err)
        return
    end
}

These are the certificate and key files i have to choose from but i notice the Lua syntax says DER and these files that was generated by my letsencrypt are all PEM format.

https://cloud.githubusercontent.com/assets/12204587/24085357/84e08b68-0cf2-11e7-860d-a2f5fb48e3bf.png

The chain file is PEM the only DER file is this one "networkflare.com-crt"

On Monday, 20 March 2017 16:03:01 UTC, rpaprocki wrote:
Hi,
On Mon, Mar 20, 2017 at 5:22 AM, c0nw0nks via openresty-en <openre...@googlegroups.com> wrote:
Hey there,

Thanks for the info and links.

I see they use this
ssl_certificate_by_lua_block {
I take it with that Lua directive I can just define ssl_cert_path_to_certificate and ssl_key_path_to_key to use that would load any dynamicly changing certificate and key from the directory i specify on the fly in a non blocking manner does anyone have a example ?
Yes, look at the documentation for this directive :) https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/balancer.md