Cosocket SSL CApath and system trusted certificates

michal · 2014-09-11T01:55:41+00:00

Hello! On Wed, Sep 10, 2014 at 1:15 PM, Michal Cichra wrote: > Agreed to both. > As we are discussing in another thread, setting lua_ssl_trusted_certi...

Cosocket SSL CApath and system trusted certificates

michal

Hi,

I'm doing http client using lua cosockets with ssl support and so far it works for me only on OSX.

For some reason on vanilla ubuntu 14.04 with compiled openresty 1.7.4.1rc2 it can't verify certificate with 'unable to get local issuer certificate'.

I briefly checked lua-nginx-module source and it delegates all the work to nginx. So in Nginx it looks like:

https://github.com/nginx/nginx/blob/1176952193ccf47078dc84b8494d0496ad1ac4a2/src/event/ngx_event_openssl.c#L508-L515

I found http://h71000.www7.hp.com/doc/83final/ba554_90007/ch04s03.html#d0e3371 explaining how to use it and it looks like Nginx is not using CApath at all.

It works on osx, maybe apple has some patches in openssl so it loads certificates all the time, even when it is not initialized.

Do you have suggestions how to make lua ssl sockets to trust os ca certificates?

Best,

Michal

michal

After investigating some openssl/libressl code. Looks like nginx should call `SSL_CTX_set_default_verify_paths` or be configured to do so.

Does it make sense to make a patch against openresty? I'm really not that good in doing all the configuration of nginx contexts and fail every time.

On Wednesday, 10 September 2014 19:55:40 UTC+2, Michal Cichra wrote:

Hi,
I'm doing http client using lua cosockets with ssl support and so far it works for me only on OSX.
For some reason on vanilla ubuntu 14.04 with compiled openresty 1.7.4.1rc2 it can't verify certificate with 'unable to get local issuer certificate'.

I briefly checked lua-nginx-module source and it delegates all the work to nginx. So in Nginx it looks like:
https://github.com/nginx/nginx/blob/1176952193ccf47078dc84b8494d0496ad1ac4a2/src/event/ngx_event_openssl.c#L508-L515

I found http://h71000.www7.hp.com/doc/83final/ba554_90007/ch04s03.html#d0e3371 explaining how to use it and it looks like Nginx is not using CApath at all.

It works on osx, maybe apple has some patches in openssl so it loads certificates all the time, even when it is not initialized.

Do you have suggestions how to make lua ssl sockets to trust os ca certificates?

Best,
Michal

agentzh

Hello!

On Wed, Sep 10, 2014 at 10:55 AM, Michal Cichra wrote:
> I'm doing http client using lua cosockets with ssl support and so far it
> works for me only on OSX.
> For some reason on vanilla ubuntu 14.04 with compiled openresty 1.7.4.1rc2
> it can't verify certificate with 'unable to get local issuer certificate'.
>

I think you should just configure the lua_ssl_trusted_certificate and
lua_ssl_verify_depth directives properly:

    https://github.com/openresty/lua-nginx-module#lua_ssl_trusted_certificate

    https://github.com/openresty/lua-nginx-module#lua_ssl_verify_depth

You can use the following tool to export selected or all built-in
certificates from NSS to a PERM-formatted file accepted by
lua_ssl_trusted_certificate, for example:

    https://github.com/openresty/nginx-devel-utils/blob/master/export-builtin-trusted-certs

I suggest you only load the CA certs you actually need to save runtime overhead.

Regards,
-agentzh

michal

Hi!

On 10 Sep 2014, at 21:14, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

I think you should just configure the lua_ssl_trusted_certificate and
lua_ssl_verify_depth directives properly:

https://github.com/openresty/lua-nginx-module#lua_ssl_trusted_certificate

https://github.com/openresty/lua-nginx-module#lua_ssl_verify_depth

Well. I need it to be multi platform. Because this can be deployed by users, it should be

It is pretty common, that software can get trusted certificates from the OS.

I made this patch to nginx https://gist.githubusercontent.com/mikz/4ce4307329095114b109/raw/97c68b3f6a4f0a6f87fe8f1b7005f19484378fcc/gistfile1.diff

that seems to work.

You can use the following tool to export selected or all built-in
certificates from NSS to a PERM-formatted file accepted by
lua_ssl_trusted_certificate, for example:

https://github.com/openresty/nginx-devel-utils/blob/master/export-builtin-trusted-certs

I suggest you only load the CA certs you actually need to save runtime overhead.

It is HTTP client capable of connecting to any server, so it should (lazy) load all the trusted certificates from the OS.

I have no idea how openssl works in the back. If adding CA path has same effect like loading all certificates, or if it is lazy loaded.

If I would try to merge it to nginx (probably with a configuration flag), would you make the same configuration for lua?

something like: `lua_ssl_trusted_certificate system;`.

Best,

Michal

agentzh

Hello!

On Wed, Sep 10, 2014 at 2:59 PM, Michal Cichra wrote:
> Well. I need it to be multi platform. Because this can be deployed by users,
> it should be
> It is pretty common, that software can get trusted certificates from the OS.
>

It's the default set in OpenSSL. AFAIK, Linux does not have a standard
trusted cert set (though some individual Linux distributions might
have their own).

> I made this patch to nginx
> https://gist.githubusercontent.com/mikz/4ce4307329095114b109/raw/97c68b3f6a4f0a6f87fe8f1b7005f19484378fcc/gistfile1.diff
> that seems to work.
>

Well, you could propose this patch (or something polished) to the
nginx-devel mailing list.

>> I suggest you only load the CA certs you actually need to save runtime
>> overhead.
> It is HTTP client capable of connecting to any server, so it should (lazy)
> load all the trusted certificates from the OS.
> I have no idea how openssl works in the back. If adding CA path has same
> effect like loading all certificates, or if it is lazy loaded.

Use of SSL_CTX_set_default_verify_paths has extra overhead at least in
the memory footprint. For the "TEST 11: www.google.com  (SSL verify
passes)" test case in ngx_lua's official test suite, I have the
following memory metrics of the nginx process, as reported by pmap:

    mapped: 55840K    writeable/private: 1276K  (just the used CA cert loaded)
    mapped: 58624K    writeable/private: 4060K
(SSL_CTX_set_default_verify_paths)

It's nontrivial for some people :)

But I agree it is convenient for many people.

> If I would try to merge it to nginx (probably with a configuration flag),
> would you make the same configuration for lua?

Yes.

Regards,
-agentzh

michal

Thank you. Will try to push it to nginx.

Link to mailing list thread for future reference http://mailman.nginx.org/pipermail/nginx/2014-September/045055.html

On Thursday, 11 September 2014 00:36:48 UTC+2, agentzh wrote:

Hello!

On Wed, Sep 10, 2014 at 2:59 PM, Michal Cichra wrote:
> Well. I need it to be multi platform. Because this can be deployed by users,
> it should be
> It is pretty common, that software can get trusted certificates from the OS.
>

It's the default set in OpenSSL. AFAIK, Linux does not have a standard
trusted cert set (though some individual Linux distributions might
have their own).

> I made this patch to nginx
> https://gist.githubusercontent.com/mikz/4ce4307329095114b109/raw/97c68b3f6a4f0a6f87fe8f1b7005f19484378fcc/gistfile1.diff
> that seems to work.
>

Well, you could propose this patch (or something polished) to the
nginx-devel mailing list.

>> I suggest you only load the CA certs you actually need to save runtime
>> overhead.
> It is HTTP client capable of connecting to any server, so it should (lazy)
> load all the trusted certificates from the OS.
> I have no idea how openssl works in the back. If adding CA path has same
> effect like loading all certificates, or if it is lazy loaded.

Use of SSL_CTX_set_default_verify_paths has extra overhead at least in
the memory footprint. For the "TEST 11: www.google.com (SSL verify
passes)" test case in ngx_lua's official test suite, I have the
following memory metrics of the nginx process, as reported by pmap:

mapped: 55840K writeable/private: 1276K (just the used CA cert loaded)
mapped: 58624K writeable/private: 4060K
(SSL_CTX_set_default_verify_paths)

It's nontrivial for some people :)

But I agree it is convenient for many people.

> If I would try to merge it to nginx (probably with a configuration flag),
> would you make the same configuration for lua?

Yes.

Regards,
-agentzh

michal

After doing some more investigation, I found that the ngx patch does not apply on ngx_lua, because ngx_ssl_trusted_certificate

is never called.

The certs are set up in https://github.com/openresty/lua-nginx-module/blob/227a5f0e5fa3314423e4430b2d6dce69224d01a4/src/ngx_http_lua_module.c#L887-L907 and if their lenght is 0, it never calls `ngx_ssl_trusted_certificate`.

Could we remove that condition and call it all the time? Then patching nginx would would fix both proxy_ssl_verify and lua verify.

But when doing these tests I've found that actually it does not fix my issue. Even when I explicitly load defaults paths, (or custom paths), they are not verified by openssl. Checked it by strace. These folders are never accessed.

Is the context object used for verification and passed to ngx_ssl_trusted_certificate the same? I was trying to debug it with gdb, found it extremely hard to see openssl source code, even when compiling it from source (with debug flags).

Best,

Michal

On Wednesday, 10 September 2014 19:55:40 UTC+2, Michal Cichra wrote:

Hi,
I'm doing http client using lua cosockets with ssl support and so far it works for me only on OSX.
For some reason on vanilla ubuntu 14.04 with compiled openresty 1.7.4.1rc2 it can't verify certificate with 'unable to get local issuer certificate'.

I briefly checked lua-nginx-module source and it delegates all the work to nginx. So in Nginx it looks like:
https://github.com/nginx/nginx/blob/1176952193ccf47078dc84b8494d0496ad1ac4a2/src/event/ngx_event_openssl.c#L508-L515

I found http://h71000.www7.hp.com/doc/83final/ba554_90007/ch04s03.html#d0e3371 explaining how to use it and it looks like Nginx is not using CApath at all.

It works on osx, maybe apple has some patches in openssl so it loads certificates all the time, even when it is not initialized.

Do you have suggestions how to make lua ssl sockets to trust os ca certificates?

Best,
Michal

michal

After more investigation, I've found that it sets verify_depth all the time to 1. Which prevents the system certificates to kick in.

Openssl does not set verify_depth unless it is set explicitly. My final hacky patch is https://gist.github.com/mikz/4dae10a0ef94de7c8139

Default verify_depth is there to limit the memory usage right? Openssl default for clients is -1. From what I checked, nginx can't parse -1 as a number. If I'll be making a patch to nginx to keep the openssl default. Would you think that `lua_ssl_verify_depth none` woud be proper name to do it? Same would apply to `proxy_ssl_verify_depth none`. Would you be ok with that?

Thanks and all the best.

Michal

On Wednesday, 10 September 2014 19:55:40 UTC+2, Michal Cichra wrote:

Hi,
I'm doing http client using lua cosockets with ssl support and so far it works for me only on OSX.
For some reason on vanilla ubuntu 14.04 with compiled openresty 1.7.4.1rc2 it can't verify certificate with 'unable to get local issuer certificate'.

I briefly checked lua-nginx-module source and it delegates all the work to nginx. So in Nginx it looks like:
https://github.com/nginx/nginx/blob/1176952193ccf47078dc84b8494d0496ad1ac4a2/src/event/ngx_event_openssl.c#L508-L515

I found http://h71000.www7.hp.com/doc/83final/ba554_90007/ch04s03.html#d0e3371 explaining how to use it and it looks like Nginx is not using CApath at all.

It works on osx, maybe apple has some patches in openssl so it loads certificates all the time, even when it is not initialized.

Do you have suggestions how to make lua ssl sockets to trust os ca certificates?

Best,
Michal