ssl session caching

sreekanthskm · 2016-02-09T00:07:14+00:00

Hello! On Tue, Feb 9, 2016 at 1:41 AM, kaspersky_us wrote: > I followed your advice and my solution still uses a single core on my > machine. Below is...

ssl session caching

sreekanthskm

Hi,

Good to see the external session caching support in formal release, 1.9.7.2.

feature: applied the ssl_cert_cb_yield patch to the bundled
version of the NGINX core to allow yielding in OpenSSL's
SSL_CTX_set_cert_cb() callbacks (needed by the ngx_lua module's
ssl_certificate_by_lua* directives, for example).

It would be great if you could share some performance test results for ssl external caching feature.

We have recently conducted some tests using nginx-1.0.10 and results does not look good. Details as below:

Test setup

We used nginx-1.0.10 and tested it for shared memory cache. Client sends around 200 Requests/sec and 50% of requests reuses ssl session.

OpenSSL version used is OpenSSL 1.0.1e-fips 11 Feb 2013.

The operational mode used for session caching is as below:

SSL_CTX_set_session_cache_mode(ssl->ctx, SSL_SESS_CACHE_SERVER | SSL_SESS_CACHE_NO_INTERNAL);

Results:

We are seeing increase in response times and it looks like an issue with OpenSSL.

Please find attached the request and connect time graph for reference.

Any feed back would be appreciated.

Thanks,

Sreekanth

Attachment: connect_request_time.png
Description: PNG image

agentzh

Hello!

On Mon, Feb 8, 2016 at 8:07 AM, sreekanth Madhavan wrote:
> Good to see the external session caching support in formal release, 1.9.7.2.
>
>  feature: applied the ssl_cert_cb_yield patch to the bundled
>      version of the NGINX core to allow yielding in OpenSSL's
>      SSL_CTX_set_cert_cb() callbacks (needed by the ngx_lua module's
>      ssl_certificate_by_lua* directives, for example).
>
> It would be great if you could share some performance test results for ssl
> external caching feature.
>

We've been profiling our online SSL gateway using this feature at
CloudFlare's production environment.

Our ssl_certificate_by_lua handler just takes less than 300us CPU time
on average for each SSL handshake request on a common production
server with real traffic, and even for the 300us CPU time, almost all
the time is spent on OpenSSL. Well, 300us is really nothing as
compared to the full SSL handshake's overhead :)

The full SSL handshake is inevitable expensive though there exists
optimizations based on leveraging special hardware instructions of
modern CPUs (like Intel AES-NI Engine of OpenSSL in your nginx.conf).
The more important optimization is to configure the SSL session cache
to maximize SSL session resumption so that re-visiting clients do not
need to do full SSL handshake at all.

The upcoming feature of ngx_lua that allows distributed SSL session
caching on the server side may help maximize SSL resumption's success
rate in a distributed environment. (Note, new SSL clients supporting
TLS session tickets do not need any server-side session caching at
all, which is superior than the SSL session ID technique).

For session resumptions, either SSL session ID or TLS session tickets,
the ssl_certificiate_by_lua handler is skipped altogether, making
things even a bit faster :)

> Results:
>
> We are seeing increase in response times and it looks like an issue with
> OpenSSL.
> Please find attached the request and connect time graph for reference.
>

For profiling, always use the various kinds of flame graph tools:

    https://openresty.org/#Profiling

Regards,
-agentzh

sreekanthskm

The upcoming feature of ngx_lua that allows distributed SSL session
caching on the server side may help maximize SSL resumption's success
rate in a distributed environment. (Note, new SSL clients supporting
TLS session tickets do not need any server-side session caching at
all, which is superior than the SSL session ID technique).

When is the distributed SSL session caching feature release planned ?

I understand that the ssl_certificate_by_la handler takes less CPU time. We found that response time increases with session caching as compared to non session caching.

Connect time goes to around 200 ms with session caching and it is just 70 ms. Please refer the attached graphs.

I think this is due to some issue with external session caching handling in OpenSSL code.

Have you faced similar issue in your testing ?

Which version of OpenSSL is being used for your testing ? Is it openSSL or some commercial one ?

Thanks,

Sreekanth

On Tuesday, February 9, 2016 at 1:06:25 AM UTC+5:30, agentzh wrote:

Hello!

On Mon, Feb 8, 2016 at 8:07 AM, sreekanth Madhavan wrote:
> Good to see the external session caching support in formal release, 1.9.7.2.
>
> feature: applied the ssl_cert_cb_yield patch to the bundled
> version of the NGINX core to allow yielding in OpenSSL's
> SSL_CTX_set_cert_cb() callbacks (needed by the ngx_lua module's
> ssl_certificate_by_lua* directives, for example).
>
> It would be great if you could share some performance test results for ssl
> external caching feature.
>

We've been profiling our online SSL gateway using this feature at
CloudFlare's production environment.

Our ssl_certificate_by_lua handler just takes less than 300us CPU time
on average for each SSL handshake request on a common production
server with real traffic, and even for the 300us CPU time, almost all
the time is spent on OpenSSL. Well, 300us is really nothing as
compared to the full SSL handshake's overhead :)

The full SSL handshake is inevitable expensive though there exists
optimizations based on leveraging special hardware instructions of
modern CPUs (like Intel AES-NI Engine of OpenSSL in your nginx.conf).
The more important optimization is to configure the SSL session cache
to maximize SSL session resumption so that re-visiting clients do not
need to do full SSL handshake at all.

The upcoming feature of ngx_lua that allows distributed SSL session
caching on the server side may help maximize SSL resumption's success
rate in a distributed environment. (Note, new SSL clients supporting
TLS session tickets do not need any server-side session caching at
all, which is superior than the SSL session ID technique).

For session resumptions, either SSL session ID or TLS session tickets,
the ssl_certificiate_by_lua handler is skipped altogether, making
things even a bit faster :)

> Results:
>
> We are seeing increase in response times and it looks like an issue with
> OpenSSL.
> Please find attached the request and connect time graph for reference.
>

For profiling, always use the various kinds of flame graph tools:

https://openresty.org/#Profiling

Regards,
-agentzh

Attachment: no_session_caching.png
Description: PNG image

Attachment: session_caching.png
Description: PNG image

erik.l.nelson

I need to interact with an SFTP server from my openresty scripts- does anyone have some tips?  I saw some conversations a couple of years ago about the idea, but nothing very recent.

Erik

----------------------------------------------------------------------
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer.   If you are not the intended recipient, please delete this message.

rpaprocki

I'm markedly curious (and terrified) about your use case- can you expand a bit more on why an HTTP server would need to directly establish a remote shell to another machine? This sounds like a security nightmare.

I would guess (if you really hated yourself) that you could write a lib using cosockets, but that sounds... awful.

Wrapping around a C lib using FFI -might- be feasible, but I would assume be blocking.

On Tue, Feb 9, 2016 at 7:30 AM, Nelson, Erik - 2 <erik.l...@bankofamerica.com> wrote:

I need to interact with an SFTP server from my openresty scripts- does anyone have some tips? I saw some conversations a couple of years ago about the idea, but nothing very recent.

Erik

----------------------------------------------------------------------
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
.

erik.l.nelson

>Robert Paprocki Sent: Tuesday, February 09, 2016 11:16 AM

>I'm markedly curious (and terrified) about your use case- can you
>expand a bit more on why an HTTP server would need to directly
>establish a remote shell to another machine? This sounds
>like a security nightmare.

It's just uploading log files over an internal network.

>>On Tue, Feb 9, 2016 at 7:30 AM, Nelson, Erik - 2 <erik.l.nelson@
>>bankofamerica.com> wrote:
>>I need to interact with an SFTP server from my openresty scripts-
>>does anyone have some tips?  I saw some conversations a couple of
>>years ago about the idea, but nothing very recent.



----------------------------------------------------------------------
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer.   If you are not the intended recipient, please delete this message.

rpaprocki

Perhaps something like https://github.com/cloudflare/lua-resty-logger-socket? It might be a bit more work to write a wrapper with authentication, but it's efficient (and sane).

On Tue, Feb 9, 2016 at 8:19 AM, Nelson, Erik - 2 <erik.l...@bankofamerica.com> wrote:

>Robert Paprocki Sent: Tuesday, February 09, 2016 11:16 AM

>I'm markedly curious (and terrified) about your use case- can you
>expand a bit more on why an HTTP server would need to directly
>establish a remote shell to another machine? This sounds
>like a security nightmare.

It's just uploading log files over an internal network.

>>On Tue, Feb 9, 2016 at 7:30 AM, Nelson, Erik - 2 <erik.l.nelson@
>>bankofamerica.com> wrote:
>>I need to interact with an SFTP server from my openresty scripts-
>>does anyone have some tips? I saw some conversations a couple of
>>years ago about the idea, but nothing very recent.

----------------------------------------------------------------------
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message..

erik.l.nelson

>Robert Paprocki Sent: Tuesday, February 09, 2016 11:22 AM
>Perhaps something like https://github.com/cloudflare/lua-resty-logger-socket? It might be a bit more work to write a wrapper with authentication, but it's efficient (and sane).

Thanks


----------------------------------------------------------------------
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer.   If you are not the intended recipient, please delete this message.

agentzh

Hello!

On Tue, Feb 9, 2016 at 1:15 AM, sreekanth Madhavan wrote:
>
> When is the distributed SSL  session caching feature release planned ?
>

Soon :) Maybe in the next few weeks or so.

> I understand that the ssl_certificate_by_la handler takes less CPU time.  We
> found that response time increases with session caching as compared to non
> session caching.
> Connect time goes to around 200 ms with session caching and it is just 70
> ms. Please refer the attached graphs.
>

Are you sure session resumption indeed succeeds? This looks wrong to
me. It's easy to do false benchmarks. See

    http://www.brendangregg.com/activebenchmarking.html

> I think this is due to some issue with external session caching handling in
> OpenSSL code.
>

Please do profiling based on flame graphs and etc. Guessing can easily
waste our time.

> Have you faced similar issue in your testing ?
>

Nope.

> Which version of OpenSSL is being used for your testing ? Is it openSSL or
> some commercial one ?
>

OpenSSL 1.0.2 with some custom patches (mostly for new features
instead of bug fixes).

Regards,
-agentzh