OpenResty as an upstream SSL proxy (Replacement for Oracle Service Bus)

rohit.c.joshi · 2014-08-16T05:01:17+00:00

This thread which landed around the same time on the list has info relevant to your inquiry:https://groups.google.com/forum/#!topic/openresty-en/QQTw0oO532QO...

OpenResty as an upstream SSL proxy (Replacement for Oracle Service Bus)

rohit.c.joshi

Hello,

Our existing architecture is as below in which I am planning to replace OSB layer with OpenResty.

Clients => Big IP/LB => Oracle Service Bus (OSB) => Web Logic application server/REST Api => Different backends

OSB Layer features we are using and need to replace with OpenResty:

Two way SSL connections (upstream and downstream). This is due to security requirements. Downstream (Clients) SSL certificates are different than upstream (toward app server)
Rewrites URL
Api Key validation against database
Creating a custom RequestId header (for end to end debugging)

I am looking at possibly using OpenResty as a replacement to Oracle Service Bus as OSB is very heavy and we are not using any OSB related features.

What I have done so far:

1. I have install and configured ngx_openresty-1.7.2.1.tar.gz

2. I have configured two way SSL for client to openresty using following config.

ssl_certificate cert.pem;

ssl_certificate_key cert-key .key;

ssl_client_certificatecert.pem;

ssl_verify_client on;

3. I am able to rewrite URL for upstream servers using following config. This will be replaced by lua script later.

location /foo/

{

rewrite ^/foo/(.*) /bar/$1/ break;

proxy_pass https://upstream:8230;

}

4. I am working on

a. integrating postgres module to validate the Api-Key which is coming as part of http header

b. populating required id into upstream http header

I need help configuring upstream two way SSL connection. Any help would be appreciated.

Thanks,

Joshi

agentzh

Hello!

On Fri, Aug 15, 2014 at 2:01 PM,  rohit.c.joshi wrote:
> OSB Layer features we are using and need to replace with OpenResty:
>
> * Two way SSL connections (upstream and downstream).   This is due to security
> requirements.  Downstream (Clients) SSL certificates are different than
> upstream (toward app server)

Nginx already supports this out of the box. Also, ngx_lua has Lua-land
SSL cosocket suppport just recently :)

> Rewrites URL

Trivial for the nginx world :)

> Api Key validation against database

What kind of database you're accessing? Hopefully it is not something
using proprietary wire protocol like the Oracle database.

> Creating a custom RequestId header (for end to end debugging)
>

Trivial for the ngx_lua or ngx_headers_more modules.

> I am looking at possibly using OpenResty as a replacement to Oracle Service
> Bus as OSB is very heavy and we are not using any OSB related features.
>

Sounds cool :)

> 3.  I am able to rewrite URL for upstream servers using following config.
> This will be replaced by lua script later.
> location /foo/
>         {
>             rewrite                 ^/foo/(.*)  /bar/$1/ break;
>             proxy_pass           https://upstream:8230;
>         }
>

You can also configure upstream SSL certificates (and etc) via the
proxy_ssl_trusted_certificate directive. Check out the official
documentation of ngx_proxy:

    http://nginx.org/en/docs/http/ngx_http_proxy_module.html

> 4.  I am working on
>     a.  integrating postgres module to validate the Api-Key which is coming
> as part of http header

Okay, you can use the ngx_postgres module with ngx_lua's
ngx.locaton.capture() + lua-resty-rds to access Pg or just choose from
those 3rd-party Lua libraries talking to Pg directly via ngx_lua's
cosocket API.

>     b.  populating required id into upstream http header
>

You can use the proxy_set_header directive provided by ngx_proxy for
it. Or just use ngx_lua's Lua API function ngx.req.set_header().

> I  need help configuring upstream two way SSL connection.  Any help would be
> appreciated.
>

See above.

Regards,
-agentzh

rohit.c.joshi

Thanks agentzh. There seem to be issue with upstream two way/mutual SSL certificate. Mutual SSL works fine with client to nginx but nginx to weblogic server gives below error.

2014/08/16 22:40:53 [debug] 33741#0: *9 SSL handshake handler: 0

2014/08/16 22:40:53 [debug] 33741#0: *9 SSL_do_handshake: -1

2014/08/16 22:40:53 [debug] 33741#0: *9 SSL_get_error: 2

2014/08/16 22:40:53 [debug] 33741#0: timer delta: 5

2014/08/16 22:40:53 [debug] 33741#0: posted events 0000000000000000

2014/08/16 22:40:53 [debug] 33741#0: worker cycle

2014/08/16 22:40:53 [debug] 33741#0: kevent timer: 59840, changes: 0

2014/08/16 22:40:53 [debug] 33741#0: kevent events: 2

2014/08/16 22:40:53 [debug] 33741#0: kevent: 7: ft:-2 fl:0025 ff:00000000 d:131520 ud:00007FF263805150

2014/08/16 22:40:53 [debug] 33741#0: *9 kevent: 7: ft:-2 fl:0025 ff:00000000 d:131520 ud:00007FF263805150

2014/08/16 22:40:53 [debug] 33741#0: *9 SSL handshake handler: 1

2014/08/16 22:40:53 [debug] 33741#0: *9 SSL_do_handshake: 0

2014/08/16 22:40:53 [debug] 33741#0: *9 SSL_get_error: 1

SSL_do_handshake() failed (SSL: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:SSL alert number 40) while SSL handshaking to upstream

client: 172.18.44.166, server: 172.18.44.166, request: "GET /customers/~/xxxx/~/xxx/health HTTP/1.1", upstream: "https://10.42.16.196:11211/customer-service/~/xxx/~/xxx/health/", host: "172.18.44.166:12121"

Here is my nginx configuration for upstream:

upstream rs_backend {

server 10.42.16.196:11211;

}

server {

server_name 172.18.44.166;

listen 172.18.44.166:12121 ssl;

ssl on;

ssl_verify_client on;

ssl_session_cache shared:SSL:1m;

ssl_session_timeout 10m;

ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;

#ssl_protocols TLSv1;

#ssl_ciphers SSL_RSA_WITH_RC4_128_MD5:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES128-SHA:AES128-GCM-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH:!CAMELLIA;

#ssl_ciphers HIGH:!MD5:!aNULL:!EDH:!CAMELLIA;

ssl_prefer_server_ciphers on;

proxy_ssl_session_reuse off;

large_client_header_buffers 4 32K;

ssl_certificate /etc/ssl/api-qaid.pem;

ssl_certificate_key /etc/ssl/api-qaid.key;

ssl_client_certificate /etc/ssl/api-qaid.pem;

location /customers/

{

rewrite ^/customers/(.*) /customer-service/$1/ break;

proxy_redirect off;

proxy_ssl_verify on;

proxy_ssl_verify_depth 4;

proxy_ssl_trusted_certificate /etc/ssl/api-qaid.pem;

proxy_pass_header Server;

proxy_http_version 1.1;

proxy_set_header Connection Keep-Alive;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header Host 172.18.44.166:11211;

proxy_set_header Accept 'application/json;v=3';

proxy_pass https://10.42.16.196:11211/;

#proxy_pass https://rs_backend;

}

On Saturday, 16 August 2014 13:45:33 UTC-4, agentzh wrote:

Hello!

On Fri, Aug 15, 2014 at 2:01 PM, rohit.c.joshi wrote:
> OSB Layer features we are using and need to replace with OpenResty:
>
> * Two way SSL connections (upstream and downstream). This is due to security
> requirements. Downstream (Clients) SSL certificates are different than
> upstream (toward app server)

Nginx already supports this out of the box. Also, ngx_lua has Lua-land
SSL cosocket suppport just recently :)

> Rewrites URL

Trivial for the nginx world :)

> Api Key validation against database

What kind of database you're accessing? Hopefully it is not something
using proprietary wire protocol like the Oracle database.

> Creating a custom RequestId header (for end to end debugging)
>

Trivial for the ngx_lua or ngx_headers_more modules.

> I am looking at possibly using OpenResty as a replacement to Oracle Service
> Bus as OSB is very heavy and we are not using any OSB related features.
>

Sounds cool :)

> 3. I am able to rewrite URL for upstream servers using following config.
> This will be replaced by lua script later.
> location /foo/
> {
> rewrite ^/foo/(.*) /bar/$1/ break;
> proxy_pass https://upstream:8230;
> }
>

You can also configure upstream SSL certificates (and etc) via the
proxy_ssl_trusted_certificate directive. Check out the official
documentation of ngx_proxy:

http://nginx.org/en/docs/http/ngx_http_proxy_module.html

> 4. I am working on
> a. integrating postgres module to validate the Api-Key which is coming
> as part of http header

Okay, you can use the ngx_postgres module with ngx_lua's
ngx.locaton.capture() + lua-resty-rds to access Pg or just choose from
those 3rd-party Lua libraries talking to Pg directly via ngx_lua's
cosocket API.

> b. populating required id into upstream http header
>

You can use the proxy_set_header directive provided by ngx_proxy for
it. Or just use ngx_lua's Lua API function ngx.req.set_header().

> I need help configuring upstream two way SSL connection. Any help would be
> appreciated.
>

See above.

Regards,
-agentzh

rohit.c.joshi

I am getting exactly same error as

http://forum.nginx.org/read.php?2,245971,245971

On Saturday, 16 August 2014 13:45:33 UTC-4, agentzh wrote:

Hello!

On Fri, Aug 15, 2014 at 2:01 PM, rohit.c.joshi wrote:
> OSB Layer features we are using and need to replace with OpenResty:
>
> * Two way SSL connections (upstream and downstream). This is due to security
> requirements. Downstream (Clients) SSL certificates are different than
> upstream (toward app server)

Nginx already supports this out of the box. Also, ngx_lua has Lua-land
SSL cosocket suppport just recently :)

> Rewrites URL

Trivial for the nginx world :)

> Api Key validation against database

What kind of database you're accessing? Hopefully it is not something
using proprietary wire protocol like the Oracle database.

> Creating a custom RequestId header (for end to end debugging)
>

Trivial for the ngx_lua or ngx_headers_more modules.

> I am looking at possibly using OpenResty as a replacement to Oracle Service
> Bus as OSB is very heavy and we are not using any OSB related features.
>

Sounds cool :)

> 3. I am able to rewrite URL for upstream servers using following config.
> This will be replaced by lua script later.
> location /foo/
> {
> rewrite ^/foo/(.*) /bar/$1/ break;
> proxy_pass https://upstream:8230;
> }
>

You can also configure upstream SSL certificates (and etc) via the
proxy_ssl_trusted_certificate directive. Check out the official
documentation of ngx_proxy:

http://nginx.org/en/docs/http/ngx_http_proxy_module.html

> 4. I am working on
> a. integrating postgres module to validate the Api-Key which is coming
> as part of http header

Okay, you can use the ngx_postgres module with ngx_lua's
ngx.locaton.capture() + lua-resty-rds to access Pg or just choose from
those 3rd-party Lua libraries talking to Pg directly via ngx_lua's
cosocket API.

> b. populating required id into upstream http header
>

You can use the proxy_set_header directive provided by ngx_proxy for
it. Or just use ngx_lua's Lua API function ngx.req.set_header().

> I need help configuring upstream two way SSL connection. Any help would be
> appreciated.
>

See above.

Regards,
-agentzh

rohit.c.joshi

If I try using openssl c_client command line, I am able to submit HTTP GET request and get 2xx response

openssl c_client -connect 10.42.16.196:11211 -cert api-qaid-nopass.pem

On Saturday, 16 August 2014 22:56:24 UTC-4, RJoshi wrote:

Thanks agentzh. There seem to be issue with upstream two way/mutual SSL certificate. Mutual SSL works fine with client to nginx but nginx to weblogic server gives below error.

2014/08/16 22:40:53 [debug] 33741#0: *9 SSL handshake handler: 0
2014/08/16 22:40:53 [debug] 33741#0: *9 SSL_do_handshake: -1
2014/08/16 22:40:53 [debug] 33741#0: *9 SSL_get_error: 2
2014/08/16 22:40:53 [debug] 33741#0: timer delta: 5
2014/08/16 22:40:53 [debug] 33741#0: posted events 0000000000000000
2014/08/16 22:40:53 [debug] 33741#0: worker cycle
2014/08/16 22:40:53 [debug] 33741#0: kevent timer: 59840, changes: 0
2014/08/16 22:40:53 [debug] 33741#0: kevent events: 2
2014/08/16 22:40:53 [debug] 33741#0: kevent: 7: ft:-2 fl:0025 ff:00000000 d:131520 ud:00007FF263805150
2014/08/16 22:40:53 [debug] 33741#0: *9 kevent: 7: ft:-2 fl:0025 ff:00000000 d:131520 ud:00007FF263805150
2014/08/16 22:40:53 [debug] 33741#0: *9 SSL handshake handler: 1
2014/08/16 22:40:53 [debug] 33741#0: *9 SSL_do_handshake: 0
2014/08/16 22:40:53 [debug] 33741#0: *9 SSL_get_error: 1
SSL_do_handshake() failed (SSL: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:SSL alert number 40) while SSL handshaking to upstream
client: 172.18.44.166, server: 172.18.44.166, request: "GET /customers/~/xxxx/~/xxx/health HTTP/1.1", upstream: "https://10.42.16.196:11211/customer-service/~/xxx/~/xxx/health/", host: "172.18.44.166:12121"

Here is my nginx configuration for upstream:

upstream rs_backend {
server 10.42.16.196:11211;
}
server {
server_name 172.18.44.166;
listen 172.18.44.166:12121 ssl;

ssl on;
ssl_verify_client on;
ssl_session_cache shared:SSL:1m;
ssl_session_timeout 10m;
ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
#ssl_protocols TLSv1;
#ssl_ciphers SSL_RSA_WITH_RC4_128_MD5:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES128-SHA:AES128-GCM-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH:!CAMELLIA;
#ssl_ciphers HIGH:!MD5:!aNULL:!EDH:!CAMELLIA;
ssl_prefer_server_ciphers on;
proxy_ssl_session_reuse off;
large_client_header_buffers 4 32K;

ssl_certificate /etc/ssl/api-qaid.pem;
ssl_certificate_key /etc/ssl/api-qaid.key;
ssl_client_certificate /etc/ssl/api-qaid.pem;

location /customers/
{
rewrite ^/customers/(.*) /customer-service/$1/ break;
proxy_redirect off;
proxy_ssl_verify on;
proxy_ssl_verify_depth 4;
proxy_ssl_trusted_certificate /etc/ssl/api-qaid.pem;

proxy_pass_header Server;
proxy_http_version 1.1;
proxy_set_header Connection Keep-Alive;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host 172.18.44.166:11211;
proxy_set_header Accept 'application/json;v=3';

proxy_pass https://10.42.16.196:11211/;
#proxy_pass https://rs_backend;
}

On Saturday, 16 August 2014 13:45:33 UTC-4, agentzh wrote:
Hello!

On Fri, Aug 15, 2014 at 2:01 PM, rohit.c.joshi wrote:
> OSB Layer features we are using and need to replace with OpenResty:
>
> * Two way SSL connections (upstream and downstream). This is due to security
> requirements. Downstream (Clients) SSL certificates are different than
> upstream (toward app server)

Nginx already supports this out of the box. Also, ngx_lua has Lua-land
SSL cosocket suppport just recently :)

> Rewrites URL

Trivial for the nginx world :)

> Api Key validation against database

What kind of database you're accessing? Hopefully it is not something
using proprietary wire protocol like the Oracle database.

> Creating a custom RequestId header (for end to end debugging)
>

Trivial for the ngx_lua or ngx_headers_more modules.

> I am looking at possibly using OpenResty as a replacement to Oracle Service
> Bus as OSB is very heavy and we are not using any OSB related features.
>

Sounds cool :)

> 3. I am able to rewrite URL for upstream servers using following config.
> This will be replaced by lua script later.
> location /foo/
> {
> rewrite ^/foo/(.*) /bar/$1/ break;
> proxy_pass https://upstream:8230;
> }
>

You can also configure upstream SSL certificates (and etc) via the
proxy_ssl_trusted_certificate directive. Check out the official
documentation of ngx_proxy:

http://nginx.org/en/docs/http/ngx_http_proxy_module.html

> 4. I am working on
> a. integrating postgres module to validate the Api-Key which is coming
> as part of http header

Okay, you can use the ngx_postgres module with ngx_lua's
ngx.locaton.capture() + lua-resty-rds to access Pg or just choose from
those 3rd-party Lua libraries talking to Pg directly via ngx_lua's
cosocket API.

> b. populating required id into upstream http header
>

You can use the proxy_set_header directive provided by ngx_proxy for
it. Or just use ngx_lua's Lua API function ngx.req.set_header().

> I need help configuring upstream two way SSL connection. Any help would be
> appreciated.
>

See above.

Regards,
-agentzh

agentzh

Hello!

On Sat, Aug 16, 2014 at 7:56 PM,  rohit.c.joshi wrote:
> Thanks agentzh.   There seem to be issue with upstream two way/mutual SSL
> certificate.  Mutual SSL works fine with client to nginx but nginx to
> weblogic server gives below error.
>
> 2014/08/16 22:40:53 [debug] 33741#0: *9 SSL_get_error: 1
> SSL_do_handshake() failed (SSL: error:14094410:SSL
> routines:SSL3_READ_BYTES:sslv3 alert handshake failure:SSL alert number 40)
> while SSL handshaking to upstream

Seems like your remote server asks for client certificate, which is
not supported by the standard ngx_proxy module AFAIK. You may consider
asking the official nginx team for this new feature on the nginx
mailing list :)

Regards,
-agentzh

rohit.c.joshi

I have added support for upstream proxy with two way/mutual SSL and sent a pull request :)

https://github.com/nginx/nginx/pull/7

Now I can configured using certificate and key.

#proxy_ssl_trusted_certificate trustedca.pem;

proxy_ssl_certificate cert.pem;

proxy_ssl_certificate_key cert.key;

On Mon, Aug 18, 2014 at 3:09 PM, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

Hello!

On Sat, Aug 16, 2014 at 7:56 PM, rohit.c.joshi wrote:
> Thanks agentzh. There seem to be issue with upstream two way/mutual SSL
> certificate. Mutual SSL works fine with client to nginx but nginx to
> weblogic server gives below error.
>

> 2014/08/16 22:40:53 [debug] 33741#0: *9 SSL_get_error: 1
> SSL_do_handshake() failed (SSL: error:14094410:SSL
> routines:SSL3_READ_BYTES:sslv3 alert handshake failure:SSL alert number 40)
> while SSL handshaking to upstream

Seems like your remote server asks for client certificate, which is
not supported by the standard ngx_proxy module AFAIK. You may consider
asking the official nginx team for this new feature on the nginx
mailing list :)

Regards,
-agentzh.

--
This e-mail and any attachment is for authorized use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.

agentzh

Hello!

On Mon, Aug 18, 2014 at 1:07 PM, Rohit Joshi wrote:
> I have added support for upstream proxy with two way/mutual SSL and sent a
> pull request :)
>
> https://github.com/nginx/nginx/pull/7
>

Awesome!

> Now  I can configured using certificate and key.
>
>  proxy_ssl_certificate cert.pem;
>  proxy_ssl_certificate_key  cert.key;
>

But IMHO proxy_ssl_client_certificate might be a better name ;) (The
same applies to proxy_ssl_certificate_key).

Regards,
-agentzh

rohit.c.joshi

Thank for your suggestion. Changed both names and committed.

On Monday, 18 August 2014 16:17:44 UTC-4, agentzh wrote:

Hello!

On Mon, Aug 18, 2014 at 1:07 PM, Rohit Joshi wrote:
> I have added support for upstream proxy with two way/mutual SSL and sent a
> pull request :)
>
> https://github.com/nginx/nginx/pull/7
>

Awesome!

> Now I can configured using certificate and key.
>
> proxy_ssl_certificate cert.pem;
> proxy_ssl_certificate_key cert.key;
>

But IMHO proxy_ssl_client_certificate might be a better name ;) (The
same applies to proxy_ssl_certificate_key).

Regards,
-agentzh

rohit.c.joshi

Here is a blog post describing how to apply patch and configure SSL certificates.

Nginx: Mutual (Two way) SSL authentication for upstream HTTPS servers

On Monday, 18 August 2014 16:57:52 UTC-4, RJoshi wrote:

Thank for your suggestion. Changed both names and committed.

On Monday, 18 August 2014 16:17:44 UTC-4, agentzh wrote:
Hello!

On Mon, Aug 18, 2014 at 1:07 PM, Rohit Joshi wrote:
> I have added support for upstream proxy with two way/mutual SSL and sent a
> pull request :)
>
> https://github.com/nginx/nginx/pull/7
>

Awesome!

> Now I can configured using certificate and key.
>
> proxy_ssl_certificate cert.pem;
> proxy_ssl_certificate_key cert.key;
>

But IMHO proxy_ssl_client_certificate might be a better name ;) (The
same applies to proxy_ssl_certificate_key).

Regards,
-agentzh