OpenResty in Kubernetes, DNS and resolv.conf issues

gmcdaid · 2019-05-03T22:40:05+00:00

Can someone give me some ideas?

OpenResty in Kubernetes, DNS and resolv.conf issues

gmcdaid

This post refers to latest OpenResty image available at:

https://github.com/openresty/docker-openresty/blob/master/stretch/Dockerfile.fat

I'm trying to use OpenResty in a Kubernetes cluster. The code relies on coroutines, which call out to proxy_pass locations.

The hosts in proxy_pass directives are Kubernetes service names, but the namespace differs depending on the environment. The code therefore needs to parse the search list in resolv.conf in order to fully expand the hostname for the relevant namespace.

proxy_pass http://my-service/api;

# cat /etc/resolv.conf
nameserver 169.254.1.1
search staging.svc.cluster.local svc.cluster.local cluster.local

So, request to

my-service

should be expanded to

my-service.staging.svc.cluster.local

I have verified from the OS layer than the search list is working as expected.

# ping my-service
PING my-service (10.231.64.104): 56 data bytes

I have tried to configure OpenResty/Nginx as follows to make it parse /etc/resolv.conf, as per docs (https://github.com/openresty/openresty/#resolvconf-parsing):

resolver 169.254.1.1 local=on local=/etc/resolv.conf ipv6=off;

However, in my Lua logs, I see the following error:

[error] 8#8: *1 my-service could not be resolved (3: Host not found)

thibaultcha

Hello,

As far as I recall, the NGINX resolver does not support search domains,
and thus, the resolver_conf_parsing patch[1] doesn't either.

I am not sure if NGINX Plus supports it. Possibly, considering the
resolver bundled with it supports SRV records[2], but I don't have
access to an NGINX Plus instance to try it.

My suggestion would be to run dnsmasq[3] alongside OpenResty and
configure the NGINX 'resolver' directive to point to it, like so:

    resolver 127.0.0.1:53

Or using a pure Lua-land DNS resolver, such as:

    https://github.com/Kong/lua-resty-dns-client

It supports search domains and various resolv.conf options (see 'man 5
resolv.conf'), even if specified as environment variables.

Best,
Thibault

[1]:
https://github.com/openresty/openresty/blob/master/patches/nginx-1.15.8-resolver_conf_parsing.patch
[2]: https://www.nginx.com/blog/dns-service-discovery-nginx-plus/
[3]: http://thekelleys.org.uk/dnsmasq/doc.html

On 5/3/19 7:40 AM, gmcdaid via openresty-en wrote:
> This post refers to latest OpenResty image available at:
> 
> https://github.com/openresty/docker-openresty/blob/master/stretch/Dockerfile.fat
> 
> I'm trying to use OpenResty in a Kubernetes cluster. The code relies on
> coroutines, which call out to proxy_pass locations.
> 
> The hosts in proxy_pass directives are Kubernetes service names, but the
> namespace differs depending on the environment. The code therefore needs
> to parse the search list in resolv.conf in order to fully expand the
> hostname for the relevant namespace.
> 
> eg
> 
> |
> proxy_pass http://my-service/api;
> |
> 
> |
> # cat /etc/resolv.conf
> nameserver 169.254.1.1
> search staging.svc.cluster.local svc.cluster.local cluster.local
> |
> 
> So, request to 
> |
> my-service
> |
> 
> should be expanded to 
> |
> my-service.staging.svc.cluster.local
> |
> 
> I have verified from the OS layer than the search list is working as
> expected.
> 
> |
> # ping my-service
> PING my-service (10.231.64.104): 56 data bytes
> |
> 
> 
> I have tried to configure OpenResty/Nginx as follows to make it parse
> /etc/resolv.conf, as per docs
> (https://github.com/openresty/openresty/#resolvconf-parsing):
> |
> resolver 169.254.1.1 local=on local=/etc/resolv.conf ipv6=off;
> |
> 
> 
> However, in my Lua logs, I see the following error:
> |
> [error] 8#8: *1 my-service could not be resolved (3: Host not found)
> |.

bianji8626

When local=on, the standard path of /etc/resolv.conf will be used. You may also specify arbitrary path to be used for parsing, for example: local=/tmp/test.conf

在 2019年5月3日星期五 UTC+8下午10:40:06，gmc...@zendesk.com写道：

This post refers to latest OpenResty image available at:

https://github.com/openresty/docker-openresty/blob/master/stretch/Dockerfile.fat

I'm trying to use OpenResty in a Kubernetes cluster. The code relies on coroutines, which call out to proxy_pass locations.

The hosts in proxy_pass directives are Kubernetes service names, but the namespace differs depending on the environment. The code therefore needs to parse the search list in resolv.conf in order to fully expand the hostname for the relevant namespace.

eg

proxy_pass http://my-service/api;

# cat /etc/resolv.conf nameserver 169.254.1.1 search staging.svc.cluster.local svc.cluster.local cluster.local

So, request to
my-service

should be expanded to
my-service.staging.svc.cluster.local

I have verified from the OS layer than the search list is working as expected.

# ping my-service PING my-service (10.231.64.104): 56 data bytes

I have tried to configure OpenResty/Nginx as follows to make it parse /etc/resolv.conf, as per docs (https://github.com/openresty/openresty/#resolvconf-parsing):
resolver 169.254.1.1 local=on local=/etc/resolv.conf ipv6=off;

However, in my Lua logs, I see the following error:
[error] 8#8: *1 my-service could not be resolved (3: Host not found)