Re: about the ngx_lua socket APIs' dependency on ngx_http_core_module.resolver directive

agentzh · 2013-07-10T02:48:49+00:00

Hello! On Thu, Jul 4, 2013 at 1:22 AM, Călin Don wrote: > Is there any way to tap into the USR1 in order to reopen the log files > as nginx woul...

Re: about the ngx_lua socket APIs' dependency on ngx_http_core_module.resolver directive

agentzh

Hello!

On Mon, Jul 8, 2013 at 11:18 PM, Glen Cao wrote:
> First thank you very much for providing such great ngx_lua.
>

Glad you like it :)

> ATT.
> We could eliminate the ngx_lua socket APIs' dependency on
> ngx_http_core_module.resolver directive, couldn't we?
>

I don't want to. Maybe you just want to support the /etc/resolv.conf
and /etc/hosts files? We *could* make the nginx core's resolver
support that.

> In the implementations of APIs of udpsocket:setpeername and
> tcpsocket:connect, we accept the first argument host in forms of both IP and
> domain name, and then call ngx_resolve_start to get the peer address.
> According to the implementations of ngx_resolve_start and ngx_inet_addr, we
> depend on the resolver derective closely.
>

If you're using IP addresses directly in setpeername() or connect(),
then you don't have to configure the "resolver" directive in
nginx.conf. Only when you use a domain name, the "resolver" directive
is required according to the current implementation.

Therefore, you can use external resolvers like lua-resty-dns to do the
DNS lookup completely in Lua and feed the IP address manually into the
setpeername() and connect() calls:

    https://github.com/agentzh/lua-resty-dns

> Could we pre-process the pass-in host argument before we use that arugment
> and call ngx_resolve_start? e.g., We could use gethostbyname or getaddrinfo
> to get the IP of host which is a domain name, and then call
> ngx_resolve_start as usual (or even call ngx_inet_addr directly, I'm not
> sure), so that we need not depend on the ngx_http_core_module.resolver
> directive.
>

No, both the gethostbyname and getaddrinfo API functions are blocking,
which is unacceptable for event-driven single-threaded applications
like Nginx. And I believe that's also why Nginx implements its own
nonblocking resolver from scratch in the first place.

BTW, you're recommended to join the openresty-en mailing list and post
such questions there:

    https://groups.google.com/group/openresty-en

Thank you for your collaborations! I'm cc'ing this list.

Best regards,
-agentzh

microwish

Thanks a lot for your reply.

I'd had headache about setpeername() and connect() receiving domain names as the first argument (actually 2nd), before Chaos suggested local DNS relay and you suggested lua-resty-dns.

I've thought about using async name resolving such as getaddrinfo_a from glibc, but Chaos reminded me of the like efficiency issue.

I also thought about making Nginx core supporting /etc/resolve.conf, but that might lead to an alone branch and become an obstacle in updating Nginx.

Thank you very much. I'll be try all of your suggestions.

Br,

Glen

On Wed, Jul 10, 2013 at 2:48 AM, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

Hello!

On Mon, Jul 8, 2013 at 11:18 PM, Glen Cao wrote:
> First thank you very much for providing such great ngx_lua.
>

Glad you like it :)

> ATT.
> We could eliminate the ngx_lua socket APIs' dependency on
> ngx_http_core_module.resolver directive, couldn't we?
>

I don't want to. Maybe you just want to support the /etc/resolv.conf
and /etc/hosts files? We *could* make the nginx core's resolver
support that.

> In the implementations of APIs of udpsocket:setpeername and
> tcpsocket:connect, we accept the first argument host in forms of both IP and
> domain name, and then call ngx_resolve_start to get the peer address.
> According to the implementations of ngx_resolve_start and ngx_inet_addr, we
> depend on the resolver derective closely.
>

If you're using IP addresses directly in setpeername() or connect(),
then you don't have to configure the "resolver" directive in
nginx.conf. Only when you use a domain name, the "resolver" directive
is required according to the current implementation.

Therefore, you can use external resolvers like lua-resty-dns to do the
DNS lookup completely in Lua and feed the IP address manually into the
setpeername() and connect() calls:

https://github.com/agentzh/lua-resty-dns

> Could we pre-process the pass-in host argument before we use that arugment
> and call ngx_resolve_start? e.g., We could use gethostbyname or getaddrinfo
> to get the IP of host which is a domain name, and then call
> ngx_resolve_start as usual (or even call ngx_inet_addr directly, I'm not
> sure), so that we need not depend on the ngx_http_core_module.resolver
> directive.
>

No, both the gethostbyname and getaddrinfo API functions are blocking,
which is unacceptable for event-driven single-threaded applications
like Nginx. And I believe that's also why Nginx implements its own
nonblocking resolver from scratch in the first place.

BTW, you're recommended to join the openresty-en mailing list and post
such questions there:

https://groups.google.com/group/openresty-en

Thank you for your collaborations! I'm cc'ing this list.

Best regards,
-agentzh

agentzh

Hello!

On Tue, Jul 9, 2013 at 8:02 PM, Glen Cao wrote:
> I'd had headache about setpeername() and connect() receiving domain names as
> the first argument (actually 2nd), before Chaos suggested local DNS relay
> and you suggested lua-resty-dns.
>

Why not just configure the "resolver" directive in your nginx.conf?

> I've thought about using async name resolving such as getaddrinfo_a from
> glibc, but Chaos reminded me of the like efficiency issue.
>

No, getaddrinfo_a sucks in that it uses pthreads internally AFAIK
which is still blocking OS threads per se, AFAIK. Not to mention it is
not portable.

> I also thought about making Nginx core supporting /etc/resolve.conf, but
> that might lead to an alone branch and become an obstacle in updating Nginx.
>

I believe the Nginx team is quite open to accepting such patches. Feel
free to submit your patches to the nginx-devel mailing list. Maxim
Dounin et al. should hold your hands there ;) So you probably don't
really have to maintain a branch in the long term. (Actually I've been
planning to write such patches myself but haven't got the time.)

Best regards,
-agentzh

microwish

Actually I have static name server addresses in files of hosts and resolv.conf managed by system administrators, but those "static" addresses may alter vary at times, 3 months, 5 months and the like, without any regularity, and I'm not able to learn the changes in time. So it may be inconvenient for set up the resolver directive. However, it's acceptable. :)

Generally speaking, getaddrinfo_a's non-portability does matter. But my platform is specific. It seems not a concern to me.
I'm not sure about the implementation of getaddrinfo_a either, also guessing it making use of signals or threads. I wanna probe into it.

agentzh, you do persuade me to enjoy patching support for hosts and resolv.conf.

On Wednesday, July 10, 2013 1:05:41 PM UTC+8, agentzh wrote:

Hello!

On Tue, Jul 9, 2013 at 8:02 PM, Glen Cao wrote:
> I'd had headache about setpeername() and connect() receiving domain names as
> the first argument (actually 2nd), before Chaos suggested local DNS relay
> and you suggested lua-resty-dns.
>

Why not just configure the "resolver" directive in your nginx.conf?

> I've thought about using async name resolving such as getaddrinfo_a from
> glibc, but Chaos reminded me of the like efficiency issue.
>

No, getaddrinfo_a sucks in that it uses pthreads internally AFAIK
which is still blocking OS threads per se, AFAIK. Not to mention it is
not portable.

> I also thought about making Nginx core supporting /etc/resolve.conf, but
> that might lead to an alone branch and become an obstacle in updating Nginx.
>

I believe the Nginx team is quite open to accepting such patches. Feel
free to submit your patches to the nginx-devel mailing list. Maxim
Dounin et al. should hold your hands there ;) So you probably don't
really have to maintain a branch in the long term. (Actually I've been
planning to write such patches myself but haven't got the time.)

Best regards,
-agentzh