Invalidate upstream DNS cache

jon.keys · 2015-03-03T01:52:29+00:00

Hello! On Mon, Mar 2, 2015 at 5:31 AM, Robert Paprocki wrote: > Out of sheer curiosity at this point, what is the mechanism that handles > this expira...

Invalidate upstream DNS cache

jon.keys

We're using an AWS elastic loadbalancer to balance upstream traffic. It is useful for dynamically adding / removing servers in a central location and supports some nice features (e.g. connection draining, auto scaling, etc...).

The problem we're having is that nginx only resolves the IP of our AWS loadbalancer at startup and then caches the IP forever (until nginx reboot). Since the AWS loadbalancer IP can change anytime this is obviously a big problem.

I found this issue / feature request on the ngx_lua_upstream module which I think could help solve our problem. However, it isn't currently implemented.

Does anyone know of any other method to invalidate an upstream DNS cache / force nginx to honor DNS TTL responses?

I'm aware of the upstream "resolve" option in nginx+ but I'd rather not pay nginx+ license fees just for this option. Also, I'd much rather use the exact nginx version that has been thoroughly tested by agentzh as part of an OpenResty release.

agentzh

Hello!

On Mon, Mar 2, 2015 at 9:52 AM, Jon Keys wrote:
> I found this issue / feature request on the ngx_lua_upstream module which I
> think could help solve our problem. However, it isn't currently implemented.
>

Yes, adding an API to ngx_lua_upstream so as to dynamically change the
peer address in the upstream configuration data structures is the
right way to go.

I'm going to review that pull request at some point.

Regards,
-agentzh

brookscunningham

Hello!

I am facing a similar issue when using the upstream feature and proxying to ELBs. Is there an update on this topic?

Cheers,

Brooks

On Monday, March 2, 2015 at 1:01:03 PM UTC-5, agentzh wrote:

Hello!

On Mon, Mar 2, 2015 at 9:52 AM, Jon Keys wrote:
> I found this issue / feature request on the ngx_lua_upstream module which I
> think could help solve our problem. However, it isn't currently implemented.
>

Yes, adding an API to ngx_lua_upstream so as to dynamically change the
peer address in the upstream configuration data structures is the
right way to go.

I'm going to review that pull request at some point.

Regards,
-agentzh

jon.keys

We ended up writing our own upstream resolver using `balancer_by_lua`. It uses a master-client model to lookup IP's only once in a master context and then cache the IP's according to the DNS TTL. Then clients in a worker context can then quickly fetch the cached IP's from the master (again, always obeying the DNS TTL values).

It actually has some nice features and it is well tested. I might have open-sourced it before I just thought we were the only ones with this issue :-)

If others are interested we could publish to GitHub or somewhere public (we currently use a private GitLab server with GitLab CI for automated testing).

If we do that then we could probably publish to OPM as well.

One thing worth noting: it currently only supports IPv4 (i.e. A records)

On Monday, June 26, 2017 at 12:22:09 PM UTC-4, Brooks Cunningham wrote:

Hello!

I am facing a similar issue when using the upstream feature and proxying to ELBs. Is there an update on this topic?

Cheers,
Brooks

On Monday, March 2, 2015 at 1:01:03 PM UTC-5, agentzh wrote:
Hello!

On Mon, Mar 2, 2015 at 9:52 AM, Jon Keys wrote:
> I found this issue / feature request on the ngx_lua_upstream module which I
> think could help solve our problem. However, it isn't currently implemented.
>

Yes, adding an API to ngx_lua_upstream so as to dynamically change the
peer address in the upstream configuration data structures is the
right way to go.

I'm going to review that pull request at some point.

Regards,
-agentzh

brookscunningham

That would be amazing Jon! Using the upstream module is pretty great for performance since NGINX doesn't have to go through the full TCP or TLS build up and tear down for nearly every new connection. However, finding out that my site is 100% down because the ELB A records have changed makes it tough to utilize the upstream module.

I am very interested in the fix even with the IPv4 only support.

Thanks Jon!

Brooks

On Monday, June 26, 2017 at 2:37:09 PM UTC-4, Jon Keys wrote:

We ended up writing our own upstream resolver using `balancer_by_lua`. It uses a master-client model to lookup IP's only once in a master context and then cache the IP's according to the DNS TTL. Then clients in a worker context can then quickly fetch the cached IP's from the master (again, always obeying the DNS TTL values).
It actually has some nice features and it is well tested. I might have open-sourced it before I just thought we were the only ones with this issue :-)

If others are interested we could publish to GitHub or somewhere public (we currently use a private GitLab server with GitLab CI for automated testing).
If we do that then we could probably publish to OPM as well.

One thing worth noting: it currently only supports IPv4 (i.e. A records)

On Monday, June 26, 2017 at 12:22:09 PM UTC-4, Brooks Cunningham wrote:
Hello!

I am facing a similar issue when using the upstream feature and proxying to ELBs. Is there an update on this topic?

Cheers,
Brooks

On Monday, March 2, 2015 at 1:01:03 PM UTC-5, agentzh wrote:
Hello!

On Mon, Mar 2, 2015 at 9:52 AM, Jon Keys wrote:
> I found this issue / feature request on the ngx_lua_upstream module which I
> think could help solve our problem. However, it isn't currently implemented.
>

Yes, adding an API to ngx_lua_upstream so as to dynamically change the
peer address in the upstream configuration data structures is the
right way to go.

I'm going to review that pull request at some point.

Regards,
-agentzh

jon.keys

I just published to GitHub: https://github.com/jkeys089/lua-resty-resolver

If I get a chance this week I'll publish to OPM as well.

I hope it helps!

On Tuesday, June 27, 2017 at 1:45:20 PM UTC-4, Brooks Cunningham wrote:

That would be amazing Jon! Using the upstream module is pretty great for performance since NGINX doesn't have to go through the full TCP or TLS build up and tear down for nearly every new connection. However, finding out that my site is 100% down because the ELB A records have changed makes it tough to utilize the upstream module.

I am very interested in the fix even with the IPv4 only support.

Thanks Jon!

Brooks

On Monday, June 26, 2017 at 2:37:09 PM UTC-4, Jon Keys wrote:
We ended up writing our own upstream resolver using `balancer_by_lua`. It uses a master-client model to lookup IP's only once in a master context and then cache the IP's according to the DNS TTL. Then clients in a worker context can then quickly fetch the cached IP's from the master (again, always obeying the DNS TTL values).
It actually has some nice features and it is well tested. I might have open-sourced it before I just thought we were the only ones with this issue :-)

If others are interested we could publish to GitHub or somewhere public (we currently use a private GitLab server with GitLab CI for automated testing).
If we do that then we could probably publish to OPM as well.

One thing worth noting: it currently only supports IPv4 (i.e. A records)

On Monday, June 26, 2017 at 12:22:09 PM UTC-4, Brooks Cunningham wrote:
Hello!

I am facing a similar issue when using the upstream feature and proxying to ELBs. Is there an update on this topic?

Cheers,
Brooks

On Monday, March 2, 2015 at 1:01:03 PM UTC-5, agentzh wrote:
Hello!

On Mon, Mar 2, 2015 at 9:52 AM, Jon Keys wrote:
> I found this issue / feature request on the ngx_lua_upstream module which I
> think could help solve our problem. However, it isn't currently implemented.
>

Yes, adding an API to ngx_lua_upstream so as to dynamically change the
peer address in the upstream configuration data structures is the
right way to go.

I'm going to review that pull request at some point.

Regards,
-agentzh

jon.keys

I just went ahead and published to OPM -- I forgot how easy it is :-D

It is available at: jkeys089/lua-resty-resolver

On Tuesday, June 27, 2017 at 6:00:09 PM UTC-4, Jon Keys wrote:

I just published to GitHub: https://github.com/jkeys089/lua-resty-resolver

If I get a chance this week I'll publish to OPM as well.

I hope it helps!

On Tuesday, June 27, 2017 at 1:45:20 PM UTC-4, Brooks Cunningham wrote:
That would be amazing Jon! Using the upstream module is pretty great for performance since NGINX doesn't have to go through the full TCP or TLS build up and tear down for nearly every new connection. However, finding out that my site is 100% down because the ELB A records have changed makes it tough to utilize the upstream module.

I am very interested in the fix even with the IPv4 only support.

Thanks Jon!

Brooks

On Monday, June 26, 2017 at 2:37:09 PM UTC-4, Jon Keys wrote:
We ended up writing our own upstream resolver using `balancer_by_lua`. It uses a master-client model to lookup IP's only once in a master context and then cache the IP's according to the DNS TTL. Then clients in a worker context can then quickly fetch the cached IP's from the master (again, always obeying the DNS TTL values).
It actually has some nice features and it is well tested. I might have open-sourced it before I just thought we were the only ones with this issue :-)

If others are interested we could publish to GitHub or somewhere public (we currently use a private GitLab server with GitLab CI for automated testing).
If we do that then we could probably publish to OPM as well.

One thing worth noting: it currently only supports IPv4 (i.e. A records)

On Monday, June 26, 2017 at 12:22:09 PM UTC-4, Brooks Cunningham wrote:
Hello!

I am facing a similar issue when using the upstream feature and proxying to ELBs. Is there an update on this topic?

Cheers,
Brooks

On Monday, March 2, 2015 at 1:01:03 PM UTC-5, agentzh wrote:
Hello!

On Mon, Mar 2, 2015 at 9:52 AM, Jon Keys wrote:
> I found this issue / feature request on the ngx_lua_upstream module which I
> think could help solve our problem. However, it isn't currently implemented.
>

Yes, adding an API to ngx_lua_upstream so as to dynamically change the
peer address in the upstream configuration data structures is the
right way to go.

I'm going to review that pull request at some point.

Regards,
-agentzh

brookscunningham

I really do appreciate the contribution. I'll take a look.

Thanks!

Brooks

On Tue, Jun 27, 2017 at 7:14 PM Jon Keys <jon...@gmail.com> wrote:

I just went ahead and published to OPM -- I forgot how easy it is :-D

It is available at: jkeys089/lua-resty-resolver

On Tuesday, June 27, 2017 at 6:00:09 PM UTC-4, Jon Keys wrote:
I just published to GitHub: https://github.com/jkeys089/lua-resty-resolver

If I get a chance this week I'll publish to OPM as well.

I hope it helps!

On Tuesday, June 27, 2017 at 1:45:20 PM UTC-4, Brooks Cunningham wrote:
That would be amazing Jon! Using the upstream module is pretty great for performance since NGINX doesn't have to go through the full TCP or TLS build up and tear down for nearly every new connection. However, finding out that my site is 100% down because the ELB A records have changed makes it tough to utilize the upstream module.

I am very interested in the fix even with the IPv4 only support.

Thanks Jon!

Brooks

On Monday, June 26, 2017 at 2:37:09 PM UTC-4, Jon Keys wrote:
We ended up writing our own upstream resolver using `balancer_by_lua`. It uses a master-client model to lookup IP's only once in a master context and then cache the IP's according to the DNS TTL. Then clients in a worker context can then quickly fetch the cached IP's from the master (again, always obeying the DNS TTL values).
It actually has some nice features and it is well tested. I might have open-sourced it before I just thought we were the only ones with this issue :-)

If others are interested we could publish to GitHub or somewhere public (we currently use a private GitLab server with GitLab CI for automated testing).
If we do that then we could probably publish to OPM as well.

One thing worth noting: it currently only supports IPv4 (i.e. A records)

On Monday, June 26, 2017 at 12:22:09 PM UTC-4, Brooks Cunningham wrote:
Hello!

I am facing a similar issue when using the upstream feature and proxying to ELBs. Is there an update on this topic?

Cheers,
Brooks

On Monday, March 2, 2015 at 1:01:03 PM UTC-5, agentzh wrote:
Hello!

On Mon, Mar 2, 2015 at 9:52 AM, Jon Keys wrote:
> I found this issue / feature request on the ngx_lua_upstream module which I
> think could help solve our problem. However, it isn't currently implemented.
>

Yes, adding an API to ngx_lua_upstream so as to dynamically change the
peer address in the upstream configuration data structures is the
right way to go.

I'm going to review that pull request at some point.

Regards,
-agentzh

.

bilal.ahmad

Hi Guys,

Can you share any link to a working example of the resolver for ELBs with lua-resty-resolver or lua-upstream-nginx-module? It would be really helpful. Thanks,
Bilal

On Wednesday, June 28, 2017 at 5:34:28 PM UTC+4, Brooks Cunningham wrote:

I really do appreciate the contribution. I'll take a look.

Thanks!
Brooks

On Tue, Jun 27, 2017 at 7:14 PM Jon Keys <jon....@gmail.com> wrote:
I just went ahead and published to OPM -- I forgot how easy it is :-D

It is available at: jkeys089/lua-resty-resolver

On Tuesday, June 27, 2017 at 6:00:09 PM UTC-4, Jon Keys wrote:
I just published to GitHub: https://github.com/jkeys089/lua-resty-resolver

If I get a chance this week I'll publish to OPM as well.

I hope it helps!

On Tuesday, June 27, 2017 at 1:45:20 PM UTC-4, Brooks Cunningham wrote:
That would be amazing Jon! Using the upstream module is pretty great for performance since NGINX doesn't have to go through the full TCP or TLS build up and tear down for nearly every new connection. However, finding out that my site is 100% down because the ELB A records have changed makes it tough to utilize the upstream module.

I am very interested in the fix even with the IPv4 only support.

Thanks Jon!

Brooks

On Monday, June 26, 2017 at 2:37:09 PM UTC-4, Jon Keys wrote:
We ended up writing our own upstream resolver using `balancer_by_lua`. It uses a master-client model to lookup IP's only once in a master context and then cache the IP's according to the DNS TTL. Then clients in a worker context can then quickly fetch the cached IP's from the master (again, always obeying the DNS TTL values).
It actually has some nice features and it is well tested. I might have open-sourced it before I just thought we were the only ones with this issue :-)

If others are interested we could publish to GitHub or somewhere public (we currently use a private GitLab server with GitLab CI for automated testing).
If we do that then we could probably publish to OPM as well.

One thing worth noting: it currently only supports IPv4 (i.e. A records)

On Monday, June 26, 2017 at 12:22:09 PM UTC-4, Brooks Cunningham wrote:
Hello!

I am facing a similar issue when using the upstream feature and proxying to ELBs. Is there an update on this topic?

Cheers,
Brooks

On Monday, March 2, 2015 at 1:01:03 PM UTC-5, agentzh wrote:
Hello!

On Mon, Mar 2, 2015 at 9:52 AM, Jon Keys wrote:
> I found this issue / feature request on the ngx_lua_upstream module which I
> think could help solve our problem. However, it isn't currently implemented.
>

Yes, adding an API to ngx_lua_upstream so as to dynamically change the
peer address in the upstream configuration data structures is the
right way to go.

I'm going to review that pull request at some point.

Regards,
-agentzh

.