upstream consul - 3 out of 12 IPs repeatedly used

vinaychennai · 2018-07-19T16:16:10+00:00

have you ever run `sudo apt-get update` to update the apt index在 2018年7月17日星期二 UTC+8下午9:20:17&...

upstream consul - 3 out of 12 IPs repeatedly used

vinaychennai

Hi, I'm facing an issue in consul dns load balancing. upstream url resolves 12 A records from consul, but openresty round-robin the requests over only 3 of them.

Is this a config issue or a bug? I'm using docker-openresty 1.13.6.1-alpine

Here is how my upstream config look like:

upstream solr_slave {

server solr-search-slave-service:8983;

}

Please help.

Thanks

hamish.forbes

Consul truncates UDP DNS replies to a maximum of 3 entries

https://www.consul.io/docs/agent/dns.html#udp-based-dns-queries

https://www.consul.io/docs/agent/options.html#enable_truncate

Normally that's not a problem because you'd re-resolve periodically and get another random set of 3 hosts.

But afaik nginx will resolve upstream addresses once at startup and never again.

I'm not sure if the nginx resolver will use TCP if it gets the truncate flag but you could try enabling that option to get the full set of 12.

On Thursday, 19 July 2018 09:16:10 UTC+1, Vinaykumar Sridharan wrote:

Hi, I'm facing an issue in consul dns load balancing. upstream url resolves 12 A records from consul, but openresty round-robin the requests over only 3 of them.
Is this a config issue or a bug? I'm using docker-openresty 1.13.6.1-alpine

Here is how my upstream config look like:

upstream solr_slave {
server solr-search-slave-service:8983;
}

Please help.
Thanks

vinaychennai

Hi Hamish,

the openresty 1.13.6.1-alpine docker image doesn't seem to have consul config file (atleast I couldn't find it)

Do you know how to set the enable_truncate to in the docker image?

Thanks

On Thursday, July 19, 2018 at 2:18:04 PM UTC+5:30, Hamish Forbes wrote:

Consul truncates UDP DNS replies to a maximum of 3 entries
https://www.consul.io/docs/agent/dns.html#udp-based-dns-queries
https://www.consul.io/docs/agent/options.html#enable_truncate

Normally that's not a problem because you'd re-resolve periodically and get another random set of 3 hosts.
But afaik nginx will resolve upstream addresses once at startup and never again.

I'm not sure if the nginx resolver will use TCP if it gets the truncate flag but you could try enabling that option to get the full set of 12.

On Thursday, 19 July 2018 09:16:10 UTC+1, Vinaykumar Sridharan wrote:
Hi, I'm facing an issue in consul dns load balancing. upstream url resolves 12 A records from consul, but openresty round-robin the requests over only 3 of them.
Is this a config issue or a bug? I'm using docker-openresty 1.13.6.1-alpine

Here is how my upstream config look like:

upstream solr_slave {
server solr-search-slave-service:8983;
}

Please help.
Thanks

hamish.forbes

It's a Consul config option, why would it be in the Openresty container?

You need to configure it on your Consul agent.

On Thursday, 19 July 2018 11:06:25 UTC+1, Vinaykumar Sridharan wrote:

Hi Hamish,
the openresty 1.13.6.1-alpine docker image doesn't seem to have consul config file (atleast I couldn't find it)
Do you know how to set the enable_truncate to in the docker image?

Thanks

On Thursday, July 19, 2018 at 2:18:04 PM UTC+5:30, Hamish Forbes wrote:
Consul truncates UDP DNS replies to a maximum of 3 entries
https://www.consul.io/docs/agent/dns.html#udp-based-dns-queries
https://www.consul.io/docs/agent/options.html#enable_truncate

Normally that's not a problem because you'd re-resolve periodically and get another random set of 3 hosts.
But afaik nginx will resolve upstream addresses once at startup and never again.

I'm not sure if the nginx resolver will use TCP if it gets the truncate flag but you could try enabling that option to get the full set of 12.

On Thursday, 19 July 2018 09:16:10 UTC+1, Vinaykumar Sridharan wrote:
Hi, I'm facing an issue in consul dns load balancing. upstream url resolves 12 A records from consul, but openresty round-robin the requests over only 3 of them.
Is this a config issue or a bug? I'm using docker-openresty 1.13.6.1-alpine

Here is how my upstream config look like:

upstream solr_slave {
server solr-search-slave-service:8983;
}

Please help.
Thanks