why ngx.balancer.get_last_failure always return nil?

xfwangcheers · 2017-03-06T23:01:07+00:00

Hello, I'm using openresty to create an API proxy. Different API has different target url. The API routing logic is implemented by lua. Target url i...

why ngx.balancer.get_last_failure always return nil?

xfwangcheers

Hello

I have successfully use the balancer to dynamically change the upstream servers with a simple round-robin strategy. I also want to implement some failover policy like the nginx builtin upstream's max_fails and fail_timeout http://nginx.org/en/docs/http/ngx_http_upstream_module.html#server, and I find the ngx.balancer.get_last_failure function. But it always returns nil even the last attempt failed, both of state_name and state_code. please suggest.

Thanks

Xiaofeng

rpaprocki

Hi,

On Mon, Mar 6, 2017 at 7:01 AM, <xfwan...@gmail.com> wrote:

Hello

I have successfully use the balancer to dynamically change the upstream servers with a simple round-robin strategy. I also want to implement some failover policy like the nginx builtin upstream's max_fails and fail_timeout http://nginx.org/en/docs/http/ngx_http_upstream_module.html#server, and I find the ngx.balancer.get_last_failure function. But it always returns nil even the last attempt failed, both of state_name and state_code. please suggest.

Can you please post a _full_, minimal, reproducible example of your config (emphasis on full ;) ) so someone can try to examine the behavior you're describing. Thanks :)

xfwangcheers

here are the codes and conf, you see, I specify the port of the second node as 18867 but it should be 8867, when the request hits the second node, the page reports 502 bad gateway but get_last_failure return nil

init_by_lua_block {
    local servers = { { "127.0.0.1", 8866 }, { "127.0.0.1", 18867 }, { "127.0.0.1", 8868 }}
    local rr = require("my.rr") -- the round-robin module
    local rrc = rr:new(servers)
    package.loaded.my_servers = servers
    package.loaded.my_rr = rrc
}

upstream dyn_ups {
    server 0.0.0.1:8866;   # just an invalid address as a place holder
    balancer_by_lua_block {
        local b = require("ngx.balancer")
        local rr = package.loaded.my_rr
        local srvId = rr:next()
        local srv = package.loaded.my_servers[srvId]
        ngx.log(ngx.WARN, "srv: " .. table.concat(srv, ":"))

        local state_name, state_code = b.get_last_failure()
        if state_name then 
            ngx.log(ngx.ERR, "last failure state " .. state_name .. "code " .. state_code)
        end       

        local ok, err = b.set_current_peer(srv[1], srv[2])
        if not ok then
            ngx.log(ngx.ERR, "failed to set the current peer: ", err)
            return ngx.exit(500)
        end
    }
    keepalive 10;
}

server {
    listen 8765;

    location /dyn_ups{
        proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
        proxy_pass http://dyn_ups;
    }
}

server {
    listen 8866;
    location /dyn_ups{
        content_by_lua 'ngx.say(ngx.var.server_port)';
    }
}

server {
    listen 8867;
    location /dyn_ups{
        content_by_lua 'ngx.say(ngx.var.server_port)';
    }
}

server {
    listen 8868;
    location /dyn_ups{
        content_by_lua 'ngx.say(ngx.var.server_port)';
    }
}

On Monday, March 6, 2017 at 11:35:21 PM UTC+8, rpaprocki wrote:

Hi,

On Mon, Mar 6, 2017 at 7:01 AM, <xfwang...@gmail.com> wrote:
Hello

I have successfully use the balancer to dynamically change the upstream servers with a simple round-robin strategy. I also want to implement some failover policy like the nginx builtin upstream's max_fails and fail_timeout http://nginx.org/en/docs/http/ngx_http_upstream_module.html#server, and I find the ngx.balancer.get_last_failure function. But it always returns nil even the last attempt failed, both of state_name and state_code. please suggest.

Can you please post a _full_, minimal, reproducible example of your config (emphasis on full ;) ) so someone can try to examine the behavior you're describing. Thanks :)

arkade

xfwangcheers

I dont think so. The document says it should work. I guess I'm missing something like configuration or initialization.

On Tuesday, March 7, 2017 at 11:15:42 AM UTC+8, Rob Kay wrote:

From the documentation:
Retrieves the failure details about the previous failed attempt (if any) when the next_upstream retrying mechanism is in action.

I believe the "next_upstream" mechanism is only active when you have multiple server directives within an upstream block and since you're using ngx.balancer to override a single server directive there is never a previous attempt.

On Tuesday, 7 March 2017 05:23:34 UTC+11, xfwang...@gmail.com wrote:
here are the codes and conf, you see, I specify the port of the second node as 18867 but it should be 8867, when the request hits the second node, the page reports 502 bad gateway but get_last_failure return nil

init_by_lua_block { local servers = { { "127.0.0.1", 8866 }, { "127.0.0.1", 18867 }, { "127.0.0.1", 8868 }} local rr = require("my.rr") -- the round-robin module local rrc = rr:new(servers) package.loaded.my_servers = servers package.loaded.my_rr = rrc } upstream dyn_ups { server 0.0.0.1:8866; # just an invalid address as a place holder balancer_by_lua_block { local b = require("ngx.balancer") local rr = package.loaded.my_rr local srvId = rr:next() local srv = package.loaded.my_servers[srvId] ngx.log(ngx.WARN, "srv: " .. table.concat(srv, ":")) local state_name, state_code = b.get_last_failure() if state_name then ngx.log(ngx.ERR, "last failure state " .. state_name .. "code " .. state_code) end local ok, err = b.set_current_peer(srv[1], srv[2]) if not ok then ngx.log(ngx.ERR, "failed to set the current peer: ", err) return ngx.exit(500) end } keepalive 10; } server { listen 8765; location /dyn_ups{ proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504; proxy_pass http://dyn_ups; } } server { listen 8866; location /dyn_ups{ content_by_lua 'ngx.say(ngx.var.server_port)'; } } server { listen 8867; location /dyn_ups{ content_by_lua 'ngx.say(ngx.var.server_port)'; } } server { listen 8868; location /dyn_ups{ content_by_lua 'ngx.say(ngx.var.server_port)'; } }

On Monday, March 6, 2017 at 11:35:21 PM UTC+8, rpaprocki wrote:
Hi,

On Mon, Mar 6, 2017 at 7:01 AM, <xfwang...@gmail.com> wrote:
Hello

I have successfully use the balancer to dynamically change the upstream servers with a simple round-robin strategy. I also want to implement some failover policy like the nginx builtin upstream's max_fails and fail_timeout http://nginx.org/en/docs/http/ngx_http_upstream_module.html#server, and I find the ngx.balancer.get_last_failure function. But it always returns nil even the last attempt failed, both of state_name and state_code. please suggest.

Can you please post a _full_, minimal, reproducible example of your config (emphasis on full ;) ) so someone can try to examine the behavior you're describing. Thanks :)

xfwangcheers

OK. I figured out this. I need invoke b.set_more_tries(10) before b.set_current_peer to explicitly tell the balancer block to retry the fail upstream request. And I also need to set the nginx proxy_next_upstream_tries directive to some value greater than zero.

For the step 2, I think it's a bug or at least it's incompatible with the nginx original upstream failover policy. Because the default 0 value indicates that nginx won't limit the retry times other than disable the retry. check more at the official document http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream_tries

On Tuesday, March 7, 2017 at 1:20:24 PM UTC+8, xfwang...@gmail.com wrote:

I dont think so. The document says it should work. I guess I'm missing something like configuration or initialization.

On Tuesday, March 7, 2017 at 11:15:42 AM UTC+8, Rob Kay wrote:
From the documentation:
Retrieves the failure details about the previous failed attempt (if any) when the next_upstream retrying mechanism is in action.

I believe the "next_upstream" mechanism is only active when you have multiple server directives within an upstream block and since you're using ngx.balancer to override a single server directive there is never a previous attempt.

On Tuesday, 7 March 2017 05:23:34 UTC+11, xfwang...@gmail.com wrote:
here are the codes and conf, you see, I specify the port of the second node as 18867 but it should be 8867, when the request hits the second node, the page reports 502 bad gateway but get_last_failure return nil

init_by_lua_block { local servers = { { "127.0.0.1", 8866 }, { "127.0.0.1", 18867 }, { "127.0.0.1", 8868 }} local rr = require("my.rr") -- the round-robin module local rrc = rr:new(servers) package.loaded.my_servers = servers package.loaded.my_rr = rrc } upstream dyn_ups { server 0.0.0.1:8866; # just an invalid address as a place holder balancer_by_lua_block { local b = require("ngx.balancer") local rr = package.loaded.my_rr local srvId = rr:next() local srv = package.loaded.my_servers[srvId] ngx.log(ngx.WARN, "srv: " .. table.concat(srv, ":")) local state_name, state_code = b.get_last_failure() if state_name then ngx.log(ngx.ERR, "last failure state " .. state_name .. "code " .. state_code) end local ok, err = b.set_current_peer(srv[1], srv[2]) if not ok then ngx.log(ngx.ERR, "failed to set the current peer: ", err) return ngx.exit(500) end } keepalive 10; } server { listen 8765; location /dyn_ups{ proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504; proxy_pass http://dyn_ups; } } server { listen 8866; location /dyn_ups{ content_by_lua 'ngx.say(ngx.var.server_port)'; } } server { listen 8867; location /dyn_ups{ content_by_lua 'ngx.say(ngx.var.server_port)'; } } server { listen 8868; location /dyn_ups{ content_by_lua 'ngx.say(ngx.var.server_port)'; } }

On Monday, March 6, 2017 at 11:35:21 PM UTC+8, rpaprocki wrote:
Hi,

On Mon, Mar 6, 2017 at 7:01 AM, <xfwang...@gmail.com> wrote:
Hello

I have successfully use the balancer to dynamically change the upstream servers with a simple round-robin strategy. I also want to implement some failover policy like the nginx builtin upstream's max_fails and fail_timeout http://nginx.org/en/docs/http/ngx_http_upstream_module.html#server, and I find the ngx.balancer.get_last_failure function. But it always returns nil even the last attempt failed, both of state_name and state_code. please suggest.

Can you please post a _full_, minimal, reproducible example of your config (emphasis on full ;) ) so someone can try to examine the behavior you're describing. Thanks :)

rpaprocki

Hi,

On Tue, Mar 7, 2017 at 6:28 AM, <xfwan...@gmail.com> wrote:

For the step 2, I think it's a bug or at least it's incompatible with the nginx original upstream failover policy. Because the default 0 value indicates that nginx won't limit the retry times other than disable the retry. check more at the official document http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream_tries

I don't think this assessment is quite accurate. The docs specifically say "The 0 value turns off this limitation.".This should mean that, by default (and depending on the configuration of proxy_next_upstream) ngx.balancer should continue to retry the request based on the value of set_more_tries (https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/balancer.md#set_more_tries). In your case, examining the debug logs would be very useful (if you have them captured and can reproduce behavior you think is buggy/undocumented, please post the debug logs so we can have a look [and use a pastebin for debug logs, it'll be pretty verbose ;)]).

Additionally, have a look at https://github.com/openresty/lua-nginx-module/blob/master/src/ngx_http_lua_balancer.c#L686. The next_upstream mechanism is only shorted by balancer by lua when the value is not 0 (which is the default, see https://github.com/nginx/nginx/blob/master/src/http/modules/ngx_http_proxy_module.c#L2946).

xfwangcheers

Confirm with agentzh that it really is a known issue and fixed for a long while. It has been patched in the latest master branch. Unfortunately it's not fixed in the latest release v1.11.2.2. You need to build from the latest master branch source code.