关于使用openresty 后端upstream 中的server 其实是没有挂或宕机，为啥访问经常出现502,错误日志显示 no live upstreams while connecting to upstream

huangmuhua180

大家好，请教一个问题问题：使用了lua-resty-upstream-healthcheck 来检测后端upstream 节点状态，看status也是已经是监控的p状态。

今天发现后端upstream 中的server 其实是完全正常的，为啥访问经常出现502呢，

查看error.log 有这么多 no live upstreams while connecting to upstream

其中： proxy_connect_timeout 120;

proxy_send_timeout 150;

proxy_read_timeout 150;

参数也设置这么多了啊

1、upstream配置如下:

upstream redmine {

server 192.168.0.163:8080;

keepalive 5;

}

2、error.log 日志如下:

2018/05/09 09:23:17 [error] 22687#22687: *28654611 no live upstreams while connecting to upstream, client: 10.8.10.81, server: redmine.etonenet.com, request: "GET /images/jstoolbar/bt_bq_remove.png HTTP/1.1", upstream: "http://redmine/images/jstoolbar/bt_bq_remove.png", host: "redmine.etonenet.com", referrer: "http://redmine.etonenet.com/issues/119"

2018/05/09 09:23:17 [error] 22684#22684: *28654620 no live upstreams while connecting to upstream, client: 10.8.10.81, server: redmine.etonenet.com, request: "GET /images/jstoolbar/bt_pre.png HTTP/1.1", upstream: "http://redmine/images/jstoolbar/bt_pre.png", host: "redmine.etonenet.com", referrer: "http://redmine.etonenet.com/issues/119"

2018/05/09 09:23:17 [error] 22687#22687: *28654611 no live upstreams while connecting to upstream, client: 10.8.10.81, server: redmine.etonenet.com, request: "GET /images/jstoolbar/bt_precode.png HTTP/1.1", upstream: "http://redmine/images/jstoolbar/bt_precode.png", host: "redmine.etonenet.com", referrer: "http://redmine.etonenet.com/issues/119"

2018/05/09 09:23:17 [error] 22687#22687: *28654609 no live upstreams while connecting to upstream, client: 10.8.10.81, server: redmine.etonenet.com, request: "GET /images/jstoolbar/bt_img.png HTTP/1.1", upstream: "http://redmine/images/jstoolbar/bt_img.png", host: "redmine.etonenet.com", referrer: "http://redmine.etonenet.com/issues/119"

2018/05/09 09:23:17 [error] 22684#22684: *28654620 no live upstreams while connecting to upstream, client: 10.8.10.81, server: redmine.etonenet.com, request: "GET /images/help.png HTTP/1.1", upstream: "http://redmine/images/help.png", host: "redmine.etonenet.com", referrer: "http://redmine.etonenet.com/issues/119"

2018/05/09 09:23:23 [error] 22684#22684: *28654750 no live upstreams while connecting to upstream, client: 10.8.10.81, server: redmine.etonenet.com, request: "GET /images/bullet_arrow_right.png HTTP/1.1", upstream: "http://redmine/images/bullet_arrow_right.png", host: "redmine.etonenet.com", referrer: "http://redmine.etonenet.com/issues/119"

3、主nginx.conf配置如下：

user nobody;

worker_processes 8;

error_log /var/log/openresty/error.log;

pid logs/nginx.pid;

worker_rlimit_nofile 655360;

events {

use epoll;

worker_connections 1000;

}

http {

log_format main '$remote_addr - $server_addr - $remote_user [$time_local][$msec] "$request $request_body" "$http_host" '

'$status $body_bytes_sent "$http_referer" $connection $connection_requests'

'"$http_user_agent" "$http_x_forwarded_for" "$request_time" "$upstream_addr" "$upstream_response_time"';

lua_package_path "/usr/local/openresty/lualib/?.lua;;";

lua_package_cpath "/usr/local/openresty/lualib/?.so;;";

access_log /var/opt/nginxlog/default/access_log main;

error_log /var/opt/nginxlog/default/error_log error;

sendfile on;

tcp_nopush on;

tcp_nodelay on;

keepalive_timeout 13 15;

keepalive_requests 5000;

types_hash_max_size 2048;

client_body_buffer_size 128k;

client_max_body_size 10m;

client_body_in_single_buffer on;

client_header_buffer_size 64k;

large_client_header_buffers 4 128k;

gzip on;

gzip_comp_level 2;

gzip_buffers 4 16k;

gzip_min_length 1k;

include mime.types;

default_type application/octet-stream;

proxy_connect_timeout 120;

proxy_send_timeout 150;

proxy_read_timeout 150;

proxy_http_version 1.1;

proxy_set_header Connection "";

proxy_next_upstream http_502 http_504 http_404 error timeout invalid_header;

proxy_set_header Host $host;

proxy_set_header X-Real-Ip $remote_addr;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

include /usr/local/openresty/nginx/conf/vhosts/*.conf;

huang_hua2012

hello，

将upstream 块里的keepalive去掉看看是否还会出现502

keepalive具体作用见官方解释 http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive

Kwanhur Huang

TL;DR

在 2018年5月9日，下午10:21，huangmuhua180 <huangm...@163.com> 写道：

大家好，请教一个问题问题：使用了lua-resty-upstream-healthcheck 来检测后端upstream 节点状态，看status也是已经是监控的p状态。

今天发现后端upstream 中的server 其实是完全正常的，为啥访问经常出现502呢，
查看error.log 有这么多 no live upstreams while connecting to upstream
其中： proxy_connect_timeout 120;
proxy_send_timeout 150;
proxy_read_timeout 150;
参数也设置这么多了啊

1、upstream配置如下:
upstream redmine {
server 192.168.0.163:8080;
keepalive 5;
}

2、error.log 日志如下:
2018/05/09 09:23:17 [error] 22687#22687: *28654611 no live upstreams while connecting to upstream, client: 10.8.10.81, server: redmine.etonenet.com, request: "GET /images/jstoolbar/bt_bq_remove.png HTTP/1.1", upstream: "http://redmine/images/jstoolbar/bt_bq_remove.png", host: "redmine.etonenet.com", referrer: "http://redmine.etonenet.com/issues/119"
2018/05/09 09:23:17 [error] 22684#22684: *28654620 no live upstreams while connecting to upstream, client: 10.8.10.81, server: redmine.etonenet.com, request: "GET /images/jstoolbar/bt_pre.png HTTP/1.1", upstream: "http://redmine/images/jstoolbar/bt_pre.png", host: "redmine.etonenet.com", referrer: "http://redmine.etonenet.com/issues/119"
2018/05/09 09:23:17 [error] 22687#22687: *28654611 no live upstreams while connecting to upstream, client: 10.8.10.81, server: redmine.etonenet.com, request: "GET /images/jstoolbar/bt_precode.png HTTP/1.1", upstream: "http://redmine/images/jstoolbar/bt_precode.png", host: "redmine.etonenet.com", referrer: "http://redmine.etonenet.com/issues/119"
2018/05/09 09:23:17 [error] 22687#22687: *28654609 no live upstreams while connecting to upstream, client: 10.8.10.81, server: redmine.etonenet.com, request: "GET /images/jstoolbar/bt_img.png HTTP/1.1", upstream: "http://redmine/images/jstoolbar/bt_img.png", host: "redmine.etonenet.com", referrer: "http://redmine.etonenet.com/issues/119"
2018/05/09 09:23:17 [error] 22684#22684: *28654620 no live upstreams while connecting to upstream, client: 10.8.10.81, server: redmine.etonenet.com, request: "GET /images/help.png HTTP/1.1", upstream: "http://redmine/images/help.png", host: "redmine.etonenet.com", referrer: "http://redmine.etonenet.com/issues/119"
2018/05/09 09:23:23 [error] 22684#22684: *28654750 no live upstreams while connecting to upstream, client: 10.8.10.81, server: redmine.etonenet.com, request: "GET /images/bullet_arrow_right.png HTTP/1.1", upstream: "http://redmine/images/bullet_arrow_right.png", host: "redmine.etonenet.com", referrer: "http://redmine.etonenet.com/issues/119"

3、主nginx.conf配置如下：
user nobody;
worker_processes 8;
error_log /var/log/openresty/error.log;
pid logs/nginx.pid;
worker_rlimit_nofile 655360;

events {
use epoll;
worker_connections 1000;
}

http {
log_format main '$remote_addr - $server_addr - $remote_user [$time_local][$msec] "$request $request_body" "$http_host" '
'$status $body_bytes_sent "$http_referer" $connection $connection_requests'
'"$http_user_agent" "$http_x_forwarded_for" "$request_time" "$upstream_addr" "$upstream_response_time"';
lua_package_path "/usr/local/openresty/lualib/?.lua;;";
lua_package_cpath "/usr/local/openresty/lualib/?.so;;";
access_log /var/opt/nginxlog/default/access_log main;
error_log /var/opt/nginxlog/default/error_log error;

sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 13 15;
keepalive_requests 5000;
types_hash_max_size 2048;

client_body_buffer_size 128k;
client_max_body_size 10m;
client_body_in_single_buffer on;
client_header_buffer_size 64k;
large_client_header_buffers 4 128k;

gzip on;
gzip_comp_level 2;
gzip_buffers 4 16k;
gzip_min_length 1k;

include mime.types;
default_type application/octet-stream;

proxy_connect_timeout 120;
proxy_send_timeout 150;
proxy_read_timeout 150;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_next_upstream http_502 http_504 http_404 error timeout invalid_header;
proxy_set_header Host $host;
proxy_set_header X-Real-Ip $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
include /usr/local/openresty/nginx/conf/vhosts/*.conf;

--
--

huangmuhua180

好的，谢谢。我试下，看说是设置连接数，尽量设置小啊。

这个去掉也有默认值么，官网上没说。

在 2018-05-09 22:34:19，"Kwanhur Huang" <huang_...@163.com> 写道：

hello，

将upstream 块里的keepalive去掉看看是否还会出现502

keepalive具体作用见官方解释 http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive

Kwanhur Huang
TL;DR

在 2018年5月9日，下午10:21，huangmuhua180 <huangm...@163.com> 写道：

大家好，请教一个问题问题：使用了lua-resty-upstream-healthcheck 来检测后端upstream 节点状态，看status也是已经是监控的p状态。

今天发现后端upstream 中的server 其实是完全正常的，为啥访问经常出现502呢，
查看error.log 有这么多 no live upstreams while connecting to upstream
其中： proxy_connect_timeout 120;
proxy_send_timeout 150;
proxy_read_timeout 150;
参数也设置这么多了啊

1、upstream配置如下:
upstream redmine {
server 192.168.0.163:8080;
keepalive 5;
}

2、error.log 日志如下:
2018/05/09 09:23:17 [error] 22687#22687: *28654611 no live upstreams while connecting to upstream, client: 10.8.10.81, server: redmine.etonenet.com, request: "GET /images/jstoolbar/bt_bq_remove.png HTTP/1.1", upstream: "http://redmine/images/jstoolbar/bt_bq_remove.png", host: "redmine.etonenet.com", referrer: "http://redmine.etonenet.com/issues/119"
2018/05/09 09:23:17 [error] 22684#22684: *28654620 no live upstreams while connecting to upstream, client: 10.8.10.81, server: redmine.etonenet.com, request: "GET /images/jstoolbar/bt_pre.png HTTP/1.1", upstream: "http://redmine/images/jstoolbar/bt_pre.png", host: "redmine.etonenet.com", referrer: "http://redmine.etonenet.com/issues/119"
2018/05/09 09:23:17 [error] 22687#22687: *28654611 no live upstreams while connecting to upstream, client: 10.8.10.81, server: redmine.etonenet.com, request: "GET /images/jstoolbar/bt_precode.png HTTP/1.1", upstream: "http://redmine/images/jstoolbar/bt_precode.png", host: "redmine.etonenet.com", referrer: "http://redmine.etonenet.com/issues/119"
2018/05/09 09:23:17 [error] 22687#22687: *28654609 no live upstreams while connecting to upstream, client: 10.8.10.81, server: redmine.etonenet.com, request: "GET /images/jstoolbar/bt_img.png HTTP/1.1", upstream: "http://redmine/images/jstoolbar/bt_img.png", host: "redmine.etonenet.com", referrer: "http://redmine.etonenet.com/issues/119"
2018/05/09 09:23:17 [error] 22684#22684: *28654620 no live upstreams while connecting to upstream, client: 10.8.10.81, server: redmine.etonenet.com, request: "GET /images/help.png HTTP/1.1", upstream: "http://redmine/images/help.png", host: "redmine.etonenet.com", referrer: "http://redmine.etonenet.com/issues/119"
2018/05/09 09:23:23 [error] 22684#22684: *28654750 no live upstreams while connecting to upstream, client: 10.8.10.81, server: redmine.etonenet.com, request: "GET /images/bullet_arrow_right.png HTTP/1.1", upstream: "http://redmine/images/bullet_arrow_right.png", host: "redmine.etonenet.com", referrer: "http://redmine.etonenet.com/issues/119"

3、主nginx.conf配置如下：
user nobody;
worker_processes 8;
error_log /var/log/openresty/error.log;
pid logs/nginx.pid;
worker_rlimit_nofile 655360;

events {
use epoll;
worker_connections 1000;
}

http {
log_format main '$remote_addr - $server_addr - $remote_user [$time_local][$msec] "$request $request_body" "$http_host" '
'$status $body_bytes_sent "$http_referer" $connection $connection_requests'
'"$http_user_agent" "$http_x_forwarded_for" "$request_time" "$upstream_addr" "$upstream_response_time"';
lua_package_path "/usr/local/openresty/lualib/?.lua;;";
lua_package_cpath "/usr/local/openresty/lualib/?.so;;";
access_log /var/opt/nginxlog/default/access_log main;
error_log /var/opt/nginxlog/default/error_log error;

sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 13 15;
keepalive_requests 5000;
types_hash_max_size 2048;

client_body_buffer_size 128k;
client_max_body_size 10m;
client_body_in_single_buffer on;
client_header_buffer_size 64k;
large_client_header_buffers 4 128k;

gzip on;
gzip_comp_level 2;
gzip_buffers 4 16k;
gzip_min_length 1k;

include mime.types;
default_type application/octet-stream;

proxy_connect_timeout 120;
proxy_send_timeout 150;
proxy_read_timeout 150;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_next_upstream http_502 http_504 http_404 error timeout invalid_header;
proxy_set_header Host $host;
proxy_set_header X-Real-Ip $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
include /usr/local/openresty/nginx/conf/vhosts/*.conf;

--
--

--

hadalsyang

我猜是你健康检查逻辑上出现问题

huangmuhua180

可以说详细点吗：nginx.conf 主配置文件如下:

user nobody;

worker_processes 8;

error_log logs/error.log debug;

pid logs/nginx.pid;

worker_rlimit_nofile 655360;

events {

use epoll;

worker_connections 1000;

}

http {

log_format main '$remote_addr - $server_addr - $remote_user [$time_local][$msec] "$request $request_body" "$http_host" '

'$status $body_bytes_sent "$http_referer" $connection $connection_requests'

'"$http_user_agent" "$http_x_forwarded_for" "$request_time" "$upstream_addr" "$upstream_response_time"';

lua_package_path "/usr/local/openresty/lualib/?.lua;;";

lua_package_cpath "/usr/local/openresty/lualib/?.so;;";

sendfile on;

tcp_nopush on;

tcp_nodelay on;

keepalive_timeout 13 15;

keepalive_requests 5000;

types_hash_max_size 2048;

client_body_buffer_size 128k;

client_max_body_size 10m;

client_body_in_single_buffer on;

client_header_buffer_size 64k;

large_client_header_buffers 4 128k;

gzip on;

gzip_comp_level 2;

gzip_buffers 4 16k;

gzip_min_length 1k;

include mime.types;

default_type application/octet-stream;

proxy_connect_timeout 120;

proxy_send_timeout 180;

proxy_read_timeout 180;

proxy_http_version 1.1;

proxy_set_header Connection "";

proxy_next_upstream http_502 http_504 http_404 error timeout invalid_header;

proxy_set_header Host $host;

proxy_set_header X-Real-Ip $remote_addr;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

include /usr/local/openresty/nginx/conf/vhosts/*.conf;

}

upstream.conf 检测配置如下：有的server是没检测，也会出现这样502

upstream boss100 {

server 10.8.12.9:9393;

keepalive 5;

}

upstream blacklist1 {

server 10.8.11.10:9040 weight=1 max_fails=1 fail_timeout=60s;

keepalive 5;

}

upstream blacklist2 {

server 10.8.11.10:9060 weight=1 max_fails=1 fail_timeout=60s;

keepalive 5;

}

# the size depends on the number of servers in upstream {}:

lua_shared_dict healthcheck 64m;

lua_socket_log_errors off;

init_worker_by_lua_block {

local hc = require "resty.upstream.healthcheck"

local ok, err = hc.spawn_checker {

shm = "healthcheck",

upstream = "hztest",

type = "http",

http_req = "GET / HTTP/1.0\r\nHost: hztest\r\n\r\n",

interval = 2000,

timeout = 1000,

fall = 3,

rise = 2,

valid_statuses = {200, 301, 302},

concurrency = 1,

}

ok, err = hc.spawn_checker {

shm = "healthcheck",

upstream = "esmsdev",

type = "http",

http_req = "GET / HTTP/1.0\r\nHost: esmsdev\r\n\r\n",

interval = 2000,

timeout = 1000,

fall = 3,

rise = 2,

valid_statuses = {200, 301, 302},

concurrency = 1,

}

ok, err = hc.spawn_checker {

shm = "healthcheck",

upstream = "polling1query1",

type = "http",

http_req = "GET /sms/mort HTTP/1.0\r\nHost: polling1query1\r\n\r\n",

interval = 2000,

timeout = 1000,

fall = 1,

rise = 3,

valid_statuses = {200, 301, 302},

concurrency = 1,

}

ok, err = hc.spawn_checker {

shm = "healthcheck",

upstream = "redmine",

type = "http",

http_req = "GET / HTTP/1.0\r\nHost: redmine\r\n\r\n",

interval = 2000,

timeout = 1000,

fall = 1,

rise = 3,

valid_statuses = {200, 301, 302},

concurrency = 1,

}

if not ok then

ngx.log(ngx.ERR, "=======> failed to spawn health checker: ", err)

return

end

}

在 2018-05-10 10:12:18，"Hadals Yang" <hada...@gmail.com> 写道：

我猜是你健康检查逻辑上出现问题

--