关于ngx_lua、resty-redis的问题

gwill.buzz · 2012-06-04T18:01:32+00:00

已经没有问题了。 On 6月4日, 下午5时14分, e wu <fenghuang...@gmail.com...

关于ngx_lua、resty-redis的问题

gwill.buzz

Hi all,

我用 resty-redis 写了一小段下载服务器流量分流的配置，但是上线以后从 ngx_status 看 writing 数飙升，平时一般维持在100左右，换了配置以后5分钟writing数就到了700多了。而且流量也上不去换了配置以后大概1 M/s 而以前是15M/s。

这里是配置，春哥帮忙看看这个是什么问题。

server {

listen 80;

server_name down.com;

access_log off;

location / {

rewrite_by_lua '

local uri = ngx.var.uri

local resty_redis = require "resty.redis"

local redis = resty_redis:new()

redis:set_timeout(500)

local ok, err = redis:connect("127.0.0.1", 6379)

if not ok then

ngx.exec("/files"..uri)

end

net_speed, err = redis:get("net_speed")

if not net_speed then

ngx.exec("/files"..uri)

end

if tonumber(net_speed) > 15728640 then
-- 流量超过15M抛给cdn

ngx.exec("/fetch/files"..uri)

else

ngx.exec("/files"..uri)

end

';

}

location /files/ {

internal;

alias /home/files/;

limit_rate 300k;

error_page 404 = /fetch$uri;

}

location /fetch/ {

internal;

rewrite ^/fetch/(.*)$ http://cdn.com/$1 permanent;

}

}

agentzh

Hello!

2012/6/4 G_will <gwil...@gmail.com>:
> 我用 resty-redis 写了一小段下载服务器流量分流的配置，但是上线以后从 ngx_status 看 writing
> 数飙升，平时一般维持在100左右，换了配置以后5分钟writing数就到了700多了。而且流量也上不去换了配置以后大概1 M/s 而 以前是15M/s。
>

几个小建议：

1. 请启用 redis 连接池（通过调用 set_keepalive 方法）；
2. 确保当前使用的是 LuaJIT，而不是标准 Lua 5.1 解释器；
3. 在 lua-resty-redis 出错时向 nginx 错误日志打印出错消息，而非简单忽略，以方便诊断问题；
4. 确保 lua_code_cache 是开启状态；
5. 在可控的测试环境中进行性能压测试验，不要在生产环境里面试验，因为不可控因素比较多；
6. 监视你的 nginx 错误日志中的错误消息，但不要开启调试日志（debugging logs）；
7. 可以考虑配置 Nginx worker 进程的 CPU 亲缘性（见
http://wiki.nginx.org/CoreModule#worker_cpu_affinity ）

Regards,
-agentzh

gwill.buzz

在 2012年6月4日下午9:09，agentzh <age...@gmail.com>写道：

Hello!

2012/6/4 G_will <gwil...@gmail.com>:

> 我用 resty-redis 写了一小段下载服务器流量分流的配置，但是上线以后从 ngx_status 看 writing
> 数飙升，平时一般维持在100左右，换了配置以后5分钟writing数就到了700多了。而且流量也上不去换了配置以后大概1 M/s 而以前是15M/s。
>

几个小建议：

1. 请启用 redis 连接池（通过调用 set_keepalive 方法）；

嗯，我配置连接池再试一下。

出现这样的问题，是不是因为我没有写close的缘故？

2. 确保当前使用的是 LuaJIT，而不是标准 Lua 5.1 解释器；

使用的是LuaJIT。

3. 在 lua-resty-redis 出错

时向 nginx 错误日志打印出错消息，而非简单忽略，以方便诊断问题；

嗯，明白了。

查error.log里有很多这样的记录：

2012/06/04 17:15:26 [alert] 5468#0: worker process 5469 exited on signal 11 (core dumped)

2012/06/04 17:15:26 [alert] 5468#0: worker process 5470 exited on signal 11 (core dumped)

2012/06/04 17:15:27 [alert] 5468#0: worker process 5471 exited on signal 11 (core dumped)

2012/06/04 17:15:27 [alert] 5468#0: worker process 5473 exited on signal 11 (core dumped)

2012/06/04 17:15:27 [alert] 5468#0: worker process 5476 exited on signal 11 (core dumped)

2012/06/04 17:15:27 [alert] 5468#0: worker process 5478 exited on signal 11 (core dumped)

4. 确保 lua_code_cache 是开启状态；

是开启的。

5. 在可控的测试环境中进行性能压测试验，不要在生产环境里面试验，因为不可控因素比较多；

对于测试，用虚拟机测试流程是通的，但是因为线上是做文件下载，我确实不知道应该怎么设计压力测试，春哥点拨一下。。

6. 监视你的 nginx 错误日志中的错误消息，但不要开启调试日志（debugging logs）；

嗯，刚查了一下error.log除了core dumped的记录就是没有找到文件的记录（这个应该是正常的，因为配置的逻辑就是找不到抛给cdn）

7. 可以考虑配置 Nginx worker 进程的 CPU 亲缘性（见
http://wiki.nginx.org/CoreModule#worker_cpu_affinity ）

确实没想这么，因为这台机子配置很低（奔腾双核），因为只做下载用一般连接数100多，cpu几乎不耗。

当时间觉得如果用流量控制cdn转向，就可以充分的利用这台机子的带宽了。

Regards,
-agentzh

agentzh

Hello!

2012/6/4 G_will <gwil...@gmail.com>:
> 嗯，我配置连接池再试一下。
>
> 出现这样的问题，是不是因为我没有写close的缘故？
>

不是，cosocket 对象会在请求结束时自动 close（如果你没有显式调用 close 方法或者 set_keepalive 方法的话）。

writing 数不断增高是因为 worker 进程崩溃导致共享内存中相应的计数器只增不减，出现数据不一致的现象。

>
> 查error.log里有很多这样的记录：
>
> 2012/06/04 17:15:26 [alert] 5468#0: worker process 5469 exited on signal 11
> (core dumped)
> 2012/06/04 17:15:26 [alert] 5468#0: worker process 5470 exited on signal 11
> (core dumped)
> 2012/06/04 17:15:27 [alert] 5468#0: worker process 5471 exited on signal 11
> (core dumped)

能否使用 gdb 检查一下 core dump 发生时的 C 调用栈？见这篇文档中的步骤：

    http://wiki.nginx.org/Debugging#Core_dump

>
> 对于测试，用虚拟机测试流程是通的，但是因为线上是做文件下载，我确实不知道应该怎么设计压力测试，春哥点拨一下。。
>

文件下载当多是 GET 请求，可以尝试使用 ab 或者 weighttp 或者 http_load 这样的工具?

Regards,
-agentzh

gwill.buzz

在 2012年6月5日上午9:06，agentzh <age...@gmail.com>写道：

Hello!

2012/6/4 G_will <gwil...@gmail.com>:
> 嗯，我配置连接池再试一下。
>
> 出现这样的问题，是不是因为我没有写close的缘故？
>

不是，cosocket 对象会在请求结束时自动 close（如果你没有显式调用 close 方法或者 set_keepalive 方法的话）。

writing 数不断增高是因为 worker 进程崩溃导致共享内存中相应的计数器只增不减，出现数据不一致的现象。

>
> 查error.log里有很多这样的记录：
>
> 2012/06/04 17:15:26 [alert] 5468#0: worker process 5469 exited on signal 11
> (core dumped)
> 2012/06/04 17:15:26 [alert] 5468#0: worker process 5470 exited on signal 11
> (core dumped)
> 2012/06/04 17:15:27 [alert] 5468#0: worker process 5471 exited on signal 11
> (core dumped)

能否使用 gdb 检查一下 core dump 发生时的 C 调用栈？见这篇文档中的步骤：

http://wiki.nginx.org/Debugging#Core_dump

参照 wiki 我配置了core dump 的调试，然后用另一台机子用ab发起get请求，一般几个请求以后ab就会提示：

# ab -n 100 -c 2 http://d1.com/file.zip

This is ApacheBench, Version 2.3 <$Revision: 655654 $>

Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/

Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking d1.com (be patient)...apr_socket_recv: Connection reset by peer (104)

Total of 9 requests completed

然后gdb里的信息如下：

# gdb nginx /var/log/nginx/cores/core

GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2) 7.4-2012.04

Copyright (C) 2012 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law. Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-linux-gnu".

For bug reporting instructions, please see:

<http://bugs.launchpad.net/gdb-linaro/>...

Reading symbols from /usr/sbin/nginx...done.

[New LWP 25730]

warning: Can't read pathname for load map: Input/output error.

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Core was generated by `nginx: worker process '.

Program terminated with signal 11, Segmentation fault.

#0 0x00000000004f0707 in ngx_http_lua_socket_finalize (r=0x1abdde0, u=0x407acf78)

at ../ngx_lua-0.5.0rc21/src/ngx_http_lua_socket.c:2028

2028 *ll = ctx->free_recv_bufs;

(gdb) bt

#0 0x00000000004f0707 in ngx_http_lua_socket_finalize (r=0x1abdde0, u=0x407acf78)

at ../ngx_lua-0.5.0rc21/src/ngx_http_lua_socket.c:2028

#1 0x00000000004f062c in ngx_http_lua_socket_cleanup (data="" at ../ngx_lua-0.5.0rc21/src/ngx_http_lua_socket.c:2002

#2 0x00000000004628b0 in ngx_http_free_request (r=0x1abdde0, rc=0) at src/http/ngx_http_request.c:2973

#3 0x00000000004627da in ngx_http_close_request (r=0x1abdde0, rc=0) at src/http/ngx_http_request.c:2948

#4 0x0000000000462348 in ngx_http_lingering_close_handler (rev=0x1b2bde0) at src/http/ngx_http_request.c:2819

#5 0x0000000000439e71 in ngx_event_process_posted (cycle=0x1aaf9b0, posted=0x760aa8) at src/event/ngx_event_posted.c:39

#6 0x0000000000437cf8 in ngx_process_events_and_timers (cycle=0x1aaf9b0) at src/event/ngx_event.c:272

#7 0x0000000000444a9f in ngx_worker_process_cycle (cycle=0x1aaf9b0, data="" at src/os/unix/ngx_process_cycle.c:801

#8 0x00000000004414d8 in ngx_spawn_process (cycle=0x1aaf9b0, proc=0x4448e3 <ngx_worker_process_cycle>, data="">
name=0x51ca73 "worker process", respawn=-3) at src/os/unix/ngx_process.c:196

#9 0x00000000004438c4 in ngx_start_worker_processes (cycle=0x1aaf9b0, n=4, type=-3) at src/os/unix/ngx_process_cycle.c:360

#10 0x0000000000442f3f in ngx_master_process_cycle (cycle=0x1aaf9b0) at src/os/unix/ngx_process_cycle.c:136

#11 0x0000000000414e42 in main (argc=3, argv=0x7fff1f1f5208) at src/core/nginx.c:411

(gdb) backtrace full

#0 0x00000000004f0707 in ngx_http_lua_socket_finalize (r=0x1abdde0, u=0x407acf78)

at ../ngx_lua-0.5.0rc21/src/ngx_http_lua_socket.c:2028

spool = 0x1000002f8

cl = 0x0

ll = 0x1b008b0

ctx = 0x0

#1 0x00000000004f062c in ngx_http_lua_socket_cleanup (data="" at ../ngx_lua-0.5.0rc21/src/ngx_http_lua_socket.c:2002

u = 0x407acf78

r = 0x1abdde0

#2 0x00000000004628b0 in ngx_http_free_request (r=0x1abdde0, rc=0) at src/http/ngx_http_request.c:2973

log = 0x1ac8c60

linger = {l_onoff = 0, l_linger = 0}

cln = 0x1b007f0

ctx = 0x1000

clcf = 0x6720202000000004

#3 0x00000000004627da in ngx_http_close_request (r=0x1abdde0, rc=0) at src/http/ngx_http_request.c:2948

c = 0x1b07f30

#4 0x0000000000462348 in ngx_http_lingering_close_handler (rev=0x1b2bde0) at src/http/ngx_http_request.c:2819

n = 0

timer = 30

c = 0x1b07f30

r = 0x1abdde0

clcf = 0x7fff1f1f3b40

buffer = "\200;\037\037\377\177\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\003\000\000\000f", '\000' <repeats 11 times>"\377, \357\377\177\000\000\000\000(z\254\001\000\000\000\000p~\260\001\000\000\000\000\060y\254\001\000\000\000\000\346\000\000\000\000\000\000\000\000\307t\000\000\000\000\000\204\000\000\000\000\000\000\000`\304t\000\000\000\000\000@\000\000\000\000\000\000\000\260>\037\037\377\177\000\000\240C\037\037\377\177\000\000\240C\037\037\377\177\000\000\220C\037\037\377\177\000\000\270D\037\037\377\177\000\000i\335\061\004f\177\000\000n\335\061\004f\177\000\000\254\033\035\003f\177", '\000' <repeats 18 times>, "n\335\061\004f\177\000\000̫+\004f\177\000\000(\026\327\002f\177\000\000\210\236+\004f\177\000\000\000\000\000\000\001\000\000\000Z\000\000\000\001", '\000' <repeats 11 times>...

#5 0x0000000000439e71 in ngx_event_process_posted (cycle=0x1aaf9b0, posted=0x760aa8) at src/event/ngx_event_posted.c:39

ev = 0x1b2bde0

#6 0x0000000000437cf8 in ngx_process_events_and_timers (cycle=0x1aaf9b0) at src/event/ngx_event.c:272

flags = 3

timer = 5000

delta = 0

#7 0x0000000000444a9f in ngx_worker_process_cycle (cycle=0x1aaf9b0, data="" at src/os/unix/ngx_process_cycle.c:801

i = 140733715534336

c = 0x0

#8 0x00000000004414d8 in ngx_spawn_process (cycle=0x1aaf9b0, proc=0x4448e3 <ngx_worker_process_cycle>, data="">
name=0x51ca73 "worker process", respawn=-3) at src/os/unix/ngx_process.c:196

on = 1

pid = 0

s = 3

#9 0x00000000004438c4 in ngx_start_worker_processes (cycle=0x1aaf9b0, n=4, type=-3) at src/os/unix/ngx_process_cycle.c:360

i = 3

ch = {command = 1, pid = 25729, slot = 2, fd = 11}

#10 0x0000000000442f3f in ngx_master_process_cycle (cycle=0x1aaf9b0) at src/os/unix/ngx_process_cycle.c:136

title = 0x1b07296 "master process /usr/sbin/nginx -c /etc/nginx/nginx.conf"

p = 0x1b072cd ""

size = 56

i = 3

n = 0

sigio = 576

set = {__val = {0 <repeats 16 times>}}

itv = {it_interval = {tv_sec = 0, tv_usec = 0}, it_value = {tv_sec = 0, tv_usec = 6}}

live = 6

delay = 11219297056050

ls = 0x870b0243f2888200

ccf = 0x1ab0978

#11 0x0000000000414e42 in main (argc=3, argv=0x7fff1f1f5208) at src/core/nginx.c:411

i = 70

log = 0x75ddc0

cycle = 0x1aaf9b0

init_cycle = {conf_ctx = 0x0, pool = 0x1aaf440, log = 0x75ddc0, new_log = {log_level = 0, file = 0x0, connection = 0,

handler = 0, data = "" action = "" files = 0x0, free_connections = 0x0, free_connection_n = 0,

reusable_connections_queue = {prev = 0x0, next = 0x0}, listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0,

pool = 0x0}, pathes = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, open_files = {last = 0x0,

part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, shared_memory = {last = 0x0,

part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, connection_n = 0, files_n = 0,

connections = 0x0, read_events = 0x0, write_events = 0x0, old_cycle = 0x0, conf_file = {len = 21,

data = "" "ss"}, conf_param = {len = 0, data = "" conf_prefix = {len = 11,

data = "" "ss"}, prefix = {len = 27, data = "" "/usr/local/openresty/nginx/"}, lock_file = {

len = 0, data = "" hostname = {len = 0, data = "">
ccf = 0x1ab0978

然后我加了redis的close代码，ab测试

# ab -n 10000 -c 100 http://d1.com/file.zip
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking d1.leaderhero.com (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

Server Software: ngx_openresty/1.0.11.28
Server Hostname: d1.com
Server Port: 80
Document Path: /file.zip
Document Length: 196 bytes
Concurrency Level: 100
Time taken for tests: 5.795 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Non-2xx responses: 10000
Total transferred: 4260000 bytes
HTML transferred: 1960000 bytes
Requests per second: 1725.64 [#/sec] (mean)
Time per request: 57.950 [ms] (mean)
Time per request: 0.579 [ms] (mean, across all concurrent requests)
Transfer rate: 717.89 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 1.0 0 13
Processing: 10 57 5.0 57 112
Waiting: 10 57 5.0 57 112
Total: 21 58 4.6 57 112
Percentage of the requests served within a certain time (ms)
50% 57
66% 60
75% 61
80% 61
90% 62
95% 63
98% 65
99% 71
100% 112 (longest request)

用curl看，跳转也是正常的。

然后我配置了连接池

rewrite_by_lua '
...
local ok, err = redis:set_keepalive(0, 100)
if not ok then
ngx.say("failed to set keepalive: ", err)
return
end
';

# ab -n 10000 -c 100 http://d1.com/file.zip
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking d1.leaderhero.com (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

Server Software: ngx_openresty/1.0.11.28
Server Hostname: d1.com
Server Port: 80
Document Path: /file.zip
Document Length: 196 bytes
Concurrency Level: 100
Time taken for tests: 6.191 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Non-2xx responses: 10000
Total transferred: 4260000 bytes
HTML transferred: 1960000 bytes
Requests per second: 1615.33 [#/sec] (mean)
Time per request: 61.907 [ms] (mean)
Time per request: 0.619 [ms] (mean, across all concurrent requests)
Transfer rate: 672.00 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.9 0 11
Processing: 12 61 3.0 62 76
Waiting: 12 61 3.0 62 76
Total: 23 62 2.4 62 76
Percentage of the requests served within a certain time (ms)
50% 62
66% 62
75% 62
80% 62
90% 63
95% 63
98% 64
99% 64
100% 76 (longest request)

看样子也正常。

所以，是不是这样我就显式的close或者配置连接池就好了？

>
> 对于测试，用虚拟机测试流程是通的，但是因为线上是做文件下载，我确实不知道应该怎么设计压力测试，春哥点拨一下。。
>

文件下载当多是 GET 请求，可以尝试使用 ab 或者 weighttp 或者 http_load 这样的工具?

Regards,
-agentzh

agentzh

Hello!

On Tue, Jun 5, 2012 at 5:18 PM, G_will <gwil...@gmail.com> wrote:
>>> Program terminated with signal 11, Segmentation fault.
>>>
>>> #0  0x00000000004f0707 in ngx_http_lua_socket_finalize (r=0x1abdde0,
>>> u=0x407acf78)
>>>
>>>     at ../ngx_lua-0.5.0rc21/src/ngx_http_lua_socket.c:2028
>>>
>>> 2028            *ll = ctx->free_recv_bufs;
>>>
>>> (gdb) bt
>>>
>>> #0  0x00000000004f0707 in ngx_http_lua_socket_finalize (r=0x1abdde0,
>>> u=0x407acf78)
>>>
>>>     at ../ngx_lua-0.5.0rc21/src/ngx_http_lua_socket.c:2028
>>>
>>> #1  0x00000000004f062c in ngx_http_lua_socket_cleanup (data=0x407acf78)
>>> at ../ngx_lua-0.5.0rc21/src/ngx_http_lua_socket.c:2002
>>>
>>> #2  0x00000000004628b0 in ngx_http_free_request (r=0x1abdde0, rc=0) at
>>> src/http/ngx_http_request.c:2973
>>>

你使用的是 ngx_lua 0.5.0rc21 这个 2 个多月前的版本。这处段错误的 bug 我应该已经在 5 月 12 日提交的一个补丁中修复了：

    https://github.com/chaoslawful/lua-nginx-module/commit/d00976d

请尝试最新的 ngx_lua 0.5.0rc29:

    https://github.com/chaoslawful/lua-nginx-module/tags

或者 ngx_openresty 1.0.15.7:

    http://openresty.org/#Download

如果问题依旧，烦请再次提供新的 gdb backtrace :) 多谢，多谢！

Best regards,
-agentzh

gwill.buzz

在 2012年6月5日下午6:30，agentzh <age...@gmail.com>写道：

Hello!

On Tue, Jun 5, 2012 at 5:18 PM, G_will <gwil...@gmail.com> wrote:
>>> Program terminated with signal 11, Segmentation fault.
>>>
>>> #0 0x00000000004f0707 in ngx_http_lua_socket_finalize (r=0x1abdde0,
>>> u=0x407acf78)
>>>
>>> at ../ngx_lua-0.5.0rc21/src/ngx_http_lua_socket.c:2028
>>>
>>> 2028 *ll = ctx->free_recv_bufs;
>>>
>>> (gdb) bt
>>>
>>> #0 0x00000000004f0707 in ngx_http_lua_socket_finalize (r=0x1abdde0,
>>> u=0x407acf78)
>>>
>>> at ../ngx_lua-0.5.0rc21/src/ngx_http_lua_socket.c:2028
>>>
>>> #1 0x00000000004f062c in ngx_http_lua_socket_cleanup (data=""> >>> at ../ngx_lua-0.5.0rc21/src/ngx_http_lua_socket.c:2002
>>>
>>> #2 0x00000000004628b0 in ngx_http_free_request (r=0x1abdde0, rc=0) at
>>> src/http/ngx_http_request.c:2973
>>>

你使用的是 ngx_lua 0.5.0rc21 这个 2 个多月前的版本。这处段错误的 bug 我应该已经在 5 月 12 日提交的一个补丁中修复了：

https://github.com/chaoslawful/lua-nginx-module/commit/d00976d

请尝试最新的 ngx_lua 0.5.0rc29:

https://github.com/chaoslawful/lua-nginx-module/tags

或者 ngx_openresty 1.0.15.7:

http://openresty.org/#Download

如果问题依旧，烦请再次提供新的 gdb backtrace :) 多谢，多谢！

原来如此，我下载的是稳定版的。

期待下一个稳定版尽快问世。

Best regards,

-agentzh

agentzh

Hello!

2012/6/5 G_will <gwil...@gmail.com>:
>> 或者 ngx_openresty 1.0.15.7:
>>
>>    http://openresty.org/#Download
>>
>> 如果问题依旧，烦请再次提供新的 gdb backtrace :) 多谢，多谢！
>
> 原来如此，我下载的是稳定版的。
> 期待下一个稳定版尽快问世。
>

按照目前的计划，我打算让 ngx_openresty 1.0.15.x 系列再发布两个开发版，即 1.0.15.9 和 1.0.15.11
之后，就发布此系列中的一个新的稳定版 1.0.15.12 :)

之后，我将着手把 ngx_openresty 中打包的 nginx 核心升级至官方最新的稳定版系列，即 1.2.x 系列。

Regards,
-agentzh