ngx_lua coroutine 分支上的性能衰退

agentzh

Hello, 晓哲老师！

我使用 mockeagain（https://github.com/agentzh/mockeagain ）运行 nginx 以摸拟 I/O
密集型的应用场景，分别使用 ngx_lua master 和 coroutine 分支运行 lua-resty-mysql 的一个用例：

    location /t {
        content_by_lua '
            local mysql = require "resty.mysql"
            local db = mysql:new()

            db:set_timeout(1000) -- 1 sec

            local ok, err, errno, sqlstate = db:connect({
                host = "$TEST_NGINX_MYSQL_HOST",
                port = $TEST_NGINX_MYSQL_PORT,
                database = "world",
                user = "ngx_test",
                password = "ngx_test"})

            if not ok then
                ngx.say("failed to connect: ", err, ": ", errno, " ", sqlstate)
                return
            end

            local res, err = db:query("select * from City limit 1")
            if not res then
                ngx.say("failed to query: ", err)
                return
            end

            local cjson = require "cjson"
            ngx.say(cjson.encode(res))

            local ok, err = db:set_keepalive(0, 1024)
            if not ok then
                ngx.say("failed to set keepalive: ", err)
                return
            end
        ';
    }

这里查询的是 MySQL 官方提供下载的 world 样例数据库中的 City 表。在我的 ThinkPad W530 (Core
i7-3920XM) 上进行 ab -k -c50 -n4000 localhost:1984/t 压测的结果如下：

    ngx_lua master
    req/sec: 594.57
    response time: 84.095ms

    ngx_lua coroutine
    req/sec: 458.16
    response time: 109.131

可以看到，coroutine 分支不幸地比 master 分支慢了 22% 以上。。。

mockeagain 的配置如下：

    export MOCKEAGAIN=wr

Nginx 一侧的其他相关配置如下：

    worker_processes  2;
    daemon on;
    master_process on;
    error_log logs/error.log warn;
    env MOCKEAGAIN;
    env LD_PRELOAD;

    http {
        postpone_output 1;

        ...
    }

如果不启用 mockeagain，则不会出现 entry lua coroutine 反复多次重入的行为，二者性能上就没有可测量的区别。

我打算着手对 entry lua coroutine 进行一些优化，比如在 ngx_http_lua_ctx_t 结构体里面加一个
entry_cc 字段，这样对于 entry coroutine，就必再查找注册表了。。。打算看看性能上会有多少改进。。。

我希望至少在不使用用户级 coroutine API 的情况下，性能上没有明显的退步 :) 这样把 coroutine 分支合并到
master 才是足够“安全”的，嘿嘿～

BTW, 抄送给 openresty 邮件列表：https://groups.google.com/group/openresty

Best regards,
-agentzh

chaoslawful

我和静龙看一下吧，这个结果挺意外的

On 08/10/2012 08:39 AM, agentzh wrote:
> Hello, 晓哲老师！
>
> 我使用 mockeagain（https://github.com/agentzh/mockeagain ）运行 nginx 以摸拟 I/O
> 密集型的应用场景，分别使用 ngx_lua master 和 coroutine 分支运行 lua-resty-mysql 的一个用例：
>
>     location /t {
>         content_by_lua '
>             local mysql = require "resty.mysql"
>             local db = mysql:new()
>
>             db:set_timeout(1000) -- 1 sec
>
>             local ok, err, errno, sqlstate = db:connect({
>                 host = "$TEST_NGINX_MYSQL_HOST",
>                 port = $TEST_NGINX_MYSQL_PORT,
>                 database = "world",
>                 user = "ngx_test",
>                 password = "ngx_test"})
>
>             if not ok then
>                 ngx.say("failed to connect: ", err, ": ", errno, " ", sqlstate)
>                 return
>             end
>
>             local res, err = db:query("select * from City limit 1")
>             if not res then
>                 ngx.say("failed to query: ", err)
>                 return
>             end
>
>             local cjson = require "cjson"
>             ngx.say(cjson.encode(res))
>
>             local ok, err = db:set_keepalive(0, 1024)
>             if not ok then
>                 ngx.say("failed to set keepalive: ", err)
>                 return
>             end
>         ';
>     }
>
> 这里查询的是 MySQL 官方提供下载的 world 样例数据库中的 City 表。在我的 ThinkPad W530 (Core
> i7-3920XM) 上进行 ab -k -c50 -n4000 localhost:1984/t 压测的结果如下：
>
>     ngx_lua master
>     req/sec: 594.57
>     response time: 84.095ms
>
>     ngx_lua coroutine
>     req/sec: 458.16
>     response time: 109.131
>
> 可以看到，coroutine 分支不幸地比 master 分支慢了 22% 以上。。。
>
> mockeagain 的配置如下：
>
>     export MOCKEAGAIN=wr
>
> Nginx 一侧的其他相关配置如下：
>
>     worker_processes  2;
>     daemon on;
>     master_process on;
>     error_log logs/error.log warn;
>     env MOCKEAGAIN;
>     env LD_PRELOAD;
>
>     http {
>         postpone_output 1;
>
>         ...
>     }
>
> 如果不启用 mockeagain，则不会出现 entry lua coroutine 反复多次重入的行为，二者性能上就没有可测量的区别。
>
> 我打算着手对 entry lua coroutine 进行一些优化，比如在 ngx_http_lua_ctx_t 结构体里面加一个
> entry_cc 字段，这样对于 entry coroutine，就必再查找注册表了。。。打算看看性能上会有多少改进。。。
>
> 我希望至少在不使用用户级 coroutine API 的情况下，性能上没有明显的退步 :) 这样把 coroutine 分支合并到
> master 才是足够“安全”的，嘿嘿～
>
> BTW, 抄送给 openresty 邮件列表：https://groups.google.com/group/openresty
>
> Best regards,
> -agentzh

chaoslawful

Hi,

我本机（Alienware M17x, i7-2720QM）加载mockeagain.so后用相同方法测试结果如下：

分支/QPS	QPS1	QPS2	QPS3
master	499.14	491.23	510.19
coroutine	483.97	486.94	496.89

coroutine分支平均比master慢2.2%

不加载mockeagain.so的测试结果如下：

分支/QPS	QPS1	QPS2	QPS3
master	6401.64	6446.04	6202.13
coroutine	6535.35	6087.72	6062.65

coroutine分支平均比master慢2%

从代码来说，对于此处的测试用例，coroutine分支并未引入太多额外处理逻辑，你所说的请求处理结束时coroutine分支在注册表中查找的逻辑也不可能产生上封信中如此之大的性能差异，加载mockeagain.s后callgrind profiling的结果可以佐证这一点：

master 分支中核心处理部分call graph如下（其中主体执行逻辑ngx_http_lua_run_thread消耗7395813个指令周期，最后 ngx_http_chunked_body_filter没有展现ngx_http_write_filter为 kcachegrind作图误差，可忽略）:

coroutine分支中核心处理部分call graph如下（其中主体执行逻辑ngx_http_lua_run_thread消耗7538251个指令周期，相比于master分支的额外开销约 (7538251-7395813)/7395813 ~ 2%）:

从上面的结果看，我这里的实测数据和代码分析预期是一致的，建议你再复查一下性能数据。

Thanks!

On 08/10/2012 08:39 AM, agentzh wrote:

Hello, 晓哲老师！

我使用 mockeagain（https://github.com/agentzh/mockeagain ）运行 nginx 以摸拟 I/O
密集型的应用场景，分别使用 ngx_lua master 和 coroutine 分支运行 lua-resty-mysql 的一个用例：

    location /t {
        content_by_lua '
            local mysql = require "resty.mysql"
            local db = mysql:new()

            db:set_timeout(1000) -- 1 sec

            local ok, err, errno, sqlstate = db:connect({
                host = "$TEST_NGINX_MYSQL_HOST",
                port = $TEST_NGINX_MYSQL_PORT,
                database = "world",
                user = "ngx_test",
                password = "ngx_test"})

            if not ok then
                ngx.say("failed to connect: ", err, ": ", errno, " ", sqlstate)
                return
            end

            local res, err = db:query("select * from City limit 1")
            if not res then
                ngx.say("failed to query: ", err)
                return
            end

            local cjson = require "cjson"
            ngx.say(cjson.encode(res))

            local ok, err = db:set_keepalive(0, 1024)
            if not ok then
                ngx.say("failed to set keepalive: ", err)
                return
            end
        ';
    }

这里查询的是 MySQL 官方提供下载的 world 样例数据库中的 City 表。在我的 ThinkPad W530 (Core
i7-3920XM) 上进行 ab -k -c50 -n4000 localhost:1984/t 压测的结果如下：

    ngx_lua master
    req/sec: 594.57
    response time: 84.095ms

    ngx_lua coroutine
    req/sec: 458.16
    response time: 109.131

可以看到，coroutine 分支不幸地比 master 分支慢了 22% 以上。。。

mockeagain 的配置如下：

    export MOCKEAGAIN=wr

Nginx 一侧的其他相关配置如下：

    worker_processes  2;
    daemon on;
    master_process on;
    error_log logs/error.log warn;
    env MOCKEAGAIN;
    env LD_PRELOAD;

    http {
        postpone_output 1;

        ...
    }

如果不启用 mockeagain，则不会出现 entry lua coroutine 反复多次重入的行为，二者性能上就没有可测量的区别。

我打算着手对 entry lua coroutine 进行一些优化，比如在 ngx_http_lua_ctx_t 结构体里面加一个
entry_cc 字段，这样对于 entry coroutine，就必再查找注册表了。。。打算看看性能上会有多少改进。。。

我希望至少在不使用用户级 coroutine API 的情况下，性能上没有明显的退步 :) 这样把 coroutine 分支合并到
master 才是足够“安全”的，嘿嘿～

BTW, 抄送给 openresty 邮件列表：https://groups.google.com/group/openresty

Best regards,
-agentzh

chaoslawful

另外，你提到的对entry coroutine判定的优化是个好主意，在这个场景下应该可以把这2%的性能也追回来，呵呵

On 08/10/2012 02:27 PM, Chaos Wang wrote:

Hi,

我本机（Alienware M17x, i7-2720QM）加载mockeagain.so后用相同方法测试结果如下：

分支/QPS
QPS1
QPS2
QPS3

master
499.14
491.23
510.19

coroutine
483.97
486.94
496.89

coroutine分支平均比master慢2.2%

不加载mockeagain.so的测试结果如下：

分支/QPS
QPS1
QPS2
QPS3

master
6401.64
6446.04
6202.13

coroutine
6535.35
6087.72
6062.65

coroutine分支平均比master慢2%

从代码来说，对于此处的测试用例，coroutine分支并未引入太多额外处理逻辑，你所说的请求处理结束时coroutine分支在注册表中查找的逻辑也不可能产生上封信中如此之大的性能差异，加载mockeagain.s后callgrind profiling的结果可以佐证这一点：

master 分支中核心处理部分call graph如下（其中主体执行逻辑ngx_http_lua_run_thread消耗7395813个指令周期，最后 ngx_http_chunked_body_filter没有展现ngx_http_write_filter为 kcachegrind作图误差，可忽略）:

coroutine分支中核心处理部分call graph如下（其中主体执行逻辑ngx_http_lua_run_thread消耗7538251个指令周期，相比于master分支的额外开销约 (7538251-7395813)/7395813 ~ 2%）:

从上面的结果看，我这里的实测数据和代码分析预期是一致的，建议你再复查一下性能数据。

Thanks!

On 08/10/2012 08:39 AM, agentzh wrote:
Hello, 晓哲老师！

我使用 mockeagain（https://github.com/agentzh/mockeagain ）运行 nginx 以摸拟 I/O
密集型的应用场景，分别使用 ngx_lua master 和 coroutine 分支运行 lua-resty-mysql 的一个用例：

    location /t {
        content_by_lua '
            local mysql = require "resty.mysql"
            local db = mysql:new()

            db:set_timeout(1000) -- 1 sec

            local ok, err, errno, sqlstate = db:connect({
                host = "$TEST_NGINX_MYSQL_HOST",
                port = $TEST_NGINX_MYSQL_PORT,
                database = "world",
                user = "ngx_test",
                password = "ngx_test"})

            if not ok then
                ngx.say("failed to connect: ", err, ": ", errno, " ", sqlstate)
                return
            end

            local res, err = db:query("select * from City limit 1")
            if not res then
                ngx.say("failed to query: ", err)
                return
            end

            local cjson = require "cjson"
            ngx.say(cjson.encode(res))

            local ok, err = db:set_keepalive(0, 1024)
            if not ok then
                ngx.say("failed to set keepalive: ", err)
                return
            end
        ';
    }

这里查询的是 MySQL 官方提供下载的 world 样例数据库中的 City 表。在我的 ThinkPad W530 (Core
i7-3920XM) 上进行 ab -k -c50 -n4000 localhost:1984/t 压测的结果如下：

    ngx_lua master
    req/sec: 594.57
    response time: 84.095ms

    ngx_lua coroutine
    req/sec: 458.16
    response time: 109.131

可以看到，coroutine 分支不幸地比 master 分支慢了 22% 以上。。。

mockeagain 的配置如下：

    export MOCKEAGAIN=wr

Nginx 一侧的其他相关配置如下：

    worker_processes  2;
    daemon on;
    master_process on;
    error_log logs/error.log warn;
    env MOCKEAGAIN;
    env LD_PRELOAD;

    http {
        postpone_output 1;

        ...
    }

如果不启用 mockeagain，则不会出现 entry lua coroutine 反复多次重入的行为，二者性能上就没有可测量的区别。

我打算着手对 entry lua coroutine 进行一些优化，比如在 ngx_http_lua_ctx_t 结构体里面加一个
entry_cc 字段，这样对于 entry coroutine，就必再查找注册表了。。。打算看看性能上会有多少改进。。。

我希望至少在不使用用户级 coroutine API 的情况下，性能上没有明显的退步 :) 这样把 coroutine 分支合并到
master 才是足够“安全”的，嘿嘿～

BTW, 抄送给 openresty 邮件列表：https://groups.google.com/group/openresty

Best regards,
-agentzh

agentzh

Hello!

2012/8/9 Chaos Wang <chaos...@gmail.com>
> coroutine分支平均比master慢2%
>

我又在我的机器上重新进行了多组测试，发现我的机器确实会随机地提升 20%
的性能，不清楚是什么原因导致的（或许是因为我使用两种不同型号的内存条，抑或是 CPU cache 完整命中？），但确实与使用 master
还是 coroutine 分支无关。

在平均条件下，二者确实几乎没有可以测量的区别，即使是在 mockeagain 模式下。

>
> 从代码来说，对于此处的测试用例，coroutine分支并未引入太多额外处理逻辑，你所说的请求处理结束时coroutine分支在注册表
> 中查找的逻辑也不可能产生上封信中如此之大的性能差异，加载mockeagain.s后callgrind profiling的结果可以佐证这一点：
>

callgrind 真是极好的工具！我喜欢它为每个函数生成的 IR (Instruction Reads) 指标！我这里使用
mockeagain 运行最初的 lua-resty-mysql 用例，同时连续请求 nginx 10 次的测量结果是：

master 分支:
    ngx_http_lua_run_thread:
          12,420  (--inclusive=no)
     7,001,944  (--inclusive=yes)

coroutine 分支：
    ngx_http_lua_run_thread:
          14,866  (--inclusive=no)
       6,922,811  (--inclusive=yes)

我在针对 entry coroutine 进行优化之后（补丁见附件或信尾，尚未提交到 GitHub），coroutine 分支的结果变为了：

    ngx_http_lua_run_thread:
          14,284  (--inclusive=no)
       6,918,866  (--inclusive=yes)

以上结果都是使用的 Nginx 默认的 gcc -O1 优化级别。

使用 gcc -O2 后，优化后的 coroutine 分支的结果是：

    ngx_http_lua_run_thread:
          13,470  (--inclusive=no)
       7,056,177  (--inclusive=yes)

而 gcc -O3 则为

    ngx_http_lua_run_thread:
          13,384  (--inclusive=no)
       7,037,605  (--inclusive=yes)

如此看来，性能上似乎已经足够好了 :)

Best regards,
-agentzh

diff --git a/src/ngx_http_lua_accessby.c b/src/ngx_http_lua_accessby.c
index 211ac2f..946dc54 100644
--- a/src/ngx_http_lua_accessby.c
+++ b/src/ngx_http_lua_accessby.c
@@ -263,6 +263,7 @@ ngx_http_lua_access_by_chunk(lua_State *L,
ngx_http_request_t *r)

     ctx->entered_access_phase = 1;

+    ctx->entry = cc;
     ctx->cc = cc;
     ctx->cc_ref = cc_ref;

diff --git a/src/ngx_http_lua_common.h b/src/ngx_http_lua_common.h
index f571900..693874c 100644
--- a/src/ngx_http_lua_common.h
+++ b/src/ngx_http_lua_common.h
@@ -210,6 +210,9 @@ typedef struct {
                                             not necessarily to be the
                                             request's entry coroutine */

+    lua_State               *entry;     /*  the entry Lua coroutine */
+
+
     int                      cc_ref;    /*  reference to anchor coroutine in
                                             the lua registry. it always
                                             ref to the entry coroutine of the
diff --git a/src/ngx_http_lua_contentby.c b/src/ngx_http_lua_contentby.c
index d727108..989bc48 100644
--- a/src/ngx_http_lua_contentby.c
+++ b/src/ngx_http_lua_contentby.c
@@ -70,6 +70,7 @@ ngx_http_lua_content_by_chunk(lua_State *L,
ngx_http_request_t *r)
     lua_rawset(cc, LUA_GLOBALSINDEX);
     /*  }}} */

+    ctx->entry = cc;
     ctx->cc = cc;
     ctx->cc_ref = cc_ref;

diff --git a/src/ngx_http_lua_rewriteby.c b/src/ngx_http_lua_rewriteby.c
index 6915747..7f0508a 100644
--- a/src/ngx_http_lua_rewriteby.c
+++ b/src/ngx_http_lua_rewriteby.c
@@ -264,6 +264,7 @@ ngx_http_lua_rewrite_by_chunk(lua_State *L,
ngx_http_request_t *r)

     ctx->entered_rewrite_phase = 1;

+    ctx->entry = cc;
     ctx->cc = cc;
     ctx->cc_ref = cc_ref;

diff --git a/src/ngx_http_lua_util.c b/src/ngx_http_lua_util.c
index 811681b..cf57121 100644
--- a/src/ngx_http_lua_util.c
+++ b/src/ngx_http_lua_util.c
@@ -913,6 +913,12 @@ ngx_http_lua_run_thread(lua_State *L,
ngx_http_request_t *r,
                  * lua_yield()
                  */
                 switch(ctx->cc_op) {
+                case NGX_HTTP_LUA_USER_CORO_NOP:
+                    dd("hit! it is the API yield");
+
+                    lua_settop(cc, 0);
+                    return NGX_AGAIN;
+
                 case NGX_HTTP_LUA_USER_CORO_RESUME:
                     ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                                    "lua coroutine: resume");
@@ -934,28 +940,19 @@ ngx_http_lua_run_thread(lua_State *L,
ngx_http_request_t *r,
                     cc = next_cc;
                     ctx->cc = cc;

-                    continue;
+                    break;
+
+                default:
+                    /* NGX_HTTP_LUA_USER_CORO_YIELD */

-                case NGX_HTTP_LUA_USER_CORO_YIELD:
                     ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                                    "lua coroutine: yield");

-                    /*
-                     * here we need to find the parent coroutine of the
-                     * yield coroutine in the weak table(in registry)
-                     */
                     ctx->cc_op = NGX_HTTP_LUA_USER_CORO_NOP;

-                    nrets = lua_gettop(cc);
-
-                    /* find parent coroutine in weak ref table */
-                    ngx_http_lua_get_coroutine_parents(L);
-                    lua_pushthread(cc);
-                    lua_xmove(cc, L, 1);
-                    lua_rawget(L, -2);
+                    if (cc == ctx->entry) {
+                        /* entry coroutine can not yield */

-                    /* entry coroutine can not yield */
-                    if (!lua_isthread(L, -1)) {
                         lua_settop(L, 0);

                         ngx_http_lua_del_thread(r, L, cc_ref);
@@ -971,6 +968,16 @@ ngx_http_lua_run_thread(lua_State *L,
ngx_http_request_t *r,
                                         NGX_HTTP_INTERNAL_SERVER_ERROR;
                     }

+                    /* being a user coroutine that has a parent */
+
+                    nrets = lua_gettop(cc);
+
+                    /* find parent coroutine in weak ref table */
+                    ngx_http_lua_get_coroutine_parents(L);
+                    lua_pushthread(cc);
+                    lua_xmove(cc, L, 1);
+                    lua_rawget(L, -2);
+
                     next_cc = lua_tothread(L, -1);
                     lua_pop(L, 2);

@@ -988,73 +995,73 @@ ngx_http_lua_run_thread(lua_State *L,
ngx_http_request_t *r,
                     cc = next_cc;
                     ctx->cc = cc;

-                    continue;
-
-                default:
-                    /* NGX_HTTP_LUA_USER_CORO_NOP */
                     break;
                 }

-                lua_settop(cc, 0);
-
-                return NGX_AGAIN;
+                /* try resuming on the new coroutine again */
+                continue;

             case 0:
 #if 0
                 ngx_http_lua_dump_postponed(r);
 #endif

-                /* check if the dead coroutine has a parent coroutine */
-                nrets = lua_gettop(cc);
+                if (cc == ctx->entry) {
+                    dd("hit! it is the entry");

-                ngx_http_lua_get_coroutine_parents(L);
-                lua_pushthread(cc);
-                lua_xmove(cc, L, 1);
-                lua_rawget(L, -2);
+                    lua_settop(L, 0);

-                /* resume the parent coroutine */
-                if (lua_isthread(L, -1)) {
-                    next_cc = lua_tothread(L, -1);
-                    lua_pop(L, 2);
-
-                    /*
-                     * ended successful, coroutine.resume returns true plus
-                     * any return values
-                     */
-                    lua_pushboolean(next_cc, 1);
+                    ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
+                                   "lua entry thread ended normally");

-                    if (nrets) {
-                        lua_xmove(cc, next_cc, nrets);
-                    }
+                    /* the entry thread was dead, delete it */
+                    ngx_http_lua_del_thread(r, L, cc_ref);
+                    ctx->cc_ref = LUA_NOREF;

-                    nrets++;
-                    ctx->cc = cc = next_cc;
+                    if (ctx->entered_content_phase) {
+                        rc = ngx_http_lua_send_chain_link(r, ctx,
+                                                          NULL /* last_buf */);

-                    ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
-                                   "lua coroutine: lua thread ended normally");
+                        if (rc == NGX_ERROR
+                            || rc >= NGX_HTTP_SPECIAL_RESPONSE)
+                        {
+                            return rc;
+                        }
+                    }

-                    continue;
+                    return NGX_OK;
                 }

-                lua_settop(L, 0);
+                /* being a user coroutine that has a parent */

-                ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
-                               "lua entry thread ended normally");
+                nrets = lua_gettop(cc);

-                /* the entry thread was dead, delete it */
-                ngx_http_lua_del_thread(r, L, cc_ref);
-                ctx->cc_ref = LUA_NOREF;
+                ngx_http_lua_get_coroutine_parents(L);
+                lua_pushthread(cc);
+                lua_xmove(cc, L, 1);
+                lua_rawget(L, -2);

-                if (ctx->entered_content_phase) {
-                    rc = ngx_http_lua_send_chain_link(r, ctx,
-                                                      NULL /* last_buf */);
+                next_cc = lua_tothread(L, -1);
+                lua_pop(L, 2);

-                    if (rc == NGX_ERROR || rc >= NGX_HTTP_SPECIAL_RESPONSE) {
-                        return rc;
-                    }
+                /*
+                 * ended successful, coroutine.resume returns true plus
+                 * any return values
+                 */
+                lua_pushboolean(next_cc, 1);
+
+                if (nrets) {
+                    lua_xmove(cc, next_cc, nrets);
                 }

-                return NGX_OK;
+                nrets++;
+                cc = next_cc;
+                ctx->cc = cc;
+
+                ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
+                               "lua coroutine: lua user thread ended
normally");
+
+                continue;

             case LUA_ERRRUN:
                 err = "runtime error";
@@ -1089,46 +1096,50 @@ ngx_http_lua_run_thread(lua_State *L,
ngx_http_request_t *r,
             trace = lua_tostring(L, -1);
             lua_pop(L, 1);

-            /* check if the dead coroutine has a parent coroutine*/
-            ngx_http_lua_get_coroutine_parents(L);
-            lua_pushthread(cc);
-            lua_xmove(cc, L, 1);
-            lua_rawget(L, -2);
+            if (cc == ctx->entry) {
+                ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
+                              "lua handler aborted: %s: %s\n%s", err, msg,
+                              trace);

-            if (lua_isthread(L, -1)) {
-                next_cc = lua_tothread(L, -1);
-                lua_pop(L, 2);
+                lua_settop(L, 0);

-                /*
-                 * ended with error, coroutine.resume returns false plus
-                 * err msg
-                 */
-                lua_pushboolean(next_cc, 0);
-                lua_xmove(cc, next_cc, 1);
-                nrets = 2;
+                ngx_http_lua_del_thread(r, L, cc_ref);
+                ctx->cc_ref = LUA_NOREF;

-                ctx->cc = cc = next_cc;
+                ngx_http_lua_request_cleanup(r);

-                ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
-                              "lua coroutine: %s: %s\n%s", err, msg, trace);
+                dd("headers sent? %d", ctx->headers_sent ? 1 : 0);

-                continue;
+                return ctx->headers_sent ? NGX_ERROR :
+                            NGX_HTTP_INTERNAL_SERVER_ERROR;
             }

-            ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
-                          "lua handler aborted: %s: %s\n%s", err, msg, trace);
+            /* being a user coroutine that has a parent */
+
+            ngx_http_lua_get_coroutine_parents(L);
+            lua_pushthread(cc);
+            lua_xmove(cc, L, 1);
+            lua_rawget(L, -2);

+            next_cc = lua_tothread(L, -1);
             lua_pop(L, 2);

-            ngx_http_lua_del_thread(r, L, cc_ref);
-            ctx->cc_ref = LUA_NOREF;
+            /*
+             * ended with error, coroutine.resume returns false plus
+             * err msg
+             */
+            lua_pushboolean(next_cc, 0);
+            lua_xmove(cc, next_cc, 1);
+            nrets = 2;

-            ngx_http_lua_request_cleanup(r);
+            cc = next_cc;
+            ctx->cc = cc;

-            dd("headers sent? %d", ctx->headers_sent ? 1 : 0);
+            ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
+                          "lua coroutine: %s: %s\n%s", err, msg, trace);

-            return ctx->headers_sent ? NGX_ERROR :
-                        NGX_HTTP_INTERNAL_SERVER_ERROR;
+            /* try resuming on the new coroutine again */
+            continue;
         }

     } NGX_LUA_EXCEPTION_CATCH {

Attachment: optimizing-entry-coroutines.patch
Description: Binary data

chaoslawful

不错，如此性能无忧矣，哈哈

On 08/11/2012 09:14 AM, agentzh wrote:
> Hello!
>
> 2012/8/9 Chaos Wang <chaos...@gmail.com>
>> coroutine分支平均比master慢2%
>>
> 我又在我的机器上重新进行了多组测试，发现我的机器确实会随机地提升 20%
> 的性能，不清楚是什么原因导致的（或许是因为我使用两种不同型号的内存条，抑或是 CPU cache 完整命中？），但确实与使用 master
> 还是 coroutine 分支无关。
>
> 在平均条件下，二者确实几乎没有可以测量的区别，即使是在 mockeagain 模式下。
>
>> 从代码来说，对于此处的测试用例，coroutine分支并未引入太多额外处理逻辑，你所说的请求处理结束时coroutine分支在注册表
>> 中查找的逻辑也不可能产生上封信中如此之大的性能差异，加载mockeagain.s后callgrind profiling的结果可以佐证这一点：
>>
> callgrind 真是极好的工具！我喜欢它为每个函数生成的 IR (Instruction Reads) 指标！我这里使用
> mockeagain 运行最初的 lua-resty-mysql 用例，同时连续请求 nginx 10 次的测量结果是：
>
> master 分支:
>     ngx_http_lua_run_thread:
>           12,420  (--inclusive=no)
>      7,001,944  (--inclusive=yes)
>
> coroutine 分支：
>     ngx_http_lua_run_thread:
>           14,866  (--inclusive=no)
>        6,922,811  (--inclusive=yes)
>
> 我在针对 entry coroutine 进行优化之后（补丁见附件或信尾，尚未提交到 GitHub），coroutine 分支的结果变为了：
>
>     ngx_http_lua_run_thread:
>           14,284  (--inclusive=no)
>        6,918,866  (--inclusive=yes)
>
> 以上结果都是使用的 Nginx 默认的 gcc -O1 优化级别。
>
> 使用 gcc -O2 后，优化后的 coroutine 分支的结果是：
>
>     ngx_http_lua_run_thread:
>           13,470  (--inclusive=no)
>        7,056,177  (--inclusive=yes)
>
> 而 gcc -O3 则为
>
>     ngx_http_lua_run_thread:
>           13,384  (--inclusive=no)
>        7,037,605  (--inclusive=yes)
>
> 如此看来，性能上似乎已经足够好了 :)
>
> Best regards,
> -agentzh
>
> diff --git a/src/ngx_http_lua_accessby.c b/src/ngx_http_lua_accessby.c
> index 211ac2f..946dc54 100644
> --- a/src/ngx_http_lua_accessby.c
> +++ b/src/ngx_http_lua_accessby.c
> @@ -263,6 +263,7 @@ ngx_http_lua_access_by_chunk(lua_State *L,
> ngx_http_request_t *r)
>
>      ctx->entered_access_phase = 1;
>
> +    ctx->entry = cc;
>      ctx->cc = cc;
>      ctx->cc_ref = cc_ref;
>
> diff --git a/src/ngx_http_lua_common.h b/src/ngx_http_lua_common.h
> index f571900..693874c 100644
> --- a/src/ngx_http_lua_common.h
> +++ b/src/ngx_http_lua_common.h
> @@ -210,6 +210,9 @@ typedef struct {
>                                              not necessarily to be the
>                                              request's entry coroutine */
>
> +    lua_State               *entry;     /*  the entry Lua coroutine */
> +
> +
>      int                      cc_ref;    /*  reference to anchor coroutine in
>                                              the lua registry. it always
>                                              ref to the entry coroutine of the
> diff --git a/src/ngx_http_lua_contentby.c b/src/ngx_http_lua_contentby.c
> index d727108..989bc48 100644
> --- a/src/ngx_http_lua_contentby.c
> +++ b/src/ngx_http_lua_contentby.c
> @@ -70,6 +70,7 @@ ngx_http_lua_content_by_chunk(lua_State *L,
> ngx_http_request_t *r)
>      lua_rawset(cc, LUA_GLOBALSINDEX);
>      /*  }}} */
>
> +    ctx->entry = cc;
>      ctx->cc = cc;
>      ctx->cc_ref = cc_ref;
>
> diff --git a/src/ngx_http_lua_rewriteby.c b/src/ngx_http_lua_rewriteby.c
> index 6915747..7f0508a 100644
> --- a/src/ngx_http_lua_rewriteby.c
> +++ b/src/ngx_http_lua_rewriteby.c
> @@ -264,6 +264,7 @@ ngx_http_lua_rewrite_by_chunk(lua_State *L,
> ngx_http_request_t *r)
>
>      ctx->entered_rewrite_phase = 1;
>
> +    ctx->entry = cc;
>      ctx->cc = cc;
>      ctx->cc_ref = cc_ref;
>
> diff --git a/src/ngx_http_lua_util.c b/src/ngx_http_lua_util.c
> index 811681b..cf57121 100644
> --- a/src/ngx_http_lua_util.c
> +++ b/src/ngx_http_lua_util.c
> @@ -913,6 +913,12 @@ ngx_http_lua_run_thread(lua_State *L,
> ngx_http_request_t *r,
>                   * lua_yield()
>                   */
>                  switch(ctx->cc_op) {
> +                case NGX_HTTP_LUA_USER_CORO_NOP:
> +                    dd("hit! it is the API yield");
> +
> +                    lua_settop(cc, 0);
> +                    return NGX_AGAIN;
> +
>                  case NGX_HTTP_LUA_USER_CORO_RESUME:
>                      ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
>                                     "lua coroutine: resume");
> @@ -934,28 +940,19 @@ ngx_http_lua_run_thread(lua_State *L,
> ngx_http_request_t *r,
>                      cc = next_cc;
>                      ctx->cc = cc;
>
> -                    continue;
> +                    break;
> +
> +                default:
> +                    /* NGX_HTTP_LUA_USER_CORO_YIELD */
>
> -                case NGX_HTTP_LUA_USER_CORO_YIELD:
>                      ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
>                                     "lua coroutine: yield");
>
> -                    /*
> -                     * here we need to find the parent coroutine of the
> -                     * yield coroutine in the weak table(in registry)
> -                     */
>                      ctx->cc_op = NGX_HTTP_LUA_USER_CORO_NOP;
>
> -                    nrets = lua_gettop(cc);
> -
> -                    /* find parent coroutine in weak ref table */
> -                    ngx_http_lua_get_coroutine_parents(L);
> -                    lua_pushthread(cc);
> -                    lua_xmove(cc, L, 1);
> -                    lua_rawget(L, -2);
> +                    if (cc == ctx->entry) {
> +                        /* entry coroutine can not yield */
>
> -                    /* entry coroutine can not yield */
> -                    if (!lua_isthread(L, -1)) {
>                          lua_settop(L, 0);
>
>                          ngx_http_lua_del_thread(r, L, cc_ref);
> @@ -971,6 +968,16 @@ ngx_http_lua_run_thread(lua_State *L,
> ngx_http_request_t *r,
>                                          NGX_HTTP_INTERNAL_SERVER_ERROR;
>                      }
>
> +                    /* being a user coroutine that has a parent */
> +
> +                    nrets = lua_gettop(cc);
> +
> +                    /* find parent coroutine in weak ref table */
> +                    ngx_http_lua_get_coroutine_parents(L);
> +                    lua_pushthread(cc);
> +                    lua_xmove(cc, L, 1);
> +                    lua_rawget(L, -2);
> +
>                      next_cc = lua_tothread(L, -1);
>                      lua_pop(L, 2);
>
> @@ -988,73 +995,73 @@ ngx_http_lua_run_thread(lua_State *L,
> ngx_http_request_t *r,
>                      cc = next_cc;
>                      ctx->cc = cc;
>
> -                    continue;
> -
> -                default:
> -                    /* NGX_HTTP_LUA_USER_CORO_NOP */
>                      break;
>                  }
>
> -                lua_settop(cc, 0);
> -
> -                return NGX_AGAIN;
> +                /* try resuming on the new coroutine again */
> +                continue;
>
>              case 0:
>  #if 0
>                  ngx_http_lua_dump_postponed(r);
>  #endif
>
> -                /* check if the dead coroutine has a parent coroutine */
> -                nrets = lua_gettop(cc);
> +                if (cc == ctx->entry) {
> +                    dd("hit! it is the entry");
>
> -                ngx_http_lua_get_coroutine_parents(L);
> -                lua_pushthread(cc);
> -                lua_xmove(cc, L, 1);
> -                lua_rawget(L, -2);
> +                    lua_settop(L, 0);
>
> -                /* resume the parent coroutine */
> -                if (lua_isthread(L, -1)) {
> -                    next_cc = lua_tothread(L, -1);
> -                    lua_pop(L, 2);
> -
> -                    /*
> -                     * ended successful, coroutine.resume returns true plus
> -                     * any return values
> -                     */
> -                    lua_pushboolean(next_cc, 1);
> +                    ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
> +                                   "lua entry thread ended normally");
>
> -                    if (nrets) {
> -                        lua_xmove(cc, next_cc, nrets);
> -                    }
> +                    /* the entry thread was dead, delete it */
> +                    ngx_http_lua_del_thread(r, L, cc_ref);
> +                    ctx->cc_ref = LUA_NOREF;
>
> -                    nrets++;
> -                    ctx->cc = cc = next_cc;
> +                    if (ctx->entered_content_phase) {
> +                        rc = ngx_http_lua_send_chain_link(r, ctx,
> +                                                          NULL /* last_buf */);
>
> -                    ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
> -                                   "lua coroutine: lua thread ended normally");
> +                        if (rc == NGX_ERROR
> +                            || rc >= NGX_HTTP_SPECIAL_RESPONSE)
> +                        {
> +                            return rc;
> +                        }
> +                    }
>
> -                    continue;
> +                    return NGX_OK;
>                  }
>
> -                lua_settop(L, 0);
> +                /* being a user coroutine that has a parent */
>
> -                ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
> -                               "lua entry thread ended normally");
> +                nrets = lua_gettop(cc);
>
> -                /* the entry thread was dead, delete it */
> -                ngx_http_lua_del_thread(r, L, cc_ref);
> -                ctx->cc_ref = LUA_NOREF;
> +                ngx_http_lua_get_coroutine_parents(L);
> +                lua_pushthread(cc);
> +                lua_xmove(cc, L, 1);
> +                lua_rawget(L, -2);
>
> -                if (ctx->entered_content_phase) {
> -                    rc = ngx_http_lua_send_chain_link(r, ctx,
> -                                                      NULL /* last_buf */);
> +                next_cc = lua_tothread(L, -1);
> +                lua_pop(L, 2);
>
> -                    if (rc == NGX_ERROR || rc >= NGX_HTTP_SPECIAL_RESPONSE) {
> -                        return rc;
> -                    }
> +                /*
> +                 * ended successful, coroutine.resume returns true plus
> +                 * any return values
> +                 */
> +                lua_pushboolean(next_cc, 1);
> +
> +                if (nrets) {
> +                    lua_xmove(cc, next_cc, nrets);
>                  }
>
> -                return NGX_OK;
> +                nrets++;
> +                cc = next_cc;
> +                ctx->cc = cc;
> +
> +                ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
> +                               "lua coroutine: lua user thread ended
> normally");
> +
> +                continue;
>
>              case LUA_ERRRUN:
>                  err = "runtime error";
> @@ -1089,46 +1096,50 @@ ngx_http_lua_run_thread(lua_State *L,
> ngx_http_request_t *r,
>              trace = lua_tostring(L, -1);
>              lua_pop(L, 1);
>
> -            /* check if the dead coroutine has a parent coroutine*/
> -            ngx_http_lua_get_coroutine_parents(L);
> -            lua_pushthread(cc);
> -            lua_xmove(cc, L, 1);
> -            lua_rawget(L, -2);
> +            if (cc == ctx->entry) {
> +                ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
> +                              "lua handler aborted: %s: %s\n%s", err, msg,
> +                              trace);
>
> -            if (lua_isthread(L, -1)) {
> -                next_cc = lua_tothread(L, -1);
> -                lua_pop(L, 2);
> +                lua_settop(L, 0);
>
> -                /*
> -                 * ended with error, coroutine.resume returns false plus
> -                 * err msg
> -                 */
> -                lua_pushboolean(next_cc, 0);
> -                lua_xmove(cc, next_cc, 1);
> -                nrets = 2;
> +                ngx_http_lua_del_thread(r, L, cc_ref);
> +                ctx->cc_ref = LUA_NOREF;
>
> -                ctx->cc = cc = next_cc;
> +                ngx_http_lua_request_cleanup(r);
>
> -                ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
> -                              "lua coroutine: %s: %s\n%s", err, msg, trace);
> +                dd("headers sent? %d", ctx->headers_sent ? 1 : 0);
>
> -                continue;
> +                return ctx->headers_sent ? NGX_ERROR :
> +                            NGX_HTTP_INTERNAL_SERVER_ERROR;
>              }
>
> -            ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
> -                          "lua handler aborted: %s: %s\n%s", err, msg, trace);
> +            /* being a user coroutine that has a parent */
> +
> +            ngx_http_lua_get_coroutine_parents(L);
> +            lua_pushthread(cc);
> +            lua_xmove(cc, L, 1);
> +            lua_rawget(L, -2);
>
> +            next_cc = lua_tothread(L, -1);
>              lua_pop(L, 2);
>
> -            ngx_http_lua_del_thread(r, L, cc_ref);
> -            ctx->cc_ref = LUA_NOREF;
> +            /*
> +             * ended with error, coroutine.resume returns false plus
> +             * err msg
> +             */
> +            lua_pushboolean(next_cc, 0);
> +            lua_xmove(cc, next_cc, 1);
> +            nrets = 2;
>
> -            ngx_http_lua_request_cleanup(r);
> +            cc = next_cc;
> +            ctx->cc = cc;
>
> -            dd("headers sent? %d", ctx->headers_sent ? 1 : 0);
> +            ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
> +                          "lua coroutine: %s: %s\n%s", err, msg, trace);
>
> -            return ctx->headers_sent ? NGX_ERROR :
> -                        NGX_HTTP_INTERNAL_SERVER_ERROR;
> +            /* try resuming on the new coroutine again */
> +            continue;
>          }
>
>      } NGX_LUA_EXCEPTION_CATCH {