[ANN] OpenResty 1.9.3.2 RC1 now available for testing

agentzh · 2015-10-28T21:12:57+00:00

Hello! On Wed, Oct 28, 2015 at 4:58 PM, 杨阳 wrote: > for example, a mobile client post some data to nginx, and nginx > proxy to c...

[ANN] OpenResty 1.9.3.2 RC1 now available for testing

agentzh

Hi guys,

I've just kicked out OpenResty 1.9.3.2 RC1 for testing:

    https://openresty.org/download/ngx_openresty-1.9.3.2rc1.tar.gz

The highlight of this release is the new *_by_lua_block {} directives
added in the ngx_http_lua module.

For example, instead of writing

    content_by_lua '
        ngx.say("Hello, OpenResty\'s world!\\n")
    ';

We can just write

    content_by_lua_block {
        ngx.say("Hello, OpenResty's world!\n")
    }

No ugly special character escaping is needed in the Lua source any
more in the latter form :)

Changes since the last (formal) release, 1.9.3.1:

 *   upgraded LuaNginxModule to 0.9.17.

     *   feature: implemented "*_by_lua_block {}" directives for all
         the existing *_by_lua directives so that we no longer have
         to escape special characters while inlining Lua source
         inside the "nginx.conf" file.

     *   feature: initial fixes when being used with the new
         "ngx_http_v2" module since nginx 1.9.5. thanks itpp16 for
         the patches.

     *   bugfix: the user specified "./configure"'s "--with-cc-opt"
         and "--with-ld-opt" might override the
         "LUAJIT_INC"/"LUAJIT_LIB" and "LUA_INC"/"LUA_LIB"
         environment settings. thanks Julian Gonggrijp for the
         report.

     *   bugfix: setting builtin request header "Upgrade" via
         ngx.req.set_header and etc might not take effect with some
         builtin nginx modules.

     *   bugfix: setting builtin request headers "Depth",
         "Destination", "Overwrite", and "Date" via
         ngx.req.set_header() and etc might not take effect at least
         with ngx_http_dav_module. thanks Igor Clark for the report.

     *   bugfix: fixed typos due to copy&paste mistakes in some error
         messages.

     *   bugfix: fixed one "-Wmaybe-uninitialized" warning when
         compiling with "gcc -Os".

     *   bugfix: use of shared dicts resulted in (unwanted)
         registrations of shared dict metatables on all the
         lightuserdata in the Lua space. thanks helloyi for the
         report and patch.

     *   bugfix: if a 3rd-party module calls
         "ngx_http_conf_get_module_srv_conf" to fetch its current
         "srv_conf" construct in its "merge_srv_conf" callback, then
         use of init_worker_by_lua might lead to segmentation faults
         (the same also applied to merge_loc_conf). thanks chiyouhen
         for the report and patch.

     *   bugfix: the "if_unmodified_since" "shortcut" field in
         "ngx_http_headers_in_t" was first added in nginx 0.9.2.

     *   bugfix: ngx.req.clear_header/ngx.req.set_header: we did not
         update the shortcut fields in "ngx_http_headers_in_t" added
         since nginx 1.3.3 which may confuse other nginx modules
         accessing them.

     *   bugfix: setting "Content-Type" response values including ";
         charset=xxx" via the ngx.header API might bypass the MIME
         type checks in other nginx modules like ngx_gzip. thanks
         Andreas Fischer for the report.

     *   bugfix: typo fixes in some debug logging messages. thanks
         doujiang for the patch.

     *   optimize: fixed the hash-table initial sizes of the cosocket
         metatables. thanks ops-dev-cn for the patch.

     *   tests: removed the useless "use lib" directives from the
         Perl test files. thanks Markus Linnala for the report.

     *   doc: various typo fixes from Lance Li.

     *   doc: ngx.exit was not disabled within the
         header_filter_by_lua* context.

     *   doc: a code example misses a "return". thanks YuanSheng Wang
         for the patch.

     *   doc: typo fix from Tatsuya Hoshino.

 *   upgraded LuaUpstreamNginxModule to 0.04.

     *   feature: "upstream.get_servers(server_name)" now returns the
         server name (if any) as well, which can be the domain name
         if the user puts it in "nginx.conf". thanks Hung Nguyen for
         the request.

 *   upgraded HeadersMoreNginxModule to 0.27.

     *   bugfix: setting (builtin) request headers "Upgrade",
         "Depth", "Destination", "Overwrite", and "Date" might not
         take effect in standard nginx modules like ngx_http_proxy
         and ngx_http_dav.

     *   bugfix: when the response header "Content-Type" contains
         parameters like "; charset=utf-8", the "-t MIME-List"
         options did not work as expected at all. thanks Joseph
         Bartels for the report.

     *   bugfix: clearing input headers "If-Unmodified-Since",
         "If-Match", and "If-None-Match" did not clear the builtin
         "shortcut" fields in "ngx_http_headers_in_t" which might
         confuse other nginx modules like
         "ngx_http_not_modified_filter_module". The first header gets
         "shortcuts" fields since nginx 0.9.2 while the latter two
         since nginx 1.3.3.

 *   upgraded IconvNginxModule to 0.12.

     *   bugfix: HTTP 0.9 requests would turn "iconv_filter" into a
         bad unrecoverable state leading to "iconv body filter
         skiped" error upon every subsequent request. thanks
         numberlife for the report. also introduced some coding style
         fixes.

     *   bugfix: lowered the error log level for HTTP 0.9 requests
         from "error" to "warn" to prevent malicious clients from
         flooding the error logs.

 *   upgraded LuaRestyRedisLibrary to 0.21.

     *   bugfix: the "attempt to call local new_tab (a table value)"
         error might happen when LuaJIT 2.0 was used and a local Lua
         module named "table.new" was visible. thanks Michael Pirogov
         for the report.

     *   doc: fixed code examples to check redis pipelined requests'
         return values more strictly. some commands (like hkeys and
         smembers) may return empty tables, which may result in "nil
         res[1]" values. thanks Dejiang Zhu for the patch.

 *   upgraded LuaJIT to v2.1-20151028:
     https://github.com/openresty/luajit2/tags

     *   imported Mike Pall's latest changes:

         *   limit number of arguments given to "io.lines()" and
             "fp:lines()".

         *   ARM64: fix "__call" metamethod handling for tail calls.

         *   FFI: Do not propagate qualifiers into subtypes of
             complex.

         *   feature: parse binary number literals ("0bxxx").

         *   fix NYICF error message.

         *   properly handle OOM in "trace_save()".

         *   ARM64: add support for saving bytecode as object files.

         *   ARM64: fix ELF bytecode saving.

         *   feature: parse Unicode string escape "\u{XX...}".

         *   FFI: add "ssize_t" declaration.

         *   fix unsinking check.

         *   feature: add "collectgarbage("isrunning")".

         *   flush symbol tables in "jit.dump" on trace flush.

Feedback welcome!

Hopefully we can cut the 1.9.3.2 formal release in the next few days
so that we can move onto the new nginx 1.9.6 core ASAP :)

Thanks!
-agentzh

igor.clark

Hello agentzh! This is awesome, thanks. The *_by_lua_block directives in particular are going to be super helpful for me, very good for prototyping and trying things out alongside nginx location / config work.

I DL'd and built 1.9.3.2-RC2 to try it out, and while the blocks are working perfectly, I have what I think is another DAV-related issue. When making requests to the DAV module via ngx.location.capture(), MOVE and DELETE requests seem to return proper HTTP codes on failure, but 0 on success.

It's not new to RC2, so I guess I should have reported it with the other DAV issue - but it didn't occur to me that it might be on the lua side rather than the DAV side, or on my side. (Which it still might be!) But it's still happening in RC2, and I can't work out what I'm doing wrong, if I am. So hopefully it's helpful to report.

Here's the config I made to test it out:

user vagrant;
worker_processes 1;

events {
worker_connections 1024;
}

http {
server {
listen 80;

  default_type text/plain;

set $file1 /file1.txt;
set $file2 /file2.txt;

location /lua-dav/move {
content_by_lua_block {
ngx.req.set_header( "Destination", ngx.var.file2 )
local mv_res = ngx.location.capture(
ngx.var.file1, { method = ngx.HTTP_MOVE }
)

  ngx.say(

"MOVE " .. ngx.var.file1 .. " -> " .. ngx.var.file2 ..

" response status: " .. mv_res.status

  )

ngx.say( ngx.location.capture( "/" ).body )
}
}

location /lua-dav/delete {
content_by_lua_block {
for _,v in ipairs( { ngx.var.file1, ngx.var.file2 } ) do
local del_res = ngx.location.capture(
v, { method = ngx.HTTP_DELETE }
)
ngx.say( "DELETE " .. v .. " response status: " .. del_res.status )
ngx.say( ngx.location.capture( "/" ).body )
end
}
}

location / {
root /home/vagrant/dav_root;
dav_methods MOVE PUT DELETE;
create_full_put_path on;
autoindex on;
autoindex_format json;
}
}
}

Here's how I tested it, and the output I got:

vagrant@debian-7-9:~/dav_root$ ls -l
total 0
vagrant@debian-7-9:~/dav_root$ curl http://127.0.0.1/lua-dav/move
MOVE response status: 404
[
]
vagrant@debian-7-9:~/dav_root$ touch file1.txt
vagrant@debian-7-9:~/dav_root$ ls -l
total 0
-rw-r--r-- 1 vagrant vagrant 0 Oct 29 22:27 file1.txt
vagrant@debian-7-9:~/dav_root$ curl http://127.0.0.1/lua-dav/move

MOVE /file1.txt -> /file2.txt response status: 0
[
{ "name":"file2.txt", "type":"file", "mtime":"Thu, 29 Oct 2015 22:27:50 GMT", "size":0 }
]
vagrant@debian-7-9:~/dav_root$ ls -l
total 0
-rw-r--r-- 1 vagrant vagrant 0 Oct 29 22:27 file2.txt
vagrant@debian-7-9:~/dav_root$ curl http://127.0.0.1/lua-dav/delete
DELETE /file1.txt response status: 404
[
{ "name":"file2.txt", "type":"file", "mtime":"Thu, 29 Oct 2015 22:27:50 GMT", "size":0 }
]
DELETE /file2.txt response status: 0
[
]
vagrant@debian-7-9:~/dav_root$ ls -l
total 0

vagrant@debian-7-9:~/dav_root$

Incidentally/separately, I also made this config to try to make lua PUT the file in the first place:

location /lua-dav/put {
content_by_lua_block {
local body_content = "blah blah blah"
local body_length = string.len( body_content )
ngx.req.set_header( "Content-type", "text/plain" )
ngx.req.set_header( "Content-length", body_length )
local put_res = ngx.location.capture(
ngx.var.file1, { method = ngx.HTTP_PUT, body = body_content }
)
ngx.say( "PUT response status: " .. put_res.status )
ngx.say( ngx.location.capture( "/" ).body )
}
}

But that reliably returned 500. Is that because the DAV module still can't read the request body as it's in a subrequest, as per https://github.com/openresty/lua-nginx-module/issues/375#issuecomment-46104177 ?

Anyway - hope it's not just me being dim!

Thanks again for openresty, it's amazing.

Cheers

Igor

On Wednesday, October 28, 2015 at 1:12:58 PM UTC, agentzh wrote:

Hi guys,

I've just kicked out OpenResty 1.9.3.2 RC1 for testing:

https://openresty.org/download/ngx_openresty-1.9.3.2rc1.tar.gz

The highlight of this release is the new *_by_lua_block {} directives
added in the ngx_http_lua module.

For example, instead of writing

content_by_lua '
ngx.say("Hello, OpenResty\'s world!\\n")
';

We can just write

content_by_lua_block {
ngx.say("Hello, OpenResty's world!\n")
}

No ugly special character escaping is needed in the Lua source any
more in the latter form :)

Changes since the last (formal) release, 1.9.3.1:

* upgraded LuaNginxModule to 0.9.17.

* feature: implemented "*_by_lua_block {}" directives for all
the existing *_by_lua directives so that we no longer have
to escape special characters while inlining Lua source
inside the "nginx.conf" file.

* feature: initial fixes when being used with the new
"ngx_http_v2" module since nginx 1.9.5. thanks itpp16 for
the patches.

* bugfix: the user specified "./configure"'s "--with-cc-opt"
and "--with-ld-opt" might override the
"LUAJIT_INC"/"LUAJIT_LIB" and "LUA_INC"/"LUA_LIB"
environment settings. thanks Julian Gonggrijp for the
report.

* bugfix: setting builtin request header "Upgrade" via
ngx.req.set_header and etc might not take effect with some
builtin nginx modules.

* bugfix: setting builtin request headers "Depth",
"Destination", "Overwrite", and "Date" via
ngx.req.set_header() and etc might not take effect at least
with ngx_http_dav_module. thanks Igor Clark for the report.

* bugfix: fixed typos due to copy&paste mistakes in some error
messages.

* bugfix: fixed one "-Wmaybe-uninitialized" warning when
compiling with "gcc -Os".

* bugfix: use of shared dicts resulted in (unwanted)
registrations of shared dict metatables on all the
lightuserdata in the Lua space. thanks helloyi for the
report and patch.

* bugfix: if a 3rd-party module calls
"ngx_http_conf_get_module_srv_conf" to fetch its current
"srv_conf" construct in its "merge_srv_conf" callback, then
use of init_worker_by_lua might lead to segmentation faults
(the same also applied to merge_loc_conf). thanks chiyouhen
for the report and patch.

* bugfix: the "if_unmodified_since" "shortcut" field in
"ngx_http_headers_in_t" was first added in nginx 0.9.2.

* bugfix: ngx.req.clear_header/ngx.req.set_header: we did not
update the shortcut fields in "ngx_http_headers_in_t" added
since nginx 1.3.3 which may confuse other nginx modules
accessing them.

* bugfix: setting "Content-Type" response values including ";
charset=xxx" via the ngx.header API might bypass the MIME
type checks in other nginx modules like ngx_gzip. thanks
Andreas Fischer for the report.

* bugfix: typo fixes in some debug logging messages. thanks
doujiang for the patch.

* optimize: fixed the hash-table initial sizes of the cosocket
metatables. thanks ops-dev-cn for the patch.

* tests: removed the useless "use lib" directives from the
Perl test files. thanks Markus Linnala for the report.

* doc: various typo fixes from Lance Li.

* doc: ngx.exit was not disabled within the
header_filter_by_lua* context.

* doc: a code example misses a "return". thanks YuanSheng Wang
for the patch.

* doc: typo fix from Tatsuya Hoshino.

* upgraded LuaUpstreamNginxModule to 0.04.

* feature: "upstream.get_servers(server_name)" now returns the
server name (if any) as well, which can be the domain name
if the user puts it in "nginx.conf". thanks Hung Nguyen for
the request.

* upgraded HeadersMoreNginxModule to 0.27.

* bugfix: setting (builtin) request headers "Upgrade",
"Depth", "Destination", "Overwrite", and "Date" might not
take effect in standard nginx modules like ngx_http_proxy
and ngx_http_dav.

* bugfix: when the response header "Content-Type" contains
parameters like "; charset=utf-8", the "-t MIME-List"
options did not work as expected at all. thanks Joseph
Bartels for the report.

* bugfix: clearing input headers "If-Unmodified-Since",
"If-Match", and "If-None-Match" did not clear the builtin
"shortcut" fields in "ngx_http_headers_in_t" which might
confuse other nginx modules like
"ngx_http_not_modified_filter_module". The first header gets
"shortcuts" fields since nginx 0.9.2 while the latter two
since nginx 1.3.3.

* upgraded IconvNginxModule to 0.12.

* bugfix: HTTP 0.9 requests would turn "iconv_filter" into a
bad unrecoverable state leading to "iconv body filter
skiped" error upon every subsequent request. thanks
numberlife for the report. also introduced some coding style
fixes.

* bugfix: lowered the error log level for HTTP 0.9 requests
from "error" to "warn" to prevent malicious clients from
flooding the error logs.

* upgraded LuaRestyRedisLibrary to 0.21.

* bugfix: the "attempt to call local new_tab (a table value)"
error might happen when LuaJIT 2.0 was used and a local Lua
module named "table.new" was visible. thanks Michael Pirogov
for the report.

* doc: fixed code examples to check redis pipelined requests'
return values more strictly. some commands (like hkeys and
smembers) may return empty tables, which may result in "nil
res[1]" values. thanks Dejiang Zhu for the patch.

* upgraded LuaJIT to v2.1-20151028:
https://github.com/openresty/luajit2/tags

* imported Mike Pall's latest changes:

* limit number of arguments given to "io.lines()" and
"fp:lines()".

* ARM64: fix "__call" metamethod handling for tail calls.

* FFI: Do not propagate qualifiers into subtypes of
complex.

* feature: parse binary number literals ("0bxxx").

* fix NYICF error message.

* properly handle OOM in "trace_save()".

* ARM64: add support for saving bytecode as object files.

* ARM64: fix ELF bytecode saving.

* feature: parse Unicode string escape "\u{XX...}".

* FFI: add "ssize_t" declaration.

* fix unsinking check.

* feature: add "collectgarbage("isrunning")".

* flush symbol tables in "jit.dump" on trace flush.

Feedback welcome!

Hopefully we can cut the 1.9.3.2 formal release in the next few days
so that we can move onto the new nginx 1.9.6 core ASAP :)

Thanks!
-agentzh

agentzh

Hello!

On Fri, Oct 30, 2015 at 6:45 AM, Igor Clark wrote:
> Hello agentzh! This is awesome, thanks. The *_by_lua_block directives in
> particular are going to be super helpful for me, very good for prototyping
> and trying things out alongside nginx location / config work.
>

Great! Glad you like it :)

> I DL'd and built 1.9.3.2-RC2 to try it out, and while the blocks are working
> perfectly,

You mean 1.9.3.2 RC1, right? RC2 is not out (yet) :)

Yeah, I do understand the frustration with so many little digits in
the version number :)

> I have what I think is another DAV-related issue. When making
> requests to the DAV module via ngx.location.capture(), MOVE and DELETE
> requests seem to return proper HTTP codes on failure, but 0 on success.
>

Thanks for the bug report and test case! Just fixed in the git master
branch of ngx_lua:

    https://github.com/openresty/lua-nginx-module/commit/4a9d4b11

It will get included in OpenResty 1.9.3.2 RC2 and ngx_lua 0.9.18 soon :)

Best regards,
-agentzh

igor.clark

Hey, thanks agentzh!

On Friday, October 30, 2015 at 7:20:00 AM UTC, agentzh wrote:

You mean 1.9.3.2 RC1, right? RC2 is not out (yet) :)

Doh. Yep. Exactly :-)

Thanks for the bug report and test case! Just fixed in the git master
branch of ngx_lua:

https://github.com/openresty/lua-nginx-module/commit/4a9d4b11

Excellent, tested and working here too, thanks very much :-)

It will get included in OpenResty 1.9.3.2 RC2 and ngx_lua 0.9.18 soon :)

Great. Really helpful to see the simple tests integrated like that too, helps me understand what Test::Nginx::Socket is up to. Which seems really cool. Does it make sense to use that testing framework for web apps/APIs written with OpenResty, or is it only really meant for testing the bundle/app server itself? Is there a preferred method for doing integration tests on apps/APIs if so?

Cheers!

Igor

lordnynex

I highly recommend busted for testing apps and apis.

http://olivinelabs.com/busted/

On Fri, Oct 30, 2015 at 3:18 AM, Igor Clark <igor...@gmail.com> wrote:

Hey, thanks agentzh!

On Friday, October 30, 2015 at 7:20:00 AM UTC, agentzh wrote:

You mean 1.9.3.2 RC1, right? RC2 is not out (yet) :)

Doh. Yep. Exactly :-)

Thanks for the bug report and test case! Just fixed in the git master
branch of ngx_lua:

https://github.com/openresty/lua-nginx-module/commit/4a9d4b11

Excellent, tested and working here too, thanks very much :-)

It will get included in OpenResty 1.9.3.2 RC2 and ngx_lua 0.9.18 soon :)

Great. Really helpful to see the simple tests integrated like that too, helps me understand what Test::Nginx::Socket is up to. Which seems really cool. Does it make sense to use that testing framework for web apps/APIs written with OpenResty, or is it only really meant for testing the bundle/app server itself? Is there a preferred method for doing integration tests on apps/APIs if so?

Cheers!
Igor

.

igor.clark

Thanks Lord Nynex, that looks tight for unit testing. I'll have a good look into it. I'm particularly interested in integration tests for API endpoints, which take into account data changes, so unit testing doesn't always seem to fit, but maybe the "async" section will help.

Any other recos for integration testing tools also v welcome :-)

Cheers!

On Friday, October 30, 2015 at 11:52:27 AM UTC, Lord Nynex wrote:

I highly recommend busted for testing apps and apis.

http://olivinelabs.com/busted/

On Fri, Oct 30, 2015 at 3:18 AM, Igor Clark <igor....@gmail.com> wrote:
Hey, thanks agentzh!

On Friday, October 30, 2015 at 7:20:00 AM UTC, agentzh wrote:

You mean 1.9.3.2 RC1, right? RC2 is not out (yet) :)

Doh. Yep. Exactly :-)

Thanks for the bug report and test case! Just fixed in the git master
branch of ngx_lua:

https://github.com/openresty/lua-nginx-module/commit/4a9d4b11

Excellent, tested and working here too, thanks very much :-)

It will get included in OpenResty 1.9.3.2 RC2 and ngx_lua 0.9.18 soon :)

Great. Really helpful to see the simple tests integrated like that too, helps me understand what Test::Nginx::Socket is up to. Which seems really cool. Does it make sense to use that testing framework for web apps/APIs written with OpenResty, or is it only really meant for testing the bundle/app server itself? Is there a preferred method for doing integration tests on apps/APIs if so?

Cheers!
Igor

.

lordnynex

I'm not sure what the actual limitation here is. If you want to use Test::Nginx::Socket to write integration tests, go for it it's your software, but what exactly do you think you can't do busted?

Here is a framework specific example of using busted for both unit testing and integration testing with their use_test_server and request http://leafo.net/lapis/reference/testing.html#using-the-test-server

Personally if I was doing integration testing (i am) to look for 'data changes' (i am) I would use busted (again, i am) because I think lua helpers are really cool and it's easy to cjson decode a json response and validate the table as opposed to messing with JSON::XS. Further, the distinction between 'unit testing' and 'integration test' kina blurs into just testing at that point.

On Fri, Oct 30, 2015 at 7:40 AM, Igor Clark <igor...@gmail.com> wrote:

Thanks Lord Nynex, that looks tight for unit testing. I'll have a good look into it. I'm particularly interested in integration tests for API endpoints, which take into account data changes, so unit testing doesn't always seem to fit, but maybe the "async" section will help.
Any other recos for integration testing tools also v welcome :-)
Cheers!

On Friday, October 30, 2015 at 11:52:27 AM UTC, Lord Nynex wrote:
I highly recommend busted for testing apps and apis.

http://olivinelabs.com/busted/

On Fri, Oct 30, 2015 at 3:18 AM, Igor Clark <igor....@gmail.com> wrote:
Hey, thanks agentzh!

On Friday, October 30, 2015 at 7:20:00 AM UTC, agentzh wrote:

You mean 1.9.3.2 RC1, right? RC2 is not out (yet) :)

Doh. Yep. Exactly :-)

Thanks for the bug report and test case! Just fixed in the git master
branch of ngx_lua:

https://github.com/openresty/lua-nginx-module/commit/4a9d4b11

Excellent, tested and working here too, thanks very much :-)

It will get included in OpenResty 1.9.3.2 RC2 and ngx_lua 0.9.18 soon :)

Great. Really helpful to see the simple tests integrated like that too, helps me understand what Test::Nginx::Socket is up to. Which seems really cool. Does it make sense to use that testing framework for web apps/APIs written with OpenResty, or is it only really meant for testing the bundle/app server itself? Is there a preferred method for doing integration tests on apps/APIs if so?

Cheers!
Igor

.

igor.clark

That's super helpful, thanks. Honestly I think the limitation is probably my (comparative) Lua & (absolute) Busted ignorance. What you describe sounds like what I want. I'll spend some time digging into it. Appreciate it!

On Oct 30, 2015, at 8:10 PM, Lord Nynex <lord...@gmail.com> wrote:

I'm not sure what the actual limitation here is. If you want to use Test::Nginx::Socket to write integration tests, go for it it's your software, but what exactly do you think you can't do busted?

Here is a framework specific example of using busted for both unit testing and integration testing with their use_test_server and request http://leafo.net/lapis/reference/testing.html#using-the-test-server

Personally if I was doing integration testing (i am) to look for 'data changes' (i am) I would use busted (again, i am) because I think lua helpers are really cool and it's easy to cjson decode a json response and validate the table as opposed to messing with JSON::XS. Further, the distinction between 'unit testing' and 'integration test' kina blurs into just testing at that point.

On Fri, Oct 30, 2015 at 7:40 AM, Igor Clark <igor...@gmail.com> wrote:
Thanks Lord Nynex, that looks tight for unit testing. I'll have a good look into it. I'm particularly interested in integration tests for API endpoints, which take into account data changes, so unit testing doesn't always seem to fit, but maybe the "async" section will help.

Any other recos for integration testing tools also v welcome :-)

Cheers!

On Friday, October 30, 2015 at 11:52:27 AM UTC, Lord Nynex wrote:
I highly recommend busted for testing apps and apis.

http://olivinelabs.com/busted/

On Fri, Oct 30, 2015 at 3:18 AM, Igor Clark <igor....@gmail.com> wrote:
Hey, thanks agentzh!

On Friday, October 30, 2015 at 7:20:00 AM UTC, agentzh wrote:

You mean 1.9.3.2 RC1, right? RC2 is not out (yet) :)

Doh. Yep. Exactly :-)

Thanks for the bug report and test case! Just fixed in the git master
branch of ngx_lua:

https://github.com/openresty/lua-nginx-module/commit/4a9d4b11

Excellent, tested and working here too, thanks very much :-)

It will get included in OpenResty 1.9.3.2 RC2 and ngx_lua 0.9.18 soon :)

Great. Really helpful to see the simple tests integrated like that too, helps me understand what Test::Nginx::Socket is up to. Which seems really cool. Does it make sense to use that testing framework for web apps/APIs written with OpenResty, or is it only really meant for testing the bundle/app server itself? Is there a preferred method for doing integration tests on apps/APIs if so?

Cheers!
Igor

.
agentzh
Hi Igor

On Fri, Oct 30, 2015 at 6:18 PM, Igor Clark wrote:
>
> Excellent, tested and working here too, thanks very much :-)
>

Thanks for your feedback.

> Great. Really helpful to see the simple tests integrated like that too,
> helps me understand what Test::Nginx::Socket is up to. Which seems really
> cool.

Glad you like it.

> Does it make sense to use that testing framework for web apps/APIs
> written with OpenResty, or is it only really meant for testing the
> bundle/app server itself?

It's good for both. For example, the Lua CDN and Lua SSL's business
systems at CloudFlare use Test::Nginx::Socket at the bottom to drive
their test suite, in a similar declarative fashion, enjoying all the
advanced testing features of Test::Nginx::Socket (memory leak testing,
hup reload mode, mockeagain testing, valgrind checks, systemtap
integration, you name it) out of the box.

The idea of organizing test cases as simple data and spec rather than
hard-coded code is very powerful. We also have tools to make the tests
update themselves easily and automatically at our will.

For higher level testing, you may want to extend the DSL offered by
Test::Nginx::Socket by subclassing it. You can invent and de-sugar
your own "sections" down to existing "sections" or implement your new
fancy sections (like those manipulating external resources, like
memcached, dns servers, and databases) yourself. For example, the test
scaffold driving ngx_stream_echo module's test suite is using this
approach:

    https://github.com/openresty/stream-echo-nginx-module/tree/master/t/

Simple and powerful.

> Is there a preferred method for doing integration
> tests on apps/APIs if so?
>

Personally I'd recommend my own Test::Nginx approach :) But I also
encourage you investigate other solutions to pick up the one you like
most.

Oh, I surely need to write more detailed tutorials on
Test::Nginx::Socket to demonstrate its power quickly accumulated over
the years (since 2009).

Regards,
-agentzh
agentzh
Hello!

On Fri, Oct 30, 2015 at 6:45 AM, Igor Clark wrote:
> Incidentally/separately, I also made this config to try to make lua PUT the
> file in the first place:
>
[...]
>
>
> But that reliably returned 500. Is that because the DAV module still can't
> read the request body as it's in a subrequest, as per
> https://github.com/openresty/lua-nginx-module/issues/375#issuecomment-46104177
> ?
>

The ngx_dav module assumes that it reads the client request body
itself and it requires the body to be always put on the file system
during the read. So if we fake this in ngx.location.capture, then we
probably shouldn't use ngx_dav's PUT method in the first place (since
there's no gain here, really, as compared to writing to the file
directly ourselves, in Lua).

Regards,
-agentzh
igor.clark

Thanks agentzh, very helpful to know it's not only used for the underlying module development. The declarative approach definitely appeals.

Cheers!

On Monday, November 2, 2015 at 2:30:11 AM UTC, agentzh wrote:
Hi Igor

On Fri, Oct 30, 2015 at 6:18 PM, Igor Clark wrote:
>
> Excellent, tested and working here too, thanks very much :-)
>

Thanks for your feedback.

> Great. Really helpful to see the simple tests integrated like that too,
> helps me understand what Test::Nginx::Socket is up to. Which seems really
> cool.

Glad you like it.

> Does it make sense to use that testing framework for web apps/APIs
> written with OpenResty, or is it only really meant for testing the
> bundle/app server itself?

It's good for both. For example, the Lua CDN and Lua SSL's business
systems at CloudFlare use Test::Nginx::Socket at the bottom to drive
their test suite, in a similar declarative fashion, enjoying all the
advanced testing features of Test::Nginx::Socket (memory leak testing,
hup reload mode, mockeagain testing, valgrind checks, systemtap
integration, you name it) out of the box.

The idea of organizing test cases as simple data and spec rather than
hard-coded code is very powerful. We also have tools to make the tests
update themselves easily and automatically at our will.

For higher level testing, you may want to extend the DSL offered by
Test::Nginx::Socket by subclassing it. You can invent and de-sugar
your own "sections" down to existing "sections" or implement your new
fancy sections (like those manipulating external resources, like
memcached, dns servers, and databases) yourself. For example, the test
scaffold driving ngx_stream_echo module's test suite is using this
approach:

https://github.com/openresty/stream-echo-nginx-module/tree/master/t/

Simple and powerful.

> Is there a preferred method for doing integration
> tests on apps/APIs if so?
>

Personally I'd recommend my own Test::Nginx approach :) But I also
encourage you investigate other solutions to pick up the one you like
most.

Oh, I surely need to write more detailed tutorials on
Test::Nginx::Socket to demonstrate its power quickly accumulated over
the years (since 2009).

Regards,
-agentzh

igor.clark

Hi!

On Monday, November 2, 2015 at 2:48:05 AM UTC, agentzh wrote:

The ngx_dav module assumes that it reads the client request body
itself and it requires the body to be always put on the file system
during the read. So if we fake this in ngx.location.capture, then we
probably shouldn't use ngx_dav's PUT method in the first place (since
there's no gain here, really, as compared to writing to the file
directly ourselves, in Lua).

Ah, OK, got it. I've organised it so clients PUT directly to ngx_dav anyway, and with this info it obviously makes sense just to leave it that way.

Thanks :-)