Things coming next (in 2015)

agentzh · 2014-12-13T07:17:51+00:00

talking about normal noise peaks: https://imgur.com/LrqmH2C this is the daily attack-level on one of our wafs, and the normal noise ranges from 500 events/...

Things coming next (in 2015)

agentzh

Hi folks

I recently updated the TODO list of ngx_lua:

    https://github.com/openresty/lua-nginx-module#todo

The newly added items will come soon (in the next year). In
particular, ssl_certificate_by_lua* are already powering our Lua SSL
gateway at CloudFlare and will be opensourced a few months later.

I personally want the TCP server and UDP server support of ngx_lua
very badly, though my own (immediate) usage is for unit testing.

In addition, the *_by_lua_block directives will be a great feature for
inlined Lua code in nginx.conf, especially excellent for unit testing
and documentation examples.

Also, the pool-based cosocket concurrency control by queueing
connect() calls on the Lua land is also a frequently-requested item in
the OpenResty community. I have a (partial) prototype in one of my
local branches.

The new command-line utility "resty" will continue its rapid
development in a separate repository, resty-cli, on GitHub:

    https://github.com/openresty/resty-cli

My colleague Guanlan Dai is adding new features to it like
command-line argument support via the special "arg" Lua table (just as
in the "lua" and "luajit" utilities). We'll have more command-line
tools written in OpenResty Lua at CloudFlare very soon :)

One focus in the next year (2015), however, would be better (online)
debugging support on the Lua land. I'm currently working on gdb
commands supporting setting breakpoints on interpreted Lua function
entries and returns. And eventually we need to have support for
probing Lua functions in dynamic tracing tools like systemtap and
ktap, both for interpreted Lua code and JITted one.

Another focus would be on a centralized package managing site (like
luaresty.org) and the surrounding toolchain like the "restypm"
command-line utility. Hopefully we can have a new openresty.org site
based on the web page templates contributed by Mashape.

If you have interest and time in any of the items listed above, you're
more than welcome to join the development work.

Thanks!
-agentzh

agentzh

Hello!

On Fri, Dec 12, 2014 at 3:17 PM, Yichun Zhang (agentzh) wrote:
> Hi folks
>
> I recently updated the TODO list of ngx_lua:
>

Sorry, I missed one nontrivial thing that is coming soon. That is,
integrating luajit-mm [1] into OpenResty's bundled LuaJIT by default.
The luajit-mm thing pushes the GC-managed memory limit in LuaJIT from
1GB to 2GB on (Linux) x86_64, which means if we have N nginx workers,
then we can use (for total) up to 2 x N GB GC-managed memory on the
Lua land. This can help many memory-intensive Lua apps survive traffic
peaks. The luajit-mm project is contributed by my colleague, Shuxin
Yang. Once we have resolved the pointer sign extension limitation in
LuaJIT's JIT compiler, we can further raise the limit to 4GB
(hopefully this can happen in the year 2015).

I am also going to bundle his new JSON library, lua-resty-json, in
OpenResty releases once it is mature enough. Its decoder is much
faster than lua-cjson for many cases and will not abort JIT
compilation.

Regards,
-agentzh

[1] https://github.com/cloudflare/luajit-mm

lordnynex

agentzh,

I've been looking forward to UDP server support for quite awhile. I have some needs and ideas around DNS. I'm not sure I can help on the core work but I'll gladly start deving against it from an implementation and tests perspective.

Additionally, (just my humble opinion) it would be an amazing addition to somehow support ZMQ and Mongo DB. There is a very vibrant mongodb community but what I've found so far doesn't seem complete. I know Nginx has plans in the next year to try and include V8/Inline JS. I think adding ZMQ and MongoDB support keeps openresty competitive with NodeJS.

Also, if I were to make a wish list, I would love lua bindings to Nginx RBTree.

On Fri, Dec 12, 2014 at 3:17 PM, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

Hi folks

I recently updated the TODO list of ngx_lua:

https://github.com/openresty/lua-nginx-module#todo

The newly added items will come soon (in the next year). In
particular, ssl_certificate_by_lua* are already powering our Lua SSL
gateway at CloudFlare and will be opensourced a few months later.

I personally want the TCP server and UDP server support of ngx_lua
very badly, though my own (immediate) usage is for unit testing.

In addition, the *_by_lua_block directives will be a great feature for
inlined Lua code in nginx.conf, especially excellent for unit testing
and documentation examples.

Also, the pool-based cosocket concurrency control by queueing
connect() calls on the Lua land is also a frequently-requested item in
the OpenResty community. I have a (partial) prototype in one of my
local branches.

The new command-line utility "resty" will continue its rapid
development in a separate repository, resty-cli, on GitHub:

https://github.com/openresty/resty-cli

My colleague Guanlan Dai is adding new features to it like
command-line argument support via the special "arg" Lua table (just as
in the "lua" and "luajit" utilities). We'll have more command-line
tools written in OpenResty Lua at CloudFlare very soon :)

One focus in the next year (2015), however, would be better (online)
debugging support on the Lua land. I'm currently working on gdb
commands supporting setting breakpoints on interpreted Lua function
entries and returns. And eventually we need to have support for
probing Lua functions in dynamic tracing tools like systemtap and
ktap, both for interpreted Lua code and JITted one.

Another focus would be on a centralized package managing site (like
luaresty.org) and the surrounding toolchain like the "restypm"
command-line utility. Hopefully we can have a new openresty.org site
based on the web page templates contributed by Mashape.

If you have interest and time in any of the items listed above, you're
more than welcome to join the development work.

Thanks!
-agentzh.

anton.heryanto

Hi,

wow, looking forward next year... lots of nice features

in addition really like to see further improvement in cosocket performance by exposed ffi based cosocket similar with resty.core
but immediate features will be package management, better if there Spec or RFC how its will be implemented so.. at least we can prepare or prototyped its
and hopefully, there will be timeline for each expected features.

On Saturday, December 13, 2014 7:17:52 AM UTC+8, agentzh wrote:

Hi folks

I recently updated the TODO list of ngx_lua:

https://github.com/openresty/lua-nginx-module#todo

The newly added items will come soon (in the next year). In
particular, ssl_certificate_by_lua* are already powering our Lua SSL
gateway at CloudFlare and will be opensourced a few months later.

I personally want the TCP server and UDP server support of ngx_lua
very badly, though my own (immediate) usage is for unit testing.

In addition, the *_by_lua_block directives will be a great feature for
inlined Lua code in nginx.conf, especially excellent for unit testing
and documentation examples.

Also, the pool-based cosocket concurrency control by queueing
connect() calls on the Lua land is also a frequently-requested item in
the OpenResty community. I have a (partial) prototype in one of my
local branches.

The new command-line utility "resty" will continue its rapid
development in a separate repository, resty-cli, on GitHub:

https://github.com/openresty/resty-cli

My colleague Guanlan Dai is adding new features to it like
command-line argument support via the special "arg" Lua table (just as
in the "lua" and "luajit" utilities). We'll have more command-line
tools written in OpenResty Lua at CloudFlare very soon :)

One focus in the next year (2015), however, would be better (online)
debugging support on the Lua land. I'm currently working on gdb
commands supporting setting breakpoints on interpreted Lua function
entries and returns. And eventually we need to have support for
probing Lua functions in dynamic tracing tools like systemtap and
ktap, both for interpreted Lua code and JITted one.

Another focus would be on a centralized package managing site (like
luaresty.org) and the surrounding toolchain like the "restypm"
command-line utility. Hopefully we can have a new openresty.org site
based on the web page templates contributed by Mashape.

If you have interest and time in any of the items listed above, you're
more than welcome to join the development work.

Thanks!
-agentzh

agentzh

Hello!

On Fri, Dec 12, 2014 at 4:08 PM, Lord Nynex wrote:
> I've been looking forward to UDP server support for quite awhile. I have
> some needs and ideas around DNS. I'm not sure I can help on the core work
> but I'll gladly start deving against it from an implementation and tests
> perspective.
>

Cool :)

> Additionally, (just my humble opinion) it would be an amazing addition to
> somehow support ZMQ and Mongo DB. There is a very vibrant mongodb community
> but what I've found so far doesn't seem complete. I know Nginx has plans in
> the next year to try and include V8/Inline JS. I think adding ZMQ and
> MongoDB support keeps openresty competitive with NodeJS.
>

I'd like to rely on the community for doing these. AFAIK, there's
already more than one Lua binding for MongoDB for OpenResty.

If the community decides one best lua-resty-* library for inclusion in
the OpenResty bundle, then I'll do it. The inclusion will become much
less important once we have the package management site and toolchain
ready anyway.

If any of the lua-resty-* libraries require any fundamental
functionalities that have to be in the ngx_lua core, feel free to let
me know :)

> Also, if I were to make a wish list, I would love lua bindings to Nginx
> RBTree.
>

The nginx rbtree is not really suitable for exposing to a dynamic
language as-is, we may need a separate C layer atop it to form a
proper ABI. This can be done in a separate nginx C module that extends
ngx_lua's Lua API, just as the ngx_lua_upstream module:

   https://github.com/openresty/lua-upstream-nginx-module

Or in other words, this does not have to be in the ngx_lua core :)
You're welcome to contribute your own ngx_lua_rbtree module to the
community ;)

Thank you for your feedback!

Regards,
-agentzh

agentzh

Hello!

On Fri, Dec 12, 2014 at 11:59 PM, Anton Heryanto wrote:
> wow, looking forward next year... lots of nice features
>

Thanks :)

> in addition really like to see further improvement in cosocket performance
> by exposed ffi based cosocket similar with resty.core

Oh yeah, sure. The lua-resty-core library will keep involving, driving
more user Lua code to get JIT compiled. I've added a TODO list to this
library's document:

    https://github.com/openresty/lua-resty-core#todo

Thanks for the reminder.

> but immediate features will be package management, better if there Spec or
> RFC how its will be implemented so.. at least we can prepare or prototyped
> its

Yeah, I'll try to draft an RFC or something like that for the package
management thing. Thanks for the suggestion :)

> and hopefully, there will be timeline for each expected features.
>

Well, all I can say is that I hope they can happen sometime in the
coming year. I give more accurate time for item that I've already done
(like the ssl_ceritficate_by_lua* thing). I have to prioritize things
according to my employer's requirements and my own mood :) One beauty
of working on opensource projects is that I usually don't have to set
a date on things ;) Thanks for your understanding! But contributions
from the community will definitely speed up things a lot. Reviewing
things take much less efforts than coding things from scratch ;)

Thanks for your feedback!

Best regards,
-agentzh

agentzh

Hello!

On Sat, Dec 13, 2014 at 1:36 PM, Yichun Zhang (agentzh) wrote:
> Oh yeah, sure. The lua-resty-core library will keep involving,

Sorry, typo here. I meant "evolving" :P

Best regards,
-agentzh

dakanji

Exciting stuff.

Hopefully this will also finally get added in 2015: https://github.com/openresty/lua-nginx-module/issues/77

;-)

zetao.wei

Hi agentzh:

I read your todo list, this is what i care about:

 tcp {
     server {
         listen 11212;
         handler_by_lua '
             -- custom Lua code implementing the special TCP server...
         ';
     }
 }

openresty will support tcp/udp proxy

add this nginx_tcp_proxy_module into project ？

在 2014年12月13日星期六 UTC+8上午7:17:52，agentzh写道：

Hi folks

I recently updated the TODO list of ngx_lua:

https://github.com/openresty/lua-nginx-module#todo

The newly added items will come soon (in the next year). In
particular, ssl_certificate_by_lua* are already powering our Lua SSL
gateway at CloudFlare and will be opensourced a few months later.

I personally want the TCP server and UDP server support of ngx_lua
very badly, though my own (immediate) usage is for unit testing.

In addition, the *_by_lua_block directives will be a great feature for
inlined Lua code in nginx.conf, especially excellent for unit testing
and documentation examples.

Also, the pool-based cosocket concurrency control by queueing
connect() calls on the Lua land is also a frequently-requested item in
the OpenResty community. I have a (partial) prototype in one of my
local branches.

The new command-line utility "resty" will continue its rapid
development in a separate repository, resty-cli, on GitHub:

https://github.com/openresty/resty-cli

My colleague Guanlan Dai is adding new features to it like
command-line argument support via the special "arg" Lua table (just as
in the "lua" and "luajit" utilities). We'll have more command-line
tools written in OpenResty Lua at CloudFlare very soon :)

One focus in the next year (2015), however, would be better (online)
debugging support on the Lua land. I'm currently working on gdb
commands supporting setting breakpoints on interpreted Lua function
entries and returns. And eventually we need to have support for
probing Lua functions in dynamic tracing tools like systemtap and
ktap, both for interpreted Lua code and JITted one.

Another focus would be on a centralized package managing site (like
luaresty.org) and the surrounding toolchain like the "restypm"
command-line utility. Hopefully we can have a new openresty.org site
based on the web page templates contributed by Mashape.

If you have interest and time in any of the items listed above, you're
more than welcome to join the development work.

Thanks!
-agentzh

agentzh

Hello!

On Mon, Jan 26, 2015 at 5:35 PM,  zetao.wei wrote:
>
> openresty will support tcp/udp proxy
>

Not only tcp/udp proxies but also more general tcp/udp endpoints (like
a nameserver speaking the DNS protocol or a TCP server emulating redis
or mysql).

>  add this nginx_tcp_proxy_module into project ？
>

This module is worth looking, but cannot be added as-is :)

The following module is actually functionally closer to what I'm
trying to build:

   https://github.com/bigplum/nginx-tcp-lua-module

Regards,
-agentzh