OpenResty roadmap for the new year (2016)

agentzh

Hi folks,

As some of you have already noticed, we have been working hard on the
Christmas formal release of OpenResty lately. Still it's also a good
time to plot big plans for the new year, 2016. Some big things in my
head for 2016:

1. ngx.semaphore API for efficient cross-request light thread synchronization,

2. Redis-style features in lua_shared_dict that support lists (or
queues) and nested hash tables.

3. ngx_stream_lua_module for using Lua to code up generic TCP servers,

4. ngx_datagra_lua_module for using Lua to code up generic UDP
servers,

5. a domain specific language named Edge for efficient and
flexible routing and rewriting in NGINX,

6. native WAF support compatible with modsecurity's rule language,

7. the new DFA-based Perl-compatible regular expression engine named
sregex that supports streaming matching and substitutions which can
beat both PCRE JIT and RE2 for speed in most cases.

8. Integrating the balancer_by_lua feature that allows Lua to write
highly flexible balancers that can use with existing nginx upstream
modules like ngx_proxy and ngx_fastcgi.

9. Integrating the ssl_certificate_by_lua feature that allows fine
control of the downstream SSL handshake for https traffic (like lazily
loading SSL certificates and private keys nonblockingly from the
remote services).

10. Integrating the ssl_session_fetch_by_lua and
ssl_session_store_by_lua that can do distributed SSL session ID
caching to reduce expensive SSL handshakes with older web clients.

11. Setting up the iresty.org website that everyone can upload and
distribute their lua-resty-* libraries and install others' libraries
easily with the iresty command-line tool.

12. The ORSQL domain-specific language that can serve as a lightweight
high-level framework for doing complete web sites or web services.

13. OpenResty port of the Perl Template Toolkit's templating language
and pluggins (there is already a client-side JavaScript port called
Jemplate).

14. OpenResty port of the Perl Pegex top-down parser generator framework.

15. Official binary packages and repositories of OpenResty for various
mainstream Linux distributions.

16. Connection-pool-based backend connection queueing and concurrency
control for cosockets.

17. Add ngx.connection API for trivially incorporating 3rd-party C
libraries that exposes a nonblocking C API (like PostgreSQL's libpq or
MySQL's libdrizzle) by registering foreign file descriptors directly
into OpenResty's I/O scheduler with just a little bit of Lua code.

The good news is that many items on the list already have pull
requests being reviewed, some of them are even running in CloudFlare's
production environment for many months. So I'd expect many of them
will happen in the first half or even the first quarter of the new
year.

As always, we need every bit of support and contributions from all of
you to make OpenResty a success :)

Merry Christmas and happy new year!

Best regards,
-agentzh

erik.l.nelson

Yichun Zhang (agentzh) Sent: Thursday, December 24, 2015 1:51 PM
> 
> As some of you have already noticed, we have been working hard on the
> Christmas formal release of OpenResty lately. Still it's also a good
> time to plot big plans for the new year, 2016. Some big things in my
> head for 2016:
> 
> 1. ngx.semaphore API for efficient cross-request light thread
> synchronization,

Lots of great features in the plan, best of luck to you and the whole team!

Erik

----------------------------------------------------------------------
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer.   If you are not the intended recipient, please delete this message.

agentzh

Hello!

On Thu, Dec 24, 2015 at 11:19 AM, Nelson, Erik - 2 wrote:
>
> Lots of great features in the plan, best of luck to you and the whole team!
>

Thanks!

Best regards,
-agentzh

igor.clark

Hey agentzh, this all sounds pretty amazing, thanks for the heads-up.

Apart from the stream-based generic TCP/UDP stuff, one of the thingsthat catches my eye most immediately is this:

5. a domain specific language named Edge for efficient and
flexible routing and rewriting in NGINX,

Is there any more information about this available yet? I'd love to knowmore about how this will work.


Thanks again, hope you have a decent break this holiday!
Igor

On 24/12/2015 18:50, Yichun Zhang (agentzh) wrote:

Hi folks,

As some of you have already noticed, we have been working hard on the
Christmas formal release of OpenResty lately. Still it's also a good
time to plot big plans for the new year, 2016. Some big things in my
head for 2016:

1. ngx.semaphore API for efficient cross-request light thread synchronization,

2. Redis-style features in lua_shared_dict that support lists (or
queues) and nested hash tables.

3. ngx_stream_lua_module for using Lua to code up generic TCP servers,

4. ngx_datagra_lua_module for using Lua to code up generic UDP
servers,

5. a domain specific language named Edge for efficient and
flexible routing and rewriting in NGINX,

6. native WAF support compatible with modsecurity's rule language,

7. the new DFA-based Perl-compatible regular expression engine named
sregex that supports streaming matching and substitutions which can
beat both PCRE JIT and RE2 for speed in most cases.

8. Integrating the balancer_by_lua feature that allows Lua to write
highly flexible balancers that can use with existing nginx upstream
modules like ngx_proxy and ngx_fastcgi.

9. Integrating the ssl_certificate_by_lua feature that allows fine
control of the downstream SSL handshake for https traffic (like lazily
loading SSL certificates and private keys nonblockingly from the
remote services).

10. Integrating the ssl_session_fetch_by_lua and
ssl_session_store_by_lua that can do distributed SSL session ID
caching to reduce expensive SSL handshakes with older web clients.

11. Setting up the iresty.org website that everyone can upload and
distribute their lua-resty-* libraries and install others' libraries
easily with the iresty command-line tool.

12. The ORSQL domain-specific language that can serve as a lightweight
high-level framework for doing complete web sites or web services.

13. OpenResty port of the Perl Template Toolkit's templating language
and pluggins (there is already a client-side JavaScript port called
Jemplate).

14. OpenResty port of the Perl Pegex top-down parser generator framework.

15. Official binary packages and repositories of OpenResty for various
mainstream Linux distributions.

16. Connection-pool-based backend connection queueing and concurrency
control for cosockets.

17. Add ngx.connection API for trivially incorporating 3rd-party C
libraries that exposes a nonblocking C API (like PostgreSQL's libpq or
MySQL's libdrizzle) by registering foreign file descriptors directly
into OpenResty's I/O scheduler with just a little bit of Lua code.

The good news is that many items on the list already have pull
requests being reviewed, some of them are even running in CloudFlare's
production environment for many months. So I'd expect many of them
will happen in the first half or even the first quarter of the new
year.

As always, we need every bit of support and contributions from all of
you to make OpenResty a success :)

Merry Christmas and happy new year!

Best regards,
-agentzh

agentzh

Hi Igor

On Thu, Dec 24, 2015 at 3:01 PM, Igor Clark wrote:
> Hey agentzh, this all sounds pretty amazing, thanks for the heads-up.
>

Glad you like it.

> Apart from the stream-based generic TCP/UDP stuff,

Well, UDP will be based on the datagram subsystem instead of "stream".

>> 5. a domain specific language named Edge for efficient and
>> flexible routing and rewriting in NGINX,
>
> Is there any more information about this available yet? I'd love to know
> more about how this will work.
>

It will be a rule based language and compiler, syntactically similar
to Prolog, but semantically similar to CLIPS [1]. It will also
incorporate a  first-class macro language to allow auto-generation of
similar rules.

The Edge language will be designed in such a way that extending and
customizing the language primitives will be easy.

The compiler will (eventually) have advanced inter-rule optimizations
like pattern merging and out-of-order optimizations (not CPU-level
out-of-order, but higher level), generating simple and compact Lua
code for OpenResty. Maybe we'll get the sregex engine or Intel's
HyperScan incorporated (Alas, the latter may be too heavy weight for
OpenResty).

It will also have support for hot code replacement (thanks to Lua) and
dynamic code loading from the remote.

The first implementation of the compiler itself will be written in
Perl 5 but it will eventually get ported over to OpenResty so that the
whole toolchain can run online (though not always necessary or
needed).

Furthermore, a WAF system can also be based upon the same compiler and
runtime used by Edge, though providing a frontend compatible with
modsecurity (and possibly others as well).

BTW, I have the domain name edgelang.org registered already though
nothing exists yet behind that domain :)

As you can tell, I'm devoted to DSLs. Just a kinda related note: I
also have plans to invent the Y language whose compilers will target
gdb python, lldb python, systemtap, and possibly other debugging
frameworks. It's hoped that with Y, advanced debugging tools (like
those in my nginx-gdb-utils, nginx-systemtap-toolkit, and stap++) will
only be required to write once, and then just compile and "run
everywhere". And yeah, I also have the domain name ylang.org
registered *grin*

> Thanks again, hope you have a decent break this holiday!

Thank you. I will :)

Best regards,
-agentzh

[1] http://clipsrules.sourceforge.net/

sparvu_71a1

A lot of new things coming along. Sweet. Many thanks for making all
these happening.

> As always, we need every bit of support and contributions from all of
> you to make OpenResty a success :)
> 
> Merry Christmas and happy new year!

lofky7

谢谢春哥~~~

在 2015年12月25日星期五 UTC+8上午2:50:58，agentzh写道：

Hi folks,

As some of you have already noticed, we have been working hard on the
Christmas formal release of OpenResty lately. Still it's also a good
time to plot big plans for the new year, 2016. Some big things in my
head for 2016:

1. ngx.semaphore API for efficient cross-request light thread synchronization,

2. Redis-style features in lua_shared_dict that support lists (or
queues) and nested hash tables.

3. ngx_stream_lua_module for using Lua to code up generic TCP servers,

4. ngx_datagra_lua_module for using Lua to code up generic UDP
servers,

5. a domain specific language named Edge for efficient and
flexible routing and rewriting in NGINX,

6. native WAF support compatible with modsecurity's rule language,

7. the new DFA-based Perl-compatible regular _expression_ engine named
sregex that supports streaming matching and substitutions which can
beat both PCRE JIT and RE2 for speed in most cases.

8. Integrating the balancer_by_lua feature that allows Lua to write
highly flexible balancers that can use with existing nginx upstream
modules like ngx_proxy and ngx_fastcgi.

9. Integrating the ssl_certificate_by_lua feature that allows fine
control of the downstream SSL handshake for https traffic (like lazily
loading SSL certificates and private keys nonblockingly from the
remote services).

10. Integrating the ssl_session_fetch_by_lua and
ssl_session_store_by_lua that can do distributed SSL session ID
caching to reduce expensive SSL handshakes with older web clients.

11. Setting up the iresty.org website that everyone can upload and
distribute their lua-resty-* libraries and install others' libraries
easily with the iresty command-line tool.

12. The ORSQL domain-specific language that can serve as a lightweight
high-level framework for doing complete web sites or web services.

13. OpenResty port of the Perl Template Toolkit's templating language
and pluggins (there is already a client-side _javascript_ port called
Jemplate).

14. OpenResty port of the Perl Pegex top-down parser generator framework.

15. Official binary packages and repositories of OpenResty for various
mainstream Linux distributions.

16. Connection-pool-based backend connection queueing and concurrency
control for cosockets.

17. Add ngx.connection API for trivially incorporating 3rd-party C
libraries that exposes a nonblocking C API (like PostgreSQL's libpq or
MySQL's libdrizzle) by registering foreign file descriptors directly
into OpenResty's I/O scheduler with just a little bit of Lua code.

The good news is that many items on the list already have pull
requests being reviewed, some of them are even running in CloudFlare's
production environment for many months. So I'd expect many of them
will happen in the first half or even the first quarter of the new
year.

As always, we need every bit of support and contributions from all of
you to make OpenResty a success :)

Merry Christmas and happy new year!

Best regards,
-agentzh

zhoujing00k

辛苦春哥，非常感谢

在 2015年12月25日星期五 UTC+8上午2:50:58，agentzh写道：

Hi folks,

As some of you have already noticed, we have been working hard on the
Christmas formal release of OpenResty lately. Still it's also a good
time to plot big plans for the new year, 2016. Some big things in my
head for 2016:

1. ngx.semaphore API for efficient cross-request light thread synchronization,

2. Redis-style features in lua_shared_dict that support lists (or
queues) and nested hash tables.

3. ngx_stream_lua_module for using Lua to code up generic TCP servers,

4. ngx_datagra_lua_module for using Lua to code up generic UDP
servers,

5. a domain specific language named Edge for efficient and
flexible routing and rewriting in NGINX,

6. native WAF support compatible with modsecurity's rule language,

7. the new DFA-based Perl-compatible regular _expression_ engine named
sregex that supports streaming matching and substitutions which can
beat both PCRE JIT and RE2 for speed in most cases.

8. Integrating the balancer_by_lua feature that allows Lua to write
highly flexible balancers that can use with existing nginx upstream
modules like ngx_proxy and ngx_fastcgi.

9. Integrating the ssl_certificate_by_lua feature that allows fine
control of the downstream SSL handshake for https traffic (like lazily
loading SSL certificates and private keys nonblockingly from the
remote services).

10. Integrating the ssl_session_fetch_by_lua and
ssl_session_store_by_lua that can do distributed SSL session ID
caching to reduce expensive SSL handshakes with older web clients.

11. Setting up the iresty.org website that everyone can upload and
distribute their lua-resty-* libraries and install others' libraries
easily with the iresty command-line tool.

12. The ORSQL domain-specific language that can serve as a lightweight
high-level framework for doing complete web sites or web services.

13. OpenResty port of the Perl Template Toolkit's templating language
and pluggins (there is already a client-side _javascript_ port called
Jemplate).

14. OpenResty port of the Perl Pegex top-down parser generator framework.

15. Official binary packages and repositories of OpenResty for various
mainstream Linux distributions.

16. Connection-pool-based backend connection queueing and concurrency
control for cosockets.

17. Add ngx.connection API for trivially incorporating 3rd-party C
libraries that exposes a nonblocking C API (like PostgreSQL's libpq or
MySQL's libdrizzle) by registering foreign file descriptors directly
into OpenResty's I/O scheduler with just a little bit of Lua code.

The good news is that many items on the list already have pull
requests being reviewed, some of them are even running in CloudFlare's
production environment for many months. So I'd expect many of them
will happen in the first half or even the first quarter of the new
year.

As always, we need every bit of support and contributions from all of
you to make OpenResty a success :)

Merry Christmas and happy new year!

Best regards,
-agentzh