Openresty + cqueues

quae · 2014-12-19T04:07:25+00:00

Hello! On Fri, Jan 9, 2015 at 11:35 AM, joe wrote: > Thank you for your responses. I'm interested in the possible locking > mechanism for a high read...

Openresty + cqueues

quae

Lately I've been a big fan of the cqueues
(http://25thandclement.com/~william/projects/cqueues.html) library.
It's the best shot I've seen at composable asynchronous lua.
I'm wondering how you could integrate it better with lua-nginx-module.

As a simple first phase, could an API be exposed to pause (i.e. yield)
a request, to be resumed when
either a file descriptor you'd like watched polls readable, or a time
out occurs.
e.g.

    local fd = 23 -- somelib:getfd()
    local timeout = 1.5 -- seconds
    local timed_out = ngx.wait_for_fd(fd, timeout)

Daurn.

agentzh

Hello!

On Thu, Dec 18, 2014 at 12:07 PM, Daurnimator wrote:
> Lately I've been a big fan of the cqueues
> (http://25thandclement.com/~william/projects/cqueues.html) library.
> It's the best shot I've seen at composable asynchronous lua.
> I'm wondering how you could integrate it better with lua-nginx-module.
>

Have you checked out the ngx.thread API and cosocket API provided by
ngx_lua? They are also composable :)

> As a simple first phase, could an API be exposed to pause (i.e. yield)
> a request, to be resumed when
> either a file descriptor you'd like watched polls readable, or a time
> out occurs.

The original design decision made in ngx_lua is to abstract away such
details on the (low) event/fd level so that we can do clever
optimizations on the C land and avoid many possibilities where things
can go wrong.

If the existing cosocket and light thread API can fulfill your needs,
then better not reinvent the wheels here.

I'm more interested in adding higher level APIs rather than exposing
low-level hard-to-get-right APIs to the Lua land.

Well, just my 2 cents.

Regards,
-agentzh

quae

On 18 December 2014 at 15:22, Yichun Zhang (agentzh) <age...@gmail.com> wrote:
> Hello!
>
> On Thu, Dec 18, 2014 at 12:07 PM, Daurnimator wrote:
>> Lately I've been a big fan of the cqueues
>> (http://25thandclement.com/~william/projects/cqueues.html) library.
>> It's the best shot I've seen at composable asynchronous lua.
>> I'm wondering how you could integrate it better with lua-nginx-module.
>>
>
> Have you checked out the ngx.thread API and cosocket API provided by
> ngx_lua? They are also composable :)

I have, but I want more than just what cosockets exposes (there's more
than tcp and udp!),
and I don't want to keep rewriting libraries to use nginx specific apis.

cqueues is great because can work almost everywhere:
it can be the main loop of your process, but it doesn't have to be.

>> As a simple first phase, could an API be exposed to pause (i.e. yield)
>> a request, to be resumed when
>> either a file descriptor you'd like watched polls readable, or a time
>> out occurs.
>
> The original design decision made in ngx_lua is to abstract away such
> details on the (low) event/fd level so that we can do clever
> optimizations on the C land and avoid many possibilities where things
> can go wrong.
>
> If the existing cosocket and light thread API can fulfill your needs,
> then better not reinvent the wheels here.

I don't believe it can.

> I'm more interested in adding higher level APIs rather than exposing
> low-level hard-to-get-right APIs to the Lua land.

One of the biggest issues I see across the wide range of lua
programmers, is the fragmentation of libraries.
You end up with 10 different libraries, all slightly different
depending on what main loop you're running under.
  - use luasockets in a blocking manner
  - only work with luaev
  - only work with nginx lua cosockets
  - only work inside of prosody's main loop
.... etc.

I see cqueues as a solution to all of this, that works wherever you are.
This means we only need to write *one* redis/postgres/user space
sctp/evdev library, and use it everywhere!
 => much fewer wheels reinvented!

> Well, just my 2 cents.
>
> Regards,
> -agentzh

I hope you can see there is a reason to expose a lower level interface
than just tcp sockets.

Having a scan through the ngx_lua source, the interface I suggested
should be possible via ngx_handle_read_event + ngx_add_timer.

agentzh

Hello!

On Thu, Dec 18, 2014 at 1:29 PM, Daurnimator wrote:
> I have, but I want more than just what cosockets exposes (there's more
> than tcp and udp!),

Tell us what's actually missing on a higher level so that we can consider that.

> and I don't want to keep rewriting libraries to use nginx specific apis.
>

Well, that's why the cosocket API keeps compatibility with LuaSocket
whenever possible. So existing libraries written for LuaSocket should
not have much problems while porting to OpenResty.

OpenResty is already a dialect of Lua (and we may extend the Lua
language syntax and semantics in the future as well). Instead of
focusing on running the same Lua code everywhere outside OpenResty,
I'd rather focus on running OpenResty everywhere :) This way I can
also get freed from supporting all those legacy ill-designed APIs done
by others.

To avoid messing things up, I'd rather focus on the one true way of
doing things instead of going further on millions of more ways of
doing similar things and keep confusing beginners.

> I see cqueues as a solution to all of this, that works wherever you are.
> This means we only need to write *one* redis/postgres/user space
> sctp/evdev library, and use it everywhere!
>  => much fewer wheels reinvented!
>

Frankly I'd rather support compatibility on a higher level. For
example, the LuaSocket API.

I'm reluctant to expose low-level APIs because

1.  it'll be very hard for me to make it fully compatible with
existing (and mostly importantly, future) high-level APIs because low
level APIs inevitably create "holes" in the overall abstraction, which
can be a pain.

2. Also it will encourage using external Lua C modules which may be
doing blocking syscalls and destroy the performance.

3. It can make future high-level optimizations hard or impossible to
add due to the lack of abstraction and encapsulation. Yes, this is
also a strength of higher-level abstractions.

Also, re-inventing wheels *can* be a good thing. I believe very few
existing Lua libraries in the wild take performance as seriously as
OpenResty :)

> I hope you can see there is a reason to expose a lower level interface
> than just tcp sockets.

I can see your reasoning here but I don't quite buy it because it
(kind of) goes against the fundamental design decisions of OpenResty
-- keeping things small, simple, easy, and fast.

Regards,
-agentzh

quae

On 18 December 2014 at 17:57, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

On Thu, Dec 18, 2014 at 1:29 PM, Daurnimator wrote:
> I have, but I want more than just what cosockets exposes (there's more
> than tcp and udp!),

Tell us what's actually missing on a higher level so that we can consider that.

The ability to use well maintained external libraries at a low level async interface.
e.g. Postgres' libpq has PQsocket and PQflush: http://www.postgresql.org/docs/9.4/static/libpq-async.html
hiredis has redisAsyncContext: https://github.com/redis/hiredis/blob/a9c21e4d48ecde49901e46e78ae87897fc182d52/async.h#L72

Everything that doesn't already have one of these lower level fd + events APIs, can simply be converted to one.

It's a fantastic point of compatibility between client library implementations and client library users.

> and I don't want to keep rewriting libraries to use nginx specific apis.
>

Well, that's why the cosocket API keeps compatibility with LuaSocket
whenever possible. So existing libraries written for LuaSocket should
not have much problems while porting to OpenResty.

Not everything is written with luasocket; and nor should it be.

Furthermore, luasocket apis become a mess as soon as you add SSL; or more than a single socket at a time.

Nested epolls/kqueues/etc fix this (which is part of what cqueues use internally)

Nevermind interfacing with non-socket pollable objects (e.g. things in /proc filesystem, which I would like to monitor via a http long polling api)

OpenResty is already a dialect of Lua (and we may extend the Lua
language syntax and semantics in the future as well).

Please do not!

Instead of
focusing on running the same Lua code everywhere outside OpenResty,
I'd rather focus on running OpenResty everywhere :) This way I can
also get freed from supporting all those legacy ill-designed APIs done
by others.

Not everything is a HTTP endpoint...

Lua runs inside (e.g.) game engines, media players and more: these are not places nginx should be.

However, cqueues can: it can let you call out in the middle of a game ai plugin.

This important thing is: cqueues is mainloop agnostic.

To avoid messing things up, I'd rather focus on the one true way of
doing things instead of going further on millions of more ways of
doing similar things and keep confusing beginners.

The current state of things immensely confuses beginners:

Common questions are:

- "Why can't I use this openresty library in project XYZ"

- "why can't I use the normal lua library XYZ inside of openresty"?

- "Why is everything so incompatible, lua programmers can't agree on anything!"

If they are one and the same (cqueues), this problem goes away.

> I see cqueues as a solution to all of this, that works wherever you are.
> This means we only need to write *one* redis/postgres/user space
> sctp/evdev library, and use it everywhere!
> => much fewer wheels reinvented!
>

Frankly I'd rather support compatibility on a higher level. For
example, the LuaSocket API.

I'm reluctant to expose low-level APIs because

1. it'll be very hard for me to make it fully compatible with
existing (and mostly importantly, future) high-level APIs because low
level APIs inevitably create "holes" in the overall abstraction, which
can be a pain.

I hope I can convince you fd + timeout is a baseline that *everything* can be expressed as.

More than one fd to watch? Combine them with epoll?

Want to watch a signal? Use a signalfd

Want to wait for a thread to join? socketpair + send

Hell, you can even use a timerfd and skip the timeout.

2. Also it will encourage using external Lua C modules which may be
doing blocking syscalls and destroy the performance.

This is already quite common for new developers that don't understand performance implications.

And blocking syscalls aren't the only problem: even long running computations are common.

This is not something that has a (reasonable) technical solution.

3. It can make future high-level optimizations hard or impossible to
add due to the lack of abstraction and encapsulation. Yes, this is
also a strength of higher-level abstractions.

See 1.

Also, re-inventing wheels *can* be a good thing. I believe very few
existing Lua libraries in the wild take performance as seriously as
OpenResty :)

> I hope you can see there is a reason to expose a lower level interface
> than just tcp sockets.

I can see your reasoning here but I don't quite buy it because it
(kind of) goes against the fundamental design decisions of OpenResty
-- keeping things small, simple, easy, and fast.

This helps keep things small; its a low level abstraction point for other modules to jump in and provide functionality.

Without it, you'll have to end up adding everything to openresty itself: dTLS, SCTP, async file io, etc. rather than as modules/plugins.

Daurn.

agentzh

Hello!

On Thu, Dec 18, 2014 at 3:34 PM, Daurnimator wrote:
> The ability to use well maintained external libraries at a low level async
> interface.
> e.g. Postgres' libpq has PQsocket and PQflush:
> http://www.postgresql.org/docs/9.4/static/libpq-async.html

After working on ngx_postgres module which integrates libpq into nginx
directly, I have to say it's quite painful. The libpq library does not
really support ET events very well and we have to workaround various
issues with dirty hacks.

> hiredis has redisAsyncContext:
>

Given the simplicity of the redis protocol, I don't think it's worth
all the trouble (and overhead) of accommodating a C library's C API.

Basically the real world is very messy. We have to eventually get
everything right anyway.

> Everything that doesn't already have one of these lower level fd + events
> APIs, can simply be converted to one.

Sorry, after years of playing with raw fd and events, I really doubt it :)

> Not everything is written with luasocket; and nor should it be.
> Furthermore, luasocket apis become a mess as soon as you add SSL;

The cosocket API already supports SSL/TLS :)

> or more
> than a single socket at a time.

Have you checked out the ngx.thread API?

> Nested epolls/kqueues/etc fix this (which is part of what cqueues use
> internally)
>

OpenResty uses epolls/kqueues/etc via the nginx event model already.
What's the point here?

> Nevermind interfacing with non-socket pollable objects (e.g. things in /proc
> filesystem, which I would like to monitor via a http long polling api)
>

I'd rather abstracting such implementation details away because I
don't want the users to deal with all the potential pitfalls here on
the Lua land.

>> OpenResty is already a dialect of Lua (and we may extend the Lua
>> language syntax and semantics in the future as well).
>
> Please do not!
>

The reason that we chose Lua from the very beginning was not because
we were Lua programmers (I started learning Lua *after* I decided to
work on the ngx_lua module) but just because the core Lua language and
the LuaJIT implementation serve OpenResty's original goals well :)

Supporting other Lua projects outside OpenResty is never a goal of
OpenResty. Though we're open to collaborations and code reuse if not
sacrificing our original vision.

> Not everything is a HTTP endpoint...

Yes, serving as an HTTP endpoint is OpenResty's main focus. But we
recently found that it can naturally extend to other use cases without
much troubles. Have you checked out the resty-cli project and the
TCP/UDP server support in ngx_lua's TODO list? See

https://github.com/openresty/resty-cli

https://github.com/openresty/lua-nginx-module#todo

I'd like to see such things happen naturally without upsetting our
existing (OpenResty) users.

> Lua runs inside (e.g.) game engines, media players and more: these are not
> places nginx should be.
> However, cqueues can: it can let you call out in the middle of a game ai
> plugin.

Yeah, I can see you are completely sold to cqueues :) But unfortunately I'm not.

My way of doing things is to focus on doing one thing and doing it
well. And I may extend things a bit to accommodate some more use cases
if I won't sacrifice the existing core features much.

Frankly I'm not a guy very a big ambition. Making OpenResty run
everywhere is actually not really a goal. I said that because it was
just a better (hypothetical) alternative to intentionally supporting a
more generic and low level Lua APIs. Because the latter will turn
OpenResty into something else which I don't quite like.

> The current state of things immensely confuses beginners:
> Common questions are:
>   - "Why can't I use this openresty library in project XYZ"
>   - "why can't I use the normal lua library XYZ inside of openresty"?
>   - "Why is everything so incompatible, lua programmers can't agree on
> anything!"
>

We will address this with our own package managing toolchain and site.
And anything outside our site or toolchain is not supposed to run in
OpenResty.

> If they are one and the same (cqueues), this problem goes away.
>

Nope, I can see a lot more (practical) problems will come in. I think
I've already listed the issues in my previous mails. So no need to
repeat them here :)

> I hope I can convince you fd + timeout is a baseline that *everything* can
> be expressed as.
> More than one fd to watch? Combine them with epoll?
> Want to watch a signal? Use a signalfd
> Want to wait for a thread to join? socketpair + send

Well, I'm not really interested in expressing everything in Lua.

I'm a C programmer and I just want Lua to be the glue. There's a ton
of implementation details involved with event-based programming, not
just fds, events, and timeout handling, and I doubt if it'll be
efficient enough to do all the heavy lifting on the Lua land instead
of C.

I can buy the idea of a portable and generic Lua APIs that various
different Lua-based systems can agree on. But I hope it to be high
level enough so that we can give enough room and freedom for the
implementors to decide the best implementation strategies. I hope we
can have the best abstraction level here. Low level APIs are certainly
not good abstractions at all IMHO. And for your cqueue API proposal, I
don't buy it as the OpenResty maintainer due to practical
considerations.

> This helps keep things small; its a low level abstraction point for other
> modules to jump in and provide functionality.
> Without it, you'll have to end up adding everything to openresty itself:
> dTLS, SCTP, async file io, etc. rather than as modules/plugins.
>

I don't mind if I can see excellent use cases for them and I can
guarantee that the implementations always work well out of the box :)

I'm not sure if you're an OpenResty user who wants to migrate his
OpenResty Lua code over to other contexts or just a Lua developer who
wants to migrate his whatever existing Lua code (running for other
contexts) over to OpenResty. For the former case, I think you can just
implement the same high level APIs (our cosockets, light threads, and
etc) in the other context, with exactly the same semantics, optimized
for your exotic new contexts, where the high level APIs can shine
again :)

Regards,
-agentzh

quae

On 19 December 2014 at 15:45, Yichun Zhang (agentzh) <age...@gmail.com> wrote:
> > Everything that doesn't already have one of these lower level fd + events
> > APIs, can simply be converted to one.
>
> Sorry, after years of playing with raw fd and events, I really doubt it :)

I hope I can prove you wrong on this one. Do you have a particular example in mind?

> > Not everything is written with luasocket; and nor should it be.
> > Furthermore, luasocket apis become a mess as soon as you add SSL;
>
> The cosocket API already supports SSL/TLS :)
>
> > or more
> > than a single socket at a time.
>
> Have you checked out the ngx.thread API?

I was referring to protocol specific libraries and the LuaSocket API.

ngx.thread is yet another (fine) solution on that pile.

However, it is not easily available in other environments, and hence makes any resty based lua modules unportable.

> > Nested epolls/kqueues/etc fix this (which is part of what cqueues use
> > internally)
> >
>
> OpenResty uses epolls/kqueues/etc via the nginx event model already.
> What's the point here?

The point was that it doesn't have to stop at the C level.

Let Lua modules have their own epoll 'trees' to abstract fds inside a library.

> Yes, serving as an HTTP endpoint is OpenResty's main focus. But we
> recently found that it can naturally extend to other use cases without
> much troubles. Have you checked out the resty-cli project and the
> TCP/UDP server support in ngx_lua's TODO list? See
>
> https://github.com/openresty/resty-cli

That's an interesting project; I will certainly be using it as part of our test suite for ngx_lua code.

> I'd like to see such things happen naturally without upsetting our
> existing (OpenResty) users.

I'm not sure how anything we've discussed would upset anyone?

This was a request to expose a lower level abstraction.

> I can buy the idea of a portable and generic Lua APIs that various

> different Lua-based systems can agree on. But I hope it to be high
> level enough so that we can give enough room and freedom for the
> implementors to decide the best implementation strategies. I hope we
> can have the best abstraction level here. Low level APIs are certainly
> not good abstractions at all IMHO. And for your cqueue API proposal, I
> don't buy it as the OpenResty maintainer due to practical
> considerations.

What are the considerations here?

> I'm not sure if you're an OpenResty user who wants to migrate his
> OpenResty Lua code over to other contexts or just a Lua developer who
> wants to migrate his whatever existing Lua code (running for other
> contexts) over to OpenResty. For the former case, I think you can just
> implement the same high level APIs (our cosockets, light threads, and
> etc) in the other context, with exactly the same semantics, optimized
> for your exotic new contexts, where the high level APIs can shine
> again :)

I write Lua in a wide range of contexts every day:

- I use ngx_lua for some high traffic http endpoints

- I write prosody plugins and maintain a significant XMPP deployment

- I maintain several lua libraries for use by others

- I regulary write lua scripts to accomplish administrative tasks

- I use embedded lua interpreters (e.g. Redis)

One of the most annoying issues I face is that there is so little overlap in the ecosystems.

If I want to use (e.g.) postgres, I have to figure out the idiosyncrasies of the particular postgres library available in that particular event loop.

They're all of varying quality => some are blocking, some use callbacks, some use coroutines, some have iterators, .... etc.

It's a hodge-podge where nothing built on top of them is portable.

When some new database comes along, each of these environments takes time to gain a client for it.

e.g. when mongo came out, I wrote https://github.com/daurnimator/mongol which uses Luasocket,

However, someone had to fork it for openresty: https://github.com/bigplum/lua-resty-mongol

But most of the changes are too nginx specific for me to take them upstream.

However, I've found that the lowest common denominator in most environments is being able to ask the mainloop to watch an fd for me;

This means I can use (e.g.) https://github.com/daurnimator/cqueues-pgsql

Furthermore, I can build things on top of it, and be assured they will work everywhere!

==> I can distribute a library that will work for the server admins, openresty users, prosody plugin writers, etc.

I'm really hoping that next time I write a client library for something, I only have to write it once.

anorov.vorona

>One of the most annoying issues I face is that there is so little overlap in the ecosystems.

The problem is that Lua is very commonly used as a domain-specific scripting engine, not its own separate framework. OpenResty isn't exactly a web framework for Lua, it's integration of Lua into nginx to improve the way you use nginx. Lua has to be small and lean and needs to be tailored specifically to the project it is being integrated with.

Libraries providing alternative event loops and abstractions unfortunately do not always apply to the environment where Lua is being used. resty-lua* is coupled pretty tightly with nginx at the moment and probably will be forever. Performance is also a major consideration here, so any abstractions really have to be written from the ground up and applied specifically to nginx, LuaJIT and FFI, the cosocket API, etc. You can't just throw in something like luvit or cqueues and expect it to integrate correctly, even though they're both great projects.

If you wanted something like this, you'd be better off writing a specific abstraction for the project. lua-resty-queues or something like that. However, I'm not quite sure how useful that would be though. If one of your Lua handlers from a request needs its own event loop, you're probably trying to do way too much at once and may not be using OpenResty properly. Better to send off that data to some background worker's socket, which itself could also be coded in Lua and use cqueues or some other event loop, or be written in any other language.

Things do seem to be changing a bit now that OpenResty will support arbitrary TCP/UDP servers and likely many more things in the coming years, but nginx is still the core and I presume always will be.

>I'm really hoping that next time I write a client library for something, I only have to write it once.

If you're a Lua coder and you use Lua embedded into other projects, then unfortunately this dream will simply never be realized, with the exception of some basic utility libraries. You kind of have to accept that in order to understand and appreciate the pragmatism of Lua. Lua is the cement in these cases, not the bricks.

On Friday, December 19, 2014 4:38:23 PM UTC-5, Daurnimator wrote:

On 19 December 2014 at 15:45, Yichun Zhang (agentzh) <age...@gmail.com> wrote:
> > Everything that doesn't already have one of these lower level fd + events
> > APIs, can simply be converted to one.
>
> Sorry, after years of playing with raw fd and events, I really doubt it :)

I hope I can prove you wrong on this one. Do you have a particular example in mind?

> > Not everything is written with luasocket; and nor should it be.
> > Furthermore, luasocket apis become a mess as soon as you add SSL;
>
> The cosocket API already supports SSL/TLS :)
>
> > or more
> > than a single socket at a time.
>
> Have you checked out the ngx.thread API?

I was referring to protocol specific libraries and the LuaSocket API.
ngx.thread is yet another (fine) solution on that pile.
However, it is not easily available in other environments, and hence makes any resty based lua modules unportable.

> > Nested epolls/kqueues/etc fix this (which is part of what cqueues use
> > internally)
> >
>
> OpenResty uses epolls/kqueues/etc via the nginx event model already.
> What's the point here?

The point was that it doesn't have to stop at the C level.
Let Lua modules have their own epoll 'trees' to abstract fds inside a library.

> Yes, serving as an HTTP endpoint is OpenResty's main focus. But we
> recently found that it can naturally extend to other use cases without
> much troubles. Have you checked out the resty-cli project and the
> TCP/UDP server support in ngx_lua's TODO list? See
>
> https://github.com/openresty/resty-cli

That's an interesting project; I will certainly be using it as part of our test suite for ngx_lua code.

> I'd like to see such things happen naturally without upsetting our
> existing (OpenResty) users.

I'm not sure how anything we've discussed would upset anyone?
This was a request to expose a lower level abstraction.

> I can buy the idea of a portable and generic Lua APIs that various
> different Lua-based systems can agree on. But I hope it to be high
> level enough so that we can give enough room and freedom for the
> implementors to decide the best implementation strategies. I hope we
> can have the best abstraction level here. Low level APIs are certainly
> not good abstractions at all IMHO. And for your cqueue API proposal, I
> don't buy it as the OpenResty maintainer due to practical
> considerations.

What are the considerations here?

> I'm not sure if you're an OpenResty user who wants to migrate his
> OpenResty Lua code over to other contexts or just a Lua developer who
> wants to migrate his whatever existing Lua code (running for other
> contexts) over to OpenResty. For the former case, I think you can just
> implement the same high level APIs (our cosockets, light threads, and
> etc) in the other context, with exactly the same semantics, optimized
> for your exotic new contexts, where the high level APIs can shine
> again :)

I write Lua in a wide range of contexts every day:
- I use ngx_lua for some high traffic http endpoints
- I write prosody plugins and maintain a significant XMPP deployment
- I maintain several lua libraries for use by others
- I regulary write lua scripts to accomplish administrative tasks
- I use embedded lua interpreters (e.g. Redis)

One of the most annoying issues I face is that there is so little overlap in the ecosystems.
If I want to use (e.g.) postgres, I have to figure out the idiosyncrasies of the particular postgres library available in that particular event loop.
They're all of varying quality => some are blocking, some use callbacks, some use coroutines, some have iterators, .... etc.
It's a hodge-podge where nothing built on top of them is portable.

When some new database comes along, each of these environments takes time to gain a client for it.
e.g. when mongo came out, I wrote https://github.com/daurnimator/mongol which uses Luasocket,
However, someone had to fork it for openresty: https://github.com/bigplum/lua-resty-mongol
But most of the changes are too nginx specific for me to take them upstream.

However, I've found that the lowest common denominator in most environments is being able to ask the mainloop to watch an fd for me;
This means I can use (e.g.) https://github.com/daurnimator/cqueues-pgsql
Furthermore, I can build things on top of it, and be assured they will work everywhere!
==> I can distribute a library that will work for the server admins, openresty users, prosody plugin writers, etc.

I'm really hoping that next time I write a client library for something, I only have to write it once.

william

On Sat, Dec 20, 2014 at 09:39:45PM -0800, Anorov wrote:
> >One of the most annoying issues I face is that there is so little overlap 
> in the ecosystems.
> 
> The problem is that Lua is very commonly used as a domain-specific 
> scripting engine, not its own separate framework. OpenResty isn't exactly a 
> web framework for Lua, it's integration of Lua into nginx to improve the 
> way you use nginx. Lua has to be small and lean and needs to be tailored 
> specifically to the project it is being integrated with.
> 
> Libraries providing alternative event loops and abstractions unfortunately 
> do not always apply to the environment where Lua is being used. resty-lua* 
> is coupled pretty tightly with nginx at the moment and probably will be 
> forever. Performance is also a major consideration here, so any 
> abstractions really have to be written from the ground up and applied 
> specifically to nginx, LuaJIT and FFI, the cosocket API, etc. You can't 
> just throw in something like luvit or cqueues and expect it to integrate 
> correctly, even though they're both great projects.

I actually use cqueues in several projects that have their own event loop,
both libevent-based and otherwise. cqueues is not like luavit, libuv,
libevent, libev, etc. Those projects are intended as mainloops and are
basically wrappers around select/poll that happen to get better performance
when an O(1) kernel polling API is available.

The thing about epoll, kqueue, and Solaris Ports that people forget is that
those descriptor objects are themselves pollable; they behave as a proxy for
events added to that object. There's a _reason_ they can do that--because it
means you no longer are stuck with choosing a _single_ mainloop. Rather, you
can mix-and-match event loops, which improves code portability across
frameworks.

Unlike luavit, libevent, or other frameworks, cqueues does five things which
make it interoperable with other systems.

1) Exports the epoll, kqueue, or Solaris Ports descriptor. By polling on
this descriptor you're polling on every event managed by that descriptor,
including other epoll/kqueue/Ports descriptors.

2) Exports the next timeout interval.

3) It uses no global state. It doesn't play with signals, threads, or
anything else behind the back of the application. That means it doesn't
interfere with mainloops that use hacks to emulate certain behaviors, such
as the pipe hack to catch signals when signalfd or kqueue isn't available.

4) The event loop doesn't block--it can step through a single set of events
before returning.

5) It doesn't rely on continuation callbacks anywhere. That means it won't
interfere with the continuation model (e.g. callbacks, restartable state
machines, etc) used elsewhere in the application, and in particular by
another event loop.

So, interoperability is actually quite easy. It was the principle driver
behind the design and implementation of cqueues. I was writing event loops
and frameworks before libevent even existed (I was also an early contributor
to libevent). I had zero desire to re-invent the wheel with cqueues. My goal
was not supplanting or replacing those event loops, but being able to write
portable Lua modules (whether in Lua-script or Lua-C) doing scalable I/O
that work in applications using any of those loops.

The remaining arguments againt permitting use of cqueues kind of boil down
to the idea that OpenResty should be a closed ecosystem using only approved
patterns, and which doesn't permit applications to do things beyond the One
True Way. That's not something I can argue with because that's just a
personal preference. I can understand that preference, but let's not pretend
it's based on technical concerns, other than perhaps concern with technical
supporting when people implement unexpected solutions within the framework.

agentzh

Hello!

On Sun, Dec 21, 2014 at 2:06 PM, William Ahern wrote:
> but let's not pretend
> it's based on technical concerns, other than perhaps concern with technical
> supporting when people implement unexpected solutions within the framework.
>

Alas. I am not pretending anything here at all. Please do not get me
wrong and keep me repeating myself. My point is that ensuring
everything (cosockets, ngx.thread, ngx.exit, ngx.timer, subrequests,
ngx.exec, and etc) always work well together in the context of
OpenResty can be VERY difficult if we open up our internals, expose a
low level API, and let the user do whatever they want to in a way we
can never control. IMHO, this is like allowing the user applications
to bypass the system call interface and call whatever kernel C
functions they like.

Ensuring the system hard to break (unintentionally) for the users is
an important goal of OpenResty's design, while helping genuine hackers
do whatever they like with the internals on the Lua land is not. One
has to admit that every design must have some trade-offs. When you
have something, you must lose something.

Well, if you really want that low level capabilities and has the
ability and strong desire to re-invent OpenResty's core features in
pure Lua yourself, then you should not use OpenResty in the first
place because you're not really among the target users of OpenResty.

At last but not least, I could be wrong here and you always have the
freedom to fork ngx_lua and show us the actual running code and all
the benefits you want to convince me. It's opensource :)

Regards,
-agentzh

quae

On 21 December 2014 at 18:09, Yichun Zhang (agentzh) <age...@gmail.com> wrote:
> Ensuring the system hard to break (unintentionally) for the users is
> an important goal of OpenResty's design, while helping genuine hackers
> do whatever they like with the internals on the Lua land is not. One
> has to admit that every design must have some trade-offs. When you
> have something, you must lose something.

OpenResty is very easy to break as soon as you bring in any C modules,
use the ffi, or even play with a couple of documented features.
This addition is much more safe than any of the above.

> Well, if you really want that low level capabilities and has the
> ability and strong desire to re-invent OpenResty's core features in
> pure Lua yourself, then you should not use OpenResty in the first
> place because you're not really among the target users of OpenResty.

This desire here is to stop reinventing lua libraries.
I would love for libraries/utilities I write to be usable anywhere lua is.
I DO use openresty, and as mentioned earlier in the thread, maintain
multiple high traffic services built on top of it.

> At last but not least, I could be wrong here and you always have the
> freedom to fork ngx_lua and show us the actual running code and all
> the benefits you want to convince me. It's opensource :)

So I've had a go at coding up the feature myself over the last few days:
https://gist.github.com/daurnimator/55acef0626e00cf66c16
The exposed function takes a file descriptor and a timeout; and the
current coroutine should be resumed when that fd polls as ready.

I have never worked with nginx from C before, and have not been able
to find much documentation (if any, besides a couple of blog posts).
I was hoping that I might be able to get your help with some of the
openresty structures, as I could not find any documentation either.
Are you able to see any egregious mistakes in the code as given above?

-----------------------------------------------------------------------------------------------------------------
The next problem I faced is actually a seperate one, and I should
probably fork the thread:

Mixing coroutine functions results in strange errors: many functions
seem to not check their input well/correctly.

e.g, this causes a SIGABRT at
https://github.com/openresty/lua-nginx-module/blob/af16d1f99441b622188e91ef64dad61da024eae1/src/ngx_http_lua_util.c#L1052

    local co = coroutine.create(function()
        coroutine._yield(10)
    end)
    coroutine.resume(co)
    ngx.say("DONE")

If you remove the argument to the _yield, the assert still passes,
and much worse internal corruption occurs.

Regards, and I hope everyone had a great Christmas!
Daurnimator.

agentzh

Hello!

On Fri, Dec 26, 2014 at 5:32 PM, Daurnimator wrote:
>
> OpenResty is very easy to break as soon as you bring in any C modules,
> use the ffi, or even play with a couple of documented features.
> This addition is much more safe than any of the above.
>

I believe you already know what I was talking about. Such argument is pointless.

> This desire here is to stop reinventing lua libraries.
> I would love for libraries/utilities I write to be usable anywhere lua is.
> I DO use openresty, and as mentioned earlier in the thread, maintain
> multiple high traffic services built on top of it.
>

Well, please come up with a good and sane Lua API proposal, as well as
a sane implementation :)

> So I've had a go at coding up the feature myself over the last few days:
> https://gist.github.com/daurnimator/55acef0626e00cf66c16
> The exposed function takes a file descriptor and a timeout; and the
> current coroutine should be resumed when that fd polls as ready.
>

The code has a lot of problems but I can get your idea. So you just
want a Lua API function to register an fd and a Lua API function to
wait for events on it? If it's just that, then I can consider it.

Your current way of doing all the initialization upon every wait()
looks VERY wasteful to me. Better introduce a dedicated registering
function and some kind of OO-like abstraction (as in the coroutine
API).

> I have never worked with nginx from C before, and have not been able
> to find much documentation (if any, besides a couple of blog posts).
> I was hoping that I might be able to get your help with some of the
> openresty structures, as I could not find any documentation either.
> Are you able to see any egregious mistakes in the code as given above?
>

Yeah, I can see a lot of issues with a quick peek. The most notable
one is the coding style.

> The next problem I faced is actually a seperate one, and I should
> probably fork the thread:
>

Yes, please.

> Mixing coroutine functions results in strange errors: many functions
> seem to not check their input well/correctly.
>

Well, feel free to create pull requests for ngx_lua :)

> e.g, this causes a SIGABRT at
> https://github.com/openresty/lua-nginx-module/blob/af16d1f99441b622188e91ef64dad61da024eae1/src/ngx_http_lua_util.c#L1052
>
>     local co = coroutine.create(function()
>         coroutine._yield(10)

Do you know that the coroutine._yield() thing is an intentionally
undocumented internal API that is never supposed to be called by the
users?

Regards,
-agentzh

quae

On 26 December 2014 at 21:04, Yichun Zhang (agentzh) <age...@gmail.com> wrote:
> On Fri, Dec 26, 2014 at 5:32 PM, Daurnimator wrote:
>> This desire here is to stop reinventing lua libraries.
>> I would love for libraries/utilities I write to be usable anywhere lua is.
>> I DO use openresty, and as mentioned earlier in the thread, maintain
>> multiple high traffic services built on top of it.
>>
>
> Well, please come up with a good and sane Lua API proposal, as well as
> a sane implementation :)

I'm putting forward cqueues as the Lua API proposal:
One of the core ideas is that 'pollable' objects expose 3 methods:
  - 'pollfd': an fd to poll
  - 'events': the type of event(s) to poll for (e.g. "r" for readable
or "w" for writable)
  - 'timeout': the maximum amount of time to wait

This is documented in section 3 of:
http://25thandclement.com/~william/projects/cqueues.pdf

Within the cqueues library you have tools that let you assemble
pollable objects in arbitrary ways.


>> So I've had a go at coding up the feature myself over the last few days:
>> https://gist.github.com/daurnimator/55acef0626e00cf66c16
>> The exposed function takes a file descriptor and a timeout; and the
>> current coroutine should be resumed when that fd polls as ready.
>>
>
> The code has a lot of problems but I can get your idea. So you just
> want a Lua API function to register an fd and a Lua API function to
> wait for events on it? If it's just that, then I can consider it.

Yes, that is what I requested in the first email in the thread.

> Your current way of doing all the initialization upon every wait()
> looks VERY wasteful to me. Better introduce a dedicated registering
> function and some kind of OO-like abstraction (as in the coroutine
> API).

Sounds great :) This is why I asked here about it in the first place!
As someone unfamiliar with either codebase (nginx or ngx_lua),
what functions are structures are available for me to use?

>> I have never worked with nginx from C before, and have not been able
>> to find much documentation (if any, besides a couple of blog posts).
>> I was hoping that I might be able to get your help with some of the
>> openresty structures, as I could not find any documentation either.
>> Are you able to see any egregious mistakes in the code as given above?
>>
>
> Yeah, I can see a lot of issues with a quick peek.

Could you enumerate them?
I would like to fix them :)


Daurn.

agentzh

Hello!

On Fri, Dec 26, 2014 at 6:31 PM, Daurnimator wrote:
> I'm putting forward cqueues as the Lua API proposal:
> One of the core ideas is that 'pollable' objects expose 3 methods:
>   - 'pollfd': an fd to poll
>   - 'events': the type of event(s) to poll for (e.g. "r" for readable
> or "w" for writable)
>   - 'timeout': the maximum amount of time to wait
>

This is too high-level IMHO and is not really necessary at all for a
primitive API. And we should never expose fds of the cosocket objects,
for example.

I suggest focusing on the poll API itself. I'd see it looks more like
the system poll/epoll API.

> This is documented in section 3 of:
> http://25thandclement.com/~william/projects/cqueues.pdf
>

I read through it and unfortunately didn't get impressed :)

> Within the cqueues library you have tools that let you assemble
> pollable objects in arbitrary ways.
>

I (personally) find the ability to assemble Lua lightweight threads
and coroutines in arbitrary ways more appealing. And that was why I
implemented the ngx.thread API and the coroutine API (with standard
semantics) in the first place :)

> Sounds great :) This is why I asked here about it in the first place!
> As someone unfamiliar with either codebase (nginx or ngx_lua),
> what functions are structures are available for me to use?
>

You'll have to look into the source code to get enlightened. Basically
the src/ngx_http_lua_socket_tcp.c file should be the most relevant.

>
> Could you enumerate them?
> I would like to fix them :)
>

I'd rather comment on github pull requests instead, which should be
much easier for me :)

Regards,
-agentzh

quae

On 2 January 2015 at 17:36, Yichun Zhang (agentzh) <age...@gmail.com> wrote:
> This is too high-level IMHO and is not really necessary at all for a
> primitive API. And we should never expose fds of the cosocket objects,
> for example.
>
> I suggest focusing on the poll API itself. I'd see it looks more like
> the system poll/epoll API.

That is much more complex than what I'm suggesting.
The API I want here only allows for waiting on a single file
descriptor (per request); not many.

> I'd rather comment on github pull requests instead, which should be
> much easier for me :)

I cleaned up the code a little bit, and created a pull request:
https://github.com/openresty/lua-nginx-module/pull/450
I'm happy if you want to move discussion there.