Writing AMQP/Lua cosocket api based lib Vs. writing nginx C module

rohit.yadav · 2013-05-17T23:56:05+00:00

Hello! On Wed, May 15, 2013 at 8:31 PM, 瞿秋丰 wrote: > nginx.conf > upstream mysql_backend { > drizzle_server 127.0.0.1:...

Writing AMQP/Lua cosocket api based lib Vs. writing nginx C module

rohit.yadav

Hi,

I'm trying to implement a fast openresty/lua based request handler which could write processed data directly to RabbitMQ. I could not find any production grade amqp/lua library which is non-blocking and reuses a connection pool or is based on cosocket API.

But I could find a good C based amqp library, so one way could be to write a C based nginx module. (can one get access to lua_shared_dicts from a C based nginx/module?)

Another way is to write my own lua library based on cosocket api and opensource it. I don't know anything about writing nginx module or lua/cosocket api based library but I've anticipated these two options and I'm going with either of these two options.

Pl. advise? Which would be the best approach for my use case.

Regards,
Rohit Yadav

brian

If you can use stomp, it is a very simple protocol and could easily be written in Lua. Amqp looks more challenging, but there are pure Lua clients for MySQL and Postgres, so it is probably doable.

rohit.yadav

Thanks Brian and Simon. I'll start with lua-resty/stomp-rabbitmq library, amqp 9-1-0 is still less complicated than 1-0-0 which 'll attempt to write later in case lua/stomp does not scale enough for my use case.

Cheers.

On Sat, May 18, 2013 at 12:53 AM, Brian Akins <br...@akins.org> wrote:

If you can use stomp, it is a very simple protocol and could easily be written in Lua. Amqp looks more challenging, but there are pure Lua clients for MySQL and Postgres, so it is probably doable.
.

agentzh

Hello!

On Wed, May 22, 2013 at 1:13 AM,  <llb...@gmail.com> wrote:
>
> Lua has rabbitmq client library  https://github.com/tcoram/rabbitmq-lua.
>
> Can it use directly ?
>

No, it will surely block the Nginx worker processes and ruin the
performance. To quote the related parts in ngx_lua's official
documentation:

"Network I/O operations in user code should only be done through the
Nginx Lua API calls as the Nginx event loop may be blocked and
performance drop off dramatically otherwise."

See http://wiki.nginx.org/HttpLuaModule#Nginx_API_for_Lua

To summarize, you should only use a client library

1. based on either ngx_lua's cosocket API or
2. implemented as an Nginx upstream C module (and accessed by Nginx
subrequests).

Best regards,
-agentzh

llbgurs

在 2013年5月17日星期五UTC+8下午11时56分05秒，Rohit Yadav写道：
> Hi,
> 
> I'm trying to implement a fast openresty/lua based request handler which could write processed data directly to RabbitMQ. I could not find any production grade amqp/lua library which is non-blocking and reuses a connection pool or is based on cosocket API.
> 
> 
> But I could find a good C based amqp library, so one way could be to write a C based nginx module. (can one get access to lua_shared_dicts from a C based nginx/module?)
> 
> Another way is to write my own lua library based on cosocket api and opensource it. I don't know anything about writing nginx module or lua/cosocket api based library but I've anticipated these two options and I'm going with either of these two options.
> 
> 
> Pl. advise? Which would be the best approach for my use case.
> 
> 
> Regards,
> Rohit Yadav

Lua has rabbitmq client library  https://github.com/tcoram/rabbitmq-lua.

Can it use directly ?

jijilu

lua:172: attempt to call method 'getreusedtimes' (a nil value)
stack traceback:
    /usr/local/webserver/lua/lib/resty/rabbitmqstomp.lua:172: in function 'connect'
    /data/rails2.3.5/biyifei/agent/01.lua:27: in main chunk
    [C]: ?

that's the errors out of lua-resty/stomp-rabbitmq library

On Sunday, May 19, 2013 9:17:30 PM UTC+8, Rohit Yadav wrote:

Thanks Brian and Simon. I'll start with lua-resty/stomp-rabbitmq library, amqp 9-1-0 is still less complicated than 1-0-0 which 'll attempt to write later in case lua/stomp does not scale enough for my use case.

Cheers.

On Sat, May 18, 2013 at 12:53 AM, Brian Akins <br...@akins.org> wrote:

If you can use stomp, it is a very simple protocol and could easily be written in Lua. Amqp looks more challenging, but there are pure Lua clients for MySQL and Postgres, so it is probably doable.
.

rohit.yadav

Hi, I'll check again with latest openresty. Can you share with us your code that caused this error?

If you can fix the issue in lua-resty-rabbitmystomp library, send a pull request on:

https://github.com/wingify/lua-resty-rabbitmqstomp

Regards.

On Mon, Jul 29, 2013 at 9:10 AM, jijilu <ji...@189.cn> wrote:

lua:172: attempt to call method 'getreusedtimes' (a nil value)
stack traceback:
    /usr/local/webserver/lua/lib/resty/rabbitmqstomp.lua:172: in function 'connect'
    /data/rails2.3.5/biyifei/agent/01.lua:27: in main chunk
    [C]: ?

that's the errors out of lua-resty/stomp-rabbitmq library

On Sunday, May 19, 2013 9:17:30 PM UTC+8, Rohit Yadav wrote:

Thanks Brian and Simon. I'll start with lua-resty/stomp-rabbitmq library, amqp 9-1-0 is still less complicated than 1-0-0 which 'll attempt to write later in case lua/stomp does not scale enough for my use case.

Cheers.

On Sat, May 18, 2013 at 12:53 AM, Brian Akins <br...@akins.org> wrote:

If you can use stomp, it is a very simple protocol and could easily be written in Lua. Amqp looks more challenging, but there are pure Lua clients for MySQL and Postgres, so it is probably doable.

.

agentzh

Hello!

On Sun, Jul 28, 2013 at 8:40 PM, jijilu wrote:
>
> lua:172: attempt to call method 'getreusedtimes' (a nil value)
>

Which version of ngx_lua (or ngx_openresty) are you using?

Best regards,
-agentzh

jijilu

hello!

the version is 1.2.86

maybe i can use lua-resty-rabbitmqstomp to connect activemq?

I try many times, it shows "timeout".

@jijilu

On 2013-7-30, at 上午2:14, "Yichun Zhang (agentzh)" <age...@gmail.com> wrote:

Which version of ngx_lua (or ngx_openresty) are you using?

agentzh

Hello!

On Mon, Jul 29, 2013 at 11:20 PM, jijilu wrote:
> the version is 1.2.86
>

Whose version are you talking about?

There's no such version of Nginx (the latest 1.2.x nginx release is 1.2.9).

And there's no such version of ngx_openresty (the latest 1.2.x
ngx_openresty release is 1.2.8.6).

Also, there's no such version of ngx_lua (the latest ngx_lua release is 0.8.5).

Or do you mean the ngx_openresty release 1.2.8.6?

> maybe i can use lua-resty-rabbitmqstomp to connect activemq?
>
> I try many times, it shows "timeout".
>

When you see "timeout" errors, it usually means that your backend
service is just too busy to catch up with the requests from your
nginx. You can use the ngx-accept-queue and ngx-recv-queue tools in my
Nginx Systemtap Toolkit to confirm this:

    https://github.com/agentzh/nginx-systemtap-toolkit

Regards,
-agentzh

jijilu

I mean the ngx_openresty release 1.2.8.6, so I checkout the protocol between stomp of activemq and AMQP of rabbitmq…

maybe the lua-resty-rabbitmqstomp can NOT use to connect the activemq?

sorry about that.


@jijilu

On 2013-7-31, at 上午2:43, "Yichun Zhang (agentzh)" <age...@gmail.com> wrote:

> Hello!
> 
> On Mon, Jul 29, 2013 at 11:20 PM, jijilu wrote:
>> the version is 1.2.86
>> 
> 
> Whose version are you talking about?
> 
> There's no such version of Nginx (the latest 1.2.x nginx release is 1.2.9).
> 
> And there's no such version of ngx_openresty (the latest 1.2.x
> ngx_openresty release is 1.2.8.6).
> 
> Also, there's no such version of ngx_lua (the latest ngx_lua release is 0.8.5).
> 
> Or do you mean the ngx_openresty release 1.2.8.6?
> 
>> maybe i can use lua-resty-rabbitmqstomp to connect activemq?
>> 
>> I try many times, it shows "timeout".
>> 
> 
> When you see "timeout" errors, it usually means that your backend
> service is just too busy to catch up with the requests from your
> nginx. You can use the ngx-accept-queue and ngx-recv-queue tools in my
> Nginx Systemtap Toolkit to confirm this:
> 
>    https://github.com/agentzh/nginx-systemtap-toolkit
> 
> Regards,
> -agentzh.
> 
>