Implementing C functionality via FFI

gmadarm · 2014-10-05T22:05:17+00:00

If you need an url parser in Lua, you can try mine :https://github.com/golgote/neturlFeel free to fork and improve.On Fri, Oct 3, 2014 at 9:37 PM, Yichun Zha...

Implementing C functionality via FFI

gmadarm

I am trying to implement gumbo-parser (https://github.com/google/gumbo-parser) in openresty via FFI. I naively tried this:

# cat gumbo.lua
local ffi = require 'ffi'

local libgumbo = (function()

local file = io.open('/usr/local/include/gumbo.h')
ffi.cdef(file:read('*a'))
file:close()

local filename = '/usr/local/src/gumbo-parser/.libs/libgumbo.so'
return ffi.load(filename)
end)()

which resulting in the error message:
2014/10/05 16:51:23 [error] 13730#0: *432 failed to run body_filter_by_lua*: /opt/openresty/nginx/conf/gumbo.lua:6: unexpected symbol near 'ifndef' at line 42
stack traceback:
        [C]: in function 'cdef'
        /opt/openresty/nginx/conf/gumbo.lua:6: in function </opt/openresty/nginx/conf/gumbo.lua:3>
        /opt/openresty/nginx/conf/gumbo.lua:11: in function </opt/openresty/nginx/conf/gumbo.lua:1> while sending response to client, client: 127.0.0.1, server: localhost, request: "GET /parse.html HTTP/1.1", host: "localhost"

Can someone point me to a step by step tutorial (for someone with limited C knowledge and who has never ported a C library via FFI before) that explain in detail what steps I need to take to properly implement this library with FFI?

lordnynex

Hello,

FFI is choking on preprocessor directives. Have a look at https://github.com/craigbarnes/lua-gumbo (I've never used it but it may work for you)

On Sun, Oct 5, 2014 at 7:05 AM, Gordon Madarm <gma...@gmail.com> wrote:

I am trying to implement gumbo-parser (https://github.com/google/gumbo-parser) in openresty via FFI. I naively tried this:

# cat gumbo.lua
local ffi = require 'ffi'

local libgumbo = (function()

local file = io.open('/usr/local/include/gumbo.h')
ffi.cdef(file:read('*a'))
file:close()

local filename = '/usr/local/src/gumbo-parser/.libs/libgumbo.so'
return ffi.load(filename)
end)()

which resulting in the error message:
2014/10/05 16:51:23 [error] 13730#0: *432 failed to run body_filter_by_lua*: /opt/openresty/nginx/conf/gumbo.lua:6: unexpected symbol near 'ifndef' at line 42
stack traceback:
        [C]: in function 'cdef'
        /opt/openresty/nginx/conf/gumbo.lua:6: in function </opt/openresty/nginx/conf/gumbo.lua:3>
        /opt/openresty/nginx/conf/gumbo.lua:11: in function </opt/openresty/nginx/conf/gumbo.lua:1> while sending response to client, client: 127.0.0.1, server: localhost, request: "GET /parse.html HTTP/1.1", host: "localhost"

Can someone point me to a step by step tutorial (for someone with limited C knowledge and who has never ported a C library via FFI before) that explain in detail what steps I need to take to properly implement this library with FFI?

.

gmadarm

On Sunday, October 5, 2014 10:44:28 PM UTC+3, Lord Nynex wrote:

Hello,

FFI is choking on preprocessor directives. Have a look at https://github.com/craigbarnes/lua-gumbo (I've never used it but it may work for you)

Thanks. I tried that but when using the following (very slightly modified) lua-gumbo example script I ended up with the error below:

# cat gumbo.lua
local gumbo = require "gumbo"
local document = assert(gumbo.parse_file(ngx.arg[1]))

for node in document:walk() do
    if node.type == "element" and node.localName == "a" then
        local href = "">        if href then
            print(href.value)
        end
    end
end

# cat /var/log/nginx/error.log
2014/10/05 23:04:24 [error] 26159#0: *436 failed to run body_filter_by_lua*: /opt/openresty/nginx/conf/gumbo.lua:4: attempt to call method 'walk' (a nil value)
stack traceback:
        /opt/openresty/nginx/conf/gumbo.lua:4: in function </opt/openresty/nginx/conf/gumbo.lua:1> while sending response to client, client: 127.0.0.1, server: localhost, request: "GET /parse.html HTTP/1.1", host: "localhost"

lordnynex

Likely because ngx.arg[1] is nil. What file are you trying to parse? Can you make your code a bit more explicit?

On Sun, Oct 5, 2014 at 1:13 PM, Gordon Madarm <gma...@gmail.com> wrote:

On Sunday, October 5, 2014 10:44:28 PM UTC+3, Lord Nynex wrote:
Hello,

FFI is choking on preprocessor directives. Have a look at https://github.com/craigbarnes/lua-gumbo (I've never used it but it may work for you)

Thanks. I tried that but when using the following (very slightly modified) lua-gumbo example script I ended up with the error below:

# cat gumbo.lua
local gumbo = require "gumbo"
local document = assert(gumbo.parse_file(ngx.arg[1]))

for node in document:walk() do
    if node.type == "element" and node.localName == "a" then
        local href = "">        if href then
            print(href.value)
        end
    end
end

# cat /var/log/nginx/error.log
2014/10/05 23:04:24 [error] 26159#0: *436 failed to run body_filter_by_lua*: /opt/openresty/nginx/conf/gumbo.lua:4: attempt to call method 'walk' (a nil value)
stack traceback:
        /opt/openresty/nginx/conf/gumbo.lua:4: in function </opt/openresty/nginx/conf/gumbo.lua:1> while sending response to client, client: 127.0.0.1, server: localhost, request: "GET /parse.html HTTP/1.1", host: "localhost"

.

gmadarm

On Sunday, October 5, 2014 11:18:24 PM UTC+3, Lord Nynex wrote:

Likely because ngx.arg[1] is nil. What file are you trying to parse? Can you make your code a bit more explicit?

I created a test file which contains several A Href attributes and then set the document variable as follows but the error message stayed the same.
local document = assert(gumbo.parse_file("/var/www/html/form.html"))

craigbarnes85

On Sunday, 5 October 2014 21:35:11 UTC+1, Gordon Madarm wrote:

I created a test file which contains several A Href attributes and then set the document variable as follows but the error message stayed the same.
local document = assert(gumbo.parse_file("/var/www/html/form.html"))

I think Gordon has overcome these errors now, but I just thought it would be worth mentioning here that gumbo.parse_file() uses standard Lua I/O and is not to be used in async environments. I believe the correct thing to do for ngx_lua/openresty would be to use sub-requests and plain gumbo.parse().

-- Craig

agentzh

Hello!

On Mon, Oct 6, 2014 at 4:57 PM, Craig Barnes wrote:
> I think Gordon has overcome these errors now, but I just thought it would be
> worth mentioning here that gumbo.parse_file() uses standard Lua I/O and is
> not to be used in async environments. I believe the correct thing to do for
> ngx_lua/openresty would be to use sub-requests and plain gumbo.parse().
>

It's worth mentioning that file IO is almost always blocking in nginx.
So it usually makes little sense to pay the extra overhead of
ngx.location.capture() (and subrequests). Manipulating files directly
from within Lua should usually be more efficient than creating a
subrequest to the nginx standard ngx_static module. The optimization
trick here is to cache things wherever possible (including the file
handler) and reduce the number of file system calls.

Regards,
-agentzh

aapo.talvensaari

On Tuesday, October 7, 2014 7:39:05 AM UTC+3, agentzh wrote:

It's worth mentioning that file IO is almost always blocking in nginx.
So it usually makes little sense to pay the extra overhead of
ngx.location.capture() (and subrequests). Manipulating files directly
from within Lua should usually be more efficient than creating a
subrequest to the nginx standard ngx_static module. The optimization
trick here is to cache things wherever possible (including the file
handler) and reduce the number of file system calls.

I was thinking about this. The use case is SQLite. Is there any more blocking or performance hit when using SQLite with OpenResty than say with PHP (e.g. Nginx + PHP-FPM)?

Say for example with one of these:

https://github.com/ColonelThirtyTwo/lsqlite3-ffi/blob/master/lsqlite3.lua

http://scilua.org/ljsqlite3.html

We could probably use a connection pool to pool SQLite connections or just have one connection cached for each nginx worker?

SQLite is more than adequate for many loads, and can handle quite a lot of concurrency with WAL-mode (and maybe :memory: (in memory) dbs).

So, is there anything special when considering using SQLite with OpenResty compared to other platforms with SQLite support (like PHP SQLite3 and PDO SQLite support)?

Regards

Aapo

aapo.talvensaari

On Tuesday, October 7, 2014 10:30:18 PM UTC+3, Aapo Talvensaari wrote:

We could probably use a connection pool to pool SQLite connections or just have one connection cached for each nginx worker?

Some other options are documented here:

http://dev.yorhel.nl/doc/sqlaccess

Also this looks usable:

https://www.sqlite.org/sharedcache.html

It also looks that this small db is getting faster and faster in each release:

http://permalink.gmane.org/gmane.comp.db.sqlite.general/90549

agentzh

Hello!

On Tue, Oct 7, 2014 at 12:30 PM, Aapo Talvensaari wrote:
> I was thinking about this. The use case is SQLite. Is there any more
> blocking or performance hit when using SQLite with OpenResty than say with
> PHP (e.g. Nginx + PHP-FPM)?
>

It depends on the actual use patterns. You have to benchmark the real things.

And a real problem is the multi-worker model used by both nginx and
php-fpm while running SQLite. When multiple workers trying to access
the same SQLite database file at the same time. You need *heavy* locks
to ensure consistency.

BTW, you can always put a network layer atop SQLite, just like
KyotoTycoon atop KyotoCabinet.

But if you really need performance, then you should probably avoid
using SQLite in the first place :)

> We could probably use a connection pool to pool SQLite connections or just
> have one connection cached for each nginx worker?
>

AFAIK, SQLite itself does not provide socket interface. Am I wrong
here? If SQLite supports sockets, then we don't have to block at all
in OpenResty for we have cosockets :)

> SQLite is more than adequate for many loads, and can handle quite a lot of
> concurrency with WAL-mode (and maybe :memory: (in memory) dbs).
>

For the in-memory database, you still need to run SQLite in a separate
backend process otherwise you cannot share the data across different
worker processes (of either nginx or php-fpm). I don't know if you can
easily plant SQLite's database into nginx's shm though.

> So, is there anything special when considering using SQLite with OpenResty
> compared to other platforms with SQLite support (like PHP SQLite3 and PDO
> SQLite support)?
>

See above.

Regards,
-agentzh

aapo.talvensaari

On Tuesday, October 7, 2014 10:49:36 PM UTC+3, agentzh wrote:

And a real problem is the multi-worker model used by both nginx and
php-fpm while running SQLite. When multiple workers trying to access
the same SQLite database file at the same time. You need *heavy* locks
to ensure consistency.

SQLite does locking on multi-process access. Usually this is only problematic with

multiple-writers scenarios, but you can always start transactions and do you work

in a single batch. Yes, other transactions will wait, but in many apps it is not a

problem to wait for a write access for a few milliseconds (in most scenarios

usually less). SSDs help as well.

BTW, you can always put a network layer atop SQLite, just like
KyotoTycoon atop KyotoCabinet.

There seems to be async, non-blocking SQLite for Node.js:

https://github.com/mapbox/node-sqlite3

So I was thinking about having one for OpenResty as well.

But if you really need performance, then you should probably avoid
using SQLite in the first place :)

I know, but I think that SQLite is great (even better than alternatives in some scenarios, even performance-wise):

http://we-love-php.blogspot.fi/2012/08/mass-inserts-updates-sqlite-vs-mysql.html (not a great test, I know, but still)

And I have found that you can get plenty of "concurrence" in many applications that I have written.

If SQLite DB commands take say a millisecond to perform you can support 1000 writing clients

in one second. Yes, the last one need to wait one second for his turn, but that is okay in many apps

as there are usually a lot more concurrent readers than writers (and readers don't block readers, and

even writers don't block readers in WAL mode).

AFAIK, SQLite itself does not provide socket interface. Am I wrong

here? If SQLite supports sockets, then we don't have to block at all
in OpenResty for we have cosockets :)

No it doesn't support sockets (officially, yet there are many projects that

have added it). What I actually meant was caching SQLite connection

_file_ handles (e.g. Lua module level, so that single worker always uses

the same persistent SQLite connection).

... just thinking about extending OpenResty support for SQLite.

Regards

Aapo

agentzh

Hello!

On Tue, Oct 7, 2014 at 1:49 PM, Aapo Talvensaari wrote:
> There seems to be async, non-blocking SQLite for Node.js:
> https://github.com/mapbox/node-sqlite3
>

Are you sure that this library is not using the internal OS thread pool of Node?

Fundamentally one cannot use epoll/poll/etc on normal (disk) file descriptors.

> So I was thinking about having one for OpenResty as well.
>

Good luck :)

> No it doesn't support sockets (officially, yet there are many projects that
> have added it). What I actually meant was caching SQLite connection
> _file_ handles (e.g. Lua module level, so that single worker always uses
> the same persistent SQLite connection).
>

As I've said, there is no such thing as "nonblocking file IO" at least
in Linux (native AIO is another story and I don't think SQLite
supports that), so even you can cache the normal file handles, the
file IO related system calls are always blocking the current OS
thread.

If we really want to embed SQLite directly into nginx, I'm afraid we
have to add an OS thread pool at the same time (and yes, an OS thread
pool has its own overhead and complexity).

Regards,
-agentzh

aapo.talvensaari

On Wednesday, October 8, 2014 12:22:44 AM UTC+3, agentzh wrote:

As I've said, there is no such thing as "nonblocking file IO" at least
in Linux (native AIO is another story and I don't think SQLite
supports that), so even you can cache the normal file handles, the
file IO related system calls are always blocking the current OS
thread.

SQLite has some support for async io::
http://www.sqlite.org/asyncvfs.html

agentzh

Hello!

On Tue, Oct 7, 2014 at 3:31 PM, Aapo Talvensaari wrote:
> SQLite has some support for async io::
> http://www.sqlite.org/asyncvfs.html
>

To quote from this web page:

"The asynchronous I/O backend is an extension that causes SQLite to
perform all write requests using a separate thread running in the
background."

So it's just using a separate OS thread. I don't see it is using the
native AIO facility provided by the kernel [1]. Feel free to prove me
wrong :)

And you can also see why I hate how the term "Asynchronous I/O" is
abused in the wild. Using extra OS threads for blocking I/O is no
different from Apache's worker mpm and it can never be a good answer
to the C10K problem [2].

Regards,
-agentzh

[1] http://lse.sourceforge.net/io/aio.html
[2] http://www.kegel.com/c10k.html

aapo.talvensaari

On Wednesday, October 8, 2014 1:38:17 AM UTC+3, agentzh wrote:

And you can also see why I hate how the term "Asynchronous I/O" is
abused in the wild. Using extra OS threads for blocking I/O is no
different from Apache's worker mpm and it can never be a good answer
to the C10K problem [2].

Yes, fair enough.

Just thinking that would SQLite with WAL mode, with NORMAL synchronous

(or even OFF), using MMAP I/O and Shared Cache-Mode be good enough

and usable in context of OpenResty? Certainly not the answer to C10K problem.

Just something that works, something that is robust, something that can handle

reasonable amount of load (not C10K amounts of).

This question may also extend to other modules in OpenResty as well. Say for

example normal file I/O for let's say image tranformations. I'm just looking for

guidance that when you really need to do blocking io, what's the preferred way

to do it in OpenResty context, as you said here:

So it usually makes little sense to pay the extra overhead of
ngx.location.capture() (and subrequests). Manipulating files directly
from within Lua should usually be more efficient than creating a
subrequest to the nginx standard ngx_static module. The optimization
trick here is to cache things wherever possible (including the file
handler) and reduce the number of file system calls.

Non-blocking I/O is considered as an overhead, when you can cache [1] things

on a first request (something I'm looking for that lua-resty-sass module). But

sometimes you do need to have blocking reads or writes. If those take only

a small amount of time, that doesn't sound too bad, may break C10K though,

and certainly does break C10M [2] ;-)

This has been an educating conversation.

[1] http://martinfowler.com/bliki/TwoHardThings.html

[2] http://highscalability.com/blog/2013/5/13/the-secret-to-10-million-concurrent-connections-the-kernel-i.html

aapo.talvensaari

> Non-blocking I/O is considered as an overhead, when you can cache things
> on a first request (something I'm looking for that lua-resty-sass module).

Hah, hard headed guy like me using non-blocking i/o here wrong. As I now understand your guidance it doesn't really make sense to do non-blocking ngx.location.capture if in the end it just makes a request to same nginx instance that has to do blocking i/o to retrieve the file with ngx_static module. I.e. better to retrieve the file with normal Lua blocking i/o and cache the results of processing that file (maybe even warm up the cache with init_by_lua if possible).