FFI bindings for libinjection

rpaprocki · 2016-04-18T05:45:44+00:00

Hello! On Mon, Apr 11, 2016 at 10:13 PM, aifei wrote: > I see that you set all *.md files of it's status "considered > experimental&quot...

FFI bindings for libinjection

rpaprocki

Hello!

I'd recently been looking at libinjection (https://github.com/client9/libinjection/), as well as its various implementations. I wasn't able to find any Lua bindings for this library from the community or other sources, so I've hacked together a first-pass attempts at bindings here:

https://github.com/p0pr0ck5/lua-resty-libinjection

I haven't worked closely with FFI before, so I would love some feedback from anyone with more experience if I've done anything shamefully misguided :p There's no complex memory management to worry about (libinjection uses a single on-stack struct for its analysis), and performance testing with this lib looks solid; a simple test shows < 10 us execution time using ngx-lua-exec-time.sxx. Hopefully someone can get some use out of this!

lordnynex

Hello,

Thanks for sharing this. This is a really great addition to the openresty/lua ecosystem. You may want to have a look at some of the available WAF modules that will provide some expanded functionality outside the scope of libinjection. Also an interesting article from cloudflare on using agentzh's systemtap tools to profile WAF performance at scale https://blog.cloudflare.com/cloudflares-new-waf-compiling-to-lua/. You may want to explore using the no-pool-patch (provided with the openresty bundle) to use valgrind to check for any leakage (it doesn't appear likely in your lib).

Admittedly my FFI skills are not great either so take this with a grain of salt.

1) I'm never quite sure on the best design practices here. Since it seems libinjection maintains state in a single struct, it's allocation/deallocation lifecycle will be closer aligned with the lifecycle of the server itself, so I think it's ok. In other types of projects where there is per request or per call allocation, I think closer integration with GC is desirable to avoid leaks. Suffice to say this line is just noise, it does not seem to apply to your module.

2) It's generally bad form to include the compiled lib in your module. Currently the norm is to state that 'libinjection' is a dependency of this module and place the burden of installation on the user. If the C code is small it can be included directly for a more 'vendorized' type of layout. Please see https://github.com/keplerproject/luarocks/wiki/Creating-a-rock#C_modules_linking_to_external_libraries for instructions or ideas about including/linking compiled libs at module build time.

Other similar modules you may draw inspiration from:

- FFI Bindings to imagemagick https://github.com/leafo/magick

- FFI Bindings for libnettle https://github.com/bungle/lua-resty-nettle

- Various FFI bindings by agentzh himself for various things. This ships with core and I often use it as a reference when FFI is invovled. https://github.com/openresty/lua-resty-core

If you manage to separate your code from shipping a compiled shared object (that is likely platform dependent), I would be interested in evaluating your module. Until then, I will not load stange compiled code :)

-Brandon

On Sun, Apr 17, 2016 at 2:45 PM, Robert Paprocki <rpap...@fearnothingproductions.net> wrote:

Hello!

I'd recently been looking at libinjection (https://github.com/client9/libinjection/), as well as its various implementations. I wasn't able to find any Lua bindings for this library from the community or other sources, so I've hacked together a first-pass attempts at bindings here:

https://github.com/p0pr0ck5/lua-resty-libinjection

I haven't worked closely with FFI before, so I would love some feedback from anyone with more experience if I've done anything shamefully misguided :p There's no complex memory management to worry about (libinjection uses a single on-stack struct for its analysis), and performance testing with this lib looks solid; a simple test shows < 10 us execution time using ngx-lua-exec-time.sxx. Hopefully someone can get some use out of this!

.

rpaprocki

Thanks for the feedback! One of the things I've really enjoyed about working on lua-resty-* projects is the great feedback that the community provides. :)

On Tue, Apr 19, 2016 at 9:44 AM, Lord Nynex <lord...@gmail.com> wrote:

Hello,

Thanks for sharing this. This is a really great addition to the openresty/lua ecosystem. You may want to have a look at some of the available WAF modules that will provide some expanded functionality outside the scope of libinjection.

Yes :) I maintain https://github.com/p0pr0ck5/lua-resty-waf, and I'm looking at getting libinjection implemented to provide a translation for ModSecurity's @detectsqli and @detectxss operators.

Also an interesting article from cloudflare on using agentzh's systemtap tools to profile WAF performance at scale https://blog.cloudflare.com/cloudflares-new-waf-compiling-to-lua/. You may want to explore using the no-pool-patch (provided with the openresty bundle) to use valgrind to check for any leakage (it doesn't appear likely in your lib).

Yep :) flamegraphs of lua stacks for lua-resty-libinjection are pretty boring, as it's essentially a direct wrapper. Haven't poked at the no-pool-patch, mostly because I'm not overly familiar with nginx memory management (welcome to the world's worst documented popular project :p), and I'l like to dive farther into it.

2) It's generally bad form to include the compiled lib in your module. Currently the norm is to state that 'libinjection' is a dependency of this module and place the burden of installation on the user. If the C code is small it can be included directly for a more 'vendorized' type of layout. Please see https://github.com/keplerproject/luarocks/wiki/Creating-a-rock#C_modules_linking_to_external_libraries for instructions or ideas about including/linking compiled libs at module build time.

Makes sense. One of the projects I modeled the FFI code on is https://github.com/cloudflare/lua-aho-corasick, which provides the source for its C shared lib directly as part of the project. I'll figure out a better way to link the libinjection source as part of the lua resty lib. It's worth noting that right now libinjection doesn't provide a .so file by default (only a static lib, see https://github.com/client9/libinjection/pull/103).

If you manage to separate your code from shipping a compiled shared object (that is likely platform dependent), I would be interested in evaluating your module. Until then, I will not load stange compiled code :)

Fair nuf ;) I'd encourage people to build libinjection from source on their own and go from there (libinjection has essentially no dependencies, which is nice). Since it tries to load libinjection.so from anywhere it can find in package.cpath, it'd be up to the user to place the lib in the right spot, meaning you don't have to use the bundled version (it's just there for convenience, but I'll yank it out).

lordnynex

Since libinjection produces a statically linked lib, you do have an (in my opinion) interesting ability to compile it directly into openresty (the same for your lua modules compiled byte code). This does not enhance the usability of your library from a users perspective, but I am personally interested dependency free builds.

Since the API for libinjection appears to be pretty simple, I encourage you to explore creating a full fledged nginx module for it (which interestingly supports DSO in recent versions). In this way, your module can be consumed as a an nginx shared object (that can include libinjection via the linker at build time). Because of the nature of your module I think building this directly into nginx certainly has attractive benefits.

Regarding the so in SCM, if you have a look at the leafo's magick module, you'll see that he's making a subshell call to pkg-config to detect the location of the lib. It's not the most error proof solution, but I appreciate that it gets the job done. (https://github.com/leafo/magick/blob/master/magick/wand/lib.moon#L88)

- Brandon

On Tue, Apr 19, 2016 at 10:04 AM, Robert Paprocki <rpap...@fearnothingproductions.net> wrote:

Thanks for the feedback! One of the things I've really enjoyed about working on lua-resty-* projects is the great feedback that the community provides. :)

On Tue, Apr 19, 2016 at 9:44 AM, Lord Nynex <lord...@gmail.com> wrote:
Hello,

Thanks for sharing this. This is a really great addition to the openresty/lua ecosystem. You may want to have a look at some of the available WAF modules that will provide some expanded functionality outside the scope of libinjection.

Yes :) I maintain https://github.com/p0pr0ck5/lua-resty-waf, and I'm looking at getting libinjection implemented to provide a translation for ModSecurity's @detectsqli and @detectxss operators.

Also an interesting article from cloudflare on using agentzh's systemtap tools to profile WAF performance at scale https://blog.cloudflare.com/cloudflares-new-waf-compiling-to-lua/. You may want to explore using the no-pool-patch (provided with the openresty bundle) to use valgrind to check for any leakage (it doesn't appear likely in your lib).

Yep :) flamegraphs of lua stacks for lua-resty-libinjection are pretty boring, as it's essentially a direct wrapper. Haven't poked at the no-pool-patch, mostly because I'm not overly familiar with nginx memory management (welcome to the world's worst documented popular project :p), and I'l like to dive farther into it.

2) It's generally bad form to include the compiled lib in your module. Currently the norm is to state that 'libinjection' is a dependency of this module and place the burden of installation on the user. If the C code is small it can be included directly for a more 'vendorized' type of layout. Please see https://github.com/keplerproject/luarocks/wiki/Creating-a-rock#C_modules_linking_to_external_libraries for instructions or ideas about including/linking compiled libs at module build time.

Makes sense. One of the projects I modeled the FFI code on is https://github.com/cloudflare/lua-aho-corasick, which provides the source for its C shared lib directly as part of the project. I'll figure out a better way to link the libinjection source as part of the lua resty lib. It's worth noting that right now libinjection doesn't provide a .so file by default (only a static lib, see https://github.com/client9/libinjection/pull/103).

If you manage to separate your code from shipping a compiled shared object (that is likely platform dependent), I would be interested in evaluating your module. Until then, I will not load stange compiled code :)

Fair nuf ;) I'd encourage people to build libinjection from source on their own and go from there (libinjection has essentially no dependencies, which is nice). Since it tries to load libinjection.so from anywhere it can find in package.cpath, it'd be up to the user to place the lib in the right spot, meaning you don't have to use the bundled version (it's just there for convenience, but I'll yank it out).

.