API disabled in the context of (unknown)

gmadarm · 2014-10-20T22:41:11+00:00

I imagine LuaJIT, with calls to C/C++ via FFI, is sufficient for the vast majority of workloads you'd use with Openresty.However, on the bright side, to my u...

API disabled in the context of (unknown)

gmadarm

I have Openresty configured as a reverse proxy and need to modify some of the proxied HTML. I'm using lua-gumbo to parse and modify the HTML. When developing the script I created a local variable containing the HTML code and was able to modify the code properly. When I updated the code to parse the proxied HTML via ngx.arg[1] and use body_filter_by_lua_file the code stopped working. I understand that body_filter_by_lua has certain API function limitations in this phase. What workarounds can I implement to run this code and modify proxied HTML on the fly?

My source code and subsequent error message are below:

local gumbo = require "gumbo"

local document = assert(gumbo.parse(ngx.arg[1]))
local Links = document:getElementsByTagName("a")

for i, element in ipairs(Links) do
    local Attr = element:getAttribute("href")
    if Attr then
        element:setAttribute("href", UpdatedHrefValue)
    end
end

ngx.say(document.documentElement.outerHTML, "\n")

====================================error.log===============================================================================
2014/10/20 17:04:30 [error] 19449#0: *67 failed to run body_filter_by_lua*: coroutine.wrap:15: API disabled in the context of (unknown)
stack traceback:
        [C]: in function 'create'
        coroutine.wrap:15: in function 'walk'
        /opt/openresty/lualib/gumbo/dom/Element.lua:50: in function 'getElementsByTagName'
        /opt/openresty/nginx/conf/abc.lua:30: in function 'main'
        /opt/openresty/nginx/conf/abc.lua:44: in function </opt/openresty/nginx/conf/abc.lua:1>, client: 127.0.0.1, server: localhost, request: "GET / HTTP/1.1", host: "localhost"
2014/10/20 17:04:30 [error] 19449#0: *67 failed to run body_filter_by_lua*: coroutine.wrap:15: API disabled in the context of (unknown)
stack traceback:
        [C]: in function 'create'
        coroutine.wrap:15: in function 'walk'
        /opt/openresty/lualib/gumbo/dom/Element.lua:50: in function 'getElementsByTagName'
        /opt/openresty/nginx/conf/abc.lua:30: in function 'main'
        /opt/openresty/nginx/conf/abc.lua:44: in function </opt/openresty/nginx/conf/abc.lua:1>, client: 127.0.0.1, server: localhost, request: "GET / HTTP/1.1", host: "localhost"

agentzh

Hello!

On Mon, Oct 20, 2014 at 7:41 AM, Gordon Madarm wrote:
> local gumbo = require "gumbo"
>
> local document = assert(gumbo.parse(ngx.arg[1]))
> local Links = document:getElementsByTagName("a")
>

I wonder if the gumbo library you're using supports parsing HTML
stream by data chunks because nginx's body filter is invoked upon each
individual data chunk with arbitrary boundaries:

https://github.com/openresty/lua-nginx-module#body_filter_by_lua

"Nginx output filters may be called multiple times for a single
request because response body may be delivered in chunks. Thus, the
Lua code specified by in this directive may also run multiple times in
the lifetime of a single HTTP request."

If it does not, then this approach won't fly.

> 2014/10/20 17:04:30 [error] 19449#0: *67 failed to run body_filter_by_lua*:
> coroutine.wrap:15: API disabled in the context of (unknown)
> stack traceback:
>         [C]: in function 'create'
>         coroutine.wrap:15: in function 'walk'

Seems like you're using an old version of ngx_lua (or OpenResty) where
the "coroutine" API is disabled. This API is already enabled in
ngx_lua 0.9.12 (or OpenResty 1.7.4.1). The following test case works
for me with the latest ngx_lua:

    location = /t {
        echo ok;
        header_filter_by_lua '
            local function f()
                print("hello")
            end
            local g = coroutine.wrap(f)
            g()
        ';
    }

> What workarounds can I implement to run this code and modify proxied HTML on the fly?
>

At CloudFlare we are currently using Ragel [1] -generated C code in an
nginx output filter C module to do such things (as well as for CSS and
etc). I'm going to migrate it to a more flexible solution based on
body_filter_by_lua and the sregex library [2]. And yeah, sregex itself
rquires significant optimizations because the focus there has been on
correctness and stability :)

BTW, you may also find this ngx_replace_filter module useful though:

https://github.com/openresty/replace-filter-nginx-module

This nginx C module is based on sregex, the streaming regex engine :)

Regards,
-agentzh

[1] http://www.colm.net/open-source/ragel/
[2] https://github.com/openresty/sregex

craigbarnes85

On Monday, 20 October 2014 20:36:54 UTC+1, agentzh wrote:

I wonder if the gumbo library you're using supports parsing HTML
stream by data chunks because nginx's body filter is invoked upon each
individual data chunk with arbitrary boundaries:
...
If it does not, then this approach won't fly.

It doesn't. libgumbo produces a tree of pointers back into the input buffer, which has to live fully in memory for at least as long as the tree is alive.

Netsurf's Hubbub[1] library supports chunked parsing but the API isn't very Lua binding friendly.

[1]: http://www.netsurf-browser.org/projects/hubbub/

gmadarm

On Monday, October 20, 2014 10:36:54 PM UTC+3, agentzh wrote:

> What workarounds can I implement to run this code and modify proxied HTML on the fly?
>

At CloudFlare we are currently using Ragel [1] -generated C code in an
nginx output filter C module to do such things (as well as for CSS and
etc). I'm going to migrate it to a more flexible solution based on
body_filter_by_lua and the sregex library [2]. And yeah, sregex itself
rquires significant optimizations because the focus there has been on
correctness and stability :)

BTW, you may also find this ngx_replace_filter module useful though:

https://github.com/openresty/replace-filter-nginx-module

Hi,

My use case requires capturing part of the regex and replacing it with an nginx variable. AFAICT this is not possible with replace-filter-nginx-module. What advantages does replace-filter-nginx-module have over using ngx.re.gmatch and ngx.re.gmatch?

thanks,
- G

gmadarm

On Monday, October 20, 2014 11:45:33 PM UTC+3, Craig Barnes wrote:

On Monday, 20 October 2014 20:36:54 UTC+1, agentzh wrote:
I wonder if the gumbo library you're using supports parsing HTML
stream by data chunks because nginx's body filter is invoked upon each
individual data chunk with arbitrary boundaries:
...
If it does not, then this approach won't fly.

It doesn't. libgumbo produces a tree of pointers back into the input buffer, which has to live fully in memory for at least as long as the tree is alive.

Hi Craig,

Assuming there is no handler that has access to the full response body, would appending each data chunk to a variable or shared dictionary variable and then parse that with lua-gumbo once the entire response has been buffered be a sane work around?

Netsurf's Hubbub[1] library supports chunked parsing but the API isn't very Lua binding friendly.

[1]: http://www.netsurf-browser.org/projects/hubbub/

On Monday, October 20, 2014 11:45:33 PM UTC+3, Craig Barnes wrote:

On Monday, 20 October 2014 20:36:54 UTC+1, agentzh wrote:
I wonder if the gumbo library you're using supports parsing HTML
stream by data chunks because nginx's body filter is invoked upon each
individual data chunk with arbitrary boundaries:
...
If it does not, then this approach won't fly.

It doesn't. libgumbo produces a tree of pointers back into the input buffer, which has to live fully in memory for at least as long as the tree is alive.

Netsurf's Hubbub[1] library supports chunked parsing but the API isn't very Lua binding friendly.

[1]: http://www.netsurf-browser.org/projects/hubbub/

agentzh

Hello!

On Tue, Oct 21, 2014 at 12:29 PM, Gordon Madarm wrote:
>
> My use case requires capturing part of the regex and replacing it with an
> nginx variable. AFAICT this is not possible with
> replace-filter-nginx-module.

Yes, supports for nginx variables in the regex patterns are still on
ngx_replace_filter's TODO list :)

> What advantages does
> replace-filter-nginx-module have over using ngx.re.gmatch and ngx.re.gmatch?
>

ngx_replace_filter uses a (small) constant memory footprint no matter
how big the response body is. For ngx.re.gmatch and etc, you need to
buffer the whole response body in memory before matching, which is way
more expensive than streaming processing.

Also, the ngx.re API uses a PCRE engine so in bad cases it will
exhaust your CPU resources while PCRE is crazy backtracking).

Streaming parsing is very useful and can be very efficient but is also
hard to implement under the hood at the same time :)

Regards,
-agentzh

craigbarnes85

On 22 October 2014 03:49, Yichun Zhang (agentzh) <age...@gmail.com> wrote:
> Streaming parsing is very useful and can be very efficient but is also
> hard to implement under the hood at the same time :)
>

To implement HTML5 parsing properly, you have to handle various
re-parenting/adoption/recovery cases that are inherently
non-streamable. Hubbub does chunked/resumable parsing, but it can't do
truly streaming, SAX-style parsing -- nor can any HTML5 conforming
parser.

This is probably the reason libxml2 still doesn't have a real HTML5
mode. It's also the reason why the validator.nu[1] parser pretends to
be "streaming" for API compatibility, while actually buffering
everything anyway, because the constraints of the spec require it. To
enable true streaming mode, you have to make all errors fatal, which
is a violation of the spec in itself.

Obviously XML/XHTML5 is streamable, but if you're proxying someone
else's site, you must parse whatever they provide with the appropriate
parser. If you're mutating HTML that will be forwarded on to a web
browser, the appropriate parser is a HTML5 parser.

[1]: http://about.validator.nu/htmlparser/apidocs/nu/validator/htmlparser/sax/HtmlParser.html