Dynamic Script Injection

alexl · 2016-04-19T17:18:59+00:00

Hello, I think you are on the right track. I am unfamiliar with SAML message formats (detached vs enveloped signatures), however, as far as I under...

Dynamic Script Injection

alexl

Hey! I'm trying to dynamicaly inject a script in a certain place,right before </head>, into the html response.

The script can either be cached locally(memcached) or if not found must be retrieved from a given url.

For some reason, I'm unable to do both, or rather the response returns empty.

this is my script:

local memcached = require("resty.memcached")
local memcachedKey = ngx.var.host
local isScriptFromMemcached = false
local replaceScript = nil

memc, err = memcached:new()
-- Try to retrieve script from Memcached
if not memc then
ngx.log(ngx.ERR, "Failed to instantiate memc: ")
else
ngx.log(ngx.ALERT, "memc successfully instantiated")
memc:set_timeout(1000)
local ok, err = memc:connect("127.0.0.1", 11211)
if not ok then
ngx.log(ngx.ALERT, "Failed to connect to Local Memcached: ", err)
else
ngx.log(ngx.ALERT, "Connected to local memcached, key: ", memcachedKey)
local scriptMemc, flags, err = memc:get(memcachedKey)
if err then
ngx.log(ngx.ALERT, "No appropriate script found in Local Memcached: ", err)
end

if scriptMemc then
ngx.log(ngx.ALERT, "Script found in memcached")
ngx.ctx.replaceScript = scriptMemc
isScriptFromMemcached = true
else
ngx.log(ngx.ALERT, "Script not found in memcached")
end
end
end
-- If script wasn't found in Memcached - retrieve script directly from URL
if not isScriptFromMemcached then
ngx.log(ngx.ALERT, "Retrieveing sript from URL")
local http = require("socket.http")
local ltn12 = require("ltn12")
local t = {}
http.request {
url = "">
method = "GET",
sink = ltn12.sink.table(t)
}
-- Concatenete recieved chunks into string
scriptCX = table.concat(t)
ngx.log(ngx.ALERT, "Script from URL: ", scriptCX)
--ngx.var.replaceScript = scriptCX .. "</head>"
replaceScript = scriptCX
ngx.log(ngx.ALERT, "memcachedKey: ", memcachedKey)
ngx.log(ngx.ALERT, "scrip: ", scriptCX)
-- local ok, err = memc:set(memcachedKey, scrip, 900, 0)
-- if not err then ngx.log(ngx.ALERT, "Script stored in Memcached, ok: ", ok)
-- else ngx.log(ngx.ALERT, "Failed to store the script in Memcached, err: ", err)
-- end
end
memc:close()
ngx.log(ngx.ALERT, "memc closed")
in nginx.conf stayed default, except for:
location / {
default_type text/plain;
root html;
index index.html index.htm;
content_by_lua_file /home/cxadmin/work/filter.lua;
}

As you can see, I'm not yet replacing anything, but still the response is empty.
I do get the index.html page if I comment out the content_by_lua_file, so I guess there lies the problem...

Is there a better way to inject my script into the page from memcached/URL?
Any input highly apreciated!

lordnynex

Hello,

I have not completely evaluated your lua code, but you do seem to have a few issues.

First, please do not use the lua socket lib. Instead you should use ngx.socket for any socket operations. For an HTTP client I highly recommend https://github.com/pintsized/lua-resty-http as it is the most feature rich module in the ecosystem currently. This module uses ngx.socket internally to ensure non-blocking operations and uses socket pooling for better efficiency.

Second, you are confusing 'content_by_lua_*' with filtering. The content_by_lua_* directives execute in the nginx content phase and are used for generating complete HTTP responses. This would bypass any other content handlers. In this case it will bypass your 'root' and 'index' directives, which is why you see no content in your response. You need to look into a body filter. Here is an example I wrote for injecting _javascript_ in the footer section of the page just before the close of the <body> tag. https://github.com/lordnynex/CLEANUP/blob/master/GISTS/openresty-bodyfilter-example/nginx.conf.

Please pay close attention to 'eof' conditional on line 8. Nginx responses are processed in a series of buffers instead of one long buffer. Consequently, you must iterate over the response in it's entirety to ensure your filter operates on the whole response.

-Brandon

On Tue, Apr 19, 2016 at 2:18 AM, <al...@chameleonx.com> wrote:

Hey! I'm trying to dynamicaly inject a script in a certain place,right before </head>, into the html response.
The script can either be cached locally(memcached) or if not found must be retrieved from a given url.
For some reason, I'm unable to do both, or rather the response returns empty.

this is my script:
local memcached = require("resty.memcached")
local memcachedKey = ngx.var.host
local isScriptFromMemcached = false
local replaceScript = nil

memc, err = memcached:new()
-- Try to retrieve script from Memcached
if not memc then
ngx.log(ngx.ERR, "Failed to instantiate memc: ")
else
ngx.log(ngx.ALERT, "memc successfully instantiated")
memc:set_timeout(1000)
local ok, err = memc:connect("127.0.0.1", 11211)
if not ok then
ngx.log(ngx.ALERT, "Failed to connect to Local Memcached: ", err)
else
ngx.log(ngx.ALERT, "Connected to local memcached, key: ", memcachedKey)
local scriptMemc, flags, err = memc:get(memcachedKey)
if err then
ngx.log(ngx.ALERT, "No appropriate script found in Local Memcached: ", err)
end

if scriptMemc then
ngx.log(ngx.ALERT, "Script found in memcached")
ngx.ctx.replaceScript = scriptMemc
isScriptFromMemcached = true
else
ngx.log(ngx.ALERT, "Script not found in memcached")
end
end
end
-- If script wasn't found in Memcached - retrieve script directly from URL
if not isScriptFromMemcached then
ngx.log(ngx.ALERT, "Retrieveing sript from URL")
local http = require("socket.http")
local ltn12 = require("ltn12")
local t = {}
http.request {
url = "" href="http://myusrl.com/testscript.txt" target="_blank">http://myusrl.com/testscript.txt",
method = "GET",
sink = ltn12.sink.table(t)
}
-- Concatenete recieved chunks into string
scriptCX = table.concat(t)
ngx.log(ngx.ALERT, "Script from URL: ", scriptCX)
--ngx.var.replaceScript = scriptCX .. "</head>"
replaceScript = scriptCX
ngx.log(ngx.ALERT, "memcachedKey: ", memcachedKey)
ngx.log(ngx.ALERT, "scrip: ", scriptCX)
-- local ok, err = memc:set(memcachedKey, scrip, 900, 0)
-- if not err then ngx.log(ngx.ALERT, "Script stored in Memcached, ok: ", ok)
-- else ngx.log(ngx.ALERT, "Failed to store the script in Memcached, err: ", err)
-- end
end
memc:close()
ngx.log(ngx.ALERT, "memc closed")
in nginx.conf stayed default, except for:
location / {
default_type text/plain;
root html;
index index.html index.htm;
content_by_lua_file /home/cxadmin/work/filter.lua;
}

As you can see, I'm not yet replacing anything, but still the response is empty.
I do get the index.html page if I comment out the content_by_lua_file, so I guess there lies the problem...

Is there a better way to inject my script into the page from memcached/URL?
Any input highly apreciated!

.

alexl

Thanks for the suggestion!

In body_filter_by_lua I'm able to download script from a url and insert it like so:

whole = string.gsub(whole, "</head>", replaceScript)
ngx.arg[1] = whole

where "whole" holds the full response body.

However, cosocket API is not available in body_filter_by_lua, meaning that I can't access memcached from here.

This is what forced me to try and do this whole operation from the content_by_lua instead.

I have also found this: http://stackoverflow.com/questions/25639445/nginx-caching-response-from-remote-server-with-memcached

It sounds close to what I need to do, so I'm gonna try to implement the memcached-part as the answear suggested(in the nginx.conf file) and maybe store the value in a variable to then pass it to body_filter_by_lua to do the insertion.

Makes sense?

On Tuesday, April 19, 2016 at 7:13:43 PM UTC+3, Lord Nynex wrote:

Hello,

I have not completely evaluated your lua code, but you do seem to have a few issues.

First, please do not use the lua socket lib. Instead you should use ngx.socket for any socket operations. For an HTTP client I highly recommend https://github.com/pintsized/lua-resty-http as it is the most feature rich module in the ecosystem currently. This module uses ngx.socket internally to ensure non-blocking operations and uses socket pooling for better efficiency.

Second, you are confusing 'content_by_lua_*' with filtering. The content_by_lua_* directives execute in the nginx content phase and are used for generating complete HTTP responses. This would bypass any other content handlers. In this case it will bypass your 'root' and 'index' directives, which is why you see no content in your response. You need to look into a body filter. Here is an example I wrote for injecting _javascript_ in the footer section of the page just before the close of the <body> tag. https://github.com/lordnynex/CLEANUP/blob/master/GISTS/openresty-bodyfilter-example/nginx.conf.

Please pay close attention to 'eof' conditional on line 8. Nginx responses are processed in a series of buffers instead of one long buffer. Consequently, you must iterate over the response in it's entirety to ensure your filter operates on the whole response.

-Brandon

On Tue, Apr 19, 2016 at 2:18 AM, <al...@chameleonx.com> wrote:
Hey! I'm trying to dynamicaly inject a script in a certain place,right before </head>, into the html response.
The script can either be cached locally(memcached) or if not found must be retrieved from a given url.
For some reason, I'm unable to do both, or rather the response returns empty.

this is my script:
local memcached = require("resty.memcached")
local memcachedKey = ngx.var.host
local isScriptFromMemcached = false
local replaceScript = nil

memc, err = memcached:new()
-- Try to retrieve script from Memcached
if not memc then
ngx.log(ngx.ERR, "Failed to instantiate memc: ")
else
ngx.log(ngx.ALERT, "memc successfully instantiated")
memc:set_timeout(1000)
local ok, err = memc:connect("127.0.0.1", 11211)
if not ok then
ngx.log(ngx.ALERT, "Failed to connect to Local Memcached: ", err)
else
ngx.log(ngx.ALERT, "Connected to local memcached, key: ", memcachedKey)
local scriptMemc, flags, err = memc:get(memcachedKey)
if err then
ngx.log(ngx.ALERT, "No appropriate script found in Local Memcached: ", err)
end

if scriptMemc then
ngx.log(ngx.ALERT, "Script found in memcached")
ngx.ctx.replaceScript = scriptMemc
isScriptFromMemcached = true
else
ngx.log(ngx.ALERT, "Script not found in memcached")
end
end
end
-- If script wasn't found in Memcached - retrieve script directly from URL
if not isScriptFromMemcached then
ngx.log(ngx.ALERT, "Retrieveing sript from URL")
local http = require("socket.http")
local ltn12 = require("ltn12")
local t = {}
http.request {
url = "" href="http://myusrl.com/testscript.txt" target="_blank" rel="nofollow" _onmousedown_="this.href="" true;" _onclick_="this.href='http://www.google.com/url?q\75http%3A%2F%2Fmyusrl.com%2Ftestscript.txt\46sa\75D\46sntz\0751\46usg\75AFQjCNENy3JVtN9jqCg3fhPocVzgihJV_Q';return true;">http://myusrl.com/testscript.txt",
method = "GET",
sink = ltn12.sink.table(t)
}
-- Concatenete recieved chunks into string
scriptCX = table.concat(t)
ngx.log(ngx.ALERT, "Script from URL: ", scriptCX)
--ngx.var.replaceScript = scriptCX .. "</head>"
replaceScript = scriptCX
ngx.log(ngx.ALERT, "memcachedKey: ", memcachedKey)
ngx.log(ngx.ALERT, "scrip: ", scriptCX)
-- local ok, err = memc:set(memcachedKey, scrip, 900, 0)
-- if not err then ngx.log(ngx.ALERT, "Script stored in Memcached, ok: ", ok)
-- else ngx.log(ngx.ALERT, "Failed to store the script in Memcached, err: ", err)
-- end
end
memc:close()
ngx.log(ngx.ALERT, "memc closed")
in nginx.conf stayed default, except for:
location / {
default_type text/plain;
root html;
index index.html index.htm;
content_by_lua_file /home/cxadmin/work/filter.lua;
}

As you can see, I'm not yet replacing anything, but still the response is empty.
I do get the index.html page if I comment out the content_by_lua_file, so I guess there lies the problem...

Is there a better way to inject my script into the page from memcached/URL?
Any input highly apreciated!

.

alexl

Consider this question solved! The solution for the future generations:

In the end I have simply avoided using memcached inside lua and instead I have this in nginx.conf:

location /memc_get {

internal;

set $memc_cmd 'get';

set $memc_key $arg_key;

memc_pass 127.0.0.1:11211;

}

and I then from within access_by_lua I do:

local memc_request = "/memc_get?key=" .. ngx.var.host

local res = ngx.location.capture(memc_request)

save res.body into global varible $memcScript and then I can access it anywhere I need through either ngx.var.memcScript or $memcScript.

agentzh

Hello!

On Mon, Apr 25, 2016 at 5:57 AM,  <al...@chameleonx.com> wrote:
> and I then from within access_by_lua I do:
>     local memc_request = "/memc_get?key=" .. ngx.var.host
>     local res = ngx.location.capture(memc_request)
>

BTW, lua-resty-memcached is preferred over ngx.location.capture +
ngx_memc for both performance and flexibility.

Regards,
-agentzh

alexl

I will look into it, thanks!

On Wednesday, April 27, 2016 at 2:57:14 AM UTC+3, agentzh wrote:

Hello!

On Mon, Apr 25, 2016 at 5:57 AM, <al...@chameleonx.com> wrote:
> and I then from within access_by_lua I do:
> local memc_request = "/memc_get?key=" .. ngx.var.host
> local res = ngx.location.capture(memc_request)
>

BTW, lua-resty-memcached is preferred over ngx.location.capture +
ngx_memc for both performance and flexibility.

Regards,
-agentzh