Nginx+Lua

alexl · 2016-03-27T03:03:30+00:00

Chris,Can you paste your config and what exact steps you're taking to better describe the issue? Simply saying "it doesn't work" won't help as we have no way...

Nginx+Lua

alexl

Hello!

I was trying to get to know Openresty, as it would possibly be required on my job, but ran into a problem. I need to be able to lookup text value from memcached(by the domain) and, if not found, retrieve it from specified location(url). Then I want to replace some text in the page(the response body) with the said string from memcached or downloaded.

I have two scripts in lua, one I call from content_by_lua and other from the body_filter_by_lua in nginx.conf file within the same location. I also use nginx variable to pass a string from one script to another. My problem is: I can't use memcached inside of body_filter_by_lua and I can't use ngx.arg API in the content_by_lua, but when I use both the response I get is just an empty page.

Logs are all clean.

Found no info on the issue, hope there is some kind of workaround!

Thanks in advance!

rpaprocki

Can you paste your code and minimal config to show a reproducible example of your problem?

On Sat, Mar 26, 2016 at 12:03 PM, Alexander Litvak <al...@chameleonx.com> wrote:

Hello!

I was trying to get to know Openresty, as it would possibly be required on my job, but ran into a problem. I need to be able to lookup text value from memcached(by the domain) and, if not found, retrieve it from specified location(url). Then I want to replace some text in the page(the response body) with the said string from memcached or downloaded.

I have two scripts in lua, one I call from content_by_lua and other from the body_filter_by_lua in nginx.conf file within the same location. I also use nginx variable to pass a string from one script to another. My problem is: I can't use memcached inside of body_filter_by_lua and I can't use ngx.arg API in the content_by_lua, but when I use both the response I get is just an empty page.

Logs are all clean.

Found no info on the issue, hope there is some kind of workaround!

Thanks in advance!

.

alexl

On Saturday, March 26, 2016 at 10:16:59 PM UTC+3, rpaprocki wrote:

Can you paste your code and minimal config to show a reproducible example of your problem?

Absolutely!

Nginx.conf part:

location / {
root html;
index index.html index.htm;
     set $replaceContent '';
  content_by_lua_file /home/cxadmin/work/fromMemcached.lua;
  body_filter_by_lua_file /home/cxadmin/work/fromUrlReplace.lua;
}

Everything else in .conf stayed default.

fromMemcached.lua :

-- Load Memcached module

local memcached = require("resty.memcached")
local memcachedKey = ngx.var.host
-- Instantiate Memcached object
local memc, err = memcached:new()
if not memc then
ngx.log(ngx.ERR, "Failed to instantiate memc")
else
ngx.log(ngx.ERR, "memc successfully instantiated")
-- Set Memcached timeout
memc:set_timeout(1000)
-- Connect to Local Memcached
local ok, err = memc:connect("10.0.0.4", 11211)
if not ok then
ngx.log(ngx.ALERT, "Failed to connect to Local Memcached: ", err)
else
ngx.log(ngx.ERR, "Connected to local memcached")
-- Try to retrieve from Memcached
fromMemc, flags, err = memc:get(memcachedKey)
if err then
ngx.log(ngx.NOTICE, "Not found in Local Memcached: ", err)
end
if not fromMemc then
memc:close()
ngx.log(ngx.ERR, "memc closed")
return
end
ngx.log(ngx.NOTICE, "Value found and passed to $replaceContent", fromMemc)
ngx.var.replaceContent = fromMemc
memc:close()
ngx.log(ngx.ERR, "memc closed")
end
end

fromUrlReplace.lua:

if not ngx.var.replaceContent then
ngx.log(ngx.NOTICE, "Retrieveing sript from URL")
-- Key will later be more complex
local http = require("socket.http")
local ltn12 = require("ltn12")
-- Request from URL
local t = {}
http.request {
url = ""> method = "GET",
sink = ltn12.sink.table(t)
}
-- Concatenete recieved chunks into string
dtext = table.concat(t)
replaceString = dtext .. "</head>"
else
ngx.log(ngx.NOTICE, "Using content from Memcached")
replaceString = ngx.var.replaceContent .. "</head>"
end
ngx.log(ngx.NOTICE, "replace string: ", replaceString)
ngx.arg[1] = ngx.re.sub(ngx.arg[1],"</head>", replaceString)

Everything is pretty much straight forward. fromMemcached.lua looks the value up in the memcached, fromUrlReplace.lua tries to download the data from a url if it wasn't found in memcached and injects it into the response body.

alexl

On Saturday, March 26, 2016 at 10:16:59 PM UTC+3, rpaprocki wrote:
Can you paste your code and minimal config to show a reproducible example of your problem?

Absolutely!
Nginx.conf part:
location / {
root html;
index index.html index.htm;
     set $replaceContent '';
  content_by_lua_file /home/cxadmin/work/fromMemcached.lua;
  body_filter_by_lua_file /home/cxadmin/work/fromUrlReplace.lua;
}
Everything else in .conf stayed default.

fromMemcached.lua :
-- Load Memcached module
local memcached = require("resty.memcached")
local memcachedKey = ngx.var.host
-- Instantiate Memcached object
local memc, err = memcached:new()
if not memc then
ngx.log(ngx.ERR, "Failed to instantiate memc")
else
ngx.log(ngx.ERR, "memc successfully instantiated")
-- Set Memcached timeout
memc:set_timeout(1000)
-- Connect to Local Memcached
local ok, err = memc:connect("10.0.0.4", 11211)
if not ok then
   ngx.log(ngx.ALERT, "Failed to connect to Local Memcached: ", err)
else
ngx.log(ngx.ERR, "Connected to local memcached")
-- Try to retrieve from Memcached
fromMemc, flags, err = memc:get(memcachedKey)
if err then
   ngx.log(ngx.NOTICE, "Not found in Local Memcached: ", err)
end
if not fromMemc then
memc:close()
ngx.log(ngx.ERR, "memc closed")
return
end
ngx.log(ngx.NOTICE, "Value found and passed to $replaceContent", fromMemc)
ngx.var.replaceContent = fromMemc
memc:close()
ngx.log(ngx.ERR, "memc closed")
end
end

fromUrlReplace.lua:
if not ngx.var.replaceContent then
ngx.log(ngx.NOTICE, "Retrieveing sript from URL")
-- Key will later be more complex
local http = require("socket.http")
local ltn12 = require("ltn12")
-- Request from URL
local t = {}
http.request {
url = "" href="https://s3-us-west-1.amazonaws.com/cxeasylistcust/luatest.txt" target="_blank" rel="nofollow" style="cursor: pointer;">https://s3-us-west-1.amazonaws.com/cxeasylistcust/luatest.txt",
method = "GET",
sink = ltn12.sink.table(t)
}
-- Concatenete recieved chunks into string
dtext = table.concat(t)
replaceString = dtext .. "</head>"
else
ngx.log(ngx.NOTICE, "Using content from Memcached")
replaceString = ngx.var.replaceContent .. "</head>"
end
ngx.log(ngx.NOTICE, "replace string: ", replaceString)
ngx.arg[1] = ngx.re.sub(ngx.arg[1],"</head>", replaceString)

Everything is pretty much straight forward. fromMemcached.lua looks the value up in the memcached, fromUrlReplace.lua tries to download the data from a url if it wasn't found in memcached and injects it into the response body.

Also I should probably mention I work on Ubuntu 14.04, just in case it matters

itamar

You could fetch the data in the content_by_lua / access_by_lua stages and store it in the request context via the ngx.ctx.* table, and use it later on when needed.

Also, instead of making the replacements in the response body in lua-land using *_filter_by_lua*, I suggest checking out the excellent replace-filter nginx module, which uses the sregex library (both by agentzh).

In case you need to use the matching result groups in your replacements, you may want to do that by setting an ngx.var.* in lua (since passing dollar signs in the nginx config file is no fun).

Best of luck!

-Itamar

On Sat, Mar 26, 2016 at 10:03 PM, Alexander Litvak <al...@chameleonx.com> wrote:

Hello!

I was trying to get to know Openresty, as it would possibly be required on my job, but ran into a problem. I need to be able to lookup text value from memcached(by the domain) and, if not found, retrieve it from specified location(url). Then I want to replace some text in the page(the response body) with the said string from memcached or downloaded.

I have two scripts in lua, one I call from content_by_lua and other from the body_filter_by_lua in nginx.conf file within the same location. I also use nginx variable to pass a string from one script to another. My problem is: I can't use memcached inside of body_filter_by_lua and I can't use ngx.arg API in the content_by_lua, but when I use both the response I get is just an empty page.

Logs are all clean.

Found no info on the issue, hope there is some kind of workaround!

Thanks in advance!

.

alexl

Thanks for the advice!

For some reason I now only get empty page in response even with replace_filter marked out.

However adding ngx.say("Something") at the bottom of the script does return Something in response...

It seems I'm touching something I don't suppose to. Do I?

Now the .conf portion looks like this:

location / {
resolver 8.8.8.8;
root html;
index index.html index.htm;
content_by_lua_file '/home/cxadmin/Work/filter.lua';
replace_filter '</head>' 'ngx.ctx.replaceScript' ig;
}

filter.lua:

local memcached = require("resty.memcached")
local memcachedKey = ngx.var.host
ngx.log(ngx.ALERT, "memcachedKey: ", memcachedKey)
local isScriptFromMemcached = false
memc, err = memcached:new()
-- Try to retrieve script from Memcached
if not memc then
ngx.log(ngx.ERR, "Failed to instantiate memc: ")
else
ngx.log(ngx.ALERT, "memc successfully instantiated")
memc:set_timeout(1000)
local ok, err = memc:connect("localhost", 11211)
if not ok then
ngx.log(ngx.ERR, "Failed to connect to Local Memcached: ", err)
else
ngx.log(ngx.ALERT, "Connected to local memcached, key: ", memcachedKey)
local scriptMemc, flags, err = memc:get(memcachedKey)
if err then
ngx.log(ngx.ALERT, "No appropriate script found in Local Memcached: ", err)
end
if scriptMemc then
ngx.log(ngx.ALERT, "Script found in memcached")
ngx.ctx.replaceScript = scriptMemc .. "</head>"
isScriptFromMemcached = true
else
ngx.log(ngx.ALERT, "Script not found in memcached")
end
end
end
-- If script wasn't found in Memcached - retrieve script directly from CX
if not isScriptFromMemcached then
ngx.log(ngx.ALERT, "Retrieveing sript from CX")
local http = require("socket.http")
local ltn12 = require("ltn12")
local t = {}
http.request {
url = "" href="http://blabla.com/luatest.txt">http://blabla.com/luatest.txt",
method = "GET",
sink = ltn12.sink.table(t)
}
-- Concatenete recieved chunks into string
scriptCX = table.concat(t)
ngx.log(ngx.ALERT, "Script from CX: ", scriptCX)
ngx.ctx.replaceScript = scriptCX .. "</head>"
ngx.log(ngx.ALERT, "memcachedKey: ", memcachedKey)
ngx.log(ngx.ALERT, "scrip: ", scriptCX)
-- local ok, err = memc:set(memcachedKey, scrip, 900)
if not err then ngx.log(ngx.ALERT, "Script stored in Memcached, ok: ", ok)
else ngx.log(ngx.ERR, "Failed to store the script in Memcached, err: ", err)
end
end
memc:close()
ngx.log(ngx.ALERT, "memc closed")
ngx.say(ngx.ctx.replaceScript)

On Sun, Mar 27, 2016 at 2:34 PM, Itamar Gilad <it...@metapacket.com> wrote:

You could fetch the data in the content_by_lua / access_by_lua stages and store it in the request context via the ngx.ctx.* table, and use it later on when needed.
Also, instead of making the replacements in the response body in lua-land using *_filter_by_lua*, I suggest checking out the excellent replace-filter nginx module, which uses the sregex library (both by agentzh).
In case you need to use the matching result groups in your replacements, you may want to do that by setting an ngx.var.* in lua (since passing dollar signs in the nginx config file is no fun).

Best of luck!
-Itamar

On Sat, Mar 26, 2016 at 10:03 PM, Alexander Litvak <al...@chameleonx.com> wrote:
Hello!

I was trying to get to know Openresty, as it would possibly be required on my job, but ran into a problem. I need to be able to lookup text value from memcached(by the domain) and, if not found, retrieve it from specified location(url). Then I want to replace some text in the page(the response body) with the said string from memcached or downloaded.

I have two scripts in lua, one I call from content_by_lua and other from the body_filter_by_lua in nginx.conf file within the same location. I also use nginx variable to pass a string from one script to another. My problem is: I can't use memcached inside of body_filter_by_lua and I can't use ngx.arg API in the content_by_lua, but when I use both the response I get is just an empty page.

Logs are all clean.

Found no info on the issue, hope there is some kind of workaround!

Thanks in advance!

.