RESTful POST &lt;FORM&gt;

mrnoas · 2013-10-16T14:01:18+00:00

I'd like to change the tcp bind address and port of a running nginx process with Lua, any hints how to do this?

RESTful POST <FORM>

mrnoas

sorry if double posting

Tried to configure RESTful POST without success. The request is sent as a FORM POST e.g {firstname:Michael, lastname:R, age:12} on behalf of a framework. It gets inserted into database as NULL. Have tried also using SET_FORM_INPUT $firstname with no luck. How do you pass the attributes so that they can be passed to psql statement? Thanks in advance

nginx.conf:

location /users {
postgres_pass database;
rds_json on;
postgres_query HEAD GET "SELECT * FROM users";

postgres_escape $firstname $arg_firstname;
postgres_escape $lastname $arg_lastname;
postgres_query
POST "INSERT INTO users (firstname, lastname, age) VALUES ($firstname,$lastname,$age) RETURNING *";
postgres_rewrite POST changes 201;
}

nginx -V:

nginx version: ngx_openresty/1.4.2.9
built by gcc 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5)
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt=-I/usr/local/include --add-module=../ngx_devel_kit-0.2.19 --add-module=../echo-nginx-module-0.48 --add-module=../xss-nginx-module-0.03rc9 --add-module=../ngx_coolkit-0.2rc1 --add-module=../set-misc-nginx-module-0.22 --add-module=../form-input-nginx-module-0.07 --add-module=../encrypted-session-nginx-module-0.03 --add-module=../ngx_postgres-1.0rc3 --add-module=../srcache-nginx-module-0.22 --add-module=../ngx_lua-0.9.0 --add-module=../headers-more-nginx-module-0.22 --add-module=../array-var-nginx-module-0.03rc1 --add-module=../memc-nginx-module-0.13 --add-module=../redis2-nginx-module-0.10 --add-module=../redis-nginx-module-0.3.6 --add-module=../auth-request-nginx-module-0.2 --add-module=../rds-json-nginx-module-0.12rc10 --add-module=../rds-csv-nginx-module-0.05rc2 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -L/usr/local/lib' --with-http_ssl_module

agentzh

Hello!

On Tue, Oct 15, 2013 at 11:01 PM,  <mr...@gmail.com> wrote:
> sorry if double posting
>

You did not join the group before sending your previous post, so your
post requires manual moderation. I've already removed your previous
post in the moderation queue for admins.

> Tried to configure RESTful POST without success. The request is sent as a
> FORM POST e.g {firstname:Michael, lastname:R, age:12} on behalf of a
> framework.

This data format looks weird to me. It looks like JSON but it is not
valid JSON. Could you use the following configuration to obtain the
raw HTTP request sent by your client?

    location = /t {
        content_by_lua '
            ngx.print(ngx.req.raw_header())
            ngx.req.read_body()
            ngx.print(ngx.req.get_body_data() or "")
        ';
    }

Of course, you can use other tools like nc or tcpdump to get the raw
HTTP request payload.

Without understanding the request, I cannot give you any suggestions
on this because it'll just be like shooting in the darkness :)

Best regards,
-agentzh

mrnoas

Hello,

Thanks for replying.

Below picture is the request being sent to the server

Or, the payload alternatively would be:

POST /users HTTP/1.1
Host: precisepangolinserver
Cache-Control: no-cache
----WebKitFormBoundaryE19zNvXGzXaLvS5C
Content-Disposition: form-data; name="firstname"
Michael
----WebKitFormBoundaryE19zNvXGzXaLvS5C
Content-Disposition: form-data; name="lastname"
R
----WebKitFormBoundaryE19zNvXGzXaLvS5C
Content-Disposition: form-data; name="age"
21
----WebKitFormBoundaryE19zNvXGzXaLvS5C

I know nothing about lua. So, here is my first attempt with rewrite_by_lua, at least whether it can get some vars from the request body but with no success

location /users {

postgres_pass database;

postgres_output rds;

rds_json on;

postgres_query HEAD GET "SELECT * FROM users";

client_body_buffer_size 8k;

client_max_body_size 8k;

set $f '';

set $lastname 'test';

set $age '99';

rewrite_by_lua '

ngx.var.f = ngx.req.get_body_data()

postgres_query

POST "INSERT INTO users (firstname,lastname,age) VALUES ('$f','$lastname','$age') RETURNING *";

postgres_rewrite POST changes 201;

On Thursday, October 17, 2013 4:14:24 AM UTC+7, agentzh wrote:

Hello!

On Tue, Oct 15, 2013 at 11:01 PM, <mrn...@gmail.com> wrote:
> sorry if double posting
>

You did not join the group before sending your previous post, so your
post requires manual moderation. I've already removed your previous
post in the moderation queue for admins.

> Tried to configure RESTful POST without success. The request is sent as a
> FORM POST e.g {firstname:Michael, lastname:R, age:12} on behalf of a
> framework.

This data format looks weird to me. It looks like JSON but it is not
valid JSON. Could you use the following configuration to obtain the
raw HTTP request sent by your client?

location = /t {
content_by_lua '
ngx.print(ngx.req.raw_header())
ngx.req.read_body()
ngx.print(ngx.req.get_body_data() or "")
';
}

Of course, you can use other tools like nc or tcpdump to get the raw
HTTP request payload.

Without understanding the request, I cannot give you any suggestions
on this because it'll just be like shooting in the darkness :)

Best regards,
-agentzh

agentzh

Hello!

On Thu, Oct 17, 2013 at 7:23 AM, <mr...@gmail.com> wrote:
>
> Below picture is the request being sent to the server
>

It's not the raw payload. Your Google Chrome Dev Tools (if that is
what you're using) display the payload in its own way. Why don't you
just use my nginx configuration in my previous email to get the raw
request?

michael.rijanto

On Friday, October 18, 2013 2:35:49 AM UTC+7, agentzh wrote:

Hello!

On Thu, Oct 17, 2013 at 7:23 AM, <mrn...@gmail.com> wrote:
>
> Below picture is the request being sent to the server
>

It's not the raw payload. Your Google Chrome Dev Tools (if that is
what you're using) display the payload in its own way. Why don't you
just use my nginx configuration in my previous email to get the raw
request?

Yes, it should be a lot easier, probably a bit overwhelmed with many other things :-)

Request of /t:

GET /t HTTP/1.1
Host: precisepangolinserver:8080
Connection: keep-alive
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Cookie: _atshc={}; __atuvc=7%7C32%2C5%7C33%2C0%7C34%2C4%7C35%2C73%7C36

Request of GET /users:

GET /users HTTP/1.1
Host: precisepangolinserver:8080
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Cookie: _atshc={}; __atuvc=7%7C32%2C5%7C33%2C0%7C34%2C4%7C35%2C73%7C36

??B%ME~?@@?:
?7
?7????4(????????
2?8??uHTTP/1.1 200 OK
Server: ngx_openresty/1.4.2.9
Date: Fri, 18 Oct 2013 00:29:28 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive

193
[{"id":45,"firstname":"Biggirl","lastname":"Subroto Mangkuwanitoudo","age":"2   "},{"id":42,"firstname":"Michael","lastname":"Rijanto R","age":"99  "},{"id":77,"firstname":"Nur","lastname":"wely","age":"52  "},{"id":44,"firstname":"Orin","lastname":"Setyadi","age":"13  "},{"id":85,"firstname":"Update test","lastname":"test","age":"99  "},{"id":86,"firstname":"test202","lastname":"test","age":"2   "}]
0

BBB%ME4c?@@S,
?7
?7????????4*Ѐ??f
8???2??`R!|??B%ME???@@??
?7
?7????????4*Ѐ??V/
8???2?GET /users HTTP/1.1
Host: precisepangolinserver:8080
Connection: keep-alive
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36
Accept: */*
Referer: http://precisepangolinserver:8080/users
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Cookie: _atshc={}; __atuvc=7%7C32%2C5%7C33%2C0%7C34%2C4%7C35%2C73%7C36

Request of POST:

POST /users HTTP/1.1
Host: precisepangolinserver:8080
Connection: keep-alive
Content-Length: 55
Accept: application/json, text/_javascript_, */*; q=0.01
Origin: http://precisepangolinserver:8080
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36
Content-Type: application/json
Referer: http://precisepangolinserver:8080/
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Cookie: _atshc={}; __atuvc=7%7C32%2C5%7C33%2C0%7C34%2C4%7C35%2C73%7C36

{"firstname":"Precise","lastname":"Pangolin","age":"1"}E?`R?BB%ME4)?@@?A
?7
?7???9??b?????|??
??8??bE?`R?00B%ME")?@@?R
?7
?7???9??b?????|??
??8??b

Then you can use the lua-cjson library to decode the JSON string in
your request body:

# create the nginx variables:
set $firstname '';
set $lastname '';
set $age '';

# assign the values to the nginx variables from the request body:
rewrite_by_lua '
local cjson = require "cjson"
ngx.req.read_body()
local data = ""
if not data then
return ngx.exit(400)
end
local tb = cjson.decode(data)
ngx.var.firstname = tb.firstname
ngx.var.lastname = tb.lastname
ngx.var.age = tb.age
';

<snip>

It seemed I am having problem with cjson. I believe it should be included in openresty as default. But it throws error 500. Then tried with just json, and it went straight to error 400.

I plan to sift the tutorial as you suggested on the weekend. Thanks for the link.

agentzh

Hello!

On Fri, Oct 18, 2013 at 2:16 AM,  michael.rijanto wrote:
> Request of POST:
>
> POST /users HTTP/1.1
[...]
>
> {"firstname":"Precise","lastname":"Pangolin","age":"1"}E?`R?BB%ME4)?@@?A
> ?7
> ?7???9??b?????|??
> ??8??bE?`R?00B%ME")?@@?R

Looks like it's doing JSON, though there's something afterwards I
don't quite understand. Encoding issue?

>
> It seemed I am having problem with cjson. I believe it should be included in
> openresty as default. But it throws error 500. Then tried with just json,
> and it went straight to error 400.
>

Have you checked out your nginx's error.log file?

Every time you see a 500 (or just something weird), you should look up
your nginx error log file :)

Also, in your Lua code, you can dump your JSON input (or any other
data in some Lua variables) into nginx error.log file directly by
calling the print() function. See
https://github.com/chaoslawful/lua-nginx-module#print

Best regards,
-agentzh

mrnoas

On Saturday, October 19, 2013 1:54:04 AM UTC+7, agentzh wrote:

Hello!

On Fri, Oct 18, 2013 at 2:16 AM, michael.rijanto wrote:
> Request of POST:
>
> POST /users HTTP/1.1
[...]
>
> {"firstname":"Precise","lastname":"Pangolin","age":"1"}E?`R?BB%ME4)?@@?A
> ?7
> ?7???9??b?????|??
> ??8??bE?`R?00B%ME")?@@?R

Looks like it's doing JSON, though there's something afterwards I
don't quite understand. Encoding issue?

I guess it's network overhead traffic that got captured. I used tcpdump

Have you checked out your nginx's error.log file?

Every time you see a 500 (or just something weird), you should look up
your nginx error log file :)

Yes, it generated one line of code the error number 500 followed with length:

10.211.55.2 - - [18/Oct/2013:19:31:33 +0700] "GET /users HTTP/1.1" 500 537 "http://precisepangolinserver/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36"

agentzh

Hello!

On Sat, Oct 19, 2013 at 3:06 AM,  <mr...@gmail.com> wrote:
>
> Yes, it generated one line of code the error number 500 followed with
> length:
>
> 10.211.55.2 - - [18/Oct/2013:19:31:33 +0700] "GET /users HTTP/1.1" 500 537
> "http://precisepangolinserver/"; "Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69
> Safari/537.36"
>

You're looking at the wrong file. It is the access log file, not the
error log file. Access logs are usually useless for trouble shooting.

See http://nginx.org/en/docs/ngx_core_module.html#error_log

-agentzh