sock:listen() & sock:accept() development questions

jaksprats · 2016-05-11T05:04:48+00:00

How to intercept response from Apache in Nginx?

sock:listen() & sock:accept() development questions

jaksprats

Hi All,

I am planning on implementing sock:listen() & sock:accept() in ngx_lua.

I need sock:listen() & sock:accept() to be able to listen & accept requests on a unique port per nginx worker, I will use these unique-per-worker ports to sticky-load-balance between worker processes.

I asked Agentzh about this topic and he said:

1.) You can take a look at how the nginx core does listening and accepting as well as cosockets' existing methods' implementation

2.) sock:listen() and sock:accept() should share the same Lua API as the LuaSocket library

3.) My (Agentzh's) hunch is that multiple light threads should be allowed to accept on the samelistening cosocket in the same handler context.

Does anyone have any more input on this?

Does anyone have any more advice on how to code this? (e.g. is there similar code that it would make sense to base these extensions on)

thanks,

Russ

romadzao

Hello, I'm trying to code similar thing (against worker's isolation) and read many sources, including nginx sources. How you want to bind workers to port if openresty doesn't choose worker on request, that does nginx, and it is made in very complicated cycle?

1.) You can take a look at how the nginx core does listening and accepting as well as cosockets' existing methods' implementation

I was get crazy while read that code. Each process goes to cycle. In that cycle process wait for connection. How it works - mystery, please tell if you understood that cycle.

jaksprats

Hi Romaboy,

It is tricky getting into both the nginx code and the open-resty code.

I did not have enough time to do the full tcpsock:accept/listen() solution :(

I came up with a solution for my exact use-case: a unique port per worker.

This is accomplished by a new lua function called unique_socket_per_worker(path) which replaces the unix-socket specified by 'path' with a unique socket per worker (i.e. path_$(PID)).

This is only avaialble in my fork: https://github.com/JakSprats/lua-nginx-module

The function is called unique_socket_per_worker(path) and is defined in this file: https://github.com/JakSprats/lua-nginx-module/blob/master/src/ngx_http_lua_unique_socket_per_worker.c

This code is experimental so use with caution. I will be adding tests to it and will discuss w/ agentzh if I should make a PR.

Read thru the C file, the same approach could be used to implement sock:accept/listen, but there is still a lot of server/stream initialization code needed, as well as openresty tcpsock code needed.

Hope this helps

- Russ

romadzao

Could you please provide usage example?

jaksprats

Here is an example: https://github.com/JakSprats/lua-nginx-module/tree/master/TEST

NOTE: This is experimental code. Also I ripped this example out of a larger code base, so it does not do much.

Start w/

nginx -c nginx_lb.conf -p $PWD

(you need to fill in working_directory and mkdir ./logs)

Test via curl:

$ curl -o - http://localhost:8080/sticky/debug

DEBUG SCRIPT RUNNING

SERVER: PID: 25051 IP: unix:/tmp/sticky_nginx_socket_25051

The basic flow is the request to 8080 goes to the upstream 'backend' which balances_by_lua_file using sticky_balancer.lua.

sticky_balancer.lua balances to a specific worker (in this case itself) by appending ngx.worker.pid() to the unix_socket_name specified in the conf file.

This unique unix_socket per worker (e.g. unix:/tmp/sticky_nginx_socket_25051) was created by the ngx.unique_socket_per_worker('unix:/tmp/sticky_nginx_socket_'); call, Since there are 2 worker processes, there is probably also unix:/tmp/sticky_nginx_socket_25052 associated w/ worker 2.

What is missing (but present in my larger code base) is sticky logic that would balance_by_lua to DIFFERENT pids.

Hope this helps