is there a way to run lua code after worker process forked ?

paynechu · 2013-08-01T15:43:39+00:00

Yichun, thanks a lot for your reminding!The most common cause for my case is that many mobile clients including mobile browers reject SSL cerf or fail to neg...

is there a way to run lua code after worker process forked ?

paynechu

Im looking for a way to run lua code after worker process forked instead of every request come in. because I need ngx.var.pid for the initialising

agentzh

Hello!

On Thu, Aug 1, 2013 at 12:43 AM, Payne Chu wrote:
> Im looking for a way to run lua code after worker process forked instead of
> every request come in. because I need ngx.var.pid for the initialising
>

There will be an init_worker_by_lua and init_worker_by_lua_file
directives. Patches welcome :)

BTW, please don't cross-post to the openresty-en, openresty mailing
lists and the github issues board again. This is very annoying.

Regards,
-agentzh

matthieu.tourne

Hi!

On Thu, Aug 1, 2013 at 11:54 AM, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

Hello!

On Thu, Aug 1, 2013 at 12:43 AM, Payne Chu wrote:
> Im looking for a way to run lua code after worker process forked instead of
> every request come in. because I need ngx.var.pid for the initialising

There will be an init_worker_by_lua and init_worker_by_lua_file
directives. Patches welcome :)

This would be a very useful mechanism!

Especially if it's possible to know if the worker is initialized from a start, reload or a crashed worker restarting.

agentzh

Hello!

On Thu, Aug 1, 2013 at 12:01 PM, Matthieu Tourne wrote:
>
> This would be a very useful mechanism!
> Especially if it's possible to know if the worker is initialized from a
> start, reload or a crashed worker restarting.
>

You can distinguish fresh server start and server reload by setting a
flag in the ngx_lua shared dictionaries because shm will not be
cleared upon server reload.

Worker crashes should always be fixed in the first place because
you'll surely have issues like a corrupted shm zone or out-of-sync
global counters. And it's not so meaningful to design for that
abnormal scenario.

Regards,
-agentzh

paynechu

oops sorry~ because I just newly learn this platform. and first time to ask question. so the first place I goto github. after that I found out that is old repos. and I found out another repos. finally I found out this group. so why I posted many times. that is kind of learning flow :)~ anyways sorry for that and will not again since I know here now. Anyways good to hear there will have init_worker_by_lua and init_worker_by_lua_file. Excited to see them ready thanks.

On Friday, August 2, 2013 2:54:39 AM UTC+8, agentzh wrote:

Hello!

On Thu, Aug 1, 2013 at 12:43 AM, Payne Chu wrote:
> Im looking for a way to run lua code after worker process forked instead of
> every request come in. because I need ngx.var.pid for the initialising
>

There will be an init_worker_by_lua and init_worker_by_lua_file
directives. Patches welcome :)

BTW, please don't cross-post to the openresty-en, openresty mailing
lists and the github issues board again. This is very annoying.

Regards,
-agentzh

agentzh

Hello!

On Thu, Aug 1, 2013 at 8:38 PM, Payne Chu wrote:
> oops sorry~ because I just newly learn this platform. and first time to ask
> question. so the first place I goto github. after that I found out that is
> old repos. and I found out another repos. finally I found out this group. so
> why I posted many times. that is kind of learning flow :)~

Okay, got it. The old Perl-based openresty repository is indeed
confusing. I've just renamed it to old-openresty:

    https://github.com/agentzh/old-openresty

> Anyways good to hear there
> will have init_worker_by_lua and init_worker_by_lua_file. Excited to see
> them ready thanks.
>

BTW, for now, you may want to consider doing your initialization at
the first request served by each nginx worker, which should be a good
approximation for init_worker_by_lua in many cases.

Regards,
-agentzh

paynechu

thanks agentzh,

if I init in the content_by_lua* I found out the table's memory address will different by each request. I know there have a cache to work on it. but is it still have some overhead? my plan is since init_by_lua or init_worker_by_lua will only load once and have a feature COW. I like to do as mush as possible in that stage. so make the runtime faster to run.

BTW, I would like to contribute some code here. but I don't find any information in openresty.org about contribution. maybe I missed can let me know? Since I cannot find contribution information. I try to explore it myself. I checkout ngx_openresty from github. seems this is kind of package manager, download through wget and pack to a bundle gzip. But what I want is a single checkout place after clone already can start to work on. better with git's submodule to manage the dependencies with other projects. So for the contributor they can easily just do `git clone $url --recursive` for checkout, `make` for build, `make test` for test `make run` for run. and since is under git's submodule. any change on the submodule can easily push to that submodule's repos. Is it have something like that already? If no I would like to help on this. or any other suggestion?

On Saturday, August 3, 2013 2:37:09 AM UTC+8, agentzh wrote:

Hello!

On Thu, Aug 1, 2013 at 8:38 PM, Payne Chu wrote:
> oops sorry~ because I just newly learn this platform. and first time to ask
> question. so the first place I goto github. after that I found out that is
> old repos. and I found out another repos. finally I found out this group. so
> why I posted many times. that is kind of learning flow :)~

Okay, got it. The old Perl-based openresty repository is indeed
confusing. I've just renamed it to old-openresty:

https://github.com/agentzh/old-openresty

> Anyways good to hear there
> will have init_worker_by_lua and init_worker_by_lua_file. Excited to see
> them ready thanks.
>

BTW, for now, you may want to consider doing your initialization at
the first request served by each nginx worker, which should be a good
approximation for init_worker_by_lua in many cases.

Regards,
-agentzh

agentzh

Hello!

On Fri, Aug 2, 2013 at 8:27 PM, Payne Chu wrote:
> if I init in the content_by_lua* I found out the table's memory address will
> different by each request. I know there have a cache to work on it. but is
> it still have some overhead? my plan is since init_by_lua or
> init_worker_by_lua will only load once and have a feature COW. I like to do
> as mush as possible in that stage. so make the runtime faster to run.
>

You can use your own Lua modules to save per-worker persistent state.
See the documentation here:

http://wiki.nginx.org/HttpLuaModule#Data_Sharing_within_an_Nginx_Worker

This is a common practice.

> BTW, I would like to contribute some code here. but I don't find any
> information in openresty.org about contribution. maybe I missed can let me
> know?

Oh, I'm sorry about that. I've just created a "Getting Involved" page
on the openresty.org site:

    http://openresty.org/#GettingInvolved

If I'm still missing anything there, please let me know :)

> Since I cannot find contribution information. I try to explore it
> myself. I checkout ngx_openresty from github. seems this is kind of package
> manager, download through wget and pack to a bundle gzip.

This is correct.

> But what I want is
> a single checkout place after clone already can start to work on. better
> with git's submodule to manage the dependencies with other projects. So for
> the contributor they can easily just do `git clone $url --recursive` for
> checkout, `make` for build, `make test` for test `make run` for run. and
> since is under git's submodule. any change on the submodule can easily push
> to that submodule's repos. Is it have something like that already? If no I
> would like to help on this. or any other suggestion?
>

Because ngx_openresty is just a bundle, each component in the bundle
is developed separately and not every component is using git. You're
very welcome to contribute to each component separately.

Best regards,
-agentzh

paynechu

Thanks,

But Im not want to share the data between worker. :)~ I just want to init my module in the beginning of the worker. Because I need pid. so I will wait for the init_worker_by_lua* or for now I can just use under content_by_lua*. Sure I will try to implement my own init_worker if I can understand the entire flow bc still too new here. Just start to play around less then one week.

> http://openresty.org/#GettingInvolved

Thanks your for adding this page to clarify. I may still consider how to improve your work flow along the way when I more depth understand them. sure for now I will try to follow your guys foot step. and try to see can I contribute something in this community.

BTW, I have another problem, please see this post http://forum.nginx.org/read.php?2,241478,241478#msg-241478

Before, I thought it's nginx problem. But after compare with the default nginx setup I cannot reproduce the same issue. seems this is OpenResty's nginx problem. any idea?

On Saturday, August 3, 2013 12:50:28 PM UTC+8, agentzh wrote:

Hello!

On Fri, Aug 2, 2013 at 8:27 PM, Payne Chu wrote:
> if I init in the content_by_lua* I found out the table's memory address will
> different by each request. I know there have a cache to work on it. but is
> it still have some overhead? my plan is since init_by_lua or
> init_worker_by_lua will only load once and have a feature COW. I like to do
> as mush as possible in that stage. so make the runtime faster to run.
>

You can use your own Lua modules to save per-worker persistent state.
See the documentation here:

http://wiki.nginx.org/HttpLuaModule#Data_Sharing_within_an_Nginx_Worker

This is a common practice.

> BTW, I would like to contribute some code here. but I don't find any
> information in openresty.org about contribution. maybe I missed can let me
> know?

Oh, I'm sorry about that. I've just created a "Getting Involved" page
on the openresty.org site:

http://openresty.org/#GettingInvolved

If I'm still missing anything there, please let me know :)

> Since I cannot find contribution information. I try to explore it
> myself. I checkout ngx_openresty from github. seems this is kind of package
> manager, download through wget and pack to a bundle gzip.

This is correct.

> But what I want is
> a single checkout place after clone already can start to work on. better
> with git's submodule to manage the dependencies with other projects. So for
> the contributor they can easily just do `git clone $url --recursive` for
> checkout, `make` for build, `make test` for test `make run` for run. and
> since is under git's submodule. any change on the submodule can easily push
> to that submodule's repos. Is it have something like that already? If no I
> would like to help on this. or any other suggestion?
>

Because ngx_openresty is just a bundle, each component in the bundle
is developed separately and not every component is using git. You're
very welcome to contribute to each component separately.

Best regards,
-agentzh

On Saturday, August 3, 2013 12:50:28 PM UTC+8, agentzh wrote:

Hello!

On Fri, Aug 2, 2013 at 8:27 PM, Payne Chu wrote:
> if I init in the content_by_lua* I found out the table's memory address will
> different by each request. I know there have a cache to work on it. but is
> it still have some overhead? my plan is since init_by_lua or
> init_worker_by_lua will only load once and have a feature COW. I like to do
> as mush as possible in that stage. so make the runtime faster to run.
>

You can use your own Lua modules to save per-worker persistent state.
See the documentation here:

http://wiki.nginx.org/HttpLuaModule#Data_Sharing_within_an_Nginx_Worker

This is a common practice.

> BTW, I would like to contribute some code here. but I don't find any
> information in openresty.org about contribution. maybe I missed can let me
> know?

Oh, I'm sorry about that. I've just created a "Getting Involved" page
on the openresty.org site:

http://openresty.org/#GettingInvolved

If I'm still missing anything there, please let me know :)

> Since I cannot find contribution information. I try to explore it
> myself. I checkout ngx_openresty from github. seems this is kind of package
> manager, download through wget and pack to a bundle gzip.

This is correct.

> But what I want is
> a single checkout place after clone already can start to work on. better
> with git's submodule to manage the dependencies with other projects. So for
> the contributor they can easily just do `git clone $url --recursive` for
> checkout, `make` for build, `make test` for test `make run` for run. and
> since is under git's submodule. any change on the submodule can easily push
> to that submodule's repos. Is it have something like that already? If no I
> would like to help on this. or any other suggestion?
>

Because ngx_openresty is just a bundle, each component in the bundle
is developed separately and not every component is using git. You're
very welcome to contribute to each component separately.

Best regards,
-agentzh

On Saturday, August 3, 2013 12:50:28 PM UTC+8, agentzh wrote:

Hello!

On Fri, Aug 2, 2013 at 8:27 PM, Payne Chu wrote:
> if I init in the content_by_lua* I found out the table's memory address will
> different by each request. I know there have a cache to work on it. but is
> it still have some overhead? my plan is since init_by_lua or
> init_worker_by_lua will only load once and have a feature COW. I like to do
> as mush as possible in that stage. so make the runtime faster to run.
>

You can use your own Lua modules to save per-worker persistent state.
See the documentation here:

http://wiki.nginx.org/HttpLuaModule#Data_Sharing_within_an_Nginx_Worker

This is a common practice.

> BTW, I would like to contribute some code here. but I don't find any
> information in openresty.org about contribution. maybe I missed can let me
> know?

Oh, I'm sorry about that. I've just created a "Getting Involved" page
on the openresty.org site:

http://openresty.org/#GettingInvolved

If I'm still missing anything there, please let me know :)

> Since I cannot find contribution information. I try to explore it
> myself. I checkout ngx_openresty from github. seems this is kind of package
> manager, download through wget and pack to a bundle gzip.

This is correct.

> But what I want is
> a single checkout place after clone already can start to work on. better
> with git's submodule to manage the dependencies with other projects. So for
> the contributor they can easily just do `git clone $url --recursive` for
> checkout, `make` for build, `make test` for test `make run` for run. and
> since is under git's submodule. any change on the submodule can easily push
> to that submodule's repos. Is it have something like that already? If no I
> would like to help on this. or any other suggestion?
>

Because ngx_openresty is just a bundle, each component in the bundle
is developed separately and not every component is using git. You're
very welcome to contribute to each component separately.

Best regards,
-agentzh

agentzh

Hello!

On Fri, Aug 2, 2013 at 10:34 PM, Payne Chu wrote:
> But Im not want to share the data between worker. :)~ I just want to init my
> module in the beginning of the worker. Because I need pid. so I will wait
> for the init_worker_by_lua* or for now I can just use under content_by_lua*.

No, you misunderstood me. it's NOT about sharing data *between* nginx workers.

It's about setting a flag on a *per-worker* basis (that is, setting it
in your own Lua module's context) so that you can know if the current
request is the first request served by the *current* worker. That way,
you can do the worker initialization properly, and for only once.

This is a common trick.

>
> BTW, I have another problem, please see this post
> http://forum.nginx.org/read.php?2,241478,241478#msg-241478
> Before, I thought it's nginx problem. But after compare with the default
> nginx setup I cannot reproduce the same issue. seems this is OpenResty's
> nginx problem. any idea?
>

Let's discuss it in a separate thread. This surely has nothing to do
with the current topic, "is there a way to run lua code after worker
process forked".

Regards,
-agentzh

paynechu

Thanks for clarification :). and I will open another thread about the new issue thanks

On Saturday, August 3, 2013 1:43:22 PM UTC+8, agentzh wrote:

Hello!

On Fri, Aug 2, 2013 at 10:34 PM, Payne Chu wrote:
> But Im not want to share the data between worker. :)~ I just want to init my
> module in the beginning of the worker. Because I need pid. so I will wait
> for the init_worker_by_lua* or for now I can just use under content_by_lua*.

No, you misunderstood me. it's NOT about sharing data *between* nginx workers.

It's about setting a flag on a *per-worker* basis (that is, setting it
in your own Lua module's context) so that you can know if the current
request is the first request served by the *current* worker. That way,
you can do the worker initialization properly, and for only once.

This is a common trick.

>
> BTW, I have another problem, please see this post
> http://forum.nginx.org/read.php?2,241478,241478#msg-241478
> Before, I thought it's nginx problem. But after compare with the default
> nginx setup I cannot reproduce the same issue. seems this is OpenResty's
> nginx problem. any idea?
>

Let's discuss it in a separate thread. This surely has nothing to do
with the current topic, "is there a way to run lua code after worker
process forked".

Regards,
-agentzh