Lua nginx compiled for nginx windows

info · 2013-09-21T04:07:31+00:00

On 21 September 2013 17:07, Brian Akins <br...@akins.org> wrote: On Sep 18, 2013, at 10:39 PM, JT Archie <jta...@gmail.com> wrote: > Can I...

Lua nginx compiled for nginx windows

info

@agentzh, A good lua test would be (as this is one of my interests in lua) an easy way to block sql attacks, ea. http://www.cvedetails.com/cve/CVE-2013-5723/
Would you be willing to share some of your cloudflare WAF lua code to get some basic WAF running in nginx/lua?

jjmutumi

Hello,

Actually managed to compile it too sometime back. Had to patch some files but it works! I didn't document it but was using Mingw with gcc version 4.6.2 (GCC). Just as a few tips:

1. Make sure you can compile nginx and lua (or luajit) separately first.

2. For some reason nginx didn't compile with some modules like gzip or ssl :(

3. nginx on windows must be compiled with select that's the default for windows

Hope this is useful info. If you are interested I could try to come up with the steps I used.

Cheers.

On Mon, Sep 23, 2013 at 3:41 PM, <i...@ecsystems.nl> wrote:

@agentzh, A good lua test would be (as this is one of my interests in lua) an easy way to block sql attacks, ea. http://www.cvedetails.com/cve/CVE-2013-5723/
Would you be willing to share some of your cloudflare WAF lua code to get some basic WAF running in nginx/lua?

.

--
Joseph N. Mutumi

Chief Technical Office
Victorium Holdings
victorium.biz | Facebook | Twitter | Blog

info

10:37 23-9-2013: nginx 1.5.6.1 Alice

Based on nginx 1.5.6 (22-9-2013) with;
+ Streaming with nginx-rtmp-module, v1.0.4 (http://nginx-rtmp.blogspot.nl/)
+ lua-nginx-module v0.8.9 (tnx to agentzh about precompiled headers!)
+ LuaJIT-2.0.2 => (lua51.dll include / lua51.lib build)
+ Added lua51.dll (is required)
+ ngx_devel_kit v0.2.15
* Additional specifications are like 10:27 10-9-2013: B02 build

Builds can be found here:
http://nginx-win.ecsds.eu/

Build process and changes are documented, when all major issues have been resolved they will be made public.

On Monday, September 23, 2013 5:50:40 PM UTC+2, Joseph Mutumi wrote:

Hello,

Actually managed to compile it too sometime back. Had to patch some files but it works! I didn't document it but was using Mingw with gcc version 4.6.2 (GCC). Just as a few tips:

agentzh

Hello!

On Mon, Sep 23, 2013 at 5:41 AM,  <i...@ecsystems.nl> wrote:
> @agentzh, A good lua test would be (as this is one of my interests in lua)
> an easy way to block sql attacks, ea.
> http://www.cvedetails.com/cve/CVE-2013-5723/

Yeah, you can do that yourself :)

> Would you be willing to share some of your cloudflare WAF lua code to get
> some basic WAF running in nginx/lua?
>

It's a big code base. Our WAF engineers built a mod_security port for
ngx_lua. And CloudFlare does not opensource it yet.

Regards,
-agentzh

agentzh

Hello!

On Sun, Sep 22, 2013 at 1:48 PM,  <i...@ecsystems.nl> wrote:
>> Also, it'll be great if you provide source line numbers in these C
>> compiler warnings or errors.
>
> I will some time tomorrow, just shut down the development machine, spend
> enough hours to get it compiling.
>

I've looked into the raw mingw compiler warnings you sent me off the
list. I've fixed most of the warnings in ngx_lua and ngx_devel_kit's
git masters:

    https://github.com/chaoslawful/lua-nginx-module/commit/585e7af
    https://github.com/simpl/ngx_devel_kit/commit/4944025

Please try out these two project's git master again on your side :)

One that in ngx_lua that has not yet been fixed for the Windows build
is an issue in the Lua/LuaJIT bytecode file loader for Windows. I've
filed a github issue for this:

    https://github.com/chaoslawful/lua-nginx-module/issues/283

Before fixing this ticket, the Lua/LuaJIT bytecode file loader in
ngx_lua does not work on Windows even that part of code is also
compiled. Fortunately people don't use this feature much and this
feature is not mandatory. See the "Lua/LuaJIT bytecode support"
section in ngx_lua's documentation for more details on this feature:
http://wiki.nginx.org/HttpLuaModule#Lua.2FLuaJIT_bytecode_support

The remaining warnings regarding ngx_hash overflow look like false
positives, for example,

    lua-nginx-module/src/ngx_http_lua_subrequest.c(1159) : warning
C4307: '*' : integral constant overflow
    lua-nginx-module/src/ngx_http_lua_control.c(258) : warning C4307:
'*' : integral constant overflow

I'm not sure how to fix these warnings (yet).

BTW, you can paste long text to the github gist site, for example:

    https://gist.github.com/

or pastebin:

    http://pastebin.com/

And just put the resulting link in your email.

Looks like I need to run a Windows box to test my code sometime in the future :)

Best regards,
-agentzh

info

>> an easy way to block sql attacks, ea.
>Yeah, you can do that yourself :)

But it would be easier with an example from you :)

>Looks like I need to run a Windows box to test my code sometime in the future :)

Get yourself virtualbox, surely you got some w7 license laying about doing nothing :)

>Please try out these two project's git master again on your side :)

Done, here are the patches and other findings, its not too big to fit here;

lua-nginx-module/src/ngx_http_lua_control.c(258) : warning C4307: '*' : integral constant overflow
see also http://msdn.microsoft.com/en-us/library/4kze989h.aspx
ngx_http_lua_control.c
line 257:
            ngx_hash(ngx_hash(ngx_hash(ngx_hash(ngx_hash(ngx_hash(
                     ngx_hash('l', 'o'), 'c'), 'a'), 't'), 'i'), 'o'), 'n');
*I count 7x ngx_hash but 8 hash values.
==============================================================================
lua-nginx-module/src/ngx_http_lua_subrequest.c(1159) : warning C4307: '*' : integral constant overflow
see also http://msdn.microsoft.com/en-us/library/4kze989h.aspx
ngx_http_lua_subrequest.c
line 1156:
    h->hash = ngx_hash(ngx_hash(ngx_hash(ngx_hash(ngx_hash(ngx_hash(ngx_hash(
            ngx_hash(ngx_hash(ngx_hash(ngx_hash(ngx_hash(
            ngx_hash('c', 'o'), 'n'), 't'), 'e'), 'n'), 't'), '-'), 'l'), 'e'),
            'n'), 'g'), 't'), 'h');
*I count 13x ngx_hash but 14 hash values.
==============================================================================
\ngx_devel_kit\src\ndk_log.h(162) :fatal error C1021: invalid preprocessor command 'warning'
-162
#warning "Nginx Devel Kit logging without variadic macros not yet implemented"
+162
/* #warning "Nginx Devel Kit logging without variadic macros not yet implemented" */
==============================================================================
/lua-nginx-module/src/ngx_http_lua_shdict.c(972) : error C2065: 'ushort' : undeclared identifier
/lua-nginx-module/src/ngx_http_lua_shdict.c(972) : error C2146: syntax error : missing ';' before identifier 'key'
/lua-nginx-module/src/ngx_http_lua_shdict.c(1079) : error C2065: 'ushort' : undeclared identifier
/lua-nginx-module/src/ngx_http_lua_shdict.c(1079) : error C2146: syntax error : missing ';' before identifier 'key'
-972
            sd->key_len = (ushort) key.len;
+972
            sd->key_len = (u_short) key.len;
-1079
    sd->key_len = (ushort) key.len;
+1079
    sd->key_len = (u_short) key.len;
==============================================================================

==============================================================================
/lua-nginx-module/src/ngx_http_lua_output.c(395) : warning C4244: '=' : conversion from 'double' to 'int', possible loss of data
==============================================================================
objs/lib/lua-nginx-module/src/ngx_http_lua_headers_out.c(147) : warning C4307: '*' : integral constant overflow
objs/lib/lua-nginx-module/src/ngx_http_lua_headers_out.c(147) : warning C4307: '*' : integral constant overflow
objs/lib/lua-nginx-module/src/ngx_http_lua_headers_out.c(538) : warning C4307: '*' : integral constant overflow
objs/lib/lua-nginx-module/src/ngx_http_lua_headers_out.c(538) : warning C4307: '*' : integral constant overflow
==============================================================================
objs/lib/lua-nginx-module/src/ngx_http_lua_clfactory.c(368) : warning C4022: 'GetFileInformationByHandle' : pointer mismatch for actual parameter 1
objs/lib/lua-nginx-module/src/ngx_http_lua_clfactory.c(368) : warning C4996: 'fileno': The POSIX name for this item is deprecated. Instead, use the ISO C++ conformant name: _fileno. See online help for details.

agentzh

Hello!

On Tue, Sep 24, 2013 at 3:48 AM,  <i...@ecsystems.nl> wrote:
>>Looks like I need to run a Windows box to test my code sometime in the
>> future :)
>
> Get yourself virtualbox, surely you got some w7 license laying about doing
> nothing :)
>

Yes, in a virtual machine. Possibly on Amazon EC2 though because I
don't want to buy Windows myself.

>>Please try out these two project's git master again on your side :)
>>
> Done, here are the patches and other findings, its not too big to fit here;

Thank you for trying it out again! I've submitted a patch to ngx_lua's
master that fixed most of these warnings:

   https://github.com/chaoslawful/lua-nginx-module/commit/c6b6b31

and also committed a patch to ngx_devel_kit's master:

    https://github.com/simpl/ngx_devel_kit/commit/4944025

> lua-nginx-module/src/ngx_http_lua_control.c(258) : warning C4307: '*' :
> integral constant overflow
> see also http://msdn.microsoft.com/en-us/library/4kze989h.aspx
> ngx_http_lua_control.c
> line 257:
>             ngx_hash(ngx_hash(ngx_hash(ngx_hash(ngx_hash(ngx_hash(
>                      ngx_hash('l', 'o'), 'c'), 'a'), 't'), 'i'), 'o'), 'n');
> *I count 7x ngx_hash but 8 hash values.
>

This is not the real issue. ngx_hash takes 2 arguments hence the difference.

>
> lua-nginx-module/src/ngx_http_lua_subrequest.c(1159) : warning C4307: '*' :
> integral constant overflow

Yeah, as the warning itself suggests, the real issue is 32-bit integer
overflow on 32-bit systems. The overflow itself is indeed the expected
behavior when calculating the string hash so all we need is to
suppress this warning.

Already fixed in my new patch.

> ngx_http_lua_subrequest.c: line 1156:
>     h->hash =
> ngx_hash(ngx_hash(ngx_hash(ngx_hash(ngx_hash(ngx_hash(ngx_hash(
>             ngx_hash(ngx_hash(ngx_hash(ngx_hash(ngx_hash(
>             ngx_hash('c', 'o'), 'n'), 't'), 'e'), 'n'), 't'), '-'), 'l'),
> 'e'),
>             'n'), 'g'), 't'), 'h');

Already fixed in my patch.

> \ngx_devel_kit\src\ndk_log.h(162) :fatal error C1021: invalid preprocessor
> command 'warning'
> -162
> #warning "Nginx Devel Kit logging without variadic macros not yet
> implemented"
> +162
> /* #warning "Nginx Devel Kit logging without variadic macros not yet
> implemented" */
>

Already fixed in my patch.

> /lua-nginx-module/src/ngx_http_lua_shdict.c(972) : error C2065: 'ushort' :
> undeclared identifier
> /lua-nginx-module/src/ngx_http_lua_shdict.c(972) : error C2146: syntax error
> : missing ';' before identifier 'key'
> /lua-nginx-module/src/ngx_http_lua_shdict.c(1079) : error C2065: 'ushort' :
> undeclared identifier
> /lua-nginx-module/src/ngx_http_lua_shdict.c(1079) : error C2146: syntax
> error : missing ';' before identifier 'key'

Nice catch. Already fixed in my patch.

> /lua-nginx-module/src/ngx_http_lua_output.c(395) : warning C4244: '=' :
> conversion from 'double' to 'int', possible loss of data
>

Already fixed in my patch.

> objs/lib/lua-nginx-module/src/ngx_http_lua_headers_out.c(147) : warning
> C4307: '*' : integral constant overflow
> objs/lib/lua-nginx-module/src/ngx_http_lua_headers_out.c(147) : warning
> C4307: '*' : integral constant overflow
> objs/lib/lua-nginx-module/src/ngx_http_lua_headers_out.c(538) : warning
> C4307: '*' : integral constant overflow
> objs/lib/lua-nginx-module/src/ngx_http_lua_headers_out.c(538) : warning
> C4307: '*' : integral constant overflow
>

They should already be fixed in my patch.

> objs/lib/lua-nginx-module/src/ngx_http_lua_clfactory.c(368) : warning C4022:
> 'GetFileInformationByHandle' : pointer mismatch for actual parameter 1
> objs/lib/lua-nginx-module/src/ngx_http_lua_clfactory.c(368) : warning C4996:
> 'fileno': The POSIX name for this item is deprecated. Instead, use the ISO
> C++ conformant name: _fileno. See online help for details.
>

These two are due to the pending compatibility issue in the Lua/LuaJIT
bytecode file loader in ngx_lua that I mentioned:

    https://github.com/chaoslawful/lua-nginx-module/issues/283

I'm waiting for jinglong to see if he'd like to fix it.

Could you please try out the new masters of ngx_lua and ngx_devel_kit
again on your side?

Thanks!
-agentzh

info

>Could you please try out the new masters of ngx_lua and ngx_devel_kit again on your side?

Done!

\ngx_devel_kit\src\ndk_log.h(162) :fatal error C1021: invalid preprocessor command 'warning'
*Fix was not committed dd. 25-9, actual fix:
Line 162 (or implement #if (WIN32)):
/* #warning "Nginx Devel Kit logging without variadic macros not yet implemented" */

/lua-nginx-module/src/ngx_http_lua_clfactory.c(368) : warning C4022: 'GetFileInformationByHandle' : pointer mismatch for actual parameter 1
*Known warning.

/lua-nginx-module/src/ngx_http_lua_clfactory.c(368) : warning C4996: 'fileno': The POSIX name for this item is deprecated. Instead, use the ISO C++ conformant name: _fileno. See online help for details.
*Renamed fileno to _fileno as suggested by compiler, warning gone.

agentzh

Hello!

On Wed, Sep 25, 2013 at 5:16 AM,  <i...@ecsystems.nl> wrote:
>
> \ngx_devel_kit\src\ndk_log.h(162) :fatal error C1021: invalid preprocessor
> command 'warning'
> *Fix was not committed dd. 25-9, actual fix:
> Line 162 (or implement #if (WIN32)):
>

Oh sorry, my bad! I forgot to push my commit to ngx_devel_kit. Just pushed:

    https://github.com/simpl/ngx_devel_kit/commit/8dd0df5

Please try the new master of ngx_devel_kit again :)

>
> /lua-nginx-module/src/ngx_http_lua_clfactory.c(368) : warning C4022:
> 'GetFileInformationByHandle' : pointer mismatch for actual parameter 1
> *Known warning.
> /lua-nginx-module/src/ngx_http_lua_clfactory.c(368) : warning C4996:
> 'fileno': The POSIX name for this item is deprecated. Instead, use the ISO
> C++ conformant name: _fileno. See online help for details.

I've applied a modified version of jinglong's patch to ngx_lua's master:

    https://github.com/chaoslawful/lua-nginx-module/commit/0387a89

which should fix these two warnings (and also the bytecode loader on
Windows). Please try it out on your side :)

Thanks!
-agentzh

info

Zero warning, zero errors !

About this Test Suite,
do you have precompiled windows perl(activestate) libraries for Test::Nginx? (not every perl source is properly windows tested)
are the nginx test modules windows tested? (they have to be or I'll be running into bugs that may originate from the module itself, the chicken-egg problem)
I don't use memcached but xcache (which is far faster/better for windows) is there a test for xcache?
Where is the master perl test script? (I can only see some bare-bone examples)

agentzh

Hello!

On Thu, Sep 26, 2013 at 12:51 AM,  <i...@ecsystems.nl> wrote:
> Zero warning, zero errors !
>

Yay!

> About this Test Suite,
> do you have precompiled windows perl(activestate) libraries for Test::Nginx?

You can use ActivePerl's "cpan" command-line utility to install
Test::Nginx from CPAN, that is, running the following command:

    cpan Test::Nginx

> (not every perl source is properly windows tested)

Because I used to do Perl development on Windows for quite a few
years, I can say that most of the CPAN modules are well tested on
Windows. Test::Nginx is one of those exceptions and as the author of
Test::Nginx, I'd love to hear what you get on Windows.

> are the nginx test modules windows tested? (they have to be or I'll be
> running into bugs that may originate from the module itself, the chicken-egg
> problem)

Again, if you run into test failures, I'd like to see those failures
and tell if it is a test driver issue or your own Windows build issue.
The test results are very detailed about failures so it's easy to
tell.

> I don't use memcached but xcache (which is far faster/better for windows) is
> there a test for xcache?

Your personal preference of caching solutions is irrelevant here.

Memcached is used by the ngx_lua test suite to test some aspects of
its features, especially the TCP cosocket API.

> Where is the master perl test script? (I can only see some bare-bone
> examples)
>

The ngx_lua test suite always resides in ngx_lua's code repository
since day #1. It's under the t/ directory. The test cases are
declarative and the test suite is data driven. The examples your see
*are* the test cases themselves. Just follow the documentation about
the test suite and run it! :) See

    http://wiki.nginx.org/HttpLuaModule#Test_Suite

Best regards,
-agentzh

info

> The test cases are declarative and the test suite is data driven. The examples your see *are* the test cases themselves. Just follow the documentation > about the test suite and run it! :) See http://wiki.nginx.org/HttpLuaModule#Test_Suite

Ok, I'll get to it next week, first the next step, the merge between Lua and naxsi, a Waf on steroids :)

info

> Zero warning, zero errors !

>

Yay!

While reading http://openresty.org/download/agentzh-nginx-tutorials-en.html I want to add array-var-nginx-module,

Got some more for you :)

/src/ngx_http_array_var_module.c(80) : warning C4152: nonstandard extension, function/data pointer conversion in _expression_
/src/ngx_http_array_var_module.c(89) : warning C4152: nonstandard extension, function/data pointer conversion in _expression_
/src/ngx_http_array_var_module.c(141) : warning C4152: nonstandard extension, function/data pointer conversion in _expression_
/src/ngx_http_array_var_module.c(258) : warning C4152: nonstandard extension, function/data pointer conversion in _expression_
/src/ngx_http_array_var_module.c(312) : warning C4152: nonstandard extension, function/data pointer conversion in _expression_
/src/ngx_http_array_var_module.c(371) : warning C4152: nonstandard extension, function/data pointer conversion in _expression_

\src\ngx_http_array_var_module.c(480) : warning C4706: assignment within conditional _expression_
\src\ngx_http_array_var_module.c(497) : warning C4701: potentially uninitialized local variable 'last' used

/src/ngx_http_array_var_util.c(73) : warning C4152: nonstandard extension, function/data pointer conversion in _expression_

agentzh

Hello!

On Mon, Sep 30, 2013 at 5:43 AM,  <i...@ecsystems.nl> wrote:
>
> Got some more for you :)
>
> /src/ngx_http_array_var_module.c(80) : warning C4152: nonstandard extension,
> function/data pointer conversion in expression
[...]
> /src/ngx_http_array_var_util.c(73) : warning C4152: nonstandard extension,
> function/data pointer conversion in expression
>

Committed a patch to master to fix these warnings:

   https://github.com/agentzh/array-var-nginx-module/commit/fbc32b4

Please try out the latest master of ngx_array_var :)

Thanks!
-agentzh

info

> Please try out the latest master of ngx_array_var :)

You missed one :)

\src\ngx_http_array_var_module.c(503) : warning C4701: potentially uninitialized local variable 'last' used

Last is not defined near line 421;

static ngx_int_t
ngx_http_array_var_split(ngx_http_request_t *r, ngx_str_t *res,
    ngx_http_variable_value_t *v, void *data)
{
    ngx_http_array_split_data_t             *conf = data;
    ngx_http_variable_value_t               *sep, *str;
    ngx_str_t                               *s;
    u_char                                  *pos, *end, *last;
    ssize_t                                  max, i, len = 4;
    ngx_array_t                             *array;

agentzh

Hello!

On Tue, Oct 1, 2013 at 2:41 AM,  <i...@ecsystems.nl> wrote:
> You missed one :)
>
> \src\ngx_http_array_var_module.c(503) : warning C4701: potentially
> uninitialized local variable 'last' used
>

Fixed in master:

    https://github.com/agentzh/array-var-nginx-module/commit/467674

Please try it again :)

Thanks!
-agentzh

info

> \src\ngx_http_array_var_module.c(503) : warning C4701: potentially

> uninitialized local variable 'last' used
>

Fixed in master:

https://github.com/agentzh/array-var-nginx-module/commit/467674

Please try it again :)

High 5 ! :)

info

On Thursday, September 26, 2013 8:33:20 PM UTC+2, agentzh wrote:

since day #1. It's under the t/ directory. The test cases are
declarative and the test suite is data driven. The examples your see
*are* the test cases themselves. Just follow the documentation about
the test suite and run it! :) See

http://wiki.nginx.org/HttpLuaModule#Test_Suite

Best regards,
-agentzh

Well I have tried to get it to compile, this is near impossible to get it working under Windows due to maaaannnnyyyy fatal errors and a lot of missing dependencies and also every config change assumes linux paths, so I wrote my own prove thing :) http://nginx-win.ecsds.eu/prove.zip feedback appreciated :)

I know you can't run windows stuff but maybe by just reading the files you could suggest which, jet missing, tests are the most important and no I'm not going to rewrite all of them, there's too many between all the modules.

agentzh

Hello!

On Tue, Dec 10, 2013 at 10:45 AM,  <i...@ecsystems.nl> wrote:
>
> Well I have tried to get it to compile, this is near impossible to get it
> working under Windows due to maaaannnnyyyy fatal errors and a lot of missing
> dependencies and also every config change assumes linux paths, so I wrote my
> own prove thing :) http://nginx-win.ecsds.eu/prove.zip feedback appreciated
> :)

If you are using ActivePerl or Strawberry Perl on Windows, then you
can just use the "cpan" utility to install all the dependencies
automatically:

    cpan Test::Nginx

Let me know if you have any particular dependency installation issues
here. I used to do a lot of Perl hacking on Windows (many years ago),
so I'd love to hear about them.

After installing everything here, you're still recommended to use the
github version of Test::Nginx because there have accumulated quite
some changes that have not yet been released to CPAN:

    https://github.com/agentzh/test-nginx

>
> I know you can't run windows stuff but maybe by just reading the files you
> could suggest which, jet missing, tests are the most important and no I'm
> not going to rewrite all of them, there's too many between all the modules.
>

No, you surely don't want to rewrite the whole test suite, which is
HUGE. And it's just a matter of porting the test scaffold Test::Nginx
over to Windows.

Regards,
-agentzh

info

I've spend an entire week trying to get this POS to compile for windows and not even mentioning all the patchwork it needs to accommodate windows paths+config files+starting stopping nginx, after 4 hours writing my own prove, which can be used for anything (not just nginx but iis and apache as well), doesn't need perl only curl which is well maintained and in compiled format and does ssl as well, I'm sticking with this one. It works, can be easily changed, anyone can understand how it works and most of all it just runs and works on any windows version.

Op dinsdag 10 december 2013 19:52:55 UTC+1 schreef agentzh:

No, you surely don't want to rewrite the whole test suite, which is
HUGE. And it's just a matter of porting the test scaffold Test::Nginx
over to Windows.

Regards,
-agentzh