Package management site and toolchain for OpenResty (Was Re: [openresty-en] [ANN] lua-resty-random, lua-resty-scrypt, and lua-scrypt 0.0.1)

agentzh · 2013-12-21T03:39:45+00:00

OK, it's really more flexible.ThanksJeromeLe vendredi 20 décembre 2013 12:12:40 UTC+1, jerome lafon a écrit :Hello,Can someone explain me what...

Package management site and toolchain for OpenResty (Was Re: [openresty-en] [ANN] lua-resty-random, lua-resty-scrypt, and lua-scrypt 0.0.1)

agentzh

Hello!

On Fri, Dec 20, 2013 at 5:19 AM, Aapo Talvensaari wrote:
> Great. I started to think about this naming conflict, and I think that we
> should have a convention in place for packages.

I'm thinking about this too, for the centralized package manager.
Maybe we can follow the github convention here, prefix the module
author's nick in the module name to install, for example,

    lresty install agentzh/resty.random

When the user simply type a Lua module name without giving the author
prefix, as in

    lresty install resty.random

then lresty will just prompt the user with a list of Lua modules with
the name "resty.random" from different authors, for example,

    $ lresty install resty.random
    I have several resty.random modules for your to choose from:
    [1] bakins/resty.random
    [2] agentzh/resty.random
    [3] leafo/resty.random
    Which one do you want to install (type a number)?

When there is only one match, lresty will just proceed without prompting.

We can also introduce a module registration mechanism, that is,
certain modules can be registered on the centralized package site.
This process has to be approved by the administrators. For registered
Lua modules, when the user simply type the module name, the registered
one will be installed without prompting even when there are other
candidates. The user can still install other unregistered modules by
explicitly specifying the author prefix, like "bakins/resty.random".

> Even without package
> management. I think that we should follow the convention you have used in
> lua-resty-core.
>
> E.g.
>
> /usr/local/openresty/lualib/resty/core.lua
> /usr/local/openresty/lualib/restycore/* (all other lib files, if any)
>
> So I propose that lua-resty-string should follow this too (this is a
> breaking change, I know, but better to to it now than later).

No. I think module names and the library name (or distribution name)
are two separate things. For the example you talk about above,
lua-resty-string is the library name, or distribution name, while
resty.random, resty.string, and etc are module names inside this
library (or distribution).

We could, however, move the resty.random module out of the
lua-resty-string library, and put it into a lua-resty-random library,
for example (and I admit it is indeed better). But having a
resty.random in a Lua library with whatever name is totally fine,
technically :)

For the package management site, we can also put some automatic code
checking hooks when the author publish his Lua code. But we should
have something running first :)

Best regards,
-agentzh

leafot

Not sure if this is useful but if you want to just reuse LuaRocks I wrote a LuaRocks package repository site in OpenResty. The one I run is called MoonRocks: http://rocks.moonscript.org/

It's opensource so you can just host your own with OpenResty packages only: https://github.com/leafo/moonrocks-site

I highly suggest re-using LuaRocks. I don't think Lua community needs another package manager. I'd rather see LuaRocks contributed it.

On Fri, Dec 20, 2013 at 11:39 AM, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

Hello!

On Fri, Dec 20, 2013 at 5:19 AM, Aapo Talvensaari wrote:
> Great. I started to think about this naming conflict, and I think that we
> should have a convention in place for packages.

I'm thinking about this too, for the centralized package manager.
Maybe we can follow the github convention here, prefix the module
author's nick in the module name to install, for example,

lresty install agentzh/resty.random

When the user simply type a Lua module name without giving the author
prefix, as in

lresty install resty.random

then lresty will just prompt the user with a list of Lua modules with
the name "resty.random" from different authors, for example,

$ lresty install resty.random
I have several resty.random modules for your to choose from:
[1] bakins/resty.random
[2] agentzh/resty.random
[3] leafo/resty.random
Which one do you want to install (type a number)?

When there is only one match, lresty will just proceed without prompting.

We can also introduce a module registration mechanism, that is,
certain modules can be registered on the centralized package site.
This process has to be approved by the administrators. For registered
Lua modules, when the user simply type the module name, the registered
one will be installed without prompting even when there are other
candidates. The user can still install other unregistered modules by
explicitly specifying the author prefix, like "bakins/resty.random".

> Even without package
> management. I think that we should follow the convention you have used in
> lua-resty-core.
>
> E.g.
>
> /usr/local/openresty/lualib/resty/core.lua
> /usr/local/openresty/lualib/restycore/* (all other lib files, if any)
>
> So I propose that lua-resty-string should follow this too (this is a
> breaking change, I know, but better to to it now than later).

No. I think module names and the library name (or distribution name)
are two separate things. For the example you talk about above,
lua-resty-string is the library name, or distribution name, while
resty.random, resty.string, and etc are module names inside this
library (or distribution).

We could, however, move the resty.random module out of the
lua-resty-string library, and put it into a lua-resty-random library,
for example (and I admit it is indeed better). But having a
resty.random in a Lua library with whatever name is totally fine,
technically :)

For the package management site, we can also put some automatic code
checking hooks when the author publish his Lua code. But we should
have something running first :)

Best regards,
-agentzh
.

agentzh

Hello leaf!

On Fri, Dec 20, 2013 at 11:44 AM, leaf corcoran wrote:
> Not sure if this is useful but if you want to just reuse LuaRocks I wrote a
> LuaRocks package repository site in OpenResty. The one I run is called
> MoonRocks: http://rocks.moonscript.org/
>

Oh this is nifty :) Thanks for sharing!

> It's opensource so you can just host your own with OpenResty packages only:
> https://github.com/leafo/moonrocks-site
>

Oh, this can also serve as a good demo site for OpenResty (or for
Lapis to be more specific) :)

> I highly suggest re-using LuaRocks.
>

Thank you for the suggestion!

To be honest I personally kinda dislike LuaRocks. I don't know why.
Maybe it's because I haven't fully understood all its magic yet. I
think I will have a closer look at LuaRocks :)

FWIW, even if we reuse luarocks's implementation, we should have
another name for the command-line utility, like lresty, to prevent
potential confusion on the user side.

> I don't think Lua community needs
> another package manager. I'd rather see LuaRocks contributed it.
>

Well, I think this is not for the Lua community in general. This is
for the OpenResty community specifically :)

Regards,
-agentzh

chase

agentzh,

Leaf beat me to listing his MoonRocks website!

On Friday, December 20, 2013 3:13:34 PM UTC-5, agentzh wrote:
>
> To be honest I personally kinda dislike LuaRocks. I don't know why. 
> Maybe it's because I haven't fully understood all its magic yet. I 
> think I will have a closer look at LuaRocks :) 
>

I also believe that adopting LuaRocks and rockspec files as the core 
package manager is the way to go.
There is no need to entirely reinvent the wheel unless there is some 
immediate benefit.

> FWIW, even if we reuse luarocks's implementation, we should have 
> another name for the command-line utility, like lresty, to prevent 
> potential confusion on the user side. 
>

What about about a general utility for OpenResty which includes package 
management?
I also imagine that it could manage Nginx modules and updating the 
OpenResty installation gracefully without shutting down the server.

> > I don't think Lua community needs 
> > another package manager. I'd rather see LuaRocks contributed it. 
> > 
>
> Well, I think this is not for the Lua community in general. This is 
> for the OpenResty community specifically :) 
>

brian

Rock spec files are a PITA to me. Why do I have to rename it to match the version? This is just silly, IMO, and makes it annoying to use.

Npm is the gold standard for me.

brian

I like this as a variation on rocks: https://github.com/leafo/heroku-buildpack-lua/

With Lua (like JavaScript) you can have more than one version of a module loaded at a time. This helps avoid "dependency hell" but you need a custom loader. Only works with Lua, not C however, however with FFI, I rarely write Lua modules in C any more.

Seems like this functionality of node/npm would be fairly easy to port to Lua.  

I think Luarocks would need significant modifications to be generally usable.

Go's dependency management is also interesting, but that's another email...

Sent from my iPhone

> On Dec 23, 2013, at 7:10 PM, Brian Akins <br...@akins.org> wrote:
> 
> Rock spec files are a PITA to me. Why do I have to rename it to match the version? This is just silly, IMO, and makes it annoying to use.
> 
> Npm is the gold standard for me.

chase

Every package manager has warts, NPM included.

Over at the Luvit mailing list, they have had a similar discussion about making an alternative Lua package manager that fit for their environment.

Regarding the Heroku Buildpack you mentioned, that only works because it is only consuming LuaRocks. The LuaRocks it consumes still require the same verbosity that you seem to dislike.

A lot of calculations will still require C, like cryptographic hashing functions as with bcrypt or scrypt. So I think that support for building native-compiled modules is necessary.

NPM uses node-gyp for native modules, but node-gyp itself requires Python v2.7 and make. I belive it used to build using CMake but then they adopted gyp.

Would it be better to just stick with normal Makefiles since cross-platform compatibility isn't a huge deal with OpenResty?

Respectfully,

Chase

On Mon, Dec 23, 2013 at 7:38 PM, Brian Akins <br...@akins.org> wrote:

I like this as a variation on rocks: https://github.com/leafo/heroku-buildpack-lua/

With Lua (like _javascript_) you can have more than one version of a module loaded at a time. This helps avoid "dependency hell" but you need a custom loader. Only works with Lua, not C however, however with FFI, I rarely write Lua modules in C any more.

Seems like this functionality of node/npm would be fairly easy to port to Lua.

I think Luarocks would need significant modifications to be generally usable.

Go's dependency management is also interesting, but that's another email...

Sent from my iPhone

> On Dec 23, 2013, at 7:10 PM, Brian Akins <br...@akins.org> wrote:
>
> Rock spec files are a PITA to me. Why do I have to rename it to match the version? This is just silly, IMO, and makes it annoying to use.
>
> Npm is the gold standard for me..

leafot

LuaRock's native code building functionality is very well done with the "builtin" build type. Typically when I see a rockspec with makefile I end up editing it due to different header locations for different versions of Lua. LuaRocks handles this automatically.

http://luarocks.org/en/Creating_a_rock

On Mon, Dec 23, 2013 at 5:00 PM, Chase Colman <ch...@colman.io> wrote:

Every package manager has warts, NPM included.

Over at the Luvit mailing list, they have had a similar discussion about making an alternative Lua package manager that fit for their environment.

Regarding the Heroku Buildpack you mentioned, that only works because it is only consuming LuaRocks. The LuaRocks it consumes still require the same verbosity that you seem to dislike.

A lot of calculations will still require C, like cryptographic hashing functions as with bcrypt or scrypt. So I think that support for building native-compiled modules is necessary.
NPM uses node-gyp for native modules, but node-gyp itself requires Python v2.7 and make. I belive it used to build using CMake but then they adopted gyp.

Would it be better to just stick with normal Makefiles since cross-platform compatibility isn't a huge deal with OpenResty?

Respectfully,
Chase

On Mon, Dec 23, 2013 at 7:38 PM, Brian Akins <br...@akins.org> wrote:

I like this as a variation on rocks: https://github.com/leafo/heroku-buildpack-lua/

With Lua (like _javascript_) you can have more than one version of a module loaded at a time. This helps avoid "dependency hell" but you need a custom loader. Only works with Lua, not C however, however with FFI, I rarely write Lua modules in C any more.

Seems like this functionality of node/npm would be fairly easy to port to Lua.

I think Luarocks would need significant modifications to be generally usable.

Go's dependency management is also interesting, but that's another email...

Sent from my iPhone

> On Dec 23, 2013, at 7:10 PM, Brian Akins <br...@akins.org> wrote:
>
> Rock spec files are a PITA to me. Why do I have to rename it to match the version? This is just silly, IMO, and makes it annoying to use.
>
> Npm is the gold standard for me.

--
.

brian

It's not the verbosity, it's the fact I have to rename the spec file
everytime I bump the version. It also will hapily load a version of a
module that doesn't met your restrictions listed in the spec file. (It's
been a while, but that's how it used to work.)   Also, module installs are
global by default and sort of a pain to use when you wan them installed on
a per project basis.

As for npm, I only know about it from an end-users perspective - both using
and creating modules - I was just using it as an example.

leafot

I think changing the rockspec per version is a tradeoff. It's a single unit of information that contains all the information necessary to install a given version regardless of how or where your sourcecode lives. I typically also have a 'dev' or 'scm' rockspec which points at the head of my repository which I don't have to update when I release new code. Don't you have to bump the version in package.json when you release a new version on npm? The only package manager I've used where it entirely pulls package versions from git is http://getcomposer.org/

I think the solution to your problem is a script that just bumps a rockspec version and where it points. That's already half way done with the luarocks new_version command. This was where I was going with the moonrocks command line tool, where it would just do everything for you and upload rockspec and packed rock. I just haven't worked on it recently though.

LuaRocks supports a --tree (or --local for ~) flag for installing your modules in any directory you want. It's very easy to bundle all your dependencies away from the system installation, then insert a new entry in package.path and package.cpath as your app starts up. That's how the lua heroku buildpack works.

LuaRocks does not work well with installing multiple versions of a package at once, but that kind of thing can mostly be avoided by doing a project local module tree.

On Tue, Dec 24, 2013 at 5:30 AM, Brian Akins <br...@akins.org> wrote:

It's not the verbosity, it's the fact I have to rename the spec file everytime I bump the version. It also will hapily load a version of a module that doesn't met your restrictions listed in the spec file. (It's been a while, but that's how it used to work.) Also, module installs are global by default and sort of a pain to use when you wan them installed on a per project basis.

As for npm, I only know about it from an end-users perspective - both using and creating modules - I was just using it as an example.

From the feedback I've gotten from my $DAYJOB peers (node and go developers mostly), openresty will have a hard time gaining more adoption if the current version of Luarocks is the dependency/package manager.

Perhaps Luarocks is a good starting place, especially WRT building modules. I need to take a look at the code, perhaps it can be used as the core for some experimentation.

.

brian

Yes, in npm you must bump the version in the package.json, but in rocks you have to do that and rename the file. It's a small thing but it's annoying and if not done correctly screws up git history.

Node/npm allows modules to depend on different versions of other modules, even in the same "project". Ie, A can depend on version 1 of B and version 2 of C, and B can depend on version 3 of C. It all works. The end user doesn't generally get sucked into "dependency hell." With a custom version of require, the same could be done for Lua.

 I have wrapper scripts I use with Luarocks, but they are a hack. If everyone is happy with Luarocks, then good for you. I think that adoption of openresty will be slowed by it. I know the luvit guys have been working on alternatives.  When I dealt with it everyday, I didn't think it was a big deal, but after stepping away for a while and working in other frameworks a lot more, I find Luarocks to be a PITA. But maybe that's just me.