Response with zipped files

bogdan · 2016-05-09T17:16:47+00:00

On 9 May 2016 at 10:15, <al...@chameleonx.com> wrote:Hey!I have a variable, holding entire web-page, another variable holding a script string.sub-ed f...

Response with zipped files

bogdan

Hello, everybody

We are using OpenResty and we are very happy with the product.Congratulations!

Currently we need to allow users to download multiple files as a singlearchive. The files are stored on disk (they are some sort of log files),and in a page we need to provide a link to download all files for thelast month. It's easy to allow downloading individual files, but allfiles at once (and archived)... I don't know how to do it.

Is it possible to use this NginX module?
https://www.nginx.com/resources/wiki/modules/zip/#

Do you have any other alternative?

Thank you

Kind regards,

Bogdan

bogdan

No answer to this?
Bummer

Bogdan Irimia

Monday, May 9, 2016 12:16 PM via Postbox

Hello, everybody

We are using OpenResty and we are very happy with the product. Congratulations!

Currently we need to allow users to download multiple files as a single archive. The files are stored on disk (they are some sort of log files), and in a page we need to provide a link to download all files for the last month. It's easy to allow downloading individual files, but all files at once (and archived)... I don't know how to do it.
Is it possible to use this NginX module?
https://www.nginx.com/resources/wiki/modules/zip/#

Do you have any other alternative?

Thank you

Kind regards,

Bogdan

aapo.talvensaari

On Thursday, 12 May 2016 17:59:34 UTC+3, bogdan wrote:

No answer to this?
Bummer

How does this question differ from this:

https://groups.google.com/forum/#!searchin/openresty-en/zip/openresty-en/-bTMc91WNkQ/laDckzgVDgAJ

Did you try any of the suggestions there?

aapo.talvensaari

On Sunday, 15 May 2016 23:48:24 UTC+3, Aapo Talvensaari wrote:

On Thursday, 12 May 2016 17:59:34 UTC+3, bogdan wrote:

No answer to this?
Bummer

How does this question differ from this:
https://groups.google.com/forum/#!searchin/openresty-en/zip/openresty-en/-bTMc91WNkQ/laDckzgVDgAJ

Did you try any of the suggestions there?

Here is a list that you may look at:

1. http://lua-users.org/wiki/CompressionAndArchiving

2. https://github.com/evanmiller/mod_zip

But then again, why make things hard, if you can just use:

1. os.execute

2. https://github.com/juce/lua-resty-shell

You could run the above to pre-build archives on:

1. ngx.timer_at (say once a day)

2. cron (say once a day)

3. on first request and then cache

Regards

Aapo

bogdan

Thank you, Aapo, for the answer

Indeed, there is no difference from the previous question, but the answers there weren't enough.
First of all, we would need stream processing, but I didn't find a module that does that and to be fully tested with OpenResty. That's why I asked, hoping that somebody actually had to do this.

The os.execute/lua-resty-shell option is another approach we thought of, but we would prefer to have the least coupling with the os. Our solution should work on other architectures (arm) and there the zip utilities might be different.

Anyway, I will take a look at https://github.com/evanmiller/mod_zip, this is the way I thought would be the best. But we will need to recompile OpenResty and I was hoping there is something written as a Lua module that doesn't need this :)

I was hoping there is a pure-lua implementation of the zip algorithm, which is non-blocking and has no external ("lib") dependencies :) I guess there isn't anything like this, though.

Thank you for the help

Bogdan

Aapo Talvensaari wrote:

On Sunday, 15 May 2016 23:48:24 UTC+3, Aapo Talvensaari wrote:

    On Thursday, 12 May 2016 17:59:34 UTC+3, bogdan wrote:

        No answer to this?
        Bummer

    How does this question differ from this:
    https://groups.google.com/forum/#!searchin/openresty-en/zip/openresty-en/-bTMc91WNkQ/laDckzgVDgAJ
    <https://groups.google.com/forum/#!searchin/openresty-en/zip/openresty-en/-bTMc91WNkQ/laDckzgVDgAJ>

    Did you try any of the suggestions there?

Here is a list that you may look at:

1. http://lua-users.org/wi ki/CompressionAndArchiving
2. https://github.com/evanmiller/mod_zip

But then again, why make things hard, if you can just use:

1. os.execute
2. https://github.com/juce/lua-resty-shell

You could run the above to pre-build archives on:

1. ngx.timer_at (say once a day)
2. cron (say once a day)
3. on first request and then cache

Regards
Aapo.

aapo.talvensaari

On Tuesday, 17 May 2016 14:36:11 UTC+3, bogdan wrote:

I was hoping there is a pure-lua implementation of the zip algorithm, which is non-blocking and has no external ("lib") dependencies :) I guess there isn't anything like this, though.

Yes, I'm afraid there isn't, and there probably will never be, because there is in many cases no such thing called non-blocking file io. Even in Nginx the file IO is blocking if AIO is not enabled (which has it's own drawbacks). There might be a solution in a distant future.

I'm not sure your case is a case where you need to stream and generate zip-files on demand at the maximum performance. Seems more like a feature that is quite rarely used, e.g. when someone wants to export the data outside the system in a raw form. But I do not know all the details. Best thing you can do is to make these zip-files outside Nginx and then just stream those premade zip-files with Nginx. If you really need to be able for user to define and build their own personalized zip-files (e.g. choosing log files and then clicking download), you cannot prebuild. But do you really need such a feature, I don't know.

Here is another option:

1. Run node.js server, and proxy there

2. On node.js server use something like this: https://github.com/archiverjs/node-zip-stream

I might look at building lua-resty-zipstream at some point. Or maybe you can do it?

aapo.talvensaari

I might look at building lua-resty-zipstream at some point. Or maybe you can do it?

I actually took a small look on this. I found this:

https://github.com/CTrabant/fdzipstream

Looks quite nice, but uses file descriptors, and that probably mean you need to patch OpenResty with this patch (not yet merged):

https://github.com/openresty/lua-nginx-module/pull/450

aapo.talvensaari

On Wednesday, 18 May 2016 00:01:59 UTC+3, Aapo Talvensaari wrote:

I might look at building lua-resty-zipstream at some point. Or maybe you can do it?

I actually took a small look on this. I found this:
https://github.com/CTrabant/fdzipstream

Looks quite nice, but uses file descriptors, and that probably mean you need to patch OpenResty with this patch (not yet merged):
https://github.com/openresty/lua-nginx-module/pull/450

Also I think I suggested this before:

1. Use ZipWriter

2. Write your own "writer" function to write to output stream (ngx.print)

3. Set your writer with this: http://moteus.github.io/ZipWriter/#ZipWriter:open_writer

I have not tested this. Looks quite easy, though.

Regards

Aapo

aapo.talvensaari

On Wednesday, 18 May 2016 00:21:02 UTC+3, Aapo Talvensaari wrote:

Also I think I suggested this before:

1. Use ZipWriter
2. Write your own "writer" function to write to output stream (ngx.print)
3. Set your writer with this: http://moteus.github.io/ZipWriter/#ZipWriter:open_writer

I have not tested this. Looks quite easy, though.

Infact I have. The simplest thing that can possibly work, here you go:

local ZipWriter = require "ZipWriter"

function make_reader(fname) 
  local f = assert(io.open(fname, 'rb')) 
  local chunk_size = 65536 
  local desc = { 
    istext = true, 
    isfile = true 
  } 
  return desc, desc.isfile and function() 
    local chunk = f:read(chunk_size) 
    if chunk then return chunk end 
    f:close() 
  end 
end 

ZipStream = ZipWriter.new() 
ZipStream:open_writer(function(chunk) 
  ngx.print(chunk) 
end) 
ZipStream:write('log-1.log', make_reader('log-1.log')) 
ZipStream:write('log-2.log', make_reader('log-2.log')) 
ZipStream:close()

Almost exactly the same as in ZipWriter docs. Now you may also want to utilize these (not sure if needed):

- http://moteus.github.io/ZipWriter/#co_writer

- http://moteus.github.io/ZipWriter/#co_reader

And you could also extend / fork ZipStream and provide support for this as well:

- https://github.com/openresty/lua-nginx-module#ngxthreadspawn

E.g. create new ngx_writer and ngx_reader functions for these:

- https://github.com/moteus/ZipWriter/blob/master/lua/ZipWriter.lua#L1133-L1149

Maybe not even needed.

But then again, the above could be all that you need. Start with that and when you hit problems etc. look for how to make it better.

bogdan

Oau, great answer!

Thank you very much! I will check and see what is the best way to proceed.

Bogdan

Aapo Talvensaari wrote:

On Wednesday, 18 May 2016 00:21:02 UTC+3, Aapo Talvensaari wrote:

    Also I think I suggested this before:

    1. Use ZipWriter
    2. Write your own "writer" function to write to output stream
    (ngx.print)
    3. Set your writer with this:
    http://moteus.github.io/ZipWriter/#ZipWriter:open_writer
    <http://moteus.github.io/ZipWriter/#ZipWriter:open_writer>

    I have not tested this. Looks quite easy, though.

Infact I have. The simplest thing that can possibly work, here you go:

||
localZipWriter=r equire"ZipWriter"

functionmake_reader(fname)
localf =assert(io.open(fname,'rb'))
localchunk_size =65536
localdesc ={
    istext =true,
    isfile =true
}
returndesc,desc.isfile andfunction()
localchunk =f:read(chunk_size)
ifchunk thenreturnchunk end
    f:close()
end
end

ZipStream=ZipWriter.new()
ZipStream:open_writer(function(chunk)
ngx.print(chunk)
end)
ZipStream:write('log-1.log',make_reader('log-1.log'))
ZipStream:write('log-2.log',make_reader('log-2.log'))
ZipStream:close()

Almost exactly the same as in ZipWriter docs. Now you may also want to
utilize these (not sure if needed):

- http://moteus.github.io/ZipWriter/#co_writer
- http://moteus.github.io/ZipWriter/#co_reader

And you could also extend / fork ZipStream and provide support for
this as well:

- https://github.com/openresty/l ua-nginx-module#ngxthreadspawn

E.g. create new ngx_writer and ngx_reader functions for these:

- https://github.com/moteus/ZipWriter/blob/master/lua/ZipWriter.lua#L1133-L1149
-
https://github.com/moteus/ZipWriter/blob/master/lua/ZipWriter.lua#L1133-L1149

Maybe not even needed.

But then again, the above could be all that you need. Start with that
and when you hit problems etc. look for how to make it better..