Hello!
On Mon, Aug 31, 2015 at 1:15 PM, Zhe Zhang wrote:
> The server I’m building will be an origin server for our media-CDN: the
> media-CDN nodes(that back our website) will fetch images from this server.
> The image server is expected to receive around 800,000 download requests per
> day from our CDN.
Thanks is not much :)
> The images range from tens of kilobytes to several
> megabytes in size.
> Upon receiving a download request, the server will first try fetching it
> from our AWS S3 instances in China. If not found, it will try fetching it
> from a more-remote server in US. In either case, if the image is found, then
> my image server will add watermark to it and send it to our media-CDN.
> I’ve also attached the Nginx conf file that summarizes its logic.
>
ngx.location.capture always fully buffers the response data (in the
Lua space) which is not suitable for large responses. There are two
options:
1. You can just use cosockets (or lua-resty-http libraries supporting
streaming) to stream the response data with a constant size buffer.
2. You can use try_files and access_by_lua as well as ngx_proxy to
control the flow.
But the water-mark thing might defeat the streaming processing effort.
> 1. Since I need to get the images watermarked, then it seems that each image
> has to be fully loaded into memory any way, which seems similar in memory
> cost to ngx.location.capture fully bufferning the images. What do you think?
Yes.
> Do you think there’s any way to reduce the memory footprint of
> downloading-and-watermarking large photos? Would try_files directive help
> here?
>
No, try_files has nothing to do with image processing.
> 2. Do you know around how much memory overhead in general does each Nginx
> connection cost? Is there any command that I can use to get the answer? The
> reason I’m asking this is for for estimating memory cost of the image
> server: for instance, assuming every image is 1MB, then the memory cost for
> 1000 concurrent connections would be 1000 * (overhead_per_conn + 1MB).
>
Yeah, that's why we need streaming processing in the first place.
Because 800,000 requests per day is really not much (assuming the
traffic distribute relatively evenly), you might never run into that
many concurrent connections in normal conditions and you can always
use the standard ngx_limit_conn module to limit the concurrency level
automatically for you (and maybe request rate too via the standard
ngx_limit_req module).
> 3. If there’s no good way to limit memory cost for download-and-watermark
> large photos, then maybe I can cache the large watermarked photos in Nginx?
> Would it be easy to code this logic?
>
Yes, it's easy to configure. See above.
Regards,
-agentzh
P.S. It's required to subscribe to the mailing list before sending mails to it.
