ngx_http_lua_send_chain_link in body filter trims data?

aviram · 2013-03-20T01:18:27+00:00

You can use luafilesystem, just keep in mind that the operation will be blocking.

ngx_http_lua_send_chain_link in body filter trims data?

aviram

Hi,

I've been using ngx_http_lua_send_chain_link in a body filter (written in C - that is the capture filter of the Lua module) in order to send chain data back straight to the user.

For some reason, the data is trimmed - for a file with a size of 2MB, only 100KB are actually downloaded.

The response doesn't contain a Content-Length field, but an ASCII zero ("0") is sent during the response (but prior to its last buffer).

When does ngx_http_lua_send_chain_link send a zero? I know that it does when the input is NULL, but I verify that it isn't before calling it...

Thanks in advance

Aviram

agentzh

Hello!

On Tue, Mar 19, 2013 at 10:18 AM,  aviram wrote:
> I've been using ngx_http_lua_send_chain_link in a body filter (written in C
> - that is the capture filter of the Lua module) in order to send chain data
> back straight to the user.
> For some reason, the data is trimmed - for a file with a size of 2MB, only
> 100KB are actually downloaded.
>
> The response doesn't contain a Content-Length field, but an ASCII zero ("0")
> is sent during the response (but prior to its last buffer).
>

Because you're coding in C right now, it is very likely that you're
having buffer management issues (or memory issues in general) here :)
Try running your code with the Valgrind's memcheck tool to see if
there's something going on. Also, systemtap is a great tool to trace
the status of chain links in various levels of output filters (even
down to the level of the writev syscall).

Since you do not provide your code, I cannot get you any further.

Best regards,
-agentzh

aviram

Hi,

Thanks for the tips, but I think I have a more basic gap in the understanding.

I'm adding here the only change I've made - the new ngx_http_lua_capture_body_filter. The that were changed are marked by "/*~~~*/".

The phenomena that happens is that when trying to download a file of 2MB, only 3 chunks of 32768 bytes are returned (96KB all in all).

static ngx_int_t

ngx_http_lua_capture_body_filter(ngx_http_request_t *r, ngx_chain_t *in)

{

/* int rc; */

ngx_http_lua_ctx_t *ctx;

ngx_http_lua_ctx_t *pr_ctx;

ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,

"lua capture body filter, uri \"%V\"", &r->uri);

if (in == NULL) {

return ngx_http_lua_next_body_filter(r, NULL);

}

ctx = ngx_http_get_module_ctx(r, ngx_http_lua_module);

if (!ctx || !ctx->capture) {

dd("no ctx or no capture %.*s", (int) r->uri.len, r->uri.data);

return ngx_http_lua_next_body_filter(r, in);

}

if (ctx->run_post_subrequest) {

ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,

"lua body filter skipped because post subrequest "

"already run");

return NGX_OK;

}

if (r->parent == NULL) {

ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,

"lua body filter skipped because no parent request "

"found");

return NGX_ERROR;

}

pr_ctx = ngx_http_get_module_ctx(r->parent, ngx_http_lua_module);

if (pr_ctx == NULL) {

return NGX_ERROR;

}

/*~~~*/

rc = ngx_http_lua_send_chain_link(r->parent, pr_ctx, in);

if (rc == NGX_ERROR || rc >= NGX_HTTP_SPECIAL_RESPONSE) {

ngx_log_error(NGX_LOG_WARN, r->connection->log, 0, "~~~ Failed sending...");

return NGX_ERROR;

}

return NGX_OK;

/*~~~*/

ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,

"lua capture body filter capturing response body, uri "

"\"%V\"", &r->uri);

rc = ngx_http_lua_add_copy_chain(r, pr_ctx, &ctx->last_body, in);

if (rc != NGX_OK) {

return NGX_ERROR;

}

ngx_http_lua_discard_bufs(r->pool, in);

return NGX_OK;

}

On Tuesday, March 19, 2013 9:23:43 PM UTC+2, agentzh wrote:

Hello!

On Tue, Mar 19, 2013 at 10:18 AM, aviram wrote:
> I've been using ngx_http_lua_send_chain_link in a body filter (written in C
> - that is the capture filter of the Lua module) in order to send chain data
> back straight to the user.
> For some reason, the data is trimmed - for a file with a size of 2MB, only
> 100KB are actually downloaded.
>
> The response doesn't contain a Content-Length field, but an ASCII zero ("0")
> is sent during the response (but prior to its last buffer).
>

Because you're coding in C right now, it is very likely that you're
having buffer management issues (or memory issues in general) here :)
Try running your code with the Valgrind's memcheck tool to see if
there's something going on. Also, systemtap is a great tool to trace
the status of chain links in various levels of output filters (even
down to the level of the writev syscall).

Since you do not provide your code, I cannot get you any further.

Best regards,
-agentzh

aviram

The line declaring the variable rc is commented out by mistake. Consider it non commented out.

On Tuesday, March 19, 2013 7:18:26 PM UTC+2, avi...@adallom.com wrote:

Hi,

I've been using ngx_http_lua_send_chain_link in a body filter (written in C - that is the capture filter of the Lua module) in order to send chain data back straight to the user.
For some reason, the data is trimmed - for a file with a size of 2MB, only 100KB are actually downloaded.

The response doesn't contain a Content-Length field, but an ASCII zero ("0") is sent during the response (but prior to its last buffer).

When does ngx_http_lua_send_chain_link send a zero? I know that it does when the input is NULL, but I verify that it isn't before calling it...

Thanks in advance
Aviram

agentzh

Hello!

On Wed, Mar 20, 2013 at 8:38 AM,  aviram wrote:
>     /*~~~*/
>     rc = ngx_http_lua_send_chain_link(r->parent, pr_ctx, in);

It is very wrong to send data downstream on behalf of the parent
request in a subrequest's output filter.

Because nginx has the notion of "active requests", that is, the
request is considered "active" if r == r->connection->data. Running
out of active requests means a lot of troubles, especially with the
standard postpone filter module.

You should always wake up the parent request from the subrequest and
then later when the parent is running, let the parent request handle
its own output. That is also exactly what ngx_lua's
ngx.location.capture is doing.

Best regards,
-agentzh

aviram

Hey,

On Thursday, March 21, 2013 1:26:37 AM UTC+2, agentzh wrote:

Hello!

On Wed, Mar 20, 2013 at 8:38 AM, aviram wrote:
> /*~~~*/
> rc = ngx_http_lua_send_chain_link(r->parent, pr_ctx, in);

It is very wrong to send data downstream on behalf of the parent
request in a subrequest's output filter.

Because nginx has the notion of "active requests", that is, the
request is considered "active" if r == r->connection->data. Running
out of active requests means a lot of troubles, especially with the
standard postpone filter module.

Yeah, that's what I thought was the problem, but I thought you'd have a trick up your sleeve :)

You should always wake up the parent request from the subrequest and
then later when the parent is running, let the parent request handle
its own output. That is also exactly what ngx_lua's
ngx.location.capture is doing.

Is there any way to wake up the parent *before* the subrequest ends (i.e. for every one or two chunks of the data), and then make the parent request wake up the subrequest?

I know that ngx_http_post_request would wake up a parent request from a subrequest, but is there a way do it the other way around?

Thanks,

Aviram

aviram

Another thing - if I call ngx_http_post_request from the capturefiler's body filter:

- Which request object should I pass to the function? The subrequest or the parent?

- Which function will called next in the context of the parent request? Is it ngx_http_lua_content_handler (I called capture from the content handler by Lua).

On Thursday, March 21, 2013 8:11:16 AM UTC+2, avi...@adallom.com wrote:

Hey,

On Thursday, March 21, 2013 1:26:37 AM UTC+2, agentzh wrote:
Hello!

On Wed, Mar 20, 2013 at 8:38 AM, aviram wrote:
> /*~~~*/
> rc = ngx_http_lua_send_chain_link(r->parent, pr_ctx, in);

It is very wrong to send data downstream on behalf of the parent
request in a subrequest's output filter.

Because nginx has the notion of "active requests", that is, the
request is considered "active" if r == r->connection->data. Running
out of active requests means a lot of troubles, especially with the
standard postpone filter module.

Yeah, that's what I thought was the problem, but I thought you'd have a trick up your sleeve :)

You should always wake up the parent request from the subrequest and
then later when the parent is running, let the parent request handle
its own output. That is also exactly what ngx_lua's
ngx.location.capture is doing.

Is there any way to wake up the parent *before* the subrequest ends (i.e. for every one or two chunks of the data), and then make the parent request wake up the subrequest?
I know that ngx_http_post_request would wake up a parent request from a subrequest, but is there a way do it the other way around?

Thanks,
Aviram

agentzh

Hello!

On Wed, Mar 20, 2013 at 11:11 PM,  aviram wrote:
>
> Is there any way to wake up the parent *before* the subrequest ends (i.e.
> for every one or two chunks of the data), and then make the parent request
> wake up the subrequest?

I've already told you to use ngx_http_post_request to wake up the
parent. It does not have to wait until the subrequest ends (as
indicated by calling the post_subrequest callback).

When you do not mark the bufs from the upstream in the subrequest's
body filter as consumed (that is, setting b->pos = b->last), then the
upstream will just stop calling the output body filter (in nonbuffered
mode) with more data and then immediately wake up the parent request
posed by you in the subrequest's body filter (by means of the
ngx_http_run_posted_requests call).

That way, you're essentially interleaving the processing of the
subrequest and its parent, which should be exactly what you want here.

> I know that ngx_http_post_request would wake up a parent request from a
> subrequest, but is there a way do it the other way around?
>

This is the only right way to go, AFAIK. Otherwise we may just ask for
troubles :)

Best regards,
-agentzh

agentzh

Hello!

On Thu, Mar 21, 2013 at 3:16 AM,  aviram wrote:
> Another thing - if I call ngx_http_post_request from the capturefiler's body
> filter:
>   - Which request object should I pass to the function? The subrequest or
> the parent?

The parent.

>   - Which function will called next in the context of the parent request? Is
> it ngx_http_lua_content_handler (I called capture from the content handler
> by Lua).
>

It is always r->write_event_handler where r is the parent request object.

r->write_event_handler in ngx_lua could be set to
ngx_http_lua_content_wev_handler (when content_by_lua* is used) or
ngx_http_core_run_phases (when rewrite/access_by_lua* are used). Both
of them resolve to ctx->resume_handler where ctx is of the type
ngx_http_lua_ctx_t. And ctx->resume_handler is usually
ngx_http_lua_wev_handler in normal contexts.

Best regards,
-agentzh

aviram

Hey,

agentzh, thanks for all the help. I got to a situation that is pretty close to working, but I think I'll still need you advice.

Here's what I've done:

- Configured the proxy module in non buffered mode.

- Made the body filter in capturefilter.c not discard the buffers and perform ngx_http_post_request so that the main request would be invoked.

- In the main request I get the buffer (passed by the ctx structure), send it to the user using ngx_http_lua_send_chain_link and then calling ngx_http_post_request.

The problem is that the body filter is called only two or three times (for every request), even for responses with large data (without my patch, it is called over 20 times).

Do you have an idea regarding what I'm doing wrong?

Thanks very much!

Aviram

On Thursday, March 21, 2013 9:39:17 PM UTC+2, agentzh wrote:

Hello!

On Wed, Mar 20, 2013 at 11:11 PM, aviram wrote:
>
> Is there any way to wake up the parent *before* the subrequest ends (i.e.
> for every one or two chunks of the data), and then make the parent request
> wake up the subrequest?

I've already told you to use ngx_http_post_request to wake up the
parent. It does not have to wait until the subrequest ends (as
indicated by calling the post_subrequest callback).

When you do not mark the bufs from the upstream in the subrequest's
body filter as consumed (that is, setting b->pos = b->last), then the
upstream will just stop calling the output body filter (in nonbuffered
mode) with more data and then immediately wake up the parent request
posed by you in the subrequest's body filter (by means of the
ngx_http_run_posted_requests call).

That way, you're essentially interleaving the processing of the
subrequest and its parent, which should be exactly what you want here.

> I know that ngx_http_post_request would wake up a parent request from a
> subrequest, but is there a way do it the other way around?
>

This is the only right way to go, AFAIK. Otherwise we may just ask for
troubles :)

Best regards,
-agentzh

agentzh

Hello!

On Wed, Mar 27, 2013 at 12:04 PM, wrote:
>
> The problem is that the body filter is called only two or three times (for
> every request), even for responses with large data (without my patch, it is
> called over 20 times).
> Do you have an idea regarding what I'm doing wrong?

You should ensure that *every time* the main request processes a
downstream write event, post the subrequest so that the subrequest can
run its own write event handler accordingly (to check if the bufs have
been consumed by the downstream).

Also, enabling and checking the nginx debugging logs should be very
helpful in tracking such issues :)

Best regards,
-agentzh