Return 400 instead of 500 for ngx_postgres error

0x6e6562 · 2014-12-03T00:27:55+00:00

Hello! On Tue, Dec 2, 2014 at 12:12 AM, Vitaly Kosenko wrote: > Hello! Do openresty have utility to get current state of openresty http > server (like...

Return 400 instead of 500 for ngx_postgres error

0x6e6562

Hi,

I'm not sure whether this is a ngx_postgres specific issue, but I was wondering whether is is possible to customize opensresty to selectively send back a 400 to the client, rather than the default of 500.

My use case is Postgres (correctly) complaining about a unique constraint violation:

2014/12/02 16:08:39 [error] 25745#0: *3 postgres: failed to receive result: PGRES_FATAL_ERROR: ERROR: duplicate key value violates unique constraint "unique_organization_name"

DETAIL: Key (name)=(x) already exists.

while waiting for result from PostgreSQL database, client: 172.16.40.1, server: localhost, request: "POST /organizations HTTP/1.1", upstream: "postgres://127.0.0.1:5432", host: "172.16.40.180"

Is there any way to match on this error and inform that client of the cause of the issue?

In this case, I'd like to return a 400 to the client and tell them that they are trying to enter duplicate data.

Many thanks,

Ben

agentzh

Hello!

On Tue, Dec 2, 2014 at 8:27 AM, Ben Hood wrote:
> PGRES_FATAL_ERROR: ERROR:  duplicate key value violates unique constraint
> "unique_organization_name"
> DETAIL:  Key (name)=(x) already exists.
>  while waiting for result from PostgreSQL database, client: 172.16.40.1,
> server: localhost, request: "POST /organizations HTTP/1.1", upstream:
> "postgres://127.0.0.1:5432", host: "172.16.40.180"
>
> Is there any way to match on this error and inform that client of the cause
> of the issue?
>

I used to have some plan to expose the Pg error codes and error
strings as nginx variables, so you can customize such things easily
yourself in nginx.conf (possibly with a little help of
header_filter_by_lua, for example). Are you willing to contribute a
patch to ngx_postgres?

Regards,
-agentzh

0x6e6562

Hi agentzh,

On Tue, Dec 2, 2014 at 10:10 PM, Yichun Zhang (agentzh)
<age...@gmail.com> wrote:
> I used to have some plan to expose the Pg error codes and error
> strings as nginx variables, so you can customize such things easily
> yourself in nginx.conf (possibly with a little help of
> header_filter_by_lua, for example). Are you willing to contribute a
> patch to ngx_postgres?

Yes, I think it would be a good idea for me to contribute a patch, for
a couple of reasons:

1. To get more control over the behavior of ngx_postgres in this
particular sceanrio
2. To improve my understanding of ngx_postgres and openresty in general

That said, I think I need to better understand the phase handling of
the underlying API and how the asynchrony works with the different
directives, so that I what I am producing is not complete garbage.

For example, I'm not 100% sure about what is happening under the
covers when you do something like this:

location ~ /rows {
      postgres_pass        database;
      rds_json                  on;
      postgres_query       HEAD GET    "SELECT * FROM generate_series(1,5)";
      add_header             X-rows      $postgres_rows;
      content_by_lua '
        ngx.say(ngx.var.postgres_rows)
      ';
}

This returns nil in the body of the response.

But if I leave the content directive out and just let the result
stream to the client, the data and the 'X-rows` header is output as
exepcted, i.e.

location ~ /rows {
      postgres_pass        database;
      rds_json                  on;
      postgres_query       HEAD GET    "SELECT * FROM generate_series(1,5)";
      add_header             X-rows      $postgres_rows;
}

So I'm assuming that in the case of the former, $postgres_rows is
scoped that is not visible via 'ngx.var.postgres_rows' in the content
directive. I assuming further that this handler is getting executed
asynchronously.

I've been looking into hacks using ngx.location.capture to gather
asynchronous results, but this exposes my ignorance of how it is
actually working.

Is there an idiomatic way to do conditional processing based on the
content of $postgres_rows in opensresty, or do I need to dig deeper
into ngx_postgres?

Cheers,

Ben

agentzh

Hello!

On Wed, Dec 3, 2014 at 10:45 AM, Ben Hood wrote:
> For example, I'm not 100% sure about what is happening under the
> covers when you do something like this:
>
> location ~ /rows {
>       postgres_pass        database;
>       rds_json                  on;
>       postgres_query       HEAD GET    "SELECT * FROM generate_series(1,5)";
>       add_header             X-rows      $postgres_rows;
>       content_by_lua '
>         ngx.say(ngx.var.postgres_rows)
>       ';

Nginx only allows one (module's) content handler in every single
location. Here you make ngx_postgres and ngx_lua register a content
handler in the same location and only one of them wins.

This is more related to the general nginx C module development. I
think it has been covered in Evan Miller's nginx module development
guides.

> I've been looking into hacks using ngx.location.capture to gather
> asynchronous results, but this exposes my ignorance of how it is
> actually working.
>

Nginx subrequests should work for your case because there're now
separate locations holding ngx_postgres's and ngx_lua's content
handler, respectively. No conflict any more. But you need to expose
subrequest-level nginx variables like $postgres_rows via the
subrequest response header, for example, by means of ngx_headers_more.
Generally, use of a cosocket-based PostgreSQL driver library like the
3rd-party lua-resty-postgres is preferred using ngx.location.capture +
ngx_postgres.

> Is there an idiomatic way to do conditional processing based on the
> content of $postgres_rows in opensresty, or do I need to dig deeper
> into ngx_postgres?
>

See above.

Regards,
-agentzh

0x6e6562

Hi agentzh,

On Thu, Dec 4, 2014 at 12:47 AM, Yichun Zhang (agentzh)
<age...@gmail.com> wrote:
> Nginx only allows one (module's) content handler in every single
> location. Here you make ngx_postgres and ngx_lua register a content
> handler in the same location and only one of them wins.

Ah, that makes sense - thanks for the heads up.

>> I've been looking into hacks using ngx.location.capture to gather
>> asynchronous results, but this exposes my ignorance of how it is
>> actually working.
>>
>
> Nginx subrequests should work for your case because there're now
> separate locations holding ngx_postgres's and ngx_lua's content
> handler, respectively. No conflict any more. But you need to expose
> subrequest-level nginx variables like $postgres_rows via the
> subrequest response header, for example, by means of ngx_headers_more.

OK, I'll look into that.

An alternative for this particular use case would be to look into
doing the conditional processing in the background, because in this
scenario, the client does not care about the result (e.g. using
ngx.timer.at). But that still wouldn't solve the fundamental issue of
trying to mix 2 content handlers in the same location.

> Generally, use of a cosocket-based PostgreSQL driver library like the
> 3rd-party lua-resty-postgres is preferred using ngx.location.capture +
> ngx_postgres.

Are you referring to https://github.com/azurewang/lua-resty-postgres ?

Cheers,

Ben

agentzh

Hello!

On Thu, Dec 4, 2014 at 2:35 AM, Ben Hood wrote:
> An alternative for this particular use case would be to look into
> doing the conditional processing in the background, because in this
> scenario, the client does not care about the result (e.g. using
> ngx.timer.at).

Well, subrequests are not allowed in the timer callback. But those
cosocket-based lua-resty-* drivers are.

>
> Are you referring to https://github.com/azurewang/lua-resty-postgres ?
>

Yes. Though I'm not really familiar with its implementation. Use it
with care if you want to.

Regards,
-agentzh

0x6e6562

Hi agentzh,

On Thu, Dec 4, 2014 at 7:55 PM, Yichun Zhang (agentzh)
<age...@gmail.com> wrote:
> On Thu, Dec 4, 2014 at 2:35 AM, Ben Hood wrote:
>> An alternative for this particular use case would be to look into
>> doing the conditional processing in the background, because in this
>> scenario, the client does not care about the result (e.g. using
>> ngx.timer.at).
>
> Well, subrequests are not allowed in the timer callback. But those
> cosocket-based lua-resty-* drivers are.

OK, it's good to know how things are supposed to work.

In the course of trying to use ngx_postgres in conjunction with some
follow on logic, I've really banged my head against the wall. So the
best thing about banging your head against the wall is stopping.

So I've re-eavluated what I'm trying to achieve and for other reasons,
I've decided that I'm going to keep my openresty locations as simple
as possible and I'm going to do this post-Postgres work via a work
queue in a separate  background application.

So on the one hand you might say that openresty in conjunction with
ngx_postgres is not flexible enough for my needs, but on reflection,
what I do like about nginx/openresty is that it forces you to keep
things simple. So the restriction of not being able to bolt together
nginx one-liners is probably a good thing - it's going to stop me from
giving myself enough rope to hang myself.

>> Are you referring to https://github.com/azurewang/lua-resty-postgres ?
>>
>
> Yes. Though I'm not really familiar with its implementation. Use it
> with care if you want to.

Because of the above, I'm going to stick with a plain ngx_postgres
handler and not try to do any funky logic on the inbound request.

But if I still wanted to stick with my original intention, I would
probably need to factor in the cost of lua-resty-postgres having
potentially bit-rotted somewhat.

Going back to my OP, I would still like to patch ngx_postgres to
translate Postgres constraint violations into HTTP 400 errors. But
I'll need to have a closer look at the ngx_postgres internals first.

Cheers,

Ben