openresty nginx+lua+kafka 发送不了数据的问题

1247652934 · 2016-11-17T15:32:50+00:00

Hello在 2016年11月16日下午7:38， <caijian...@gmail.com>写道：各位朋&#x53...

openresty nginx+lua+kafka 发送不了数据的问题

1247652934

使用lua-resty-kafka库

在nginx中写lua脚本产生producer时，

报错：

1.not foundd topic

2.37 [lua] content_by_lua(nginx.conf:242):39: kafka send err:not foundd topic, client: 192.168.0.143, server: 192.168.0.78, request: "POST /test2 HTTP/1.1", host: "master1:8001"

2016/10/05 03:30:54 [error] 96080#0: *138 [lua] client.lua:150: _fetch_metadata(): all brokers failed in fetch topic metadata, client: 192.168.0.143, server: 192.168.0.78, request: "POST /test2 HTTP/1.1", host: "master1:8001"

2016/10/05 03:30:54 [error] 96080#0: *138 [lua] content_by_lua(nginx.conf:242):39: kafka send err:not foundd topic, client: 192.168.0.143, server: 192.168.0.78, request: "POST /test2 HTTP/1.1", host: "master1:8001

但是kafka服务是正常的，topic也有存在的，我通过java api推送消息给kafka是可以的

一直找不出原因，请大家帮忙解决下

doujiang24

Hello

在 2016年11月17日下午3:32， <1247...@qq.com>写道：

使用lua-resty-kafka库
在nginx中写lua脚本产生producer时，
报错：
1.not foundd topic
2.37 [lua] content_by_lua(nginx.conf:242):39: kafka send err:not foundd topic, client: 192.168.0.143, server: 192.168.0.78, request: "POST /test2 HTTP/1.1", host: "master1:8001"
2016/10/05 03:30:54 [error] 96080#0: *138 [lua] client.lua:150: _fetch_metadata(): all brokers failed in fetch topic metadata, client: 192.168.0.143, server: 192.168.0.78, request: "POST /test2 HTTP/1.1", host: "master1:8001"

这行错误意味着，获取 metadata 失败

你可以把错误日志调到 INFO 级别，看看更具体的错误日志

2016/10/05 03:30:54 [error] 96080#0: *138 [lua] content_by_lua(nginx.conf:242):39: kafka send err:not foundd topic, client: 192.168.0.143, server: 192.168.0.78, request: "POST /test2 HTTP/1.1", host: "master1:8001
但是kafka服务是正常的，topic也有存在的，我通过java api推送消息给kafka是可以的
一直找不出原因，请大家帮忙解决下

--

1247652934

按你说的配置nginx log级别为info后，看到了这样的问题

2016/11/18 13:55:37 [error] 25226#0: *45 [lua] client.lua:150: _fetch_metadata(): all brokers failed in fetch topic metadata, client: 192.168.0.143, server: localhost, request: "POST /test HTTP/1.1", host: "192.168.0.54:8111"

2016/11/18 13:55:37 [info] 25226#0: *45 [lua] client.lua:141: _fetch_metadata(): broker fetch metadata failed, err:no resolver defined to resolve "master1"master19092, client: 192.168.0.143, server: localhost, request: "POST /test HTTP/1.1", host: "192.168.0.54:8111"

master1 kafka 是与openrety安装在不同机器上的

然后我配置了nginx中的resolver 也不行

doujiang24

Hello

在 2016年11月18日下午2:00， <1247...@qq.com>写道：

按你说的配置nginx log级别为info后，看到了这样的问题

2016/11/18 13:55:37 [error] 25226#0: *45 [lua] client.lua:150: _fetch_metadata(): all brokers failed in fetch topic metadata, client: 192.168.0.143, server: localhost, request: "POST /test HTTP/1.1", host: "192.168.0.54:8111"
2016/11/18 13:55:37 [info] 25226#0: *45 [lua] client.lua:141: _fetch_metadata(): broker fetch metadata failed, err:no resolver defined to resolve "master1"master19092, client: 192.168.0.143, server: localhost, request: "POST /test HTTP/1.1", host: "192.168.0.54:8111"

你可以看看这个

https://github.com/doujiang24/lua-resty-kafka/issues/5#issuecomment-70195763

2016/11/18 13:55:37 [error] 25226#0: *45 [lua] client.lua:150: _fetch_metadata(): all brokers failed in fetch topic metadata, client: 192.168.0.143, server: localhost, request: "POST /test HTTP/1.1", host: "192.168.0.54:8111

master1 kafka 是与openrety安装在不同机器上的
然后我配置了nginx中的resolver 也不行

--

1247652934

嗯嗯，非常感谢，问题解决了

yuansheng

2016-11-18 14:09 GMT+08:00 DeJiang Zhu <douj...@gmail.com>:

Hello

在 2016年11月18日下午2:00， <1247...@qq.com>写道：
按你说的配置nginx log级别为info后，看到了这样的问题

2016/11/18 13:55:37 [error] 25226#0: *45 [lua] client.lua:150: _fetch_metadata(): all brokers failed in fetch topic metadata, client: 192.168.0.143, server: localhost, request: "POST /test HTTP/1.1", host: "192.168.0.54:8111"
2016/11/18 13:55:37 [info] 25226#0: *45 [lua] client.lua:141: _fetch_metadata(): broker fetch metadata failed, err:no resolver defined to resolve "master1"master19092, client: 192.168.0.143, server: localhost, request: "POST /test

@德江

关键错误信息，应当抛到外层记录到 error 中，比如这里的 “no resolver defined to resolve” ，对用户有明显提示意义。

HTTP/1.1", host: "192.168.0.54:8111"

你可以看看这个
https://github.com/doujiang24/lua-resty-kafka/issues/5#issuecomment-70195763

2016/11/18 13:55:37 [error] 25226#0: *45 [lua] client.lua:150: _fetch_metadata(): all brokers failed in fetch topic metadata, client: 192.168.0.143, server: localhost, request: "POST /test HTTP/1.1", host: "192.168.0.54:8111

master1 kafka 是与openrety安装在不同机器上的
然后我配置了nginx中的resolver 也不行

--

--

YuanSheng Wang

---------------------------------------

My Github: https://github.com/membphis

OpenResty lover ^_^

doujiang24

Hello

在 2016年11月22日下午10:00，YuanSheng Wang <yuan...@openresty.org>写道：

关键错误信息，应当抛到外层记录到 error 中，比如这里的 “no resolver defined to resolve” ，对用户有明显提示意义。

在同步模式下是有返回错误信息的

然而在异步模式下，send 只是先放入缓冲队列中，发送时的错误并不能很友善的提供给调用方，还是打个日志比较方便 :)

HTTP/1.1", host: "192.168.0.54:8111"

你可以看看这个
https://github.com/doujiang24/lua-resty-kafka/issues/5#issuecomment-70195763

2016/11/18 13:55:37 [error] 25226#0: *45 [lua] client.lua:150: _fetch_metadata(): all brokers failed in fetch topic metadata, client: 192.168.0.143, server: localhost, request: "POST /test HTTP/1.1", host: "192.168.0.54:8111

master1 kafka 是与openrety安装在不同机器上的
然后我配置了nginx中的resolver 也不行

--

--

--

YuanSheng Wang
---------------------------------------
My Github: https://github.com/membphis
OpenResty lover ^_^

--

lichunliang1984

怎么解决的，你的qq多少，求~

在 2016年11月17日星期四 UTC+8下午3:32:50，12476...@qq.com写道：

使用lua-resty-kafka库
在nginx中写lua脚本产生producer时，
报错：
1.not foundd topic
2.37 [lua] content_by_lua(nginx.conf:242):39: kafka send err:not foundd topic, client: 192.168.0.143, server: 192.168.0.78, request: "POST /test2 HTTP/1.1", host: "master1:8001"
2016/10/05 03:30:54 [error] 96080#0: *138 [lua] client.lua:150: _fetch_metadata(): all brokers failed in fetch topic metadata, client: 192.168.0.143, server: 192.168.0.78, request: "POST /test2 HTTP/1.1", host: "master1:8001"
2016/10/05 03:30:54 [error] 96080#0: *138 [lua] content_by_lua(nginx.conf:242):39: kafka send err:not foundd topic, client: 192.168.0.143, server: 192.168.0.78, request: "POST /test2 HTTP/1.1", host: "master1:8001
但是kafka服务是正常的，topic也有存在的，我通过java api推送消息给kafka是可以的
一直找不出原因，请大家帮忙解决下