请教 openresty +proxy_pss + srcache + redis

bill · 2015-12-08T14:12:47+00:00

第一次感受到了这个 API 的用途，嘿嘿2015-12-08 14:29 GMT+08:00 DeJia...

请教 openresty +proxy_pss + srcache + redis

bill

用openresty做反向代理， proxy_pass 指向内部网站，内部网站返回的内容是gzip过的，但是redis 里面的内容是ungziｐ的，　我想直接把gzip的内容直接存入redis，

问题可能出在

redis2_query set $key $echo_request_body;

$echo_request_body可能是已经ungzip过了，请教怎么直接拿到gzip的内容

chen.wei37

/nginx-systemtap-toolkit # stap -ve 'probe begin { log("hello world") exit() }'
Pass 1: parsed user script and 110 library script(s) using 55264virt/31796res/2064shr/30324data kb, in 240usr/40sys/289real ms.
Pass 2: analyzed script: 1 probe(s), 2 function(s), 0 embed(s), 0 global(s) using 55924virt/32856res/2444shr/30984data kb, in 10usr/0sys/9real ms.
Pass 3: translated to C into "/tmp/stapO6NFip/stap_5819a3d9b4695613734d172f88377aa5_1126_src.c" using 56060virt/33224res/2776shr/31120data kb, in 0usr/0sys/1real ms.
Pass 4: compiled C into "stap_5819a3d9b4695613734d172f88377aa5_1126.ko" in 890usr/180sys/1272real ms.
Pass 5: starting run.
hello world
Pass 5: run completed in 0usr/20sys/572real ms.

请问为何很多脚本都运行失败，举例如下：

# ./sample-bt-off-cpu -p 29764 -t 5
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_10.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_11.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_12.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_13.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_14.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_15.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_16.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_17.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_18.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_19.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_2.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_20.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_21.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_22.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_23.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_24.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_25.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_26.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_27.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_28.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_3.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_4.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_5.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_6.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_7.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_8.o debuginfo: No DWARF information found [man warning::debuginfo]
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_9.o debuginfo: No DWARF information found [man warning::debuginfo]
semantic error: while resolving probe point: identifier 'kernel' at /chenw/system_tap/share/systemtap/tapset/linux/scheduler.stp:37:2
source: kernel.function("context_switch")
^

semantic error: missing x86_64 kernel/module debuginfo [man warning::debuginfo] under '/lib/modules/2.6.32.59-0.7-default/build'

semantic error: while resolving probe point: identifier 'scheduler' at <input>:12:7
source: probe scheduler.cpu_off {
^

semantic error: no match

Pass 2: analysis failed. [man error::pass2]
Number of similar error messages suppressed: 1.
Rerun with -v to see them.

发件人: Bill Liao <b...@ncix.com>
收件人: openresty <openresty@googlegroups.com>,
日期: 2015/12/08 14:13
主题: [openresty] 请教 openresty +proxy_pss + srcache + redis
发件人: openresty@googlegroups.com

用openresty做反向代理， proxy_pass 指向内部网站，内部网站返回的内容是gzip过的，但是redis 里面的内容是ungziｐ的，　我想直接把gzip的内容直接存入redis，

问题可能出在

redis2_query set $key $echo_request_body;

$echo_request_body可能是已经ungzip过了，请教怎么直接拿到gzip的内容


--------------------------------------------------------
ZTE Information Security Notice: The information contained in this mail (and any attachment transmitted herewith) is privileged and confidential and is intended for the exclusive use of the addressee(s).  If you are not an intended recipient, any disclosure, reproduction, distribution or other dissemination or use of the information contained is strictly prohibited.  If you have received this mail in error, please delete it and notify us immediately.

doujiang24

Hello,

2015-12-08 15:29 GMT+08:00 <chen...@zte.com.cn>:

/nginx-systemtap-toolkit # stap -ve 'probe begin { log("hello world") exit() }'
Pass 1: parsed user script and 110 library script(s) using 55264virt/31796res/2064shr/30324data kb, in 240usr/40sys/289real ms.
Pass 2: analyzed script: 1 probe(s), 2 function(s), 0 embed(s), 0 global(s) using 55924virt/32856res/2444shr/30984data kb, in 10usr/0sys/9real ms.
Pass 3: translated to C into "/tmp/stapO6NFip/stap_5819a3d9b4695613734d172f88377aa5_1126_src.c" using 56060virt/33224res/2776shr/31120data kb, in 0usr/0sys/1real ms.
Pass 4: compiled C into "stap_5819a3d9b4695613734d172f88377aa5_1126.ko" in 890usr/180sys/1272real ms.
Pass 5: starting run.
hello world
Pass 5: run completed in 0usr/20sys/572real ms.

这只是表明 systemtap 安装成功

请问为何很多脚本都运行失败，举例如下：

# ./sample-bt-off-cpu -p 29764 -t 5
WARNING: cannot find module /tmp/stapJDGC1C/tracequery_kmod_1/tracequery_kmod_1_9.o debuginfo: No DWARF information found [man warning::debuginfo]

这个告警是说，没找到调试符信息，比如你是不是没装 kernel-debuginfo

个人推荐的做法是，自己编译 kernel

semantic error: while resolving probe point: identifier 'kernel' at /chenw/system_tap/share/systemtap/tapset/linux/scheduler.stp:37:2
source: kernel.function("context_switch") ^
semantic error: missing x86_64 kernel/module debuginfo [man warning::debuginfo] under '/lib/modules/2.6.32.59-0.7-default/build'
semantic error: while resolving probe point: identifier 'scheduler' at <input>:12:7
source: probe scheduler.cpu_off {

可能是你的 systemtap 版本有点老了，最好安装最新版的

^

semantic error: no match

Pass 2: analysis failed. [man error::pass2]
Number of similar error messages suppressed: 1.
Rerun with -v to see them.

发件人: Bill Liao <b...@ncix.com>
收件人: openresty <openresty@googlegroups.com>,
日期: 2015/12/08 14:13
主题: [openresty] 请教 openresty +proxy_pss + srcache + redis
发件人: openresty@googlegroups.com

用openresty做反向代理， proxy_pass 指向内部网站，内部网站返回的内容是gzip过的，但是redis 里面的内容是ungziｐ的，　我想直接把gzip的内容直接存入redis，

问题可能出在

redis2_query set $key $echo_request_body;

$echo_request_body可能是已经ungzip过了，请教怎么直接拿到gzip的内容

--
--------------------------------------------------------
ZTE Information Security Notice: The information contained in this mail (and any attachment transmitted herewith) is privileged and confidential and is intended for the exclusive use of the addressee(s).  If you are not an intended recipient, any disclosure, reproduction, distribution or other dissemination or use of the information contained is strictly prohibited.  If you have received this mail in error, please delete it and notify us immediately.
--

yuansheng_2f16

使用 resty-**** 代码劫持一下 proxy_pass 返回内容，在 lua 中解压缩，然后调用 resty-redis 的方式完成 redis 的写入。

相对的，resty-*** 将会获得更大的自由度，尤其是内部有比较复杂的业务逻辑。

> proxy_pass 指向内部网站，内部网站返回的内容是gzip过的

这里可以添加 HTTP 头指名不允许 gzip 压缩。

2015-12-08 14:12 GMT+08:00 Bill Liao <b...@ncix.com>:

用openresty做反向代理， proxy_pass 指向内部网站，内部网站返回的内容是gzip过的，但是redis 里面的内容是ungziｐ的，　我想直接把gzip的内容直接存入redis，

问题可能出在

redis2_query set $key $echo_request_body;

$echo_request_body可能是已经ungzip过了，请教怎么直接拿到gzip的内容
--

Membhis

My github: https://github.com/membphis

Our Book: OpenResty Best Practices

chen.wei37

lua-resty-http测试似乎只适合小文件，如果文件超过300M，会概率出现程序挂死。

陈伟 Chen Wei 多媒体业务产品部
	云计算及IT产品经营部
	Cloud Computing & IT Product Operation
1/F, Building No. 3, ZTE Corporation, Yuhuatai District, , Nanjing, P.R.China, 210012 Mobile: +86-18913979100 Email: chen...@zte.com.cn More Information: VAS 、Cloud

发件人: YuanSheng Wang <yuan...@iresty.org>
收件人: openresty@googlegroups.com,
日期: 2015/12/08 18:33
主题: Re: [openresty] 请教 openresty +proxy_pss + srcache + redis
发件人: openresty@googlegroups.com

使用 resty-**** 代码劫持一下 proxy_pass 返回内容，在 lua 中解压缩，然后调用 resty-redis 的方式完成 redis 的写入。

相对的，resty-*** 将会获得更大的自由度，尤其是内部有比较复杂的业务逻辑。

> proxy_pass 指向内部网站，内部网站返回的内容是gzip过的

这里可以添加 HTTP 头指名不允许 gzip 压缩。

2015-12-08 14:12 GMT+08:00 Bill Liao <b...@ncix.com>:
用openresty做反向代理， proxy_pass 指向内部网站，内部网站返回的内容是gzip过的，但是redis 里面的内容是ungziｐ的，　我想直接把gzip的内容直接存入redis，

问题可能出在

redis2_query set $key $echo_request_body;

$echo_request_body可能是已经ungzip过了，请教怎么直接拿到gzip的内容

--

--

Membhis
My github: https://github.com/membphis
Our Book: OpenResty Best Practices


--------------------------------------------------------
ZTE Information Security Notice: The information contained in this mail (and any attachment transmitted herewith) is privileged and confidential and is intended for the exclusive use of the addressee(s).  If you are not an intended recipient, any disclosure, reproduction, distribution or other dissemination or use of the information contained is strictly prohibited.  If you have received this mail in error, please delete it and notify us immediately.

chen.wei37

多用户请求且单用户执行任务时间较长得场景，看了lua-resty-http的代码，用到了协程，但是协程同一时刻只能允许一个在执行，是不是这是导致不稳定的本质？

陈伟 Chen Wei 多媒体业务产品部
	云计算及IT产品经营部
	Cloud Computing & IT Product Operation
1/F, Building No. 3, ZTE Corporation, Yuhuatai District, , Nanjing, P.R.China, 210012 Mobile: +86-18913979100 Email: chen...@zte.com.cn More Information: VAS 、Cloud

发件人: chen...@zte.com.cn
收件人: openresty@googlegroups.com,
日期: 2015/12/09 08:33
主题: [probably forge email可能是仿冒邮件]答复: Re: [openresty] 请教 openresty +proxy_pss + srcache + redis
发件人: openresty@googlegroups.com

lua-resty-http测试似乎只适合小文件，如果文件超过300M，会概率出现程序挂死。

陈伟 Chen Wei 多媒体业务产品部
	云计算及IT产品经营部
	Cloud Computing & IT Product Operation
1/F, Building No. 3, ZTE Corporation, Yuhuatai District, , Nanjing, P.R.China, 210012 Mobile: +86-18913979100 Email: chen...@zte.com.cn More Information: VAS 、Cloud

--
size=3>

--

Membhis
My github: https://github.com/membphis
Our Book: OpenResty Best Practices

--
size=3>

-------------------------------------------------------- ZTE Information Security Notice: The information contained in this mail (and any attachment transmitted herewith) is privileged and confidential and is intended for the exclusive use of the addressee(s). If you are not an intended recipient, any disclosure, reproduction, distribution or other dissemination or use of the information contained is strictly prohibited. If you have received this mail in error, please delete it and notify us immediately.


--------------------------------------------------------
ZTE Information Security Notice: The information contained in this mail (and any attachment transmitted herewith) is privileged and confidential and is intended for the exclusive use of the addressee(s).  If you are not an intended recipient, any disclosure, reproduction, distribution or other dissemination or use of the information contained is strictly prohibited.  If you have received this mail in error, please delete it and notify us immediately.

doujiang24

Hello,

在 2015年12月9日上午9:07， <chen...@zte.com.cn>写道：

多用户请求且单用户执行任务时间较长得场景，看了lua-resty-http的代码，用到了协程，但是协程同一时刻只能允许一个在执行，是不是这是导致不稳定的本质？

并不是的，这里的协程只是为了实现生产者 -> 消费者模型，从来达到流式处理

最好能提供方便复现的用例出来，出现问题应该解决才好

陈伟 Chen Wei
多媒体业务产品部
云计算及IT产品经营部
Cloud Computing & IT Product Operation
1/F, Building No. 3, ZTE Corporation, Yuhuatai District, ,
Nanjing, P.R.China, 210012
Mobile: +86-18913979100
Email: chen...@zte.com.cn
More Information: VAS 、Cloud

发件人: chen...@zte.com.cn
收件人: openresty@googlegroups.com,
日期: 2015/12/09 08:33
主题: [probably forge email可能是仿冒邮件]答复: Re: [openresty] 请教 openresty +proxy_pss + srcache + redis
发件人: openresty@googlegroups.com

lua-resty-http测试似乎只适合小文件，如果文件超过300M，会概率出现程序挂死。

陈伟 Chen Wei
多媒体业务产品部
云计算及IT产品经营部
Cloud Computing & IT Product Operation
1/F, Building No. 3, ZTE Corporation, Yuhuatai District, ,
Nanjing, P.R.China, 210012
Mobile: +86-18913979100
Email: chen...@zte.com.cn
More Information: VAS 、Cloud

发件人: YuanSheng Wang <yuan...@iresty.org>
收件人: openresty@googlegroups.com,
日期: 2015/12/08 18:33
主题: Re: [openresty] 请教 openresty +proxy_pss + srcache + redis
发件人: openresty@googlegroups.com

使用 resty-**** 代码劫持一下 proxy_pass 返回内容，在 lua 中解压缩，然后调用 resty-redis 的方式完成 redis 的写入。

相对的，resty-*** 将会获得更大的自由度，尤其是内部有比较复杂的业务逻辑。

> proxy_pass 指向内部网站，内部网站返回的内容是gzip过的

这里可以添加 HTTP 头指名不允许 gzip 压缩。

2015-12-08 14:12 GMT+08:00 Bill Liao <b...@ncix.com>:
用openresty做反向代理， proxy_pass 指向内部网站，内部网站返回的内容是gzip过的，但是redis 里面的内容是ungziｐ的，　我想直接把gzip的内容直接存入redis，

问题可能出在

redis2_query set $key $echo_request_body;

$echo_request_body可能是已经ungzip过了，请教怎么直接拿到gzip的内容

--
size="3">

--

Membhis
My github: https://github.com/membphis
Our Book: OpenResty Best Practices
--
size="3">

-------------------------------------------------------- ZTE Information Security Notice: The information contained in this mail (and any attachment transmitted herewith) is privileged and confidential and is intended for the exclusive use of the addressee(s). If you are not an intended recipient, any disclosure, reproduction, distribution or other dissemination or use of the information contained is strictly prohibited. If you have received this mail in error, please delete it and notify us immediately.

--
--------------------------------------------------------
ZTE Information Security Notice: The information contained in this mail (and any attachment transmitted herewith) is privileged and confidential and is intended for the exclusive use of the addressee(s).  If you are not an intended recipient, any disclosure, reproduction, distribution or other dissemination or use of the information contained is strictly prohibited.  If you have received this mail in error, please delete it and notify us immediately.
--