Data aggregation architecture advice

lordnynex · 2015-10-27T04:51:47+00:00

Yichun Zhang Sent: Monday, October 26, 2015 8:40 PM > On Tue, Oct 27, 2015 at 12:54 AM, Nelson, Erik - 2 wrote: > > Do echo_location and echo_async_...

Data aggregation architecture advice

lordnynex

Hello,

I have a growing number of projects that that require storing small bits of time series data. Something similar to RRD. Basically, I'd like to present a REST endpoint that just reads POST body and writes it to some sort of db. I don't think time series data like this is a good candidate for a relational DB. I also don't know if RRD is a good format because my understanding is it can be a bit lossy.

My only requirements are:

- Data integrity (no lost or lossy data)

- Respect nginx IO sensitivity

- Reasonably compact like RRD

- Data will only periodically be summarized (graphing for example)

So I'm curious if anyone out there does something similar? Or if they have any ideas about good alternatives to RRD? Redis is one alternative, but I'd prefer if the format was a bit more portable. Frankly, archiving RDB files does not seem like a good approach. I've been toying with the idea of porting graphites whisper format to lua, but I wanted to see if anyone knew of anything better first.

-Brandon

sparvu_71a1

On 10/26/15 22:51, Lord Nynex wrote:
> Hello,
> 
> I have a growing number of projects that that require storing small bits
> of time series data. Something similar to RRD. Basically, I'd like to
> present a REST endpoint that just reads POST body and writes it to some
> sort of db. I don't think time series data like this is a good candidate
> for a relational DB.

And indeed it is not. A relational DB would be overkill for time series
data. We been successfully, but with our own type of problems, using:

 * OpenResty
 * Redis
 * Lua

for time series data analytics written entirely in Lua. We keep all
stats in Redis and raw data on flat files CSV. As said we are growing
rapidly and we are developing now more and more on top of OpenResty and
Redis.

We been evaluating dedicated time series databases like InfluxDB and
others but we ended up with Redis for its simplicity and data rich
features and easy connectivity to OpenResty.

Hope it helps,

-- 
Stefan Parvu <sp...@kronometrix.org>

perusio

May I suggest another alternative: tarantool http://tarantool.org.

I'm currently working on an API that will relies heavily on it. I created a resty library to talk to it [1]. It has persistence out of the box, you just select the sophia storage engine, transactions, sharding, master-master replication and an expiration daemon is already coded for it [2] if you need it. Most importantly it is fully scriptable in Lua which means you can do whatever you want with it inside OpenResty/nginx + Lua.

Grüße,

Antonio

[1] - https://github.com/perusio/lua-resty-tarantool

[2] - https://github.com/tarantool/expirationd

On Mon, Oct 26, 2015 at 9:51 PM Lord Nynex <lord...@gmail.com> wrote:

Hello,

I have a growing number of projects that that require storing small bits of time series data. Something similar to RRD. Basically, I'd like to present a REST endpoint that just reads POST body and writes it to some sort of db. I don't think time series data like this is a good candidate for a relational DB. I also don't know if RRD is a good format because my understanding is it can be a bit lossy.

My only requirements are:

- Data integrity (no lost or lossy data)
- Respect nginx IO sensitivity
- Reasonably compact like RRD
- Data will only periodically be summarized (graphing for example)

So I'm curious if anyone out there does something similar? Or if they have any ideas about good alternatives to RRD? Redis is one alternative, but I'd prefer if the format was a bit more portable. Frankly, archiving RDB files does not seem like a good approach. I've been toying with the idea of porting graphites whisper format to lua, but I wanted to see if anyone knew of anything better first.

-Brandon

.

lordnynex

Hi Stefan,

Thanks for the reply. When you say you store data in flat CSV files does this mean you just have some periodic job that exports all keys into a csv? Also, isn't this much more inefficient for storage compared to rrd? I will research influxdb as well.

On Mon, Oct 26, 2015 at 2:02 PM, Stefan Parvu <sp...@kronometrix.org> wrote:

On 10/26/15 22:51, Lord Nynex wrote:
> Hello,
>
> I have a growing number of projects that that require storing small bits
> of time series data. Something similar to RRD. Basically, I'd like to
> present a REST endpoint that just reads POST body and writes it to some
> sort of db. I don't think time series data like this is a good candidate
> for a relational DB.

And indeed it is not. A relational DB would be overkill for time series
data. We been successfully, but with our own type of problems, using:

* OpenResty
* Redis
* Lua

for time series data analytics written entirely in Lua. We keep all
stats in Redis and raw data on flat files CSV. As said we are growing
rapidly and we are developing now more and more on top of OpenResty and
Redis.

We been evaluating dedicated time series databases like InfluxDB and
others but we ended up with Redis for its simplicity and data rich
features and easy connectivity to OpenResty.

Hope it helps,

--
Stefan Parvu <sp...@kronometrix.org.

sparvu_71a1

On 10/27/15 00:09, Lord Nynex wrote:
> Hi Stefan,
> 
> Thanks for the reply. When you say you store data in flat CSV files does
> this mean you just have some periodic job that exports all keys into a
> csv? Also, isn't this much more inefficient for storage compared to rrd?

We store CSV flat files for exploratory data analysis as an interactive
way of diving into files and analyze certain things, if we really need
to go down back to original data or we use the CSV files to exchange raw
data with other systems. CSV is a very simple format which works with
any type of backend or type of other processing system.

We dont periodically export it but rather as soon as the raw data
updates come in we store them on disk(s) as CSV, before any redis and
numerical processing is taken place.

Everything what we been doing previously in RRD we are doing now in
Redis. CSV flat files, the raw data have nothing to do with the
aggregates or summary statistics.

-- 
Stefan Parvu <sp...@kronometrix.org>