Appending data to a log file

marco · May 29, 2015

Hi,

I am building a custom logger that appends data to a specific file on disk, and I was wondering if the following approach would be recommended:

log_by_lua '

local function log(premature, message)

local f = io.open("/some/path.log", "a+")

f:write(message.."\n")

f:close()

end

local ok, err = ngx.timer.at(0, log, "Test Message")

if not ok then

ngx.log(ngx.ERR, "Failed to create timer: ", err)

end

';

Specifically I am wondering if under heavy load opening and closing the file every time would be recommended, or a different approach should be taken because it would be too expensive.

Thanks

manchev · May 29, 2015

I guess you're better off with lua-resty-logger-socket [0] for non-blocking logging.

Opening and closing the file would involve syscalls that might be the most expensive part of your code above, so maybe you should reconsider your approach.

[0]: https://github.com/cloudflare/lua-resty-logger-socket

Best,

Vladislav

On Fri, May 29, 2015 at 10:16 AM, Marco Palladino <ma...@mashape.com> wrote:

Hi,

I am building a custom logger that appends data to a specific file on disk, and I was wondering if the following approach would be recommended:

log_by_lua '
local function log(premature, message)
local f = io.open("/some/path.log", "a+")
f:write(message.."\n")
f:close()
end

local ok, err = ngx.timer.at(0, log, "Test Message")
if not ok then
ngx.log(ngx.ERR, "Failed to create timer: ", err)
end
';

Specifically I am wondering if under heavy load opening and closing the file every time would be recommended, or a different approach should be taken because it would be too expensive.

Thanks

.

marco · May 29, 2015

Unfortunately that module doesn't support logging to a third party file on disk. I was wondering if I opening the file once, and storing the handler in a global variable to be re-used every time, would work?

On Friday, May 29, 2015 at 12:23:21 AM UTC-7, Vladislav Manchev wrote:

I guess you're better off with lua-resty-logger-socket [0] for non-blocking logging.

Opening and closing the file would involve syscalls that might be the most expensive part of your code above, so maybe you should reconsider your approach.

[0]: https://github.com/cloudflare/lua-resty-logger-socket

Best,
Vladislav

On Fri, May 29, 2015 at 10:16 AM, Marco Palladino <ma...@mashape.com> wrote:
Hi,

I am building a custom logger that appends data to a specific file on disk, and I was wondering if the following approach would be recommended:

log_by_lua '
local function log(premature, message)
local f = io.open("/some/path.log", "a+")
f:write(message.."\n")
f:close()
end

local ok, err = ngx.timer.at(0, log, "Test Message")
if not ok then
ngx.log(ngx.ERR, "Failed to create timer: ", err)
end
';

Specifically I am wondering if under heavy load opening and closing the file every time would be recommended, or a different approach should be taken because it would be too expensive.

Thanks

.

agentzh · May 29, 2015

Hello!

On Fri, May 29, 2015 at 3:32 PM, Marco Palladino wrote:
> Unfortunately that module doesn't support logging to a third party file on
> disk. I was wondering if I opening the file once, and storing the handler in
> a global variable to be re-used every time, would work?
>

Yes, you could open the file in append mode and use LuaJITFFI to call
the write() or writev() syscalls directly without going through libc's
buffered I/O layer. You could cache and share the file descriptor on
the nginx worker level via the following technique:

https://github.com/openresty/lua-nginx-module#data-sharing-within-an-nginx-worker

Just my 2 cents :)

Best regards,
-agentzh

manchev · May 29, 2015

It won't be possible to share a file descriptor between workers, so it's a dead end I think.

Also, not sure what's stopping you from logging to a third party file on disk with lua-resty-logger-socket. You can just log to a localhost syslog-ng instance (although I never tried that).

Best,

Vladislav

On Fri, May 29, 2015 at 10:32 AM, Marco Palladino <ma...@mashape.com> wrote:

Unfortunately that module doesn't support logging to a third party file on disk. I was wondering if I opening the file once, and storing the handler in a global variable to be re-used every time, would work?

On Friday, May 29, 2015 at 12:23:21 AM UTC-7, Vladislav Manchev wrote:
I guess you're better off with lua-resty-logger-socket [0] for non-blocking logging.

Opening and closing the file would involve syscalls that might be the most expensive part of your code above, so maybe you should reconsider your approach.

[0]: https://github.com/cloudflare/lua-resty-logger-socket

Best,
Vladislav

On Fri, May 29, 2015 at 10:16 AM, Marco Palladino <ma...@mashape.com> wrote:
Hi,

I am building a custom logger that appends data to a specific file on disk, and I was wondering if the following approach would be recommended:

log_by_lua '
local function log(premature, message)
local f = io.open("/some/path.log", "a+")
f:write(message.."\n")
f:close()
end

local ok, err = ngx.timer.at(0, log, "Test Message")
if not ok then
ngx.log(ngx.ERR, "Failed to create timer: ", err)
end
';

Specifically I am wondering if under heavy load opening and closing the file every time would be recommended, or a different approach should be taken because it would be too expensive.

Thanks

.

agentzh · May 29, 2015

Hello!

On Fri, May 29, 2015 at 3:55 PM, Vladislav Manchev wrote:
> It won't be possible to share a file descriptor between workers, so it's a
> dead end I think.
>

We don't need to share the log file descriptor across workers in the
first place. Even nginx's builtin logger does not do this. Thanks to
modern operating systems' atomic file appending feature ;)

> Also, not sure what's stopping you from logging to a third party file on
> disk with lua-resty-logger-socket. You can just log to a localhost syslog-ng
> instance (although I never tried that).
>

Sure, AFAIK, syslog-ng server can further log into the file system on
our behalf.

Regards,
-agentzh

marco · Jun 2, 2015

I decided to use "fopen", "fprintf" and "fclose":

log_by_lua '

local ffi = require "ffi"

ffi.cdef[[

typedef struct {

char *fpos;

void *base;

unsigned short handle;

short flags;

short unget;

unsigned long alloc;

unsigned short buffincrement;

} FILE;

FILE *fopen(const char *filename, const char *mode);

int fprintf(FILE *stream, const char *format, ...);

int fclose(FILE *stream);

]]

local function log(premature, message)

local f = ffi.C.fopen("/some/path.log", "a+")

ffi.C.fprintf(f, message)

ffi.C.fclose(f)

end

local ok, err = ngx.timer.at(0, log, "Test Message")

if not ok then

ngx.log(ngx.ERR, "Failed to create timer: ", err)

end

';

The example above still opens and closes the file on every request, I will go ahead and implement the @agentzh suggestion for sharing the file handler.

On Friday, May 29, 2015 at 12:59:27 AM UTC-7, agentzh wrote:

Hello!

On Fri, May 29, 2015 at 3:55 PM, Vladislav Manchev wrote:
> It won't be possible to share a file descriptor between workers, so it's a
> dead end I think.
>

We don't need to share the log file descriptor across workers in the
first place. Even nginx's builtin logger does not do this. Thanks to
modern operating systems' atomic file appending feature ;)

> Also, not sure what's stopping you from logging to a third party file on
> disk with lua-resty-logger-socket. You can just log to a localhost syslog-ng
> instance (although I never tried that).
>

Sure, AFAIK, syslog-ng server can further log into the file system on
our behalf.

Regards,
-agentzh

agentzh · Jun 13, 2015

Hello!

On Tue, Jun 2, 2015 at 9:15 AM, Marco Palladino wrote:
> I decided to use "fopen", "fprintf" and "fclose":
>

As mentioned previously, these libc functions are not safe at all in
this context because of its own userland buffering. Use the thin libc
wrappers for the syscalls like write() and writev() directly.

Regards,
-agentzh

Appending data to a log file

Mmarco

Mmanchev

Mmarco

Aagentzh

Mmanchev

Aagentzh

Mmarco

Aagentzh