Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.sys.acorn.programmer > #121 > unrolled thread

Re: Limiting the the size of log files

Back to article view | Back to comp.sys.acorn.programmer

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Limiting the the size of log files Bob Latham <bob@sick-of-spam.invalid> - 2011-03-29 12:59 +0100
    Re: Limiting the the size of log files Ian Hamilton <Ian.Hamilton@AAUG.net> - 2011-03-29 13:26 +0100
      Re: Limiting the the size of log files Dr Peter Young <pnyoung@ormail.co.uk> - 2011-03-29 15:02 +0100
      Re: Limiting the the size of log files Bob Latham <bob@sick-of-spam.invalid> - 2011-03-29 15:37 +0100
      Re: Limiting the the size of log files Steve Fryatt <news@stevefryatt.org.uk> - 2011-03-29 22:24 +0100
        Re: Limiting the the size of log files Ian Hamilton <Ian.Hamilton@AAUG.net> - 2011-03-30 06:34 +0100
    Re: Limiting the the size of log files Jeremy Nicoll - news posts <jn.nntp.scrap007@wingsandbeaks.org.uk> - 2011-03-29 13:51 +0100
      Re: Limiting the the size of log files Bob Latham <bob@sick-of-spam.invalid> - 2011-03-29 15:43 +0100
        Re: Limiting the the size of log files Jeremy Nicoll - news posts <jn.nntp.scrap007@wingsandbeaks.org.uk> - 2011-03-29 16:10 +0100
        Re: Limiting the the size of log files "Ste (news)" <steve@revi11.plus.com> - 2011-03-31 13:24 +0100
          Re: Limiting the the size of log files Bob Latham <bob@sick-of-spam.invalid> - 2011-03-31 15:13 +0100
            Re: Limiting the the size of log files Dave Higton <davehigton@dsl.pipex.com> - 2011-03-31 20:14 +0100
              Re: Limiting the the size of log files Matthew Phillips <mnews@sinenomine.freeserve.co.uk> - 2011-03-31 21:27 +0100
              Re: Limiting the the size of log files Jeremy Nicoll - news posts <jn.nntp.scrap007@wingsandbeaks.org.uk> - 2011-03-31 21:45 +0100
                Re: Limiting the the size of log files Dave Higton <davehigton@dsl.pipex.com> - 2011-04-01 21:41 +0100
                  Re: Limiting the the size of log files Jeremy Nicoll - news posts <jn.nntp.scrap007@wingsandbeaks.org.uk> - 2011-04-02 00:33 +0100

#121 — Re: Limiting the the size of log files

In article <mpro.litgto0037k7o033o@wingsandbeaks.org.uk.invalid>,
   Jeremy Nicoll - news posts <jn.nntp.scrap007@wingsandbeaks.org.uk>
wrote:

> Yes... There's no point in throwing away log info; if you do you'll then
> find you need it.  I've always run apps that have configurable logging
> with as much detail in the logs as is possible.  The speed impact is
> negligible compared with the difficulty (with no/inadequate logs) in
> diagnosing problems.

I take your point. I just find that if I don't keep trimming these files
by hand backups take ages backing up huge log files. In addition, I've
never found I needed more than a couple of days of logging. I'm also
concerned about my wife's machine ceasing to function whilst I'm not there
due to a humungous log file. The application in question does not appear
to offer any log file management facilities.
  

Cheers,

Bob.

[toc] | [next] | [standalone]

#122

In article <51bb8c2ee0bob@sick-of-spam.invalid>, Bob Latham
<bob@sick-of-spam.invalid> wrote:
> In article <mpro.litgto0037k7o033o@wingsandbeaks.org.uk.invalid>,
>    Jeremy Nicoll - news posts <jn.nntp.scrap007@wingsandbeaks.org.uk>
> wrote:

> > Yes... There's no point in throwing away log info; if you do you'll
> > then find you need it.  

[Snip]

> I take your point. I just find that if I don't keep trimming these
> files by hand backups take ages backing up huge log files.

[Snip]

> The application in question does not appear to offer any log file
> management facilities.
>   

If by any chance the application is !Hermes, you could look at my
!Debatcher program which includes trimming the logs as one of the options.

Ian

-- 
Ian Hamilton (Iyonix RO5)  http://www.hamiltoni.pwp.blueyonder.co.uk/

[toc] | [prev] | [next] | [standalone]

#124

On 29 Mar 2011  Ian Hamilton <Ian.Hamilton@AAUG.net> wrote:

> In article <51bb8c2ee0bob@sick-of-spam.invalid>, Bob Latham
> <bob@sick-of-spam.invalid> wrote:
>> In article <mpro.litgto0037k7o033o@wingsandbeaks.org.uk.invalid>,
>>    Jeremy Nicoll - news posts <jn.nntp.scrap007@wingsandbeaks.org.uk>
>> wrote:

>>> Yes... There's no point in throwing away log info; if you do you'll
>>> then find you need it.

> [Snip]

>> I take your point. I just find that if I don't keep trimming these
>> files by hand backups take ages backing up huge log files.

> [Snip]

>> The application in question does not appear to offer any log file
>> management facilities.
>> 

> If by any chance the application is !Hermes, you could look at my
> !Debatcher program which includes trimming the logs as one of the options.

And which works excellently. Many thanks for it.

With best wishes,

Peter.

-- 
Peter   \  /      zfc Lu     \     Prestbury, Cheltenham,  Glos. GL52
and      \/ __            __  \                              England.
family   / /  \ | | |\ | /  _  \      http://pnyoung.orpheusweb.co.uk
        /  \__/ \_/ | \| \__/   \______________ pnyoung@ormail.co.uk

[toc] | [prev] | [next] | [standalone]

#125

In article <51bb8e9e6dIan.Hamilton@AAUG.Net>,
   Ian Hamilton <Ian.Hamilton@AAUG.net> wrote:

> If by any chance the application is !Hermes, 

It might be. ;-)

> you could look at my !Debatcher program which includes trimming the logs
> as one of the options.

I'll have a look. Thanks.

Cheers,

Bob.

[toc] | [prev] | [next] | [standalone]

#128

On 29 Mar, Ian Hamilton wrote in message
    <51bb8e9e6dIan.Hamilton@AAUG.Net>:

> If by any chance the application is !Hermes, you could look at my
> !Debatcher program which includes trimming the logs as one of the options.

Doesn't Hermes use SysLog, then?

-- 
Steve Fryatt - Leeds, England             Wakefield Acorn & RISC OS Show
                                              Saturday 16 April 2011
http://www.stevefryatt.org.uk/           http://www.wakefieldshow.org.uk/

[toc] | [prev] | [next] | [standalone]

#129

In article <mpro.liu8ti00xgchc01hz.news@stevefryatt.org.uk>,
   Steve Fryatt <news@stevefryatt.org.uk> wrote:
> On 29 Mar, Ian Hamilton wrote in message
>     <51bb8e9e6dIan.Hamilton@AAUG.Net>:

> > If by any chance the application is !Hermes, you could look at my
> > !Debatcher program which includes trimming the logs as one of the
> > options.

> Doesn't Hermes use SysLog, then?

No. Logs just keep growing.

-- 
Ian Hamilton (Iyonix RO5)  http://www.hamiltoni.pwp.blueyonder.co.uk/

[toc] | [prev] | [next] | [standalone]

#123

Bob Latham <bob@sick-of-spam.invalid> wrote:

> In article <mpro.litgto0037k7o033o@wingsandbeaks.org.uk.invalid>,
>    Jeremy Nicoll - news posts <jn.nntp.scrap007@wingsandbeaks.org.uk>
> wrote:
> 
> > Yes... There's no point in throwing away log info; if you do you'll then
> > find you need it.  I've always run apps that have configurable logging
> > with as much detail in the logs as is possible.  The speed impact is
> > negligible compared with the difficulty (with no/inadequate logs) in
> > diagnosing problems.
> 
> I take your point. I just find that if I don't keep trimming these files
> by hand backups take ages backing up huge log files.

Sure but if the file is archived every day, the current one won't ever get
huge.  And you don't need to make multiple backups of those already
archived.

> In addition, I've never found I needed more than a couple of days of
> logging.

You surprise me.  You use every app every day?  The reason to keep lots of
logs is to give you a good chance that you still have a log from when
something was last used ok - which could be weeks or months ago.

Even in normal use if you're ill or on holiday your machine might not get
used for a few weeks.  You don't want the first thing that happens when it's
next booted is that all the old logs are deleted.

> I'm also concerned about my wife's machine ceasing to function whilst I'm
> not there due to a humungous log file. The application in question does
> not appear to offer any log file management facilities.

It's too much like hard work using per-app log managers.  It's far easier to
use your own code to move any app's log to an archive and (if necessary)
place an empty log file where the app can then add to it.



-- 
Jeremy C B Nicoll - my opinions are my own.

Email sent to my from-address will be deleted. Instead, please reply
to newsreplyaaa@wingsandbeaks.org.uk replacing "aaa" by "284".  

[toc] | [prev] | [next] | [standalone]

#126

In article <mpro.litl1z006h2um033o@wingsandbeaks.org.uk.invalid>,
   Jeremy Nicoll - news posts <jn.nntp.scrap007@wingsandbeaks.org.uk>
wrote:
> Bob Latham <bob@sick-of-spam.invalid> wrote:

> > In article <mpro.litgto0037k7o033o@wingsandbeaks.org.uk.invalid>,
> >    Jeremy Nicoll - news posts <jn.nntp.scrap007@wingsandbeaks.org.uk>
> > wrote:

[Snip]

> > I take your point. I just find that if I don't keep trimming these
> > files by hand backups take ages backing up huge log files.

> Sure but if the file is archived every day, 

Some chance of that. :-)

> the current one won't ever get huge.  And you don't need to make
> multiple backups of those already archived.

Archiving is done at best once a month. Don't have the time for more than
that.

> > In addition, I've never found I needed more than a couple of days of
> > logging.

> You surprise me.  You use every app every day?  The reason to keep lots
> of logs is to give you a good chance that you still have a log from when
> something was last used ok - which could be weeks or months ago.

> Even in normal use if you're ill or on holiday your machine might not
> get used for a few weeks.  You don't want the first thing that happens
> when it's next booted is that all the old logs are deleted.

Which is exactly why I said I would trim the log based on its size not the
dates contained within.

> > I'm also concerned about my wife's machine ceasing to function whilst
> > I'm not there due to a humungous log file. The application in question
> > does not appear to offer any log file management facilities.

> It's too much like hard work using per-app log managers.  It's far
> easier to use your own code to move any app's log to an archive and (if
> necessary) place an empty log file where the app can then add to it.

We log to an NSLU2 which takes about 40 minutes from switch on to become
useable. Yes, I would like something better but that applies to just about
everything I own.


Cheers,

Bob.

[toc] | [prev] | [next] | [standalone]

#127

Bob Latham <bob@sick-of-spam.invalid> wrote:

> > the current one won't ever get huge.  And you don't need to make
> > multiple backups of those already archived.
> 
> Archiving is done at best once a month. Don't have the time for more than
> that.

I think you're conflating two issues.  Archiving non-current logs is very
quick; you're only renaming them out of their in-use locations to somewhere
else.  Daily, or - if logs fill & swap to an alternate - whenever that
happens is easy to organise.

I presume what you "don't have time for" is some sort of major backup
process.  That can look at the archived logs as part of its processing. 
 

> > Even in normal use if you're ill or on holiday your machine might not
> > get used for a few weeks.  You don't want the first thing that happens
> > when it's next booted is that all the old logs are deleted.
> 
> Which is exactly why I said I would trim the log based on its size not the
> dates contained within.

Yes, but you said you'd trim to 10K.  Most of the apps I log from here
generate many MB every day.  Trimming to 10K is tantamount to deleting all
the old data.


-- 
Jeremy C B Nicoll - my opinions are my own.

Email sent to my from-address will be deleted. Instead, please reply
to newsreplyaaa@wingsandbeaks.org.uk replacing "aaa" by "284".  

[toc] | [prev] | [next] | [standalone]

#130

In article <51bb9b357dbob@sick-of-spam.invalid>,
   Bob Latham <bob@sick-of-spam.invalid> wrote:
> We log to an NSLU2 which takes about 40 minutes from switch on to become
> useable.

Run that by me again: this device takes 40 minutes to boot?! Surely there's
something catastropically broken there. I wouldn't trust any backups on a
machine that broken.

Or are you saying your backups take about 40 minutes to complete before you
feel safe to use your machine again?

Ta,

Steve

-- 
Steve Revill @ Home
Note: All opinions expressed herein are my own.

[toc] | [prev] | [next] | [standalone]

#131

In article <51bc9619f8steve@revi11.plus.com>,
   Ste (news) <steve@revi11.plus.com> wrote:
> In article <51bb9b357dbob@sick-of-spam.invalid>,
>    Bob Latham <bob@sick-of-spam.invalid> wrote:
> > We log to an NSLU2 which takes about 40 minutes from switch on to
> > become useable.

> Run that by me again: this device takes 40 minutes to boot?! Surely
> there's something catastropically broken there. I wouldn't trust any
> backups on a machine that broken.

I have one of these at work and another at home. When the drive is empty,
they boot in a reasonable time. However, stick any data on the discs and
they take forever to allow read/write access. Yes, around 40 minutes
before I can read it or write it. They've both been like that for a couple
of years now.

I don't think they are faulty, my guess is that it runs through the disc
and catalogues the contents before allowing anything to be changed but
that's just a guess.

I would like something better but.....


Cheers,

Bob.

[toc] | [prev] | [next] | [standalone]

#134

In message <51bca01fcebob@sick-of-spam.invalid>
          Bob Latham <bob@sick-of-spam.invalid> wrote:

> In article <51bc9619f8steve@revi11.plus.com>,
>    Ste (news) <steve@revi11.plus.com> wrote:
> > In article <51bb9b357dbob@sick-of-spam.invalid>,
> >    Bob Latham <bob@sick-of-spam.invalid> wrote:
> > > We log to an NSLU2 which takes about 40 minutes from switch on to
> > > become useable.
> 
> > Run that by me again: this device takes 40 minutes to boot?! Surely
> > there's something catastropically broken there. I wouldn't trust any
> > backups on a machine that broken.
> 
> I have one of these at work and another at home. When the drive is empty,
> they boot in a reasonable time. However, stick any data on the discs and
> they take forever to allow read/write access. Yes, around 40 minutes
> before I can read it or write it. They've both been like that for a couple
> of years now.
> 
> I don't think they are faulty, my guess is that it runs through the disc
> and catalogues the contents before allowing anything to be changed but
> that's just a guess.

Yes, they verify the integrity of the disc(s) before allowing
access.  Lots of files -> a long time.

Dave

[toc] | [prev] | [next] | [standalone]

#135

In message <52a5bbbc51.davehigton@dsl.pipex.com>
 on 31 Mar 2011 Dave Higton  wrote:

> In message <51bca01fcebob@sick-of-spam.invalid>
>           Bob Latham <bob@sick-of-spam.invalid> wrote:
> 
> > I have one of these at work and another at home. When the drive is empty,
> > they boot in a reasonable time. However, stick any data on the discs and
> > they take forever to allow read/write access. Yes, around 40 minutes
> > before I can read it or write it. They've both been like that for a
> > couple of years now.
> > 
> > I don't think they are faulty, my guess is that it runs through the disc
> > and catalogues the contents before allowing anything to be changed but
> > that's just a guess.
> 
> Yes, they verify the integrity of the disc(s) before allowing
> access.  Lots of files -> a long time.

I cannot remember what OS the NSLU2 comes with by default: it's Linux-based
and locked down.  We reflashed ours, and so run a fairly recent Debian on it
instead.  The hard drive we boot off doesn't have a vast amount on.  I expect
it runs fsck on boot every so often, but it doesn't take too long.  The drive
with our backups on is removed and kept elsewhere, and mounted manually when
required.  I don't think it appears to run fsck for that, so we don't have
any trouble.

The processor in the NSLU2 is much the same as in the Iyonix, I believe, but
the USB system is faster, I suspect.  Nice little machine, though it's
incredibly hair-raising doing an upgrade of the distribution when you have no
keyboard or screen attached.  The only means of access to the machine is
through the network.  We were wanting to use the NSLU2 to run a script to
build an Ubuntu boot disc for the BeagleBoard, but the Debian distribution
was too old to run the script.  In the course of upgrading the NSLU2
something went wrong and we lost network access, which might nearly have been
a disaster.  Fortunately I could plug the NSLU2's boot drive into the
Beagleboard and use the Angstrom distribution which came with the board to
access the drive and fix the faulty config file.  Rebooting the NSLU2 we were
relieved to regain network access to log in.  Then we proceeded to run the
script to produce the Ubuntu boot card for the BeagleBoard!  Convoluted, but
good fun in the end.

I mainly use it for web development, with Apache, PHP and MySQL.  Apart from
the 486 processor in the 2nd hand RISC PC with non-functioning PSU over in
the corner of the room, all of our computers are 100% ARM.  Two Z80-based
machines in boxes in the garage as well.

-- 
Matthew Phillips
Durham

[toc] | [prev] | [next] | [standalone]

#136

Dave Higton <davehigton@dsl.pipex.com> wrote:


> Yes, they verify the integrity of the disc(s) before allowing
> access.  Lots of files -> a long time.

So... zipping lots of older stuff up would make this process faster?

-- 
Jeremy C B Nicoll - my opinions are my own.

Email sent to my from-address will be deleted. Instead, please reply
to newsreplyaaa@wingsandbeaks.org.uk replacing "aaa" by "284".  

[toc] | [prev] | [next] | [standalone]

#138

In message <mpro.lixwba00hv6md00c0@wingsandbeaks.org.uk.invalid>
          Jeremy Nicoll - news posts <jn.nntp.scrap007@wingsandbeaks.org.uk>
wrote:

> Dave Higton <davehigton@dsl.pipex.com> wrote:
> 
> 
> > Yes, they verify the integrity of the disc(s) before allowing access. 
> > Lots of files -> a long time.
> 
> So... zipping lots of older stuff up would make this process faster?

Have you ever tried zipping things on a network drive?  Does it
work for you?

Dave

[toc] | [prev] | [next] | [standalone]

#140

Dave Higton <davehigton@dsl.pipex.com> wrote:

> Have you ever tried zipping things on a network drive?  Does it
> work for you?

I haven't got a network drive.  

I'd zip the stuff elsewhere then place the zips on the drive.

-- 
Jeremy C B Nicoll - my opinions are my own.

Email sent to my from-address will be deleted. Instead, please reply
to newsreplyaaa@wingsandbeaks.org.uk replacing "aaa" by "284".  

[toc] | [prev] | [standalone]

Back to top | Article view | comp.sys.acorn.programmer

csiph-web