Sorting Qmail Log files with message ID

Saturday, September 8, 2007

Qmail does not use message ID when sending emails and instead uses inode numbers which leads to a lot of duplicate numbering and then ultimately when you go through month old logs you know that logging in Qmail sucks big time, most of the time you can just grep and complete your investigation but at times when you need to submit a official report you get into mess. Well here's script that will parse all the Qmail logs and sort them according to Message ID's (inode) but per email wise, so all generation of email to end of email (delivery / bounce) gets into 4-5 lines which is much faster to read and do a quick find than the normal stuff. Use this script or sorting.

#!/usr/bin/perl

$version = '0.25' ;

use Getopt::Long ;
BEGIN { eval "require IO::Page" ; }

# Change the variables according to your system.
$DEFLOG = '/var/log/qmail/current';
$TAIDIR = '/command';

%uncompress = (
'.Z' => 'uncompress -c',
'.gz' => 'gunzip -c',
'.bz2' => 'bzip2 -dc',
) ;

$res = GetOptions(
"grep=s", \$grep,
"nogrep=s", \$nogrep,
"follow", \$follow,
"help", \$help,
"tell", \$tell,
"seek=i", \$seek,
) ;
if ( $res == 0 || $help == 1 ) {
print STDERR "\n" if $res == 0 ;
( $0 ) = $0 =~ m!([^/]+)$! ;
die <<"USAGE" ; usage: $0 [--grep pattern] emit records that do contain pattern [--nogrep pattern] emit records that don't contain pattern [--follow] follow last log file in list as it grows [--tell] adds a file position for each record [--seek n] where n is an integer from tell: line [--help] this message [file1 ...] list of files to search automatically uncompresses @{[ sort keys %uncompress ]} if systems supports it version: $version USAGE } # set autoflush $| = 1 ; @ARGV = ( $DEFLOG ) unless @ARGV ; $zipexts = join "|", map { quotemeta } keys %uncompress ; @ARGV = map { /($zipexts)$/o ? "$uncompress{$1} $_ |" : $_ } @ARGV ; # process the tai64n timestamps open TAI64N, "|-", "$TAIDIR/tai64nlocal" or die $!; FOLLOW: while (1) { if ( @ARGV > 0 ) {
$FILE = shift @ARGV ;
open FILE
or die "open $FILE: $!\n" ;
}

if ( defined $seek ) {
seek FILE, $seek, 0 ;
$lasttell = $seek ;
} else {
$lasttell = 0 ;
}

while () {
# for syslog maillog => next unless / qmail:/ ;

if ( ( $type, $msgnum ) = /(\w+) msg (\d+)/ ) {
if ( $type eq 'new' ) {
$hash{$msgnum}{MSG} = $_ ;
$hash{$msgnum}{TELL} = $tellbefore ;
} elsif ( $type eq 'end' ) {
printmsg( $msgnum, $_ ) ;
} else {
$hash{$msgnum}{MSG} .= $_ ;
}
} elsif ( ($delivery, $msgnum ) = /delivery (\d+): msg (\d+)/ ) {
$d2m{$delivery} = $msgnum ;
$hash{$msgnum}{$delivery} .= $_ ;
push @{ $hash{$msgnum}{DELI} }, $delivery ;
} elsif ( ( $delivery ) = /delivery (\d+)/ ) {
$hash{$d2m{$delivery}}{$delivery} .= $_ ;
}

# don't do syscall unless needed
$tellbefore = $lasttell ;
$lasttell = tell FILE if $tell ;
}

if ( $follow && @ARGV == 0 ) {
# wait three seconds
sleep 3 ;
# and reset eof!
seek FILE, 0, 1 or die "bad seek on FILE: $!\n" ;
} else {
close FILE ;
while ( ($key,$_) = each %hash ) {
printmsg( $key ) ;
}
undef %hash ;
last FOLLOW if @ARGV == 0 ;
}

} # while(1)
close TAI64N ;

sub printmsg {
my( $msgnum, $txt, ) = @_ ;
my( $delivery, $tellpos, ) ;

$txt = "$hash{$msgnum}{MSG}$txt" ;
foreach $delivery ( @{ $hash{$msgnum}{DELI} } ) {
$txt .= $hash{$msgnum}{$delivery} ;
}
$tellpos = $hash{$msgnum}{TELL} ;

delete $hash{$msgnum} ;

next if $grep && $txt !~ /$grep/sio ;
next if $nogrep && $txt =~ /$nogrep/sio ;

print TAI64N ( ( $tell ? "tell: $tellpos\n" : ""), "$txt\n" );
}

0 comments:

  © Blogger templates Newspaper by Ourblogtemplates.com 2008

Back to TOP