Softpanorama
(slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

Google   


Perl HTTP Servers Logs Analyzers

News Recommended Links Articles Perl web log Proxy log analysers Syslog tools Graphic  
AWstats WWWstat  Webalizer Analog Calamaris
(various proxy logs)
C-based Products Commercial Products Etc

A lot can be done using just Perl and pipes. Most useful reports can be generated this way.

There are a lot of free http log file analysis written in Perl out there that haven't been updated since the mid 90's, awstats however few are both free, and up to date. 

There are a lot of free http log file analysis tools out there that haven't been updated since the mid 90's, awstats however is both free, and up to date. It looks a bit like web trends (though I haven't used web trends in several years). Here's an online demo. awstats can be used on several web servers including IIS, and Apache. You can either have generate static html files, or run with a perl script in the cgi-bin.

Here's a quick rundown of setting it up on unix/apache

Each virtual web site you want to track stats for should have a file /etc/awstats.sitename.conf the directives for the configuration file can be found here: http://awstats.sourceforge.net/docs/awstats_config.html they also provide a default conf file in cgi-bin/awstats.model.conf you can use this as a base.

Make sure your log files are using NCSA combined format, this is usually done in apache by saying CustomLog /logs/access.log combined you can use other formats but you have to customize the conf file.

You will probably want to edit the LogFile directive to point to where your logfile is stored, SiteDomain this is the main domain for the site, HostAliases lets you put in other domains for the site, and the DirData directive lets you specify where the awstats databases will be stored (each site will have its own file in the directory).

Once that is setup you will want to update the database this is done from the command line by running

perl awstats.pl –config=sitename –update 

Now copy everything in the wwwroot folder to a web root, and visit http://sitename.com/cgi-bin/awstats.pl if you want to view other domains use /cgi-bin/awstats.pl?config=othersitename

Where sitename would be the name of your config file awstats.sitename.conf

If you want to generate static html files run the awstats_buildstaticpages.pl script found in the tools folder. You have to give it the path to the awstats.pl perl script, and a directory to put the static html files in.

perl awstats_buildstaticpages.pl -config=sitename -awstatsprog=/web/cgi-bin/awstats.pl 
  -dir=/web/stats/sitename/
More setup info can be found here: http://awstats.sourceforge.net/docs/index.html 

If one is looking for Squid logs analyser in Perl, Calamaris is one variant to try.  See also Proxy log analysers

Note: This page is outdated and some links are broken. In such cases please use Goggle or other search engine to find an actual link.  There is plenty of simple Perl scripts for log processing on the Web nowadays and this page is no longer relevant. See for example Unix Log Analysis

Name Platform Cost Available from Notes
3Dstats UNIX free Netstore  
Analog Mac, Windows Unix free http://www.analog.cx/ Mac version also available from summary.net  
BrowserCounter UNIX free Benjamin "Snowhare" Franz  
eXTReMe Tracking Any (online service) Free eXTReMe Tracking Unique visitors, referrers, browser, geographical location... No traffic limitation
FTPWebLog UNIX free Benjamin "Snowhare" Franz  
iisstat UNIX free Lotus Development Corporation  
pwebstat Unix (requires perl5 + fly) free Martin Gleeson  
RefStats UNIX free Benjamin "Snowhare" Franz  
Relax Unix, Windows free (GPL) ktmatu Perl 5 script for referrer and search engine keyword analysis.
Webalizer Unix free (GPL) Bradford L Barrett Supports common logfile format & variations of combined logfile format, partial logs & multiple languages.
wwwstat UNIX free Roy Fielding  
W3Perl Unix, Windows free (GPL) Laurent Domisse  

 



Notes:
  • Those pages are written by people for whom English is not a native language. Some amount of grammar and spelling errors should be expected.
  • This is a Spartan WHYFF (We Help You For Free) site. It cannot replace the best teachers and the best books.
  • The site contain some obsolete pages as it develops like a living tree... Some links on older pages are broken. Please try to use Google, Open directory, etc. to find a replacement link (see HOWTO search the WEB for details). We would appreciate if you can mail us a correct link.

Search Amazon by keywords:

Google   
Open directory

Research Index

 

Old News ;-)

[Nov 22, 2006] freshmeat.net Project details for W3Perl

W3Perl is a Web logfile analyser. All major Web stats are available (referer, agent, session, error, etc.). Reports are fully customisable via configuration files, and there is an administration interface control available.

Release focus: Major feature enhancements

[Mar 30,2006] open.itworld.com - Building and Using Analog on Solaris by Sandra Henry-Stocker

Analog is a free web traffic analysis tool that prepares reports on activity on your web sites, including graphs that summarize hourly, daily, file size file type, visiting site, return codes and numerous other statistics that illustrate how your web sites are being used. I recently compiled and deployed Analog on a couple of Solaris 9 servers. Today's column is a how-to on building Analog and a quick introduction to how it works.

To compile Apache on a Solaris system, you should first grab a copy of the source code. I went to http://www.analog.cx/download.html and downloaded analog-6.0.tar.gz. This command should work on the command line if you have wget installed:

wget http://www.analog.cx/analog-6.0.tar.gz

I then gunzipped and extracted the contents of the downloaded file and attempted to compile the application: $ gunzip analog-6.0.tar.gz $ tar xf analog-6.0.tar $ cd analog-6.0 $ make

My attempt to compile Analog ran into some problems -- notably undefined symbols.
        $ make
        cd src && make
        make[1]: Entering directory `/export/home/henrystocker/analog-6.0/src'
        gcc            -O2      -DUNIX          -c alias.c
        gcc            -O2      -DUNIX          -c analog.c
        gcc            -O2      -DUNIX          -c cache.c
        ... omitted output ...
        Undefined                       first referenced
         symbol                             in file
        gethostbyaddr                       alias.o
        inet_addr                           alias.o
        ld: fatal: Symbol referencing errors. No output written to ../analog
        collect2: ld returned 1 exit status
        make[1]: *** [analog] Error 1
        make[1]: Leaving directory `/export/home/henrystocker/analog-6.0/src'
        make: *** [analog] Error 2

I soon figured out that I needed to make a small change to one of my Makefiles. I made the change with this perl command, adding the network services library after noting that the man pages for the undefined symbols both referenced -lnsl.
        $ cd src
        $ perl -i -p -e "s/LIBS = -lm/LIBS = -lnsl -lm/" Makefile

My LIBS line then looked like this: 

        LIBS = -lnsl -lm (added -lnsl)

After this change, Analog compiled without a hitch: 

        $ cd ..
        $ make
        cd src && make
        make[1]: Entering directory `/export/home/shs/analog-6.0/src'
        gcc            -O2      -DUNIX          -c alias.c
        gcc            -O2      -DUNIX          -c analog.c
        gcc            -O2      -DUNIX          -c cache.c
        ... omitted output ...
        gcc            -O2     -o ../analog alias.o analog.o cache.o dates.o
        globals.o hash.o init.o init2.o input.o macinput.o macstuff.o output.o
        output2.o outcro.o outhtml.o outlatex.o outplain.o outxhtml.o outxml.o
        process.o settings.o sort.o tree.o utils.o win32.o libgd/gd.o
        libgd/gd_io.o libgd/gd_io_file.o libgd/gd_png.o libgd/gdfontf.o
        libgd/gdfonts.o libgd/gdtables.o libpng/png.o libpng/pngerror.o
        libpng/pngmem.o libpng/pngset.o libpng/pngtrans.o libpng/pngwio.o
        libpng/pngwrite.o libpng/pngwtran.o libpng/pngwutil.o pcre/pcre.o
        zlib/adler32.o zlib/compress.o zlib/crc32.o zlib/deflate.o zlib/gzio.o
        zlib/infblock.o zlib/infcodes.o zlib/inffast.o zlib/inflate.o
        zlib/inftrees.o zlib/infutil.o zlib/trees.o zlib/uncompr.o zlib/zutil.o
        unzip/ioapi.o unzip/unzip.o bzip2/bzlib.o bzip2/blocksort.o
        bzip2/compress.o bzip2/crctable.o bzip2/decompress.o bzip2/huffman.o
        bzip2/randtable.o -lnsl -lm
        make[1]: Leaving directory `/export/home/shs/analog-6.0/src'

        $ ls -l analog
        -rwxr-xr-x   1 root     other     577568 Mar 29 19:34 analog
Once Analog was compiled, I moved it into /usr/local/bin (there was no "make install" option) and ran a "make clean" to remove object files. At this point, I had switched over to root.

The next step was setting up a configuration file to give Analog some directions on how I wanted it to work. Analog comes with example configuration files and there are numerous options that can be used to customize your reports, but I wanted to start with something simple, so I set up a handful of options and installed the file as /usr/local/bin/analog.cfg:
        # cat > /usr/local/bin/analog.cfg << EOF
        > LANGFILE usa.lng
        > HOSTNAME boson.particles.org
        > HOSTURL "http://boson.particles.org"
        > DAILYSUM ON
        > DAILYREP ON
        > LOGFILE /opt/apache/logs/access_log
        > OUTFILE /opt/apache/htdocs/webstats.html
        > DOMAINSFILE usdom.tab
        > EOF
I also had to create a directory named /usr/local/bin/lang and copy the usa.lng file and usdom.tab files from my lang directory into it.
        # mkdir /usr/local/bin/lang
        # cp lang/usa.lng /usr/local/bin/lang
        # cp lang/usdom.tab /usr/local/bin/lang
I then ran the report like this:

# analog /opt/apache/logs/access_log

My processed report appeared in my /opt/apache/htdocs directory along with four image files containing pie charts for some of my statistics. You can see a sample Analog report here.
Here's a list of features that you can turn off or on: 
MONTHLY ON       # one line for each month
WEEKLY ON        # one line for each week
DAILYREP ON      # one line for each day
DAILYSUM ON      # one line for each day of the week
HOURLYREP ON     # one line for each hour of the day
GENERAL ON       # the General Summary at the top
REQUEST ON       # which files were requested
FAILURE ON       # which files were not found
DIRECTORY ON     # Directory Report
HOST ON          # which computers requested files
ORGANISATION ON  # which organisations they were from
DOMAIN ON        # which countries they were in
REFERRER ON      # where people followed links from
FAILREF ON       # where people followed broken links from
SEARCHQUERY ON   # the phrases and words they used...
SEARCHWORD ON    # ...to find you from search engines
BROWSERSUM ON    # which browser types people were using
OSREP ON         # and which operating systems
FILETYPE ON      # types of file requested
SIZE ON          # sizes of files requested
STATUS ON        # number of each type of success and failure

Internet Access Monitor for Squid 2.4d - Monitoring of company's Internet access usage

Download Internet Access Monitor for Squid free - 2.01 Mb

Internet Access Monitor is a comprehensive Internet use monitoring and reporting utility for corporate networks. The program takes advantage of the fact that most corporations provide Internet access through proxy servers, like MS ISA Server, WinGate, WinRoute, MS Proxy, WinProxy, EServ, Squid, Proxy Plus and others. Each time any user accesses any website, downloads files or images, these actions are logged. Internet Access Monitor processes these log files to offer system administrators wealth of report building options. The program can build reports for individual users, showing the list of websites he or she visited, along with a detailed break down of internet activity (downloading, reading text, viewing pictures, watching movies, listening to music, working). Plus, the program can create comprehensive reports with analysis of overall bandwidth consumption, building easy to comprehend visual charts that suggest the areas where wasteful bandwidth consumption may be eliminated.

A good and free Perl http log analysis script - awstats

There are a lot of free http log file analysis tools out there that haven't been updated since the mid 90's, awstats however few are both free, and up to date.  AWSTAT is one of the latter.

There are a lot of free http log file analysis tools out there that haven't been updated since the mid 90's, awstats however is both free, and up to date. It looks a bit like web trends (though I haven't used web trends in several years). Here's an online demo. awstats can be used on several web servers including IIS, and Apache. You can either have generate static html files, or run with a perl script in the cgi-bin.

Here's a quick rundown of setting it up on unix/apache

Each virtual web site you want to track stats for should have a file /etc/awstats.sitename.conf the directives for the configuration file can be found here: http://awstats.sourceforge.net/docs/awstats_config.html they also provide a default conf file in cgi-bin/awstats.model.conf you can use this as a base.

Make sure your log files are using NCSA combined format, this is usually done in apache by saying CustomLog /logs/access.log combined you can use other formats but you have to customize the conf file.

You will probably want to edit the LogFile directive to point to where your logfile is stored, SiteDomain this is the main domain for the site, HostAliases lets you put in other domains for the site, and the DirData directive lets you specify where the awstats databases will be stored (each site will have its own file in the directory).

Once that is setup you will want to update the database this is done from the command line by running

perl awstats.pl –config=sitename –update 

Now copy everything in the wwwroot folder to a web root, and visit http://sitename.com/cgi-bin/awstats.pl if you want to view other domains use /cgi-bin/awstats.pl?config=othersitename

Where sitename would be the name of your config file awstats.sitename.conf

If you want to generate static html files run the awstats_buildstaticpages.pl script found in the tools folder. You have to give it the path to the awstats.pl perl script, and a directory to put the static html files in.

perl awstats_buildstaticpages.pl -config=sitename -awstatsprog=/web/cgi-bin/awstats.pl 
  -dir=/web/stats/sitename/
More setup info can be found here: http://awstats.sourceforge.net/docs/index.html 

AWStats - Free log file analyzer for advanced statistics (GNU GPL).

Flexible and robust Perl written WEB log analyser. Industrial strength ! Highly recommended..

AWStats is a free powerful and featureful tool that generates advanced web, ftp or mail server statistics, graphically. This log analyzer works as a CGI or from command line and shows you all possible information your log contains, in few graphical web pages. It uses a partial information file to be able to process large log files, often and quickly. It can analyze log files from IIS (W3C log format), Apache log files (NCSA combined/XLF/ELF log format or common/CLF log format), WebStar and most of all web, proxy, wap, streaming servers, mail servers (and some ftp).
Take a look at this comparison table for an idea on differences between most famous statistics tools (AWStats, Analog, Webalizer,...).
AWStats is a free software distributed under the GNU General Public License. You can have a look at this license chart to know what you can/can't do.
As AWStats works from the command line but also as a CGI, it can work with major web hosting provider that allows CGI and log access.

You can browse AWStats demo (Real-time feature to update stats from web has been disabled on demos) to see a sample of most important information AWStats shows you...

[Aug 5, 2001] Squid2MySQL

Squid2MySQL is an accounting system for squid. It includes monthly, daily, and timed detail levels, and uses MySQL for log storage.

Eugene V. Chernyshev <evc (at) chat (dot) ru> [contact developer]

[Aug 29, 2000 ] Recommendations for log analysis software

I'm looking for access log analysis software that will run independently from the ACS. Hopefully the thing would read an AOLserver or Apache access log and generate some graphs and tables in a configurable way. Any specific suggestions? The goal is usage analysis for marketing purposes (i.e., not performance analysis). Probably running on Solaris and it doesn't have to be freeware.

-- S. Y., August 29, 2000

I've been using webalizer (http://www.webalizer.com). It's real easy to set up and configure. For a sample of what it does, check out http://www.badgertronics.com/reports
-- Mark Dalrymple, August 29, 2000

And if you don't mind paying, NetTracker (from http://www.sane.com) has a ton of reports, handles big log files (a site we know that's using it has 600 meg daily access logs), handles cobranding, etc
-- Mark Dalrymple, August 29, 2000

Nettracker rules! We've been using it for a couple of years now and everyone loves it. It has no knowledge of the ACS or it's users, of coures, so real clickstream tracking isn't really possible, but for basic log analysis with lots of information it's great.
-- Janine Sisk, August 29, 2000

I worked quite a while with NetAnalysis by the Boston-based company NetGen. It's database-backed log file analysis software, i .e. usually once per day you stuff your log file information into your db. This way you can keep track of your homepage's success over time. It also enables you to connect your webserver log information with information from other databases and come up with really powerful information. For example it has pretty damn good adaptors to Intershop and to Vignette StoryServer and you can implement your own adaptors with a Perl-based (soon to be Java) API. (Why do I know these adaptors are good? I made the one for Intershop work properly and I specified the one for my former company's Vignette StoryServer standard installation ;)

The software to run analysis on your log is very smartly configurable and you have good metrics and I was promised some really powerful metrics for a future release (that should be out as of now).

It runs on Solaris and you'll need Oracle and a pretty heavy machine (way bigger than your webserver usually).

Other products I know about are Accrue (used to be based on sniffing TCP/IP packets) and iDecide by Informix (has another name these days) that are db-backed as well. Oh, and then I once met with a Canadian company that use JavaScript to spy out some information about the client. It was probably the most expensive software regarding costs/line of code - tho they had very, very, very good metrics. You can head over to playboy.com and look for a JS call on their entry site ;)

regards Dirk
-- Dirk Gómez, August 30, 2000

I have had great success with 'analog' a free utility for log parsing and analysis. It can generate simple reports and also machine readable reports (good for DB backed summary generation). There is also a perl based package called 'Report Magic' which takes the machine readable output and makes pretty pictures and tables for you.

The problem with many Log analysis tools is that they cannot be consumate and definately cannot reflect the structure of your particular site especially if it has 'interesting' mechanisms by which content is rendered - frames, multiple urls for a given page etc. Also, when you get ~ 1GB of log data a day, some cjust croak big time. In this case, rolling-your-own is one solution whih I have been implementing (using perl -regexp is you friend- and DB) which follows some of the principles discussed at ASJ/. It also allows for simple reporting on only the bits n pieces I feel necessary - quick with HUGE sets of data.

This is a good technique for allowing ad-hoc queries on data but make sure you have a reasonably hefty machine (Dual Xeon with 2GB RAM, 100GB RAID).

I guess if there is a package out there which is extensible and modular (plugin support as mentioned above) then that may also be quite useful.

It's important to define exactly what info is most important in analysing logs else you can open up a huge can of worms - ie. mapping all relationships between everything which whilst interesting, is quite difficult and reasonalby useless unless it is quick and can add value in some way.

Sorry if I've stated the obvious, just my $0.05
-- geoff webb, September 4, 2000

Look carefully before you choose NetTracker. It scans the log files and saves the data into its own flat file database. If you ever lose files or decide that you want to add another report, it needs to regenerate all of that data. For a month's worth of logs (the 600 MB+ logs referred to by Mark Dalrymple above), it can take 3+ days.
-- Doug Harris, September 11, 2000

HTTPd Log Analyzers

Log Analyzer Tools

PAGE-STATS

page-stats.pl will examine the acceslog of a http daemon and search it for occurrences of certain references. These references are then counted and put into a HTML file that is ready to be displayed to the outside world as a "Page Statistics" page. Each page can be selected from the statistics page.

The Big Brother Log Analyzer (BBLA) Perl and C

Big Brother Log Analyzer, or BBLA for short, is a package comprised of two components: a logger, which logs all accesses to selected web pages, and a log analyzer, which nicely formats the logs into an HTML page. The generated HTML is fully W3C compliant (HTML 4.01/Transitional), which guarantees that it will be rendered the way it should under any compliant browser. Another interesting feature of BBLA is that it is tag-based (you put a tag in each page you want to track): this allows for tracking pages hosted on different servers. For instance, I track accesses to my pages in the School of Information Management and Systems at the University of California, Berkeley, along with these pages hosted on SourceForge in a single file. See the demo for more information.

A lot of HTML log analyzers exist on the market, but most of them are either targeted at systems administrators (with full access to httpd log files, for instance), or require general users to display an advertising banner on their pages, or (even worse) limit the number of pages you can track for free. Most of the time, the pages generated do not even follow the W3C consortium recommendations for writing proper HTML, yielding unpredictable results when viewed with different browsers.

BBLA is free, doesn't require you to have a banner on your web page, uses W3C-compliant HTML and PNG images (hence, no licensing issues with GIFs), allows for tracking pages hosted on different servers, and is actually completely transparent. So, unless your visitors look into your HTML source, they won't notice that you are tracking them.

Last but not least, BBLA is extremely light-weight: the current tarball is roughly 30KB, making it much more easier to install on platforms with scarce disk space than some of its counterparts. As an added bonus, it doesn't take ages to compile, even on really ancient hardware.

freshmeat.net Project details for Relax log analyzer

Relax is a multi-platform Web server log analyzer written in Perl. It can be used to track which search engines, search keywords, and referring URLs led visitors to the Web site. It can also track down bad links and analyze which keywords to bid for at pay-per-click search engines. The parser module in Relax recognizes several hundred search engines and is capable of extracting the keywords used. Generated HTML reports can be configured to include links to other Web-based keyword analysis tools, making it easier to further improve the ranking of web pages in search engines.

freshmeat.net Project details for proxy-report.pl

proxy-report.pl generates a list of requested server addresses (simplified URLs) from your Squid proxy server log files. Requests for each URL are summarized on a per day basis. This script can generate reports based on the IP of the user. It also automatically handles gzipped files. URL exclusion patterns are supported. A sample report is available on the home page.

Calamaris Home Page

parses logfiles from Squid, NetCache, Inktomi Traffic Server, Oops! proxy server, Novell Internet Caching System, Compaq Tasksmart or Netscape/iplanet Web Proxy Server and generates a report. Written in perl5.

About: Calamaris parses the logfiles of a wide variety of Web proxy servers and generates reports about peak-usage, request-methods, status-report of incoming and outgoing requests, second and top-level destinations, content-types, and performance.

Changes: There are bugfixes regarding %-URLs and syntactically wrong HTML output. A function to use Calamaris' output in server side includes has been added.

[Mar 11, 2000] O'Reilly Network: Log Rhythms

"In this column I'll give you a gentle introduction to Apache web server logs and their place in monitoring, security, marketing, and feedback."

stoic Freeware web server log analysis tool Jul 05th 1999, 05:52
stable: 1.2 - devel: none license: Freeware

Stoic is a small Perl script that examines Apache or Netscape web server access logs. Reports include logfile totals, domains visiting, top documents requested, browser agent statistics, platform statistics and a bunch of other stuff.

[May, 26, 1999] Webalizer scoop - January 11th 1998, 20:47 EST

Download: ftp://ftp.mrunix.net/pub/webalizer/
Alternate Download: ftp://samhain.unix.cslab.tuwien.ac.at/webalizer/
Homepage: http://www.mrunix.net/webalizer/
Changelog: ftp://ftp.mrunix.net/pub/webalizer/CHANGES

The Webalizer is a web server log analysis program. It is designed to scan web server log files in various formats and produce usage statistics in HTML format for viewing through a browser. Very good output, good charts and very fast. Just missing special mode for a total statistics (all virtual webservers) without that many details.

[Mar 26, 1999] Information about Web Trends -- a commercial package was added to etc.


Recommended Links

freshmeat.net Browse project tree - Topic Internet Log Analysis

Google Directory - Computers Software Internet Site Management Log Analysis

Produces highly detailed, easily configurable, incremental HTML usage reports in many languages, from multiple log formats, with builds for Linux, Solaris, Mac, OS/2, Cobalt, OpenVMS, Netware, and BeOS. [Open Source, GPL]

Computers: Software: Internet: Site Management: Log Analysis: Commercial   (20 matches)
 

Computers: Software: Internet: Site Management: Log Analysis: Freeware and Open Source   (5)
 

 


Articles

SunWorld Online - March - Webmaster

SunWorld Online - May - Webmaster -- Analyzing your referrer log

The PIPER Letter Making Sense of Web Usage Statistics

Selected Perl log processing scripts

logmanage

logmanage is a program which is designed to perform flexible management of web statistics for a variety of users and main server logs. In its current configuration it is designed to work with http-analyze but should work with any web stats program that takes log input on STDIN and can be configured for the output directory on the command line. Manages a large collection of pipes to the stats program with inclusion and exclusion regular expressions. Can generate stats for lots of different users from one log file or from many log files.

recycle-logs

Recycle-logs is a logfile manager written in Perl that attempts to overcome the limitations of other system log utilities. File rotation and other customization is based on control information specified in one or several configuration files.

W3Perl

W3Perl is a Web logfile analyzer. All major Web stats are available (referrer, agent, session, error, etc.). Reports are fully customizable via configuration files, and there is an administration interface control available.

AWstats

AWStats - Free log file analyzer for advanced statistics (GNU GPL). -- extremely flexible and robust Perl=written log analyser. Industrial strength ! Highly recommended..

AWStats is a free powerful and featureful tool that generates advanced web, ftp or mail server statistics, graphically. This log analyzer works as a CGI or from command line and shows you all possible information your log contains, in few graphical web pages. It uses a partial information file to be able to process large log files, often and quickly. It can analyze log files from IIS (W3C log format), Apache log files (NCSA combined/XLF/ELF log format or common/CLF log format), WebStar and most of all web, proxy, wap, streaming servers, mail servers (and some ftp).

Take a look at this comparison table for an idea on differences between most famous statistics tools (AWStats, Analog, Webalizer,...).

AWStats is a free software distributed under the GNU General Public License. You can have a look at this license chart to know what you can/can't do.

As AWStats works from the command line but also as a CGI, it can work with major web hosting provider that allows CGI and log access.

You can browse AWStats demo (Real-time feature to update stats from web has been disabled on demos) to see a sample of most important information AWStats shows you...

A good and free log analysis tool - awstats

There are a lot of free http log file analysis tools out there that haven't been updated since the mid 90's, awstats however is both free, and up to date. It looks a bit like web trends (though I haven't used web trends in several years). Here's an online demo. awstats can be used on several web servers including IIS, and Apache. You can either have generate static html files, or run with a perl script in the cgi-bin.

Here's a quick rundown of setting it up on unix/apache

Each virtual web site you want to track stats for should have a file /etc/awstats.sitename.conf the directives for the configuration file can be found here: http://awstats.sourceforge.net/docs/awstats_config.html they also provide a default conf file in cgi-bin/awstats.model.conf you can use this as a base.

Make sure your log files are using NCSA combined format, this is usually done in apache by saying CustomLog /logs/access.log combined you can use other formats but you have to customize the conf file.

You will probably want to edit the LogFile directive to point to where your logfile is stored, SiteDomain this is the main domain for the site, HostAliases lets you put in other domains for the site, and the DirData directive lets you specify where the awstats databases will be stored (each site will have its own file in the directory).

Once that is setup you will want to update the database this is done from the command line by running

perl awstats.pl –config=sitename –update 

Now copy everything in the wwwroot folder to a web root, and visit http://sitename.com/cgi-bin/awstats.pl if you want to view other domains use /cgi-bin/awstats.pl?config=othersitename

Where sitename would be the name of your config file awstats.sitename.conf

If you want to generate static html files run the awstats_buildstaticpages.pl script found in the tools folder. You have to give it the path to the awstats.pl perl script, and a directory to put the static html files in.

perl awstats_buildstaticpages.pl -config=sitename -awstatsprog=/web/cgi-bin/awstats.pl 
  -dir=/web/stats/sitename/
More setup info can be found here: http://awstats.sourceforge.net/docs/index.html 

Webalizer

Webalizer The Webalizer is a fast, free web server log file analysis program. It produces highly detailed, easily configurable usage reports in HTML format, for easy viewing with a standard web browser.

wwwstat

wwwstat HTTPd Logfile Analysis Software

The wwwstat program will process a sequence of HTTPd common logfile format (CLF) access_log files and output a log summary in HTML format suitable for publishing on a website.

The splitlog program will process a sequence of CLF (or CLF with a prefix) access_log files and split the entries into separate files according to the requested URL and/or vhost prefix.

Both programs are written in Perl and, once customized for your site, should work on any UNIX-based system with Perl 4.036, 5.002, or better.

Qiegang Long, formerly at UMass, has released a program called gwstat that takes the output from wwwstat and generates a set of graphs to illustrate your httpd server traffic by hour, day, week or calling country/domain.

A mailing list, now shut down, was created for discussion and support of wwwstat development.

Etc Perl Scripts

Log Scanner 0.9b

Log Scanner was written to watch for anomalies in log files. Upon finding them, it can notify you in a variety of ways. It was designed to be very modular and configurable. Unlike most other log scanners, this one has more than single pattern matches. It will allow you to trigger notifications on multiple occurrences of one or several events.


Graphic


C-based Products


Commercial Products

FlashStats Mac
Unix
Windows
$99/$249 Maximized Software  
HTTP-Analyze Unix
Windows
free/326 euro/388 euro/1470 euro http://www.http-analyze.org/  
Lumberjack Unix $1250 BitWrench Inc  
NetTracker Unix, Windows from $495 Sane Solutions  
WebTrends Windows
Solaris
Red Hat
$499 - $1999 NetIQ  
Sawmill Mac
Unix
Windows
$999 Substantial discounts for edu & small organisations sawmill.net  

Web Trends -- actually a pretty limited commercial package. Decent prepackaged reporting capabilities, but not very flexible (reports for dummies)


Etc

EasyLog version 1.2
This is a simple Server Side Includes script that can "watch" a given page, and add entries to a HTML file reporting what browser they are using, when they accessed your page, and a few other pieces of information. Language: Perl   Platform: Unix View Product Homepage

Checklog version 1.0
Perl script that analyzes HTTP server logs.

Language: Perl   Platform: Unix, Windows

View Product Homepage

Download Complete Source Code, 0.010M bytes

Click file name to view online:
checklog.pl, 14504 bytes

FTPWebLog version 1.0.3
Perl script that analyses WWW and FTP logs and produces graphical reports.

Language: Perl   Platform: Unix, Windows

View Product Homepage

Download Complete Source Code, 0.096M bytes

Relax version 2.0
Relax is a free reference log analysis program written in Perl, which can be used to analyse how people are finding your web site, what keywords they use in the search engines, and how they move within the site.

Language: Perl   Platform: Unix, Windows

View Product Homepage

Download Complete Source Code, 0.010M bytes

Log Reverse Domain Name System (lrdns) version 1.1 -- Converts numeric IP addresses in accesss log files into textual domain names. Language: Perl   Platform: Unix

View Product Homepage

Download Complete Source Code, 0.010M bytes

See also



Copyright © 1996-2007 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. Submit comments This document is an industrial compilation designed and created exclusively for educational use and is placed under the copyright of the Open Content License(OPL). Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

Standard disclaimer: The statements, views and opinions presented on this web page are those of the author and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: March 15, 2008