|
Softpanorama |
May the source be with you, but remember the KISS principle ;-)
Softpanorama Search
|
A lot can be done using just Perl and pipes. Most useful reports can be generated this way.
There are a lot of free http log file analysis written in Perl out there that haven't been updated since the mid 90's. However awstats is one of the few well-written script that is both free, and up to date.
There are a lot of free http log file analysis tools out there that haven't been updated since the mid 90's, awstats however is both free, and up to date. It looks a bit like web trends (though I haven't used web trends in several years). Here's an online demo. awstats can be used on several web servers including IIS, and Apache. You can either have generate static html files, or run with a perl script in the cgi-bin.
Here's a quick rundown of setting it up on unix/apache
Each virtual web site you want to track stats for should have a file /etc/awstats.sitename.conf the directives for the configuration file can be found here: http://awstats.sourceforge.net/docs/awstats_config.html they also provide a default conf file in cgi-bin/awstats.model.conf you can use this as a base.
Make sure your log files are using NCSA combined format, this is usually done in apache by saying CustomLog /logs/access.log combined you can use other formats but you have to customize the conf file.
You will probably want to edit the LogFile directive to point to where your logfile is stored, SiteDomain this is the main domain for the site, HostAliases lets you put in other domains for the site, and the DirData directive lets you specify where the awstats databases will be stored (each site will have its own file in the directory).
Once that is setup you will want to update the database this is done from the command line by running
perl awstats.pl –config=sitename –updateNow copy everything in the wwwroot folder to a web root, and visit http://sitename.com/cgi-bin/awstats.pl if you want to view other domains use /cgi-bin/awstats.pl?config=othersitename
Where sitename would be the name of your config file awstats.sitename.conf
If you want to generate static html files run the
awstats_buildstaticpages.plscript found in the tools folder. You have to give it the path to theawstats.plperl script, and a directory to put the static html files in.perl awstats_buildstaticpages.pl -config=sitename -awstatsprog=/web/cgi-bin/awstats.pl -dir=/web/stats/sitename/More setup info can be found here: http://awstats.sourceforge.net/docs/index.html
If one is looking for Squid logs analyser in Perl, Calamaris is one variant to try. See also Proxy log analysers
Note: This page is outdated and some links are broken. In such cases please use Goggle or other search engine to find an actual link. There is plenty of simple Perl scripts for log processing on the Web nowadays and this page is no longer relevant. See for example Unix Log Analysis
| Name | Platform | Cost | Available from | Notes |
|---|---|---|---|---|
| 3Dstats | UNIX | free | Netstore | |
| Analog | Mac, Windows Unix | free | http://www.analog.cx/ Mac version also available from summary.net | |
| BrowserCounter | UNIX | free | Benjamin "Snowhare" Franz | |
| eXTReMe Tracking | Any (online service) | Free | eXTReMe Tracking | Unique visitors, referrers, browser, geographical location... No traffic limitation |
| FTPWebLog | UNIX | free | Benjamin "Snowhare" Franz | |
| iisstat | UNIX | free | Lotus Development Corporation | |
| pwebstat | Unix (requires perl5 + fly) | free | Martin Gleeson | |
| RefStats | UNIX | free | Benjamin "Snowhare" Franz | |
| Relax | Unix, Windows | free (GPL) | ktmatu | Perl 5 script for referrer and search engine keyword analysis. |
| Webalizer | Unix | free (GPL) | Bradford L Barrett | Supports common logfile format & variations of combined logfile format, partial logs & multiple languages. |
| wwwstat | UNIX | free | Roy Fielding | |
| W3Perl | Unix, Windows | free (GPL) | Laurent Domisse |
|
|||||||
W3Perl is a Web logfile analyser. All major Web stats are available (referer, agent, session, error, etc.). Reports are fully customisable via configuration files, and there is an administration interface control available.Release focus: Major feature enhancements
Analog is a free web traffic analysis tool that prepares reports on activity on your web sites, including graphs that summarize hourly, daily, file size file type, visiting site, return codes and numerous other statistics that illustrate how your web sites are being used. I recently compiled and deployed Analog on a couple of Solaris 9 servers. Today's column is a how-to on building Analog and a quick introduction to how it works.
To compile Apache on a Solaris system, you should first grab a copy of the source code. I went to http://www.analog.cx/download.html and downloaded analog-6.0.tar.gz. This command should work on the command line if you have wget installed:
wget http://www.analog.cx/analog-6.0.tar.gz
I then gunzipped and extracted the contents of the downloaded file and attempted to compile the application: $ gunzip analog-6.0.tar.gz $ tar xf analog-6.0.tar $ cd analog-6.0 $ make
My attempt to compile Analog ran into some problems -- notably undefined symbols.$ make cd src && make make[1]: Entering directory `/export/home/henrystocker/analog-6.0/src' gcc -O2 -DUNIX -c alias.c gcc -O2 -DUNIX -c analog.c gcc -O2 -DUNIX -c cache.c ... omitted output ... Undefined first referenced symbol in file gethostbyaddr alias.o inet_addr alias.o ld: fatal: Symbol referencing errors. No output written to ../analog collect2: ld returned 1 exit status make[1]: *** [analog] Error 1 make[1]: Leaving directory `/export/home/henrystocker/analog-6.0/src' make: *** [analog] Error 2I soon figured out that I needed to make a small change to one of my Makefiles. I made the change with this perl command, adding the network services library after noting that the man pages for the undefined symbols both referenced -lnsl.$ cd src $ perl -i -p -e "s/LIBS = -lm/LIBS = -lnsl -lm/" Makefile My LIBS line then looked like this: LIBS = -lnsl -lm (added -lnsl) After this change, Analog compiled without a hitch: $ cd .. $ make cd src && make make[1]: Entering directory `/export/home/shs/analog-6.0/src' gcc -O2 -DUNIX -c alias.c gcc -O2 -DUNIX -c analog.c gcc -O2 -DUNIX -c cache.c ... omitted output ... gcc -O2 -o ../analog alias.o analog.o cache.o dates.o globals.o hash.o init.o init2.o input.o macinput.o macstuff.o output.o output2.o outcro.o outhtml.o outlatex.o outplain.o outxhtml.o outxml.o process.o settings.o sort.o tree.o utils.o win32.o libgd/gd.o libgd/gd_io.o libgd/gd_io_file.o libgd/gd_png.o libgd/gdfontf.o libgd/gdfonts.o libgd/gdtables.o libpng/png.o libpng/pngerror.o libpng/pngmem.o libpng/pngset.o libpng/pngtrans.o libpng/pngwio.o libpng/pngwrite.o libpng/pngwtran.o libpng/pngwutil.o pcre/pcre.o zlib/adler32.o zlib/compress.o zlib/crc32.o zlib/deflate.o zlib/gzio.o zlib/infblock.o zlib/infcodes.o zlib/inffast.o zlib/inflate.o zlib/inftrees.o zlib/infutil.o zlib/trees.o zlib/uncompr.o zlib/zutil.o unzip/ioapi.o unzip/unzip.o bzip2/bzlib.o bzip2/blocksort.o bzip2/compress.o bzip2/crctable.o bzip2/decompress.o bzip2/huffman.o bzip2/randtable.o -lnsl -lm make[1]: Leaving directory `/export/home/shs/analog-6.0/src' $ ls -l analog -rwxr-xr-x 1 root other 577568 Mar 29 19:34 analogOnce Analog was compiled, I moved it into /usr/local/bin (there was no "make install" option) and ran a "make clean" to remove object files. At this point, I had switched over to root.
The next step was setting up a configuration file to give Analog some directions on how I wanted it to work. Analog comes with example configuration files and there are numerous options that can be used to customize your reports, but I wanted to start with something simple, so I set up a handful of options and installed the file as /usr/local/bin/analog.cfg:# cat > /usr/local/bin/analog.cfg << EOF > LANGFILE usa.lng > HOSTNAME boson.particles.org > HOSTURL "http://boson.particles.org" > DAILYSUM ON > DAILYREP ON > LOGFILE /opt/apache/logs/access_log > OUTFILE /opt/apache/htdocs/webstats.html > DOMAINSFILE usdom.tab > EOFI also had to create a directory named /usr/local/bin/lang and copy the usa.lng file and usdom.tab files from my lang directory into it.# mkdir /usr/local/bin/lang # cp lang/usa.lng /usr/local/bin/lang # cp lang/usdom.tab /usr/local/bin/langI then ran the report like this:
# analog /opt/apache/logs/access_log
My processed report appeared in my /opt/apache/htdocs directory along with four image files containing pie charts for some of my statistics. You can see a sample Analog report here.Here's a list of features that you can turn off or on: MONTHLY ON # one line for each month WEEKLY ON # one line for each week DAILYREP ON # one line for each day DAILYSUM ON # one line for each day of the week HOURLYREP ON # one line for each hour of the day GENERAL ON # the General Summary at the top REQUEST ON # which files were requested FAILURE ON # which files were not found DIRECTORY ON # Directory Report HOST ON # which computers requested files ORGANISATION ON # which organisations they were from DOMAIN ON # which countries they were in REFERRER ON # where people followed links from FAILREF ON # where people followed broken links from SEARCHQUERY ON # the phrases and words they used... SEARCHWORD ON # ...to find you from search engines BROWSERSUM ON # which browser types people were using OSREP ON # and which operating systems FILETYPE ON # types of file requested SIZE ON # sizes of files requested STATUS ON # number of each type of success and failure
Download Internet Access Monitor for Squid free - 2.01 MbInternet Access Monitor is a comprehensive Internet use monitoring and reporting utility for corporate networks. The program takes advantage of the fact that most corporations provide Internet access through proxy servers, like MS ISA Server, WinGate, WinRoute, MS Proxy, WinProxy, EServ, Squid, Proxy Plus and others. Each time any user accesses any website, downloads files or images, these actions are logged. Internet Access Monitor processes these log files to offer system administrators wealth of report building options. The program can build reports for individual users, showing the list of websites he or she visited, along with a detailed break down of internet activity (downloading, reading text, viewing pictures, watching movies, listening to music, working). Plus, the program can create comprehensive reports with analysis of overall bandwidth consumption, building easy to comprehend visual charts that suggest the areas where wasteful bandwidth consumption may be eliminated.
There are a lot of free http log file analysis tools out there that haven't been updated since the mid 90's, awstats however is both free, and up to date. It looks a bit like web trends (though I haven't used web trends in several years). Here's an online demo. awstats can be used on several web servers including IIS, and Apache. You can either have generate static html files, or run with a perl script in the cgi-bin.
Here's a quick rundown of setting it up on unix/apache
Each virtual web site you want to track stats for should have a file /etc/awstats.sitename.conf the directives for the configuration file can be found here: http://awstats.sourceforge.net/docs/awstats_config.html they also provide a default conf file in cgi-bin/awstats.model.conf you can use this as a base.
Make sure your log files are using NCSA combined format, this is usually done in apache by saying CustomLog /logs/access.log combined you can use other formats but you have to customize the conf file.
You will probably want to edit the LogFile directive to point to where your logfile is stored, SiteDomain this is the main domain for the site, HostAliases lets you put in other domains for the site, and the DirData directive lets you specify where the awstats databases will be stored (each site will have its own file in the directory).
Once that is setup you will want to update the database this is done from the command line by running
perl awstats.pl –config=sitename –updateNow copy everything in the wwwroot folder to a web root, and visit http://sitename.com/cgi-bin/awstats.pl if you want to view other domains use /cgi-bin/awstats.pl?config=othersitename
Where sitename would be the name of your config file awstats.sitename.conf
If you want to generate static html files run the
awstats_buildstaticpages.plscript found in the tools folder. You have to give it the path to theawstats.plperl script, and a directory to put the static html files in.perl awstats_buildstaticpages.pl -config=sitename -awstatsprog=/web/cgi-bin/awstats.pl -dir=/web/stats/sitename/More setup info can be found here: http://awstats.sourceforge.net/docs/index.html
AWStats is a free powerful and featureful tool that generates advanced web, ftp or mail server statistics, graphically. This log analyzer works as a CGI or from command line and shows you all possible information your log contains, in few graphical web pages. It uses a partial information file to be able to process large log files, often and quickly. It can analyze log files from IIS (W3C log format), Apache log files (NCSA combined/XLF/ELF log format or common/CLF log format), WebStar and most of all web, proxy, wap, streaming servers, mail servers (and some ftp).
Take a look at this comparison table for an idea on differences between most famous statistics tools (AWStats, Analog, Webalizer,...).
AWStats is a free software distributed under the GNU General Public License. You can have a look at this license chart to know what you can/can't do.
As AWStats works from the command line but also as a CGI, it can work with major web hosting provider that allows CGI and log access.
You can browse AWStats demo (Real-time feature to update stats from web has been disabled on demos) to see a sample of most important information AWStats shows you...
Squid2MySQL is an accounting system for squid. It includes monthly, daily, and timed detail levels, and uses MySQL for log storage.
Eugene V. Chernyshev <evc (at) chat (dot) ru> [contact developer]
I'm looking for access log analysis software that will run independently from the ACS. Hopefully the thing would read an AOLserver or Apache access log and generate some graphs and tables in a configurable way. Any specific suggestions? The goal is usage analysis for marketing purposes (i.e., not performance analysis). Probably running on Solaris and it doesn't have to be freeware.
-- S. Y., August 29, 2000
I've been using webalizer (http://www.webalizer.com). It's real easy to set up and configure. For a sample of what it does, check out http://www.badgertronics.com/reports
-- Mark Dalrymple, August 29, 2000And if you don't mind paying, NetTracker (from http://www.sane.com) has a ton of reports, handles big log files (a site we know that's using it has 600 meg daily access logs), handles cobranding, etc
-- Mark Dalrymple, August 29, 2000Nettracker rules! We've been using it for a couple of years now and everyone loves it. It has no knowledge of the ACS or it's users, of coures, so real clickstream tracking isn't really possible, but for basic log analysis with lots of information it's great.
-- Janine Sisk, August 29, 2000
I worked quite a while with NetAnalysis by the Boston-based company NetGen. It's database-backed log file analysis software, i .e. usually once per day you stuff your log file information into your db. This way you can keep track of your homepage's success over time. It also enables you to connect your webserver log information with information from other databases and come up with really powerful information. For example it has pretty damn good adaptors to Intershop and to Vignette StoryServer and you can implement your own adaptors with a Perl-based (soon to be Java) API. (Why do I know these adaptors are good? I made the one for Intershop work properly and I specified the one for my former company's Vignette StoryServer standard installation ;)
The software to run analysis on your log is very smartly configurable and you have good metrics and I was promised some really powerful metrics for a future release (that should be out as of now).
It runs on Solaris and you'll need Oracle and a pretty heavy machine (way bigger than your webserver usually).
Other products I know about are Accrue (used to be based on sniffing TCP/IP packets) and iDecide by Informix (has another name these days) that are db-backed as well. Oh, and then I once met with a Canadian company that use JavaScript to spy out some information about the client. It was probably the most expensive software regarding costs/line of code - tho they had very, very, very good metrics. You can head over to playboy.com and look for a JS call on their entry site ;)
regards Dirk
-- Dirk Gómez, August 30, 2000
I have had great success with 'analog' a free utility for log parsing and analysis. It can generate simple reports and also machine readable reports (good for DB backed summary generation). There is also a perl based package called 'Report Magic' which takes the machine readable output and makes pretty pictures and tables for you.
The problem with many Log analysis tools is that they cannot be consumate and definately cannot reflect the structure of your particular site especially if it has 'interesting' mechanisms by which content is rendered - frames, multiple urls for a given page etc. Also, when you get ~ 1GB of log data a day, some cjust croak big time. In this case, rolling-your-own is one solution whih I have been implementing (using perl -regexp is you friend- and DB) which follows some of the principles discussed at ASJ/. It also allows for simple reporting on only the bits n pieces I feel necessary - quick with HUGE sets of data.
This is a good technique for allowing ad-hoc queries on data but make sure you have a reasonably hefty machine (Dual Xeon with 2GB RAM, 100GB RAID).
I guess if there is a package out there which is extensible and modular (plugin support as mentioned above) then that may also be quite useful.
It's important to define exactly what info is most important in analysing logs else you can open up a huge can of worms - ie. mapping all relationships between everything which whilst interesting, is quite difficult and reasonalby useless unless it is quick and can add value in some way.
Sorry if I've stated the obvious, just my $0.05
-- geoff webb, September 4, 2000
Look carefully before you choose NetTracker. It scans the log files and saves the data into its own flat file database. If you ever lose files or decide that you want to add another report, it needs to regenerate all of that data. For a month's worth of logs (the 600 MB+ logs referred to by Mark Dalrymple above), it can take 3+ days.
-- Doug Harris, September 11, 2000
page-stats.pl will examine the acceslog of a http daemon and search it for occurrences of certain references. These references are then counted and put into a HTML file that is ready to be displayed to the outside world as a "Page Statistics" page. Each page can be selected from the statistics page.
Big Brother Log Analyzer, or BBLA for short, is a package comprised of two components: a logger, which logs all accesses to selected web pages, and a log analyzer, which nicely formats the logs into an HTML page. The generated HTML is fully W3C compliant (HTML 4.01/Transitional), which guarantees that it will be rendered the way it should under any compliant browser. Another interesting feature of BBLA is that it is tag-based (you put a tag in each page you want to track): this allows for tracking pages hosted on different servers. For instance, I track accesses to my pages in the School of Information Management and Systems at the University of California, Berkeley, along with these pages hosted on SourceForge in a single file. See the demo for more information.
A lot of HTML log analyzers exist on the market, but most of them are either targeted at systems administrators (with full access to httpd log files, for instance), or require general users to display an advertising banner on their pages, or (even worse) limit the number of pages you can track for free. Most of the time, the pages generated do not even follow the W3C consortium recommendations for writing proper HTML, yielding unpredictable results when viewed with different browsers.
BBLA is free, doesn't require you to have a banner on your web page, uses W3C-compliant HTML and PNG images (hence, no licensing issues with GIFs), allows for tracking pages hosted on different servers, and is actually completely transparent. So, unless your visitors look into your HTML source, they won't notice that you are tracking them.
Last but not least, BBLA is extremely light-weight: the current tarball is roughly 30KB, making it much more easier to install on platforms with scarce disk space than some of its counterparts. As an added bonus, it doesn't take ages to compile, even on really ancient hardware.
Relax is a multi-platform Web server log analyzer written in Perl. It can be used to track which search engines, search keywords, and referring URLs led visitors to the Web site. It can also track down bad links and analyze which keywords to bid for at pay-per-click search engines. The parser module in Relax recognizes several hundred search engines and is capable of extracting the keywords used. Generated HTML reports can be configured to include links to other Web-based keyword analysis tools, making it easier to further improve the ranking of web pages in search engines.
proxy-report.pl generates a list of requested server addresses (simplified URLs) from your Squid proxy server log files. Requests for each URL are summarized on a per day basis. This script can generate reports based on the IP of the user. It also automatically handles gzipped files. URL exclusion patterns are supported. A sample report is available on the home page.
parses logfiles from Squid, NetCache, Inktomi Traffic Server, Oops! proxy server, Novell Internet Caching System, Compaq Tasksmart or Netscape/iplanet Web Proxy Server and generates a report. Written in perl5.
About: Calamaris parses the logfiles of a wide variety of Web proxy servers and generates reports about peak-usage, request-methods, status-report of incoming and outgoing requests, second and top-level destinations, content-types, and performance.
Changes: There are bugfixes regarding %-URLs and syntactically wrong HTML output. A function to use Calamaris' output in server side includes has been added.
"In this column I'll give you a gentle introduction to Apache web server logs and their place in monitoring, security, marketing, and feedback."stoic Freeware web server log analysis tool Jul 05th 1999, 05:52
stable: 1.2 - devel: none license: FreewareStoic is a small Perl script that examines Apache or Netscape web server access logs. Reports include logfile totals, domains visiting, top documents requested, browser agent statistics, platform statistics and a bunch of other stuff.
Download: ftp://ftp.mrunix.net/pub/webalizer/ Alternate Download: ftp://samhain.unix.cslab.tuwien.ac.at/webalizer/ Homepage: http://www.mrunix.net/webalizer/ Changelog: ftp://ftp.mrunix.net/pub/webalizer/CHANGES The Webalizer is a web server log analysis program. It is designed to scan web server log files in various formats and produce usage statistics in HTML format for viewing through a browser. Very good output, good charts and very fast. Just missing special mode for a total statistics (all virtual webservers) without that many details.
freshmeat.net Browse project tree - Topic Internet Log Analysis
Google Directory - Computers Software Internet Site Management Log Analysis
Produces highly detailed, easily configurable, incremental HTML usage reports in many languages, from multiple log formats, with builds for Linux, Solaris, Mac, OS/2, Cobalt, OpenVMS, Netware, and BeOS. [Open Source, GPL]
Computers: Software: Internet: Site Management: Log Analysis: Commercial
(20
matches)
Computers: Software: Internet: Site Management: Log Analysis: Freeware and Open
Source (5)
SunWorld Online - March - Webmaster
SunWorld Online - May - Webmaster -- Analyzing your referrer log
The PIPER Letter Making Sense of Web Usage Statistics
logmanage is a program which is designed to perform flexible management of web statistics for a variety of users and main server logs. In its current configuration it is designed to work with http-analyze but should work with any web stats program that takes log input on STDIN and can be configured for the output directory on the command line. Manages a large collection of pipes to the stats program with inclusion and exclusion regular expressions. Can generate stats for lots of different users from one log file or from many log files.
Recycle-logs is a logfile manager written in Perl that attempts to overcome the limitations of other system log utilities. File rotation and other customization is based on control information specified in one or several configuration files.
W3Perl is a Web logfile analyzer. All major Web stats are available (referrer, agent, session, error, etc.). Reports are fully customizable via configuration files, and there is an administration interface control available.
AWStats - Free log file analyzer for advanced statistics (GNU GPL). -- extremely flexible and robust Perl=written log analyser. Industrial strength ! Highly recommended..
AWStats is a free powerful and featureful tool that generates advanced web, ftp or mail server statistics, graphically. This log analyzer works as a CGI or from command line and shows you all possible information your log contains, in few graphical web pages. It uses a partial information file to be able to process large log files, often and quickly. It can analyze log files from IIS (W3C log format), Apache log files (NCSA combined/XLF/ELF log format or common/CLF log format), WebStar and most of all web, proxy, wap, streaming servers, mail servers (and some ftp).
Take a look at this comparison table for an idea on differences between most famous statistics tools (AWStats, Analog, Webalizer,...).
AWStats is a free software distributed under the GNU General Public License. You can have a look at this license chart to know what you can/can't do.
As AWStats works from the command line but also as a CGI, it can work with major web hosting provider that allows CGI and log access.
You can browse AWStats demo (Real-time feature to update stats from web has been disabled on demos) to see a sample of most important information AWStats shows you...
A good and free log analysis tool - awstats
There are a lot of free http log file analysis tools out there that haven't been updated since the mid 90's, awstats however is both free, and up to date. It looks a bit like web trends (though I haven't used web trends in several years). Here's an online demo. awstats can be used on several web servers including IIS, and Apache. You can either have generate static html files, or run with a perl script in the cgi-bin.
Here's a quick rundown of setting it up on unix/apache
Each virtual web site you want to track stats for should have a file /etc/awstats.sitename.conf the directives for the configuration file can be found here: http://awstats.sourceforge.net/docs/awstats_config.html they also provide a default conf file in cgi-bin/awstats.model.conf you can use this as a base.
Make sure your log files are using NCSA combined format, this is usually done in apache by saying CustomLog /logs/access.log combined you can use other formats but you have to customize the conf file.
You will probably want to edit the LogFile directive to point to where your logfile is stored, SiteDomain this is the main domain for the site, HostAliases lets you put in other domains for the site, and the DirData directive lets you specify where the awstats databases will be stored (each site will have its own file in the directory).
Once that is setup you will want to update the database this is done from the command line by running
perl awstats.pl –config=sitename –updateNow copy everything in the wwwroot folder to a web root, and visit http://sitename.com/cgi-bin/awstats.pl if you want to view other domains use /cgi-bin/awstats.pl?config=othersitename
Where sitename would be the name of your config file awstats.sitename.conf
If you want to generate static html files run the
awstats_buildstaticpages.plscript found in the tools folder. You have to give it the path to theawstats.plperl script, and a directory to put the static html files in.perl awstats_buildstaticpages.pl -config=sitename -awstatsprog=/web/cgi-bin/awstats.pl -dir=/web/stats/sitename/More setup info can be found here: http://awstats.sourceforge.net/docs/index.html
Webalizer The Webalizer is a fast, free web server log file analysis program. It produces highly detailed, easily configurable usage reports in HTML format, for easy viewing with a standard web browser.
The wwwstat program will process a sequence of HTTPd common logfile format (CLF) access_log files and output a log summary in HTML format suitable for publishing on a website.
The splitlog program will process a sequence of CLF (or CLF with a prefix) access_log files and split the entries into separate files according to the requested URL and/or vhost prefix.
Both programs are written in Perl and, once customized for your site, should work on any UNIX-based system with Perl 4.036, 5.002, or better.
Qiegang Long, formerly at UMass, has released a program called gwstat that takes the output from wwwstat and generates a set of graphs to illustrate your httpd server traffic by hour, day, week or calling country/domain.
A mailing list, now shut down, was created for discussion and support of wwwstat development.
Log Scanner was written to watch for anomalies in log files. Upon finding them, it can notify you in a variety of ways. It was designed to be very modular and configurable. Unlike most other log scanners, this one has more than single pattern matches. It will allow you to trigger notifications on multiple occurrences of one or several events.
The problem with many Log analysis tools is that they cannot be consumate and definately cannot reflect the structure of your particular site especially if it has 'interesting' mechanisms by which content is rendered - frames, multiple urls for a given page etc. Also, when you get ~ 1GB of log data a day, some cjust croak big time. In this case, rolling-your-own is one solution whih I have been implementing (using perl -regexp is you friend- and DB) which follows some of the principles discussed at ASJ/. It also allows for simple reporting on only the bits n pieces I feel necessary - quick with HUGE sets of data.
| FlashStats | Mac Unix Windows |
$99/$249 | Maximized Software | |
| HTTP-Analyze | Unix Windows |
free/326 euro/388 euro/1470 euro | http://www.http-analyze.org/ | |
| Lumberjack | Unix | $1250 | BitWrench Inc | |
| NetTracker | Unix, Windows | from $495 | Sane Solutions | |
| WebTrends | Windows Solaris Red Hat |
$499 - $1999 | NetIQ | |
| Sawmill | Mac Unix Windows |
$999 Substantial discounts for edu & small organisations | sawmill.net |
Web Trends -- actually a pretty limited commercial package. Decent prepackaged reporting capabilities, but not very flexible (reports for dummies)
EasyLog version 1.2
This is a simple Server Side Includes script that can "watch" a given page,
and add entries to a HTML file reporting what browser they are using, when they
accessed your page, and a few other pieces of information. Language: Perl
Platform: Unix
View Product Homepage
Checklog version 1.0
Perl script that analyzes HTTP server logs.
Language: Perl Platform: Unix, Windows
Download Complete Source Code, 0.010M bytes
Click file name to view online:
checklog.pl,
14504 bytes
FTPWebLog version 1.0.3
Perl script that analyses WWW and FTP logs and produces graphical reports.
Language: Perl Platform: Unix, Windows
Download Complete Source Code, 0.096M bytes
Relax version 2.0
Relax is a free reference log analysis program written in Perl, which can
be used to analyse how people are finding your web site, what keywords they use
in the search engines, and how they move within the site.
Language: Perl Platform: Unix, Windows
Download Complete Source Code, 0.010M bytes
Log Reverse Domain Name System (lrdns) version 1.1 -- Converts numeric IP addresses in accesss log files into textual domain names. Language: Perl Platform: Unix
Download Complete Source Code, 0.010M bytes
See also
Copyright © 1996-2009 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. Submit comments This document is an industrial compilation designed and created exclusively for educational use and is placed under the copyright of the Open Content License(OPL). Site uses AdSense so you need to be aware of Google privacy policy. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
Disclaimer:
Last modified: August 15, 2009