May the source be with you,
but remember the KISS principle ;-)
Key Softpanorama Topics
|About||Contents||Top Updates||Top Visited|
|Bulletin||Selected Papers||Softpanorama Bookshelf||History|
|News||See also||Recommended Books||Recommended Links||Tutorials||Course Descriptions and Outlines||Papers|
|Debugging||Typical errors||Typical Errors in C||Notorious Bugs||Tools||Expect||DejaGnu|
|Test automation||Web Testing||Application testing||Sysadmin Horror Stories||Humor||Debugging Humor||Random Findings||Etc|
To succeed in the world, it is much more necessary to possess the penetration to discern who is a fool, than to discover who is a clever man.
Charles-Maurice de Talleyrand
As software is becoming more and more complex and budgets are more and more limited the role of automatic testing increases. Moreover development of the test suit should be part of software development specification and initial test suit should be developed at least partially by software developers. Often that stimulated redesign that made software more "testing-friendly". Given the current economic times and having to do more with less regression testing cycles should be by and large automated and developers should be presses to do their best to make use of authentic tools simpler and making the spectrum of testing with such tools more comprehensive. . Debugging and testing can easily consume lion share of the total development costs.
Testing is time-consuming, about intensive area even with maximum use of automatic tools. The situation can be made even worse with the adoption of a fashionable but questionable methodology. Excessive zeal here hurts.
From my point of view general purpose scripting languages are often better testing tools than expensive testing suits (see Expect/TCL, DejaGnu, Perl, etc.). I would like to warn against testing tools with their own (and often inferior) specialized mini-scripting languages. In his column Hey Vendors, Give Us Real Scripting Languages Bret Pettichord aptly wrote:
Most test tools come bundled with vendor-specific scripting languages that I call vendorscripts. They are hard to learn, weakly implemented, and most important, they discourage collaboration between testers and developers. Testers deserve full-featured, standardized languages for their test development.
For most of the test tools available today, you have to write test scripts in the tool’s scripting language. What drives me crazy is that most vendors have their own special language that you must learn in order to use their tool. They often have recorders to help you generate code, but invariably you need to go in and make changes and write code yourself. And you end up having to learn another language—another vendorscript—another dialect that is good for nothing but the one tool.
Because a vendorscript is a specialized language, developers are less likely to know it and little inclined to learn it. I frequently counsel testers and developers to work together on test automation projects. There are many reasons why this is a productive collaboration, but vendorscripts get in the way. They split testers from developers and from each other into tool-specific language isolation. This reduces the opportunities to share, collaborate, and improve the craft.
We have enough trouble bringing developers and testers together in the testing process. The last thing we need is an inherent obstacle from the get-go in our testing tools. Ask a developer to learn Visual Basic? No problem. They can always use that knowledge in the future. Ask a developer to learn a specialized language unique to one tool? You're dreaming. You may already be having trouble getting the developer to pay attention to testing, and now you’re asking for more.
Why spend money on a tool that’s going to undermine the collaboration efforts between testers and developers?
I would to reiterate the major point:
Testers deserve full-featured, standardized scripting language for their test development
Building "testability" into application is equaly important. Among the simplest requirements that increase testability are:
Dr. Nikolai Bezroukov
|Bulletin||Latest||Past week||Past month||
About: Tcpreplay is a set of Unix tools which allows the editing and replaying of captured network traffic in pcap (tcpdump) format. It can be used to test a variety of passive and inline network devices, including IPS's, UTM's, routers, firewalls, and NIDS.
Changes: This release dramatically improves packet timing, introduces full fragroute support in tcprewrite, and improves Windows/Cygwin and FreeBSD support. Additionally, a number of smaller enhancements have been made and user discovered bugs have been resolved. All users are strongly encouraged to update.
About: Plash is a sandbox for running GNU/Linux programs with minimum privileges. It is suitable for running both command line and GUI programs. It can dynamically grant Gtk-based GUI applications access rights to individual files that you want to open or edit. This happens transparently through the Open/Save file chooser dialog box, by replacing GtkFileChooserDialog. Plash virtualizes the file namespace and provides per-process/per-sandbox namespaces. It can grant processes read-only or read-write access to specific files and directories, mapped at any point in the filesystem namespace. It does not require modifications to the Linux kernel.
Changes: The build system for PlashGlibc has been changed to integrate better with glibc's normal build process. As a result, it is easier to build Plash on architectures other than i386, and this is the first release to support AMD-64. The forwarding of stdin/stdout/stderr that was introduced in the previous release caused a number of bugs that should now be fixed.
I have a friend who is new to unit tests and therefore has never done Test-Drive Development. His management has decided that developers should now write unit tests. But there has been some debate about what should be tested. Since his "application" is really a class library, a lot of the functionality is hidden inside the library (the public API is just a small part).
Some of the people he talked to suggested not writing unit tests for the implementation (just test the public APIs) because they were nervous about making the library "brittle". As my friend said, "they define brittle as in it's really easy for someone to make a change and break stuff."
The comment about brittle unit tests is actually very interesting. I’ve encountered that myself. It’s frustrating when I have to go back and update unit tests after doing a large refactoring. It certainly makes the unit tests “feel” brittle. But I think this feeling is a result of changing my approach to development rather than a real issue. I think it can be broken into two cases:
- Unit tests fail, showing bugs in my code. This is obviously the “good” case.
- Unit tests fail because assumptions have changed. This makes unit test feel brittle. But actually it’s a good thing. They fail because my assumptions or “specifications” have changed. That means I really do need to rewrite the tests to reflect the new specifications that resulted from design changes.
There is another way to look at item 2 above. Often the broken unit tests are a result of bugs in the specifications, so what you're really faced with is having to interrupt what you were working on to fix bugs. This can certainly be unsettling, because it's a brick wall you have to scale before you can continue with your thread of thought. However, since you're fixing bugs as you go, your code will be much more reliable.
Web Performance Suite 3.4 Update 4
by Michael Czeiszperger - Wed, Jul 25th 2007 11:01 PDT
About: The Web Performance Suite is a suite of testing tools for Web servers, including load testing, Web page performance analysis, and operating system tuning. The statistical analysis in the Load Testing Module takes your site's performance criteria, and determines the number of users your Web site can handle. Because data is collected at the URL level, it not only identifies slow Web pages, but identifies the particular part of the Web page that caused the problem. It includes hard-to-find features such as IP spoofing, client certificates, multiple simultaneous test cases, authentication, SSL recording and playback, automated analysis, AJAX support, SOAP support, and more.
Changes: This release adds new modules to automatically detect performance problems in servers running either Windows or Linux. Up to 23 different OS parameters are tracked and then evaluated for performance issues.
Learn about test tools worth thousands of dollars, all available to you for free! Open Testware Reviews is a source of information about freely available test tools. Whether your motivation is a limited budget at work, a desire to learn new kinds of tools without hassling with tool vendors, or having access to the source code for your tools, Open Testware Reviews will help you select the most useful of the hundreds of available freeware test tools.
Open Testware Reviews is closed to new subscribers. For information about buying a hard copy of all past content, contact Danny Faught.
Free content - Some feature articles and blotter entries are now available for free.
The scope of Open Testware Reviews is freely available tools that are useful for software testers and developers, focusing on open source tools. Content includes detailed reviews of specific tools, high-level surveys of test tool categories, and brief tool writeups. In the style of the Tejas Software Consulting Newsletter, subscribers are encouraged to send feedback to be published, and also to suggest topics for future issues.
If you are not satisfied within the first three months of your subscription for any reason, you may cancel and receive a full refund, or after three months you may receive a refund on a pro-rated basis.
Open Testware Reviews is registered with the US Library of Congress. ISSN: 1545-8857.
If testers start looking at developers as vital resource of information, there is a lot better testing that can happen over that place. Rapid Software Testing says, "As a tester be a service and not an obstacle to the project" and isn't that amazing? One dimension to being an obstacle to the project is about having a bad relationship with developers.
It is not just a tester-developer relationship that goes bad in an organization. Speaking from my experience, a developer-developer relationship has gone bad and a tester-tester relationship is no exception but the highlight of all is tester-developer. I infer from the results of a bad, developer-tester relationship that the resultant was more expensive than a developer-developer or tester-tester relationship going bad.
The ownership problem
It appears to me that if you ask a developer and a tester, a question of who contributed to the relationship going bad, a testers finger would point at the developer and the developer would be a victim of Newton's 3rd law. It is a rare occasion that a tester or a developer would point his finger at himself. Sometimes, the developer or a tester is not aware that management is also a reason behind it.
In India, I have heard so many stories of management being a major contributing factor to the developer-tester relationship going bad. I must admit that there are certain companies in India who encourage a tester developer relationship but I am not sure where they are.
Fear of university
I am happy of having done this detail research in which I explored the mindset of a tester when he works for a developer from a better university than where he graduated from. In India IIT and IISC are considered to be the most fundoo universities. Start ups, usually hire fresh graduates from IIT and IISC for development and conduct a walk in for testers and pay them 1/10 th of what an IIT Developer is paid. Some testers start feeling inferior and their confidence at the start of the career goes down. They turn out to look for development jobs.
Here is an interesting story: In one of the company I worked for, I was one tester for 3 developers who were from IIT and IISC. They welcomed me with a look as if , "Is this guy going to find bugs in my code? ha" and it took me little time to understand that they too were humans who could make mistakes. After a couple of months, the three developers from top universities called me, "Crash specialist" and I enjoyed a good relationship with them. There was another tester on a similar project in the same company working for 3 developers from premier organizations I mentioned. That tester always exhibited discomfort working with those three as he kept feeling inferior. He quit the company as he could not live with inferiority complex for a long time.
Now, I know the secret of killing inferiority complex when it is caused by someone who claims to be the best programmer on earth. As you are my blog reader, I want to share the secret with you but keep it within yourself. The secret is ...FIND BUGS IN HIS CODE!
Self assumed slave!
I am not sure why such an assumption enters a testers mind but I have seen some of my co-workers running to a developers seat when a developer calls the tester over his extension and say, "Can you come to my seat?". A self assumed slave tester runs towards a developers cubicle dropping the test he was doing and also without asking the reason for the developer calling him.
Here is another story: In one of the companies I worked for, I saw a tester who looked to my eyes as sitting idle and I *requested* him to join me for the test content generation so that I could get better ideas. I said, "Hey let's generate this file as a test input" and started doing it. The tester interrupted, "Pradeep, can I ask the developer and come whether we can test with this clip?". Jeez! why should he ask a developer whether we testers can use a clip for testing? On probing him, I got to know that he was once scolded by a developer for doing a test without informing him. That's funny!
Most of the developers I have worked with never hesitate to ask for information from testers when they need it and I admire them for that. On a contrary some testers who were co-workers to me have feared to ask for information from the developers. This class of testers feel that developers might consider the questions as *silly*. What this class of tester is not aware is, questioning needs to be his tool to solve any testing problems he has in hand. If you aren't questioning, you aren't testing!
Finding bugs to bug a developer
Some testers are eager to find more bugs to keep the developer busy in case the relationship is in a bad state. Such a tester feels great after doing so and he doesn't realize that he isn't concentrating on learning a product while testing it. He keeps thinking, the crash and the hang that crashes a developers weekend plans.
Another dimension is, the severity and priority of the bug found is for no reason kept high by a tester to bug the developer and only a chain of mails flow to bring the levels down.
The foreign trip trauma
Some developers who come back from an on site assignment behave with testers in a way that makes the tester feel deprived of certain things. Are you thinking I would say, "This is a developers mistake?"
Nah! It is still a mistake of a tester who fails to recognize, being in India he is in a fastest growing economy of the world and the most watched nation of the world.
The jargon menace
Developers seem to talk jargons that a tester might not be aware of. Under some situations, a tester gets to feel inferior but the tester does not realize the fact that he need not know all the jargons the developer knows. A tester ideally should know a product better. In my experience, I worked with a developer who was writing code for an MP3 application whose jargons and lingo I did not understand but came up to me once, to ask in a soft voice; can I know how to do a Fast Forward in this application?.
Do you now think, I am trying to say a tester is better than a developer in this context?
Look at them, they do come out and admit mistakes/ask for information in front of a tester!
Testers need to realize that it is not just by talking and knowing the jargons of a developer helps him become a better tester but *skills* do make them better tester.
March 2005 (DDJ) Language features like closures and native syntax for list and maps mean less typing and code that's easier to read
David Black, Unit testing is a Groovy Thing DDJ March 2005 p 12
PagePoker is a Perl package that defines a browser agent with many powerful features for monitoring and testing Web sites, including elaborate failure handling that can send email and trigger SNMP traps. It comes with three scripts that implement it for different uses: poke.pl, for single agents; pokes.pl, for many parallel agents, and pokehard.pl, for loadtesting and benchmarking.
The meditation Perl in the Microsoft Research Center seems to have made me more sensitive to noticing mentions of "Microsoft" and "Perl" in close proximity.
My latest delightful discovery is that the C# compiler Test team uses a perl script to execute and validate their test suite (more than 18,000 dictinct tests) several times every day. This is particularly apposite for me as our test processes are very much in the spotlight right now where I work. If only we could ditch the GUI </wry smirk>
No surprise, just a pleasant resonance.
Here is a quote from the web log of Gus Perez, the C# Compiler Test Team Lead:
Our test harness (tool we use to execute and validate results of each test) is a Perl script developed several years ago, before C#, in the C++ team (which is where the C# team was born before we had our own product unit) that has proven to a lot of us that simple is often better when it comes to test harnesses. Every now and then we get the urge to start working on a new, all-powerful harness (of course written in C#), but somehow we always end up holding off on that adventure as what we have has demonstrated it's capable time and again and there's always other work that needs to get done.
PureTest is an application which is primarily used to setup scenarios of tasks, execute and debug them. Even though it supports testing a variety of applications it is especially useful for debugging and snooping of web applications. PureTest includes a HTTP Recorder and Web Crawler which makes it useful for generic verification of HTTP requests and web content checking.
The normal way to access web sites is via a browser; however, there are times when it is desirable to bypass the browser and access a site from a program, including:
- Debugging of HTTP requests and responses
- Automated web site testing
The HTTP Recorder simplifies the process of capturing all requests that are exchanged between a browser and the web server. Then use PureTest to replay each request in order to carefully watch the HTTP data that is transferred on the wire (HTTP headers, request parameters, response headers and response content). The Web Crawler is useful to pro-actively verify the consistence of a static web structure. It reports various metrics, broken links and the structure of the crawled web.
Test scenarios that be saved to file and later be repeated, to verify that you server applictaion works as expected. This can be done using the PureTest debugger in the grapical user interface, but also using a command line interface.
View the full feature table.
About: stress is a simple tool to impose certain types of stress on a POSIX system, including CPU load, I/O subsystem load, RAM load, and HDD load.
Changes: A --loadavg option has been added which allows the user to bring the system load average up to an arbitrary value. Use with caution.
INSTALL 1. Uncompress and untar the archive. 2. Compile by invoking make. 3. Move stress to a directory in your path. USAGE The following is the usage statement printed by invoking stress with no arguments. % stress stress $Revision: 1.13 $ (firstname.lastname@example.org) usage: stress [flag [arguments]] flags: --hogio [n] (make n sync(2) calls) --hogcpu [n] (make n sqrt(3) calls) --hogmemory [n s] (malloc(3) n pages of s bytes) --hogdisk [n f] (fputs(3) n bytes to file f NOTES This program works really well for me, but it might not have some of the features that you want. If you would like, please extend the code and send me the patch. Enjoy the program :-) Amos Waterland Norman, Oklahoma 27 NOV 2001
This suite of tools was used in my papers More than a Gigabuck: Estimating GNU/Linux's Size and Estimating Linux's Size to measure the SLOC of entire GNU/Linux distributions. Others have measured Debian GNU/Linux using this tool suite. SLOCCount runs on GNU/Linux, FreeBSD, Windows, and hopefully on other systems too. To run on Windows, you have to install Cygwin first to create a Unix-like environment for SLOCCount (Cygwin users: be sure to use ``Unix'' newlines, not ``DOS'' newlines, when you install Cygwin).
SLOCCount has a number of ease-of-use features. You can easily install it, particularly on RPM-based GNU/Linux systems. For most situations, once it's installed all you need to do is type this to measure all the code in a given directory (including its descendants):sloccount directoryname
SLOCCount is released under the General Public License (GPL), and can measure the following languages (common extensions for the language are listed in parentheses):
- Ada (.ada, .ads, .adb)
- Assembly (.s, .S, .asm)
- awk (.awk)
- Bourne shell and variants (.sh)
- C (.c)
- C++ (.C, .cpp, .cxx, .cc)
- C shell (.csh)
- COBOL (.cob, .cbl) as of version 2.10
- C# (.cs) as of version 2.11
- Expect (.exp)
- Fortran (.f)
- Haskell (.hs) as of version 2.11
- Java (.java)
- lex/flex (.l)
- LISP/Scheme (.el, .scm, .lsp, .jl)
- Makefile (makefile) - not normally shown.
- Modula-3 (.m3, .i3) as of version 2.07
- Objective-C (.m)
- Pascal (.p)
- Perl (.pl, .pm, .perl)
- PHP (.php, .php, .inc) as of version 2.05
- Python (.py)
- Ruby (.rb) as of version 2.09
- sed (.sed)
- SQL (.sql) - not normally shown.
- TCL (.tcl, .tk, .itk)
- Yacc/Bison (.y)
SLOCCount includes a number of heuristics, so it can automatically detect file types, even those that don't use the ``standard'' extensions, and conversely, it can detect many kinds of files that have a standard extension but aren't really of that type. The SLOC counters have enough smarts to handle oddities of several languages. For example, SLOCCount examines assembly language files, determines the comment scheme, and then correctly counts the lines automatically. It also correctly handles language constructs that are often mishandled by other tools, such as Python's constant strings when used as comments and Perl's "perlpod" documentation.
SLOCCount comes with extensive documentation - you should be able to just pick it up and use it.
Through a four-phase approach, the author shows why eliminating bugs is tricky and why testing is a constant trade-off.
This release features much better handling of SMTP errors in the internal mailer.
The Test Environment Toolkit (TET), is a multi-platform uniform test scaffold, into which non-distributed and distributed test suites can be incorporated. Its source is freely available under the Artistic License. TET supports tests written in C, C++, Perl, Tcl, Shell (sh , bash), and Korn Shell.
This is a maintenance release that now includes support and portability fixes for Solaris 7.
(developerWorks) "Function testing is the phase during a development cycle in which the software application is tested to ensure that the functionality is working as desired and that any errors in the code are properly handled. It is usually done after the unit testing of individual modules, and before a more thorough system test of the entire product under load/stress conditions."
"There are many testing tools in the marketplace that offer a lot of functionality to help with the testing efforts. However, they need to be obtained, installed, and configured, which could take up valuable time and effort. Bash can help to speed things along."
Mr. Cluey's Kludge Page reincarnation on the WEB (new home).
Thanks "Keith Zambelich" <email@example.com> for the information)
Testing vs. QA
Pattern Languages (a lot of snake oil here ;-)
Some ideas on how to organize software development.
DejaGnu is a popular Expect-based framework for testing other programs. The latest version is 1.4.3. It is used as a testing framework by the GCC, binutils, and gdb projects. Its purpose is to provide a single front end for all tests. If you are starting out and feel overwhelmed by the capabilities of Expect or would just like some guidance on how to structure a test suite, check out DejaGnu.
Think of it as a custom library of Tcl procedures crafted to support writing a test harness. A Test Harness is the testing infrastructure that is created to support a specific program or tool. Each program can have multiple testsuites, all supported by a single test harness. DejaGnu is written in Expect, which in turn uses Tcl -- Tool command language. There is more information on Tcl at the Tcl web site on SourceForge, and the Expect web site is at NIST.
DejaGnu is used by many standards testing organizations.
DejaGnu has an excellent documentation.
You may also find more information about DejaGnu by looking at /usr/doc/dejagnu/ or /usr/local/doc/dejagnu/ or by looking at man pages (man runtest at the shell prompt) on your system.
Remote Testing with Dejagnu
The newsgroup comp.software.testing is main newsgroup on the subject. This is a place where testers and testing managers go to for advice when they get squeezed particularly hard. Many experienced and helpful software testers frequent this newsgroup.
The moderated newsgroup comp.risks contains descriptions of software failures, especially when loss of life is involved. This is an excellent place to learn about the kinds of systemic problems that plague software.
- The Voyager bug (sent the probe into the sun).
- The AT&T bug that took out 1/3 of US telephones.
- The DCS bug that took out the other 1/3 a few months later.
- The Intel Pentium chip bug (it was software, not hardware).
- The Ariane V bug.
In a later letter, he agreed that the Therac 25 bug was also notorious, but not really a unit bug.
Careers in Software Testing
Other Commercial Sites
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : C++ Humor : ARE YOU A BBS ADDICT? : Object oriented programmers of all nations : C Humor : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor: Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : The Most Comprehensive Collection of Editor-related Humor : Microsoft plans to buy Catholic Church : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor : Best Russian Programmer Humor : Russian Musical Humor : The Perl Purity Test : Politically Incorrect Humor : GPL-related Humor : OFM Humor : IDS Humor : Real Programmers Humor : Scripting Humor : Web Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor :
The Last but not Least
|You can use PayPal to make a contribution, supporting hosting of this site with different providers to distribute and speed up access. Currently there are two functional mirrors: softpanorama.info (the fastest) and softpanorama.net.|
The statements, views and opinions presented on this web page are those of the author and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.
Last modified: July, 07, 2013