Software Testing in the Product

¶ … Software Testing in the Product Life Cycle

THE PLACE of SOFTWARE TESTING in the PRODUCT LIFECYCLE

Definitions & Terms in the Research

Alpha Test - test phase of the PLC where code is complete and product is testable and stable.

Automated Testing - individual testing without direct tester involvement.

Beta Test - portion of test phase of PLC where testing for integration is completed.

Black Box Test - test in which the tester cannot see into the software

Code Complete - phase of the PLC where coding of functionality is completed

Coding Phase - phase of the PLC where coding is accomplished for specs of function and architecture.

Compatibility Test - testing for compatibility of software being developed with other software and hardware.

Concept Phase - phase of the PLC where the product 'idea' is developed and defined.

Data Validation - verifying data for correctness.

Debug - to seek and rid the software of any elements, errors or malfunctions.

Product Life Cycle (PLC) - phases of product design, development and revision.

Real World Testing - testing for integration which seeks to copy the real world use of the product.

White Box Test - testing of areas that cannot be enacted from the black box testing level. (also referred to as 'glass box' testing)

THE PLACE of SOFTWARE TESTING in the PRODUCT LIFECYCLE

Objective

The objective of this work is to provide an overview of software testing as well as the need for testing and how testing fits into the software development lifecycle. The research component will be how to formulate a software testing strategy prior to deploying and release a software product.

Introduction

Software testing is a critical aspect of software product development. In fact testing of software consumes approximately 50% of the total time spent in development of software. Software testing holds an inevitable place in the product lifecycle.

I. Software Testing Overview

According to a work of Carnegie Mellon University, Jiantao Pan (1999) in the document entitled: "Software Testing" the testing of software is an activity aimed at the evaluation of "an attribute or capability of a program or system and determining that it meets its required results." Software testing is not only critical for quality issues related to software but as well, "software testing still remains an art..." (Pan, 1999) the reason that is true is due to the limited comprehension held by most of the principles of software. Testing, according to Pan (1999) is much more than a simple debug process but as well testing is for the purpose of: "...quality assurance, verification and validation, or reliability estimation." (Pan, 1999)

Two major areas of software testing are:

1) correctness testing; and 2) reliability testing. (Pan, 1999)

Pan states that software testing in actuality represents a "trade-off between budget, time and quality." (Pan, 1999) Defects that are found in software are generally caused by errors in design instead of errors in the manufacturing process. Pan (1999) states that software:

Does not suffer from corrosion or wear-and-tear;

Generally does not change until upgrades are released or it becomes obsolete;

Bugs are often found existing in practically any "software module with moderate size" and this is stated to be due to the "complexity of the software [being] generally intractable" added to the fact that human beings possess ability only in limited amounts to "manage complexity." (Pan, 1999)

Complete testing is ideal however, it has not been feasible to testing completely and it is further complicated by the "dynamic nature of programs." (Pan, 1999) Software could be ensured better without the complexity however, as pointed out in the work of Pan: "Society seems to be unwilling to limit complexity because we all want that extra bell, whistle and feature interaction. Thus, our users always push us to the complexity barrier and how close we can approach that barrier is largely determined by the strength of the techniques we can wield against more complex and subtle bugs." (Beizer, 1990; as cited in Pan, 1999) While limitations certainly exist: "testing is an integral part of software development t. It is broadly deployed in the very phase in the software development cycle." (Pan, 1999) in fact, more than 1/2 of time spent in product development is spent in testing.

In the software product development initiative quality may be understood to mean: "...the conformance to the specified design requirement." (Pan, 1999)

Testing plays a key role in the 'Verification and Validation' phase of software development in that testing can: "...make claims based on interpretations of the testing results...[and] it is also possible to "compare the quality among different products under the same specification, based on results from the same test." (Pan, 1999) While quality cannot be tested in a direct manner, related factors may be tested in order to "...make quality visible." (Pan, 1999)

Quality contains three sets of factors which are those of:

1) functionality;

2) engineering; and 3) adaptability." (Pan, 19999)

The chart below illustrates the quality considerations, which are cited most often.

Typical Software Quality Factors

Functionality (exterior quality)

Engineering (interior quality)

Adaptability (future quality)

Correctness

Efficiency

Flexibility

Reliability

Testability

Reusability

Usability

Documentation

Maintainability

Integrity

Structure

Source: Pan (1999)

When tests are conducted for the express reason of validation then the "product works are named clean tests, or positive tests." (Pan, 1999) Pan points out the fact that software testing is still in its' infancy and that software testing is still "an art." (1999) While testing software is a costly pursuit, failure to test software could result in astronomical costs. Various and sundry methods and techniques of testing software exist and all of these serve "multiple purposes in different life cycle." (Pan, 1999)

II. Purpose of Software Testing Categories

Categorized according to purpose of testing software may be grouped into:

1) Correctness testing;

2) Performance testing;

3) Reliability testing; and 4) Security testing. (Pan, 1999)

III. Life-cycle Phase Testing Categories

When grouped according to the life-cycle phase testing of software may be categorized as:

1) Requirement phase testing;

2) Design phase testing;

3) Program phase testing;

4) Evaluating test results;

5) Component testing;

6) Integration testing; and 7) System testing." (Pan, 1999)

The following is a list with accompanying descriptions of each:

Correctness testing - the minimal requirement of testing software;

Black-box testing - a method of testing whereby derivation of data is from "the specified functional requirements without regard to the final program structure." (Pan, 1999) This is also known as 'functional testing' because the focus is 'functionality'. (Pan, 1999) This functionality is sought through the method of maximizes testing effectiveness yet with minimal costs.

Blackbox testing has also been referred to as a) "Data-driven, b) Input/output drive, or Requirements-based testing.

White-box testing' or also referred to as a "glass-box" testing since the "structure and flow of the software" undergoing testing is clearly visible. This type of testing is also referred to as:

a) glass-box testing;

b) logic-driven testing; or design-based testing." (Pan, 1999)

Performance-Testing' - tests for problems in design that causes degradation in the software. Performance testing includes:

a) Resource usage;

b) Throughput;

Stimulus-response time; and d) Queue lengths, which detail the "average of maximum number of tasks waiting to be serviced by selected resources."(Pan, 1999)

Reliability-testing' - just as it sounds referring to "the probability of failure-free operation of a system." (Pan, 1999)

Two types of reliability testing are:

a) Robustness testing - Robustness refers to the degree to which a component can correctly function while receiving exceptional inputs or while under conditions of a stressful environment. (Pan, 1999) and b) Stress-testing - Robustness refers to the degree to which a component can correctly function while receiving exceptional inputs or while under conditions of a stressful environment. (Pan, 1999) Stress testing is also referred to as "load testing" and is generally the method used in whole-system testing instead of only testing of software. Stress is inclusive of:

1) resource exhaustion;

2) burst of activities; and 3) sustained high loads."

Security testing looks for software flaws that open the door to malicious attacks. Pan (1999) states that "software quality, reliability, and security are tightly coupled." Software applications and services generally have security measures that have been integrated against intruders. Security testing of these software systems have the purpose of: (a) identification and removal of flaws in software that might open the way to violations of security; and (b) Validation of effectiveness of measures of security. 'Automation of testing' is used in cutting on time and cost. (Pan, 1999; paraphrased)

IV. Software Testing Tools

Testing as already stated is "a trade-off between budget, time and quality and the drivers of software testing is that of 'profit models.' A great supply of software testing tools are in existence including:

Mothara - an automated mutations testing tool-set which was a Purdue University development and is one in which the "tester can create and execute test cases, measure test adequacy, determine input-output correctness, locate and remove faults or bugs, and control and document the test." (Pan, 1999)

MuMega's Boundschecker and Rational's Purify - Both of these are "run-time checking and debugging aids and both are able to check and protect the system from leaks in memory and pointer problems. (Pan, 1999; paraphrased)

The point at which it is generally considered acceptable to stop testing has as its basis two criteria for stop-testing criteria which are those of: (1) when a threshold has been reached with the reliability; and (2) when the testing costs are not justified by reliability gains.

V. Test Automation Overview

The work of Carl J. Nagle states the fact that: "When developing our test strategy, we must minimize the impact caused by changes in the applications we are testing, and changes in the tools we use to test them." (nd) Nagle additionally relates in the work: "Test Automation Frameworks" that in the present environment: "...of plummeting cycle times, test automatic becomes an increasingly critical and strategic necessity." (nd) Even making the assumption that the historical levels of testing was deemed to be sufficient, and rarely ever has this been the actual case, the question is presented as to how the pace of new development and deployment can be maintained while simultaneously ensuring the reduction of risk and the retention of satisfactory testing results?

In the past test automation has not been as successful as it positional might have been due to the early death of test automation development. Limiting the potential of test automation as well is the fact that: "otherwise fully capable testers are seldom given the time required to gain the appropriate software development skills. For the most part, tests have been testers, not programmers. Consequently, the 'simple' commercial solutions have been far to complex to implement and maintain; and they become shelfware." The statement of Nagle (nd) that is critically important to comprehend is the fact that: "Test automation MUST be approached as a full-blown software development effort in its own right. Without this it is most likely destined to failure in the long-term." (Nagle,

Some of the lessons learned relating to test automation are the following principles to guide the test development strategy:

Test automation is a fulltime effort, not a sideline.

The test design and the test framework are totally separate entities.

The test framework should be application-independent.

The test framework must be easy to expand, maintain, and perpetuate.

The test strategy/design vocabulary should be framework independent.

The test strategy/design should remove most testers from the complexities of the test framework." (Nagel, nd)

VI. Product Life Cycle Test Automation

The work of Dave Kelly entitled: "Software Test Automation and the Product Life Cycle: Implementing Software Test in the Product Life Cycle" states that the product life cycle (PLC) is the stages of development through which the product transitions. Kelly affirms other reports concerning the debate surrounding the usefulness of automated testing stating that that is where the 'product life cycle' (PLC) "comes in because the effectiveness of ones use of PLC will generally be "dependent upon...programming resources and the length of the PLC.' (Kelly, 2007))

VII. Product Life Cycle

The first phase of the product life cycle (PLC) is the "Design Phase" which is the phase for planning and formulation of ideas of the product. The second phase is the 'code complete phase' which is the phase in which the code is likely to be written but the bugs have not been fixed in the system as of yet. One important aspect of the product life cycle is the 'Automation Checklist' in this area it is related that affirmation to the following indicates the need for serious consideration of automation of the test:

Can the test sequence of actions be defined?

Is it necessary to repeat the sequence of actions many times?

Is it possible to automate the sequence of actions?

Is it possible to "semi-automate" a test?

Is the behavior of the software under test the same with automation as without?

Are you testing non-UI aspects of the program?

Do you need to run the same tests on multiple hardware configurations? (Kelly, 2007)

The next phase is referred to as the 'Alpha Phase' which identifies the precise time when the product is adjudged stable and in its' complete form by development and quality assurance teams. The product has transitioned from only 'basically functional' to 'a finely tuned product' in the Alpha Phase. In order to attain the status of the Alpha Stage the components of compatibility, interoperability, and performance tests are all in a state of being complete and a state of automation to the greatest extent possible. The next phase is the 'Beta Phase' which is a stage characterized by the system being for the most part, 'bug-free'. Kelly states that bugs not yet fixed at this point or in the 'Beta Phase' will "almost definitely be a slip in the shipping schedule." (Kelly, 2007)

The 'Zero Defect Build Phase' is a period "of stability where no new serious defects are discovered" with the product being stabilized and ready for shipment. The potential of automation during the 'zero defect build stage' includes regression tests which serve to make verification of correction of system bugs. The Green Master phase is the final inspection prior to shipping. Automation during the Green Master Phase includes: "...running general acceptance tests, running regression tests" (Kelly, 2007) saving time during this phase. The work entitled: "Software Performance and Testing" published by LioNBRIDGE states that qualify may be improved through the entire process of the application development lifecycle.

The work of Ram Chillarege entitled: "Software Testing Best Practices" states that the report identifies 28 'best practices' that provide contribution to improvement in testing of software. Listed as 'best practices' are the following:

Functional Specifications;

Reviews and Inspection

Formal entry and exit criteria;

Multi-platform testing

Internal Betas

Automated Test Execution;

Beta Programs; and Nightly Builds (1999)

Functional specifications are considered 'key' components in the development processes. Inspection and reviews " are one of the most efficient methods of debugging code." Formal entry and exit criteria expresses a process in which the idea is that:..."every process step, be it inspection, functional test, or software design, has a precise entry and precise exit criteria." (Chillarege, 1999)

VIII. Application Test Tools

Application test tools are inclusive of the following examples in Figure 2.

Examples of Application Test Tools

Product Vendor

Comments

AdaTEST IPL

Coverage, static and dynamic software testing for Ada Apps

CodeWizard

ParaSoft

C++ analysis tool

Logiscope DMS Source code analysis tools

PolySpace Suite PolySpace Detects run-time errors and non-deterministic constructs in applications at compilation time

TCMON Testwell Test coverage and execution profiling tool for Ada code.

Source: Software QA Testing and Test Tool Resources (2007)

IX. Classic Testing Mistakes

The work of Brian Marick (1997) entitled: "Classic Testing Mistakes" states that the mistakes of testing "cluster usefully into five groups" which are those of:

The Role of Testing: who does the testing team serve, and how does it do that?

Planning the Testing Effort: how should the whole team's work be organized?

Personnel Issues: who should test? And the Tester at Work: designing, writing, and maintaining individual tests (Marick, 1997)

X. Developing a Team of Testers is Key in Software Development and Testing

The team that is responsible for testing is a team that must critically possess certain qualities and characteristics in order for the maximum potential to be realized in the area of software development in the testing phase. In regards to the testing team staff, Marick states that a good tester has the following qualities and characteristics:

good tester is "methodical and systematic" good tester is "tactful and diplomatic" good tester is "skeptical, especially about assumptions, and wants to see concrete evidence;

good tester is able to notice and pursue odd details;

good tester has "good written and verbal skills (for explaining bugs clearly and concisely)" good tester has: "a knack for anticipating what others are likely to misunderstand"; and good tester has "a willingness to get one's hands dirty, to experiment, to try something to see what happens." (Marick, 1997)

Marick states that one should be particularly careful "to avoid the trap of testers who are not domain experts." (1997) the work of Ben-Yaacov and Gazlay entitled; "Real World Software Testing at a Silicon Valley High-Tech Software Company" states that the: "Silicon Valley high-tech software products teams face a troubling paradox on a daily basis - How to introduce new technology and features faster than ever, while simultaneously improving product quality and responsiveness to customer quality issues." (2001) Stated additionally are the following goals of the Silicon Valley high-tech software team:

Minimize time-to-market by delivering new technology as soon as possible

Maximize customer satisfaction by delivering a specific set of features

Minimize number of known defect in the shipped product releases ("built-in quality")

Maximize level of support to customer quality issues. (Ben-Yaacov and Gazlay, 2001)

Developed from customer input concerning satisfaction criteria are the following, which are stated to "...drive the development and quality assurance strategies are stated to be:

The ultra-rapidly evolving technology market. This is characterized by the fast-paced introduction of fundamental technologies, which are otherwise not available to customers.

It is further characterized by the introduction of successive revisions very quickly.

Customers are willing to put up with low quality to obtain this technology

The high-end technology market. Quality is more important but not as much as new, robust features.

The main-stream technology market. Quality and stability are of paramount importance." (Ben-Yaacov and Gazlay, 2001)

According to Ben-Yaacov and Gazlay before making a decision about the area of quality or the business priority to focus attention upon the questions asked include: (1) What would customers value most at this specific stage of the product life cycle? (2) Are there features that would influence customer's decisions to buy or not buy the product? And (3) What aspects of the product do customer perceive as drivers of their success?' (2001) Four key measurements of quality in software products are stated to include:

1) technology;

2) features;

3) freedom from bugs (QA and testing), and;

4) responsive support. (Ben-Yaacov and Gazlay, 2001)

The work of Elizabeth Hendrickson entitled: "Evaluating Tools" published in the Testing and Quality Journal Volume I and Issue I relates that process of evaluating tools on the market and lists initial requirements in choosing those tools. First listed are the following issues in compatibility. Any testing tool will need to be compatible with the following:

1) "the operating system(s) your application supports;

2) the development environment(s) used to create your application; and 3) third party software with which your application integrates." (Hendrickson, 1999)

Stated as questions that must be answered in identifying the initial requirements are the following while keeping "the audience for the tool in mind":

1) "Who will be using the tool on a day-to-day basis?

2) Is your organization willing to invest time and energy in training? And 3) Does your organization expect test automation to pay off right away? " (Hendrickson, 1999)

The skill level of the users of the tool will be important in determining the best tool however, one must remember that "powerful tools have higher potential benefits is used effectively" however these tools are most expensive and budgetary constraints may have to be considered in the choice of tool. The two questions that must be addressed are: (1) How much can the organization afford; and (2) How much is this tool worth to the organization? (Hendrickson, 1999; paraphrased) Other necessary questions include: (1) What is the hoped-for benefit to the organization from the tool?; and (2) " Are the benefit quantifiable and measurable?" (Hendrickson, 1999)

When questioning sales concerning the competition vs. The chosen tool Hendrickson states the following questions are being relevant:

1) How can I compare your product with other products on the market?;

2) Under what situations is your tool the best choice?;

3) Under what situations is your tool probably not the best choice?;

4) What features differentiate your tool from the competition?; and 5) What don't you like about your tool?" (Hendrickson, 1999)

At this point, the organization should refine their requirements, which will require effectively communicating the impact of implementation of the tools "to everyone affected" by the tools and their accompanying impacts. Hendrickson provides the example of when the tester is threatened by the automation process wondering if their job will become obsolete. This is counterproductive and everyone in the organization should feel comfortable with the change occurring, in other words there may be concerns that "seem to have nothing to do with the tool" that must be overcome in the phase of choosing the tools used for testing. After the list has been narrowed down it may very well be, according to Hendrickson, that there is only one tool that appears to meet the testing needs and even in this case it is important that one make sure that this tool will address all requirements in testing needs.