Software testing's role in the product development lifecycle

¶ … Software Testing in the Product Life Cycle

THE PLACE of SOFTWARE TESTING in the PRODUCT LIFECYCLE

Definitions & Terms in the Research

Alpha Test - test phase of the PLC where code is complete and product is testable and stable.

Automated Testing - individual testing without direct tester involvement.

Beta Test - portion of test phase of PLC where testing for integration is completed.

Black Box Test - test in which the tester cannot see into the software

Code Complete - phase of the PLC where coding of functionality is completed

Coding Phase - phase of the PLC where coding is accomplished for specs of function and architecture.

Compatibility Test - testing for compatibility of software being developed with other software and hardware.

Concept Phase - phase of the PLC where the product 'idea' is developed and defined.

Data Validation - verifying data for correctness.

Debug - to seek and rid the software of any elements, errors or malfunctions.

Product Life Cycle (PLC) - phases of product design, development and revision.

Real World Testing - testing for integration which seeks to copy the real world use of the product.

White Box Test - testing of areas that cannot be enacted from the black box testing level. (also referred to as 'glass box' testing)

THE PLACE of SOFTWARE TESTING in the PRODUCT LIFECYCLE

Objective

The objective of this work is to provide an overview of software testing as well as the need for testing and how testing fits into the software development lifecycle. The research component will be how to formulate a software testing strategy prior to deploying and release a software product.

Introduction

Software testing is a critical aspect of software product development. In fact testing of software consumes approximately 50% of the total time spent in development of software. Software testing holds an inevitable place in the product lifecycle.

I. Software Testing Overview

According to a work of Carnegie Mellon University, Jiantao Pan (1999) in the document entitled: "Software Testing" the testing of software is an activity aimed at the evaluation of "an attribute or capability of a program or system and determining that it meets its required results." Software testing is not only critical for quality issues related to software but as well, "software testing still remains an art..." (Pan, 1999) the reason that is true is due to the limited comprehension held by most of the principles of software. Testing, according to Pan (1999) is much more than a simple debug process but as well testing is for the purpose of: "...quality assurance, verification and validation, or reliability estimation." (Pan, 1999)

Two major areas of software testing are:

IV. Software Testing Tools

Testing as already stated is "a trade-off between budget, time and quality and the drivers of software testing is that of 'profit models.' A great supply of software testing tools are in existence including:

Mothara - an automated mutations testing tool-set which was a Purdue University development and is one in which the "tester can create and execute test cases, measure test adequacy, determine input-output correctness, locate and remove faults or bugs, and control and document the test." (Pan, 1999)

MuMega's Boundschecker and Rational's Purify - Both of these are "run-time checking and debugging aids and both are able to check and protect the system from leaks in memory and pointer problems. (Pan, 1999; paraphrased)

The point at which it is generally considered acceptable to stop testing has as its basis two criteria for stop-testing criteria which are those of: (1) when a threshold has been reached with the reliability; and (2) when the testing costs are not justified by reliability gains.

V. Test Automation Overview

The work of Carl J. Nagle states the fact that: "When developing our test strategy, we must minimize the impact caused by changes in the applications we are testing, and changes in the tools we use to test them." (nd) Nagle additionally relates in the work: "Test Automation Frameworks" that in the present environment: "...of plummeting cycle times, test automatic becomes an increasingly critical and strategic necessity." (nd) Even making the assumption that the historical levels of testing was deemed to be sufficient, and rarely ever has this been the actual case, the question is presented as to how the pace of new development and deployment can be maintained while simultaneously ensuring the reduction of risk and the retention of satisfactory testing results?

In the past test automation has not been as successful as it positional might have been due to the early death of test automation development. Limiting the potential of test automation as well is the fact that: "otherwise fully capable testers are seldom given the time required to gain the appropriate software development skills. For the most part, tests have been testers, not programmers. Consequently, the 'simple' commercial solutions have been far to complex to implement and maintain; and they become shelfware." The statement of Nagle (nd) that is critically important to comprehend is the fact that: "Test automation MUST be approached as a full-blown software development effort in its own right. Without this it is most likely destined to failure in the long-term." (Nagle,

Some of the lessons learned relating to test automation are the following principles to guide the test development strategy:

Test automation is a fulltime effort, not a sideline.

The test design and the test framework are totally separate entities.

The test framework should be application-independent.

The test framework must be easy to expand, maintain, and perpetuate.

The test strategy/design vocabulary should be framework independent.

The test strategy/design should remove most testers from the complexities of the test framework." (Nagel, nd)

VI. Product Life Cycle Test Automation

The work of Dave Kelly entitled: "Software Test Automation and the Product Life Cycle: Implementing Software Test in the Product Life Cycle" states that the product life cycle (PLC) is the stages of development through which the product transitions. Kelly affirms other reports concerning the debate surrounding the usefulness of automated testing stating that that is where the 'product life cycle' (PLC) "comes in because the effectiveness of ones use of PLC will generally be "dependent upon...programming resources and the length of the PLC.' (Kelly, 2007))

VII. Product Life Cycle

The first phase of the product life cycle (PLC) is the "Design Phase" which is the phase for planning and formulation of ideas of the product. The second phase is the 'code complete phase' which is the phase in which the code is likely to be written but the bugs have not been fixed in the system as of yet. One important aspect of the product life cycle is the 'Automation Checklist' in this area it is related that affirmation to the following indicates the need for serious consideration of automation of the test:

Can the test sequence of actions be defined?

Is it necessary to repeat the sequence of actions many times?

Is it possible to automate the sequence of actions?

Is it possible to "semi-automate" a test?

Is the behavior of the software under test the same with automation as without?

Are you testing non-UI aspects of the program?

Do you need to run the same tests on multiple hardware configurations? (Kelly, 2007)

The next phase is referred to as the 'Alpha Phase' which identifies the precise time when the product is adjudged stable and in its' complete form by development and quality assurance teams. The product has transitioned from only 'basically functional' to 'a finely tuned product' in the Alpha Phase. In order to attain the status of the Alpha Stage the components of compatibility, interoperability, and performance tests are all in a state of being complete and a state of automation to the greatest extent possible. The next phase is the 'Beta Phase' which is a stage characterized by the system being for the most part, 'bug-free'. Kelly states that bugs not yet fixed at this point or in the 'Beta Phase' will "almost definitely be a slip in the shipping schedule." (Kelly, 2007)

The 'Zero Defect Build Phase' is a period "of stability where no new serious defects are discovered" with the product being stabilized and ready for shipment. The potential of automation during the 'zero defect build stage' includes regression tests which serve to make verification of correction of system bugs. The Green Master phase is the final inspection prior to shipping. Automation during the Green Master Phase includes: "...running general acceptance tests, running regression tests" (Kelly, 2007) saving time during this phase. The work entitled: "Software Performance and Testing" published by LioNBRIDGE states that qualify may be improved through the entire process of the application development lifecycle.

The work of Ram Chillarege entitled: "Software Testing Best Practices" states that the report identifies 28 'best practices' that provide contribution to improvement in testing of software. Listed as 'best practices' are the following:

Functional Specifications;

Reviews and Inspection

Formal entry and exit criteria;

Multi-platform testing

Internal Betas

Automated Test Execution;

Beta Programs; and Nightly Builds (1999)

Functional specifications are considered 'key' components in the development processes. Inspection and reviews " are one of the most efficient methods of debugging code." Formal entry and exit criteria expresses a process in which the idea is that:..."every process step, be it inspection, functional test, or software design, has a precise entry and precise exit criteria." (Chillarege, 1999)

VIII. Application Test Tools

Application test tools are inclusive of the following examples in Figure 2.

Examples of Application Test Tools

Product Vendor

Comments

AdaTEST IPL

Coverage, static and dynamic software testing for Ada Apps

CodeWizard

ParaSoft

C++ analysis tool

Logiscope DMS Source code analysis tools

PolySpace Suite PolySpace Detects run-time errors and non-deterministic constructs in applications at compilation time

TCMON Testwell Test coverage and execution profiling tool for Ada code.

Source: Software QA Testing and Test Tool Resources (2007)

IX. Classic Testing Mistakes

The work of Brian Marick (1997) entitled: "Classic Testing Mistakes" states that the mistakes of testing "cluster usefully into five groups" which are those of:

The Role of Testing: who does the testing team serve, and how does it do that?

Planning the Testing Effort: how should the whole team's work be organized?

Personnel Issues: who should test? And the Tester at Work: designing, writing, and maintaining individual tests (Marick, 1997)

X. Developing a Team of Testers is Key in Software Development and Testing

The team that is responsible for testing is a team that must critically possess certain qualities and characteristics in order for the maximum potential to be realized in the area of software development in the testing phase. In regards to the testing team staff, Marick states that a good tester has the following qualities and characteristics:

good tester is "methodical and systematic" good tester is "tactful and diplomatic" good tester is "skeptical, especially about assumptions, and wants to see concrete evidence;

good tester is able to notice and pursue odd details;

good tester has "good written and verbal skills (for explaining bugs clearly and concisely)" good tester has: "a knack for anticipating what others are likely to misunderstand"; and good tester has "a willingness to get one's hands dirty, to experiment, to try something to see what happens." (Marick, 1997)

Marick states that one should be particularly careful "to avoid the trap of testers who are not domain experts." (1997) the work of Ben-Yaacov and Gazlay entitled; "Real World Software Testing at a Silicon Valley High-Tech Software Company" states that the: "Silicon Valley high-tech software products teams face a troubling paradox on a daily basis - How to introduce new technology and features faster than ever, while simultaneously improving product quality and responsiveness to customer quality issues." (2001) Stated additionally are the following goals of the Silicon Valley high-tech software team:

Minimize time-to-market by delivering new technology as soon as possible

Maximize customer satisfaction by delivering a specific set of features

Minimize number of known defect in the shipped product releases ("built-in quality")

Maximize level of support to customer quality issues. (Ben-Yaacov and Gazlay, 2001)

Developed from customer input concerning satisfaction criteria are the following, which are stated to "...drive the development and quality assurance strategies are stated to be:

The ultra-rapidly evolving technology market. This is characterized by the fast-paced introduction of fundamental technologies, which are otherwise not available to customers.

It is further characterized by the introduction of successive revisions very quickly.

Customers are willing to put up with low quality to obtain this technology

The high-end technology market. Quality is more important but not as much as new, robust features.

The main-stream technology market. Quality and stability are of paramount importance." (Ben-Yaacov and Gazlay, 2001)

According to Ben-Yaacov and Gazlay before making a decision about the area of quality or the business priority to focus attention upon the questions asked include: (1) What would customers value most at this specific stage of the product life cycle? (2) Are there features that would influence customer's decisions to buy or not buy the product? And (3) What aspects of the product do customer perceive as drivers of their success?' (2001) Four key measurements of quality in software products are stated to include:

1) technology;

2) features;

3) freedom from bugs (QA and testing), and;

4) responsive support. (Ben-Yaacov and Gazlay, 2001)

The work of Elizabeth Hendrickson entitled: "Evaluating Tools" published in the Testing and Quality Journal Volume I and Issue I relates that process of evaluating tools on the market and lists initial requirements in choosing those tools. First listed are the following issues in compatibility. Any testing tool will need to be compatible with the following:

1) "the operating system(s) your application supports;

2) the development environment(s) used to create your application; and 3) third party software with which your application integrates." (Hendrickson, 1999)

Stated as questions that must be answered in identifying the initial requirements are the following while keeping "the audience for the tool in mind":

1) "Who will be using the tool on a day-to-day basis?

2) Is your organization willing to invest time and energy in training? And 3) Does your organization expect test automation to pay off right away? " (Hendrickson, 1999)

The skill level of the users of the tool will be important in determining the best tool however, one must remember that "powerful tools have higher potential benefits is used effectively" however these tools are most expensive and budgetary constraints may have to be considered in the choice of tool. The two questions that must be addressed are: (1) How much can the organization afford; and (2) How much is this tool worth to the organization? (Hendrickson, 1999; paraphrased) Other necessary questions include: (1) What is the hoped-for benefit to the organization from the tool?; and (2) " Are the benefit quantifiable and measurable?" (Hendrickson, 1999)

When questioning sales concerning the competition vs. The chosen tool Hendrickson states the following questions are being relevant:

1) How can I compare your product with other products on the market?;

2) Under what situations is your tool the best choice?;

3) Under what situations is your tool probably not the best choice?;

4) What features differentiate your tool from the competition?; and 5) What don't you like about your tool?" (Hendrickson, 1999)

At this point, the organization should refine their requirements, which will require effectively communicating the impact of implementation of the tools "to everyone affected" by the tools and their accompanying impacts. Hendrickson provides the example of when the tester is threatened by the automation process wondering if their job will become obsolete. This is counterproductive and everyone in the organization should feel comfortable with the change occurring, in other words there may be concerns that "seem to have nothing to do with the tool" that must be overcome in the phase of choosing the tools used for testing. After the list has been narrowed down it may very well be, according to Hendrickson, that there is only one tool that appears to meet the testing needs and even in this case it is important that one make sure that this tool will address all requirements in testing needs.

The work of Gupta and Sinha states that software testability "has been defined as provisions that can be made in the software during the design and coding phases, so that test-plans can be executed easily and systematically." (1994) "Controllability and observability " are identified by Gupta and Sinha as being the measures that the incorporation of enables the facilitation of internal states observation. Incorporation of measures of controllability enable the software "to attain states required for execution of various tests (normally difficult to achieve through user interface commands)." (Gupta and Sinha, 1994) the testability measures will depend upon the plan for testing making the pre-testing phase dependent upon the pre-testing phase of software development.

The work entitled: "Leveraging the Role of Testing and Quality Across the Lifecycle to Cut Costs and Drive it/Business Responsiveness" published by Microsoft Visual Studio Team System states that Microsoft's Visual Studio Team System (VSTS) addresses testing from two perspectives - developer centric testing which is part of the developer role and tester-centric testing which targets Quality Assurance and Quality Control personnel with the test role." (Microsoft Visual Studio Team System, 2005) Testing should be consistent and frequent and the earlier problems are found and fixed the less cost is realized in the project overall.

Summary and Conclusion

This work has reviewed the work of many experts in the software testing and development field and has clearly demonstrated that testing of software is not something that is done within a specific phase of development but indeed, is an ongoing process throughout all phases of development and the entire lifecycle of the software product. As stated in the introduction of this work testing of software accounts for approximately 50% of the time spent in development of the software product.

Testing is more than just getting the bugs out because testing is the method of assuring quality and as well of verification and validation of the product. (Pan, 1999) Software testing includes testing for correctness and testing for reliability. (Pan, 1999) Errors are rarely the case of software defects but are generally errors in the process of manufacturing the software. Testing of software is deployed in each phase of the development cycle. (Pan, 1999) in order to ensure product quality it is necessary to address functionality, engineering and adaptability. (Pan, 1999)

There are several phases of testing within the product lifecycle, which includes requirement, design, program, evaluation, component, integration, and system testing. (Pan, 1999) There are various testing tools available and the organization should shop around before choosing the testing tool that they will rely upon. Automation in software testing is generally not the preferred method as testers are generally the method used for testing software. One reason for this is due to the lack of commitment toward development of automated software testing tools. This work further reviewed the 'best practices' in software testing which include functional specifications, review and inspection, formal entry and exit criteria, multi-platform testing; internal betas, automated text execution, beta programs and nightly builds. (Chillarege, 1999)

Classic mistakes made in testing include the areas of 'the role' of testing, 'planning' of the testing, 'personnel issues' and the tester's performance. (Marick, 1997) This work has examined the importance of development of the tester team and the qualities and characteristics that the testers must have to be good at what they do. Included in these characteristics are hat the tester be 'methodical and systematic', 'tactful and diplomatic', skeptical, detail-oriented, in possession of good skills in communication verbally and in the written form, have a talent for anticipation of what others tend to have trouble understanding, and finally the tester must be willing to work in a hands-on approach and be willing to test for the purpose of seeing what the results might be. Marick (1997)

The introduction of technology and features being developed in the rapid action which this occurs and to, at the same time, make improvement to the quality of the product and as well to respond to the issues of customers in product development is quite an initiative to undertake. Driving development and quality assurance of software are the technology market which is characterized by an extremely fast past and the revisions are released just as quickly. While quality is considered to be important in the high-tech marketplace new and robust features are considered even more important. (Ben-Yaacov and Gazlay, 2001) Four key measurements of the software product quality have been identified to be the technology, features, freedom from bugs and responsive support. (Ben-Yaacov and Gazlay, 2001)

With the rapid development and release of software in today's high-technology market, it appears that the organization producing software products will be extremely wise to focus attention upon the development of a testing team that can enable the organization through ensuring quality products through in-house testing of the product prior to the product being released into the consumer market. The focus on tester team development is crucial in this initiative. Software testing is an ongoing effort that has a place within each and every phase of the product development and product lifecycle.