{{Short description |Use of purpose-built software to control test execution}} {{Redirect|Automated QA|the company|AutomatedQA}} {{Software development process}} {{More footnotes|date=February 2009}} '''Test automation''' is the use of software (separate from the software being tested) for controlling the execution of tests and comparing actual outcome with predicted.<ref>{{cite book | last = Kolawa | first = Adam |author2=Huizinga, Dorota | title = Automated Defect Prevention: Best Practices in Software Management | year = 2007 | publisher = Wiley-IEEE Computer Society Press | page=74| isbn = 978-0-470-04212-0 }}</ref> Test automation supports testing the system under test (SUT) without manual interaction which can lead to faster test execution and testing more often. Test automation is a key aspect of continuous testing and often for continuous integration and continuous delivery (CI/CD).<ref>{{Cite book|last1=O’Connor|first1=Rory V.|url=https://books.google.com/books?id=2xOcCgAAQBAJ&q=Systems%2C+Software+and+Services+Process+Improvement%3A+27th+European+Conference&pg=PA71|title=Systems, Software and Services Process Improvement: 22nd European Conference, EuroSPI 2015, Ankara, Turkey, September 30 -- October 2, 2015. Proceedings|last2=Akkaya|first2=Mariye Umay|last3=Kemaneci|first3=Kerem|last4=Yilmaz|first4=Murat|last5=Poth|first5=Alexander|last6=Messnarz|first6=Richard|date=2015-10-15|publisher=Springer|isbn=978-3-319-24647-5|language=en}}</ref>
==Compared to manual testing== Automation provides many benefits over manual testing.
===API testing=== For API testing, tests drive the SUT via its application programming interface (API). Compared to manual testing, automated API testing often can execute a relatively large number of cases in a relatively short time.
===GUI testing=== For GUI testing, tests drive the SUT via its graphical user interface (GUI) by generating events such as keystrokes and mouse clicks. Automated GUI testing can be challenging to develop, but can run much faster than a human could perform the same testing. Specializations include:
* Record & playback testing {{endash}} Some GUI testing tools provide a feature that allows for interactively recording user actions and replaying them later as a test, comparing actual results to expected. An advantage of this approach is that it requires little or no coding. However, some claim that such tests suffer from reliability, maintainability and accuracy issues. For example, changing the label of a button or moving it to another part of the view may require tests to be re-recorded, and such tests often are inefficient and incorrectly record unimportant activities.{{Citation needed|date=March 2013}}
* For testing a web site, the GUI is the browser and interaction is via DOM events and HTML. A headless browser or solutions based on Selenium Web Driver are normally used for this purpose.<ref>Headless Testing with Browsers; https://docs.travis-ci.com/user/gui-and-headless-browsers/</ref><ref name="Headless Testing with Browsers">Headless Testing with PhantomJS;http://phantomjs.org/headless-testing.html</ref><ref>Automated User Interface Testing; https://www.devbridge.com/articles/automated-user-interface-testing/</ref>
===Regression testing=== When automated testing is in place, regression testing can be a relatively quick and easy operation. Instead of a significant outlay of human time and effort, a regression test run could require nothing more than a push of a button and even starting the run can be automated.
==Automated techniques== The following are notable testing techniques categorized as test automation.
===Continuous testing=== Continuous testing is the process of executing automated tests as part of the software delivery pipeline to assess the business risk of releasing the SUT.<ref name="essential">[https://www.techwell.com/techwell-insights/2015/08/part-pipeline-why-continuous-testing-essential Part of the Pipeline: Why Continuous Testing Is Essential], by Adam Auerbach, TechWell Insights August 2015</ref><ref name="stickym">[http://www.stickyminds.com/interview/relationship-between-risk-and-continuous-testing-interview-wayne-ariola The Relationship between Risk and Continuous Testing: An Interview with Wayne Ariola], by Cameron Philipp-Edmonds, Stickyminds December 2015</ref> The scope of testing extends from validating bottom-up requirements or user stories to assessing the system requirements associated with overarching business goals.<ref name="pnsqc">[http://uploads.pnsqc.org/2015/papers/t-007_Ariola_paper.pdf DevOps: Are You Pushing Bugs to Clients Faster], by Wayne Ariola and Cynthia Dunlop, PNSQC October 2015</ref>
===Model-based testing=== For model-based testing, the SUT is modeled and test cases can be generated from it to support no code test development. Some tools support the encoding of test cases as plain English that can be used on multiple operating systems, browsers, and smart devices.<ref>{{cite book|title=Proceedings from the 5th International Conference on Software Testing and Validation (ICST). Software Competence Center Hagenberg. "Test Design: Lessons Learned and Practical Implications.|isbn=978-0-7381-5746-7|doi=10.1109/IEEESTD.2008.4578383}}</ref>
===Test-driven development=== Test-driven development (TDD) inherently includes the generation of automation test code. Unit test code is written while the SUT code is written. When the code is complete, the tests are complete as well.<ref name="Learning TDD">{{cite journal|doi=10.1109/ms.2007.80|title=Learning Test-Driven Development by Counting Lines|year=2007|last1=Vodde|first1=Bas|last2=Koskela|first2=Lasse|journal=IEEE Software|volume=24|issue=3|pages=74–79|s2cid=30671391}}</ref>
===Other=== Other test automation techniques include:
* Data-driven testing * Modularity-driven testing * Keyword-driven testing * Behavior driven development
==Considerations== A review of 52 practitioner and 26 academic sources found that five main factors to consider in test automation decision are: system under test (SUT), scope of testing, test toolset, human and organizational topics, cross-cutting factors. The factors most frequently identified were: need for regression testing, economic factors, and maturity of SUT.<ref>{{Cite journal|last1=Garousi|first1=Vahid|last2=Mäntylä|first2=Mika V.|date=2016-08-01|title=When and what to automate in software testing? A multi-vocal literature review|journal=Information and Software Technology|volume=76|pages=92–117|doi=10.1016/j.infsof.2016.04.015}}</ref><ref> {{cite web|url=http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=ART&ObjectId=2010|title=When Should a Test Be Automated?|author=Brian Marick|publisher=StickyMinds.com|access-date=2009-08-20}}</ref>
While the reusability of automated tests is valued by software development companies, this property can also be viewed as a disadvantage as it leads to a plateau effect, where repeatedly executing the same tests stops detecting errors.
Testing tools can help automate tasks such as product installation, test data creation, GUI interaction, problem detection (consider parsing or polling agents equipped with test oracles), defect logging, etc., without necessarily automating tests in an end-to-end fashion.
Considerations when developing automated tests include: * Platform and operating system independence * Data-driven testing * Reporting (database, Crystal Reports) * Ease of debugging * Logging * Version control * Extension and customization (e.g., APIs for integrating with other tools) * Integration with developer tools (e.g., using Ant or Maven for Java development) * Unattended test runs for integration with build processes and batch runs * Email notifications (i.e., bounce messages) * Distributed test execution
==Roles== To support coded automated testing, the test engineer or software quality assurance person must have software coding ability. Some testing techniques such as table-driven and no-code can lessen or alleviate the need for programming skill.
==Framework== A test automation framework provides a programming environment that integrates test logic, test data, and other resources. The framework provides the basis of test automation and simplifies the automation effort. Using a framework can lower the cost of test development and maintenance. If there is change to any test case then only the test case file needs to be updated and the driver script and startup script will remain the same.
A framework is responsible for defining the format in which to express expectations, providing a mechanism to hook into or drive the SUT, executing the tests, and reporting results.<ref>{{cite web | url = https://www.youtube.com/watch?v=qf2i-xQ3LoY | title = Selenium Meet-Up 4/20/2010 Elisabeth Hendrickson on Robot Framework 1of2 | website = YouTube | date = 28 April 2010 | access-date = 2010-09-26 }}</ref>
Various types of frameworks are available:
* Linear {{endash}} procedural code, possibly generated by tools like those that use record and playback * Structured {{endash}} uses control structures - typically ‘if-else’, ‘switch’, ‘for’, ‘while’ conditions/ statements * Data-driven {{endash}} data is persisted outside of tests in a database, spreadsheet, or other mechanism * Keyword-driven * Hybrid {{endash}} multiple types are used * Agile automation framework * Unit testing {{endash}} some frameworks are intended primarily for unit testing such as xUnit, JUnit and NUnit
==Test automation interface== A test automation interface is a platform that provides a workspace for incorporating multiple testing tools and frameworks for system/integration testing. A test automation interface may simplify the process of mapping tests to business criteria without coding. A test automation interface may improve the efficiency and flexibility of maintaining tests.<ref name="Interface">{{cite web | url = http://www.qualitycow.com/Docs/ConquestInterface.pdf | title = Conquest: Interface for Test Automation Design | access-date = 2011-12-11 | archive-date = 2012-04-26 | archive-url = https://web.archive.org/web/20120426044210/http://www.qualitycow.com/Docs/ConquestInterface.pdf | url-status = dead }}</ref>
thumb|Test Automation Interface Model
A test automation interface consists of the following aspects:
; Interface engine: Consists of a parser and a test runner. The parser is present to parse the object files coming from the object repository into the test specific scripting language. The test runner executes the test scripts using a test harness.<ref name="Interface" />
; Object repository: Collection of UI/Application object data recorded by the testing tool while exploring the SUT.<ref name="Interface" />
== See also == * {{Annotated link |Comparison of GUI testing tools}} * {{Annotated link |List of web testing tools}} * {{Annotated link |Fuzzing}}
==References== {{Reflist|30em}}
===General references=== {{refbegin}} * {{cite book | author = Elfriede Dustin | title = Automated Software Testing | year = 1999 | publisher = Addison Wesley | isbn = 978-0-201-43287-9 | display-authors = etal | url-access = registration | url = https://archive.org/details/automatedsoftwar00elfr }} * {{cite book | author = Elfriede Dustin| title = Implementing Automated Software Testing | publisher = Addison Wesley | isbn =978-0-321-58051-1 |display-authors=etal| year = 2009 }} * {{cite book | author = Mark Fewster & Dorothy Graham | title = Software Test Automation | year = 1999 | publisher = ACM Press/Addison-Wesley | isbn = 978-0-201-33140-0}} * Roman Savenkov: ''How to Become a Software Tester.'' Roman Savenkov Consulting, 2008, {{ISBN|978-0-615-23372-7}} * {{cite book | author = Hong Zhu | title = AST '08: Proceedings of the 3rd International Workshop on Automation of Software Test | url = http://portal.acm.org/citation.cfm?id=1370042# | year = 2008 | publisher = ACM Press | isbn = 978-1-60558-030-2 | display-authors = etal | doi = 10.1145/1370042 }} * {{cite book |last1=Mosley |first1=Daniel J. |last2=Posey |first2=Bruce |title=Just Enough Software Test Automation |isbn=978-0130084682 |year=2002 |publisher=Prentice Hall Professional }} * Hayes, Linda G., "Automated Testing Handbook", Software Testing Institute, 2nd Edition, March 2004 * Kaner, Cem, "[http://www.kaner.com/pdfs/testarch.pdf Architectures of Test Automation] {{Webarchive|url=https://web.archive.org/web/20210126100156/http://www.kaner.com/pdfs/testarch.pdf |date=2021-01-26 }}", August 2000 {{refend}}
{{Software testing}} {{Authority control}}
{{DEFAULTSORT:Test Automation}} Automation Category:Automation software