Non-functional testing refers to software testing that is carried out to test the non-functional aspects of a software application.
There are lots of different types of non-functional testing, and some types of software testing can be considered both functional testing and non-functional at the same time.
Non-functional testing is necessary because it evaluates essential user criteria such as performance and usability and verifies whether the software runs as expected outside of its basic functionality.
In this article, we explore the non-functional testing definition and characteristics alongside types of non-functional testing, approaches to non-functional testing, and testing tools that can help optimize and improve your own non-functional testing processes.
What is non-functional testing?
Non-functional testing is any type of software testing where non-functional aspects of the software build are tested.
Examples of non-functional testing include testing designed to assess capacity, performance, usability, recovery, and portability.
Verifying the quality and status of each of these non-functional criteria is just as important as verifying the functions of a piece of software, but these parameters aren’t tested in standard functional testing.
Essentially, non-functional testing means testing ‘how’ software functions work rather than testing ‘if’ they work.
1. When do you need non-functional testing?
Non-functional testing is carried out during the system testing phase of software testing after unit testing and integration testing have taken place.
During system testing, testers will carry out both functional and non-functional testing, starting with functional testing.
Once testers have established that the software is functioning as expected, they carry out non-functional testing to assess whether it also meets non-functional parameters.
It’s usually necessary to carry out functional testing before non-functional testing because it’s impossible to test the reliability or performance of functions that don’t work at all. Non-functional testing is one of the last stages of software testing before user acceptance testing and final product release.
2. When you don’t need non-functional testing
It’s always important to test non-functional aspects of software unless they have already been tested and found to be adequate.
Even if you’ve carried out non-functional testing on software before, it may be necessary to test non-functional parameters again, for example, if new features have been added to the software or if changes have been made to the code that could affect performance and reliability.
The objectives of non-functional testing
The objectives of non-functional testing are to check that the product meets user expectations and to optimize the product before release.
It can also help developers and testers to understand the software better and use this knowledge in future optimisations.
1. Quality control
Non-functional testing aims to test factors that impact the product’s usability, reliability, maintainability, portability, and efficiency.
Testing these elements ensures that the product that’s released to the market is of suitably high quality and meets user expectations in regard to performance, load times, and user capacity.
2. Risk management
Non-functional testing also reduces the risk and cost associated with releasing a product to the market by maximizing the chances of the team releasing a satisfactory product.
By checking the non-functional parameters of the software build, it’s possible to reduce the costs of releasing the product because the need for further development and software changes later is reduced.
3. Optimization
Non-functional testing helps testers and developers to optimize the software build and optimize performance during installation, setup, execution, and use.
You can also use non-functional testing to optimize the way that you manage and monitor the software build.
4. Data collection
Non-functional testing allows testers to collect and produce measurements and metrics that can be used by testing teams for internal research and development.
You can use the data that you gather from non-functional testing to understand how your product works and how you can optimize it more effectively for users.
5. Knowledge enhancement
Non-functional testing improves and enhances the testing team’s knowledge of the product behavior and the technologies that it utilizes.
Not only does this help testing teams understand the software they’re working on better, but it may also provide useful knowledge that helps testers to understand future builds better.
Who is involved in non-functional testing?
Non-functional testing is usually carried out by testers in the QA environment, but sometimes developers may carry out non-functional testing during development.
System testing is almost always carried out by testers, and this is the phase of testing where most of the non-functional testing takes place.
If non-functional tests fail, testers will send the software back to developers to correct the errors in performance before testing again.
The benefits of non-functional testing
There are many benefits to carrying out non-functional testing, and non-functional testing is an essential step in system testing.
Without non-functional testing, testing teams would not be able to verify that the software actually meets the requirements of the client or that it meets the requirements set out in the software development plan.
1. Improve software performance
Non-functional testing can help testers and developers to improve the overall performance of software applications. Non-functional tests identify areas where software performance is lacking, for example in terms of loading speed or processing capacity, and prompt software teams to make changes to correct these defects.
This ensures that software teams only release software to the public when it’s ready, and when its performance is good enough.
2. Keep software secure
Non-functional testing includes security testing, which is critical in ensuring that a software build is secure and safeguarded from external threats and attacks.
Security testing enables testers and developers to check that software adequately protects confidential data and has sufficient security in place to guard against contemporary cyber attacks.
3. Increase the user-friendliness of the software
Non-functional testing is the best way to make your software more user-friendly, especially by carrying out usability testing that assesses how easy it is for users to learn how to use and operate your software.
User-friendliness is very important because it determines how satisfied users are with your software and ensures that users are capable of taking full advantage of everything your software offers.
4. Ensure that the software meets user needs
Ensuring that software meets the needs of users should be one of the top priorities of all software development and testing teams. As well as expecting the software to be functional, users expect that software performs well, runs smoothly, and safeguards confidential data.
Non-functional testing is one of the only ways to ensure that your software meets these requirements.
The challenges of non-functional testing
There are some disadvantages to carrying out non-functional testing. While non-functional testing is essential during the system testing phase of software testing, the process of non-functional testing can pose challenges to software teams that don’t have ample resources and tools.
1. Repetition
Non-functional testing in software testing must be carried out every time that software is updated by developers or every time code is changed. This means that non-functional testing can be very repetitive, which not only takes time but also tires testers out.
Tired testers carrying out very repetitive tasks are also more likely to get distracted and make mistakes.
2. Cost
Because non-functional testing is so repetitive, it can also be quite costly, especially for testing teams that rely on manual non-functional testing.
Software teams must allocate time and budget for frequent non-functional testing, and software developers will have to pay extra for this additional testing.
What do we test in non-functional testing?
Non-functional testing can be used to test many different non-functional parameters, each of which affects the quality and usability of the system. Each of these parameters is tested during system testing against criteria set out in the testing plan.
1. Security
Security testing is a type of non-functional testing that measures how well a system is safeguarded against external threats and attacks. These include deliberate security violations as well as data leaks and other common breaches.
Security testing is an important step in non-functional testing because it provides end users and clients with the reassurance that their data is secure.
2. Reliability
Testers use non-functional testing to assess the reliability of software and to ensure that the software can continually perform its specified functions without fail.
While functional testing ensures that software carries out its key functions, only non-functional testing actually tests the reliability and repeatability of these results.
3. Survivability
Survivability describes how a software system responds in the case of failure to function, and survivability testing ensures that if errors and failures do occur the system can recover itself.
Survivability testing may check whether the software is capable of saving data to minimize data loss in the case of sudden failure, for example.
4. Availability
The availability of the software refers to the degree to which the user can depend on the system during its operation. This is also called stability, and it’s tested by stability testing.
Stability testing bears some resemblance to reliability testing because it checks whether the system can perform to expected standards consistently.
5. Usability
Usability testing is another important type of non-functional testing in software testing. This type of testing assesses how well the user can learn, operate, and use the software system by following the instructions provided on the screen and other basic guides.
Usability testing is important because if the software isn’t highly usable, most users will simply abandon it or choose to use something else.
6. Scalability
Scalability testing tests to what degree a software application can expand its processing capacity to meet increasing demand.
For example, if the software is designed to be used by multiple users on a single network at once, how does it work when ten users log on at the same time? Do higher user counts significantly affect performance or loading times?
7. Interoperability
Interoperability testing is a type of non-functional testing that checks how well a software system interfaces with other software systems.
This is particularly important when software is designed as part of a suite of products that all integrate with one another.
8. Efficiency
Efficiency in software testing refers to the extent to which a software system can handle capacity, quantity, and response time.
For example, testers may assess how many users can log into the system at once, how long it takes to retrieve data from the database, or how quickly the software can perform basic tasks.
9. Flexibility
Flexibility measures the degree to which a software system can work with different types of hardware and peripherals.
For example, how much RAM the software requires or whether it requires a particular amount of CPU. The lower the requirements for the software application, the more flexible the software is.
10. Portability
Portability testing is used to test how flexibly the software can be transferred from its current hardware or software environment, and how easy this is to do.
Portability is important because it affects how easily end users can manage software and move it between different systems.
11. Reusability
Reusability testing is a type of non-functional testing that tests whether parts of the software system can be converted for reuse within another application.
While reusability testing doesn’t usually affect clients and end users, it is a good reflection of how effectively developers are creating components that can be reused in the future.
The characteristics of non-functional tests
Understanding what non-functional tests are involves understanding the characteristics of non-functional tests. These characteristics define non-functional testing in software testing.
1. Measurable
Non-functional testing is always quantitative and measurable, which means that testers don’t use subjective phrases like ‘nice’ or ‘good’, instead using numbers and facts to describe the outputs of non-functional testing.
For example, rather than describing loading times as ‘fast’ or ‘slow’, non-functional testing should result in specific figures showing the number of times.
2. Specific
When carrying out non-functional testing, the purpose of the tests should be specific to the design specifications of the software.
For example, if the software project plan refers to the number of users that should be able to log in at once, this should be prioritized when non-functional testing.
3. Unknown
While non-functional testing can be specifically designed to measure attributes set out in project plans, in many cases these attributes won’t be specified in advance.
In this case, testers should simply perform non-functional testing to assess the software based on each parameter and later compare these with expectations.
The life-cycle of non-functional tests
Since non-functional testing does not refer to a specific phase in the software testing life cycle but simply to a type of testing that usually occurs during the system testing phase of software testing, the life cycle of non-functional testing can vary a great deal between projects.
In general, it follows a similar life cycle to other types of software testing that starts with analyzing project requirements and ends with test execution and cycle fulfillment.
1. Software requirement analysis
The first stage of the life cycle for non-functional testing is the analysis of software requirements. Software teams work towards specific criteria when they’re building and testing applications, and these criteria should dictate what type of tests need to be carried out.
2. Test planning
The next stage of the life cycle is test planning. During the test planning stage, the QA lead will put together a detailed test plan that details what will be tested, who will carry out the testing, and what testing approaches, methods, and tools will be used.
The test plan should include all the necessary details that testers need to create and execute test cases.
3. Test case creation
Test case creation is the next stage of non-functional testing. This stage involves developing non-functional test cases that testers will execute at a later stage to test the non-functional requirements of the system.
Test cases describe what will be tested, how it will be tested, and what the expected outcome of the test is.
4. Test environment setup
The next stage in the non-functional testing life cycle is setting up the test environment before testing begins.
The testing environment is where all testing takes place, and it is home to resources and tools that you’ll use to execute non-functional tests.
The testing team prepares the test environment set up before test execution.
5. Test execution
Test execution is the next phase of the non-functional testing life cycle. It involves executing the test cases created previously to test different aspects of software applications including security, loading times, capacity, and portability.
The test team executes each case individually and checks the outcome of each test against the expected outcome.
6. Cycle repeat
The final stage of the non-functional testing life cycle is cycle fulfillment and repeat. After executing all the test cases, testers check to see which tests passed and which tests failed.
Tests that fail usually indicate that there is a defect that must be fixed by developers. Once developers have patched or edited the code, the software testing cycle repeats again until no defects are found.
Clearing up some confusion:
Non-functional testing vs functional testing
Functional testing and non-functional testing are two different but equally important types of software testing that, together, are used to assess whether a software application meets the requirements of users as set out in the project brief.
While they’re both necessary types of testing that allow software teams to identify defects within software builds, functional and non-functional testing are completely distinct from each other.
1. What is the difference between functional and non-functional testing?
The difference between functional and non-functional testing is in what they test. Functional testing tests the functions of the application and checks whether they work as expected. Non-functional testing tests other aspects of the application that affect user satisfaction and application quality.
Functional and non-functional testing occur in different phases of software testing, but both types of testing are usually carried out during the system testing phase.
Both functional and non-functional testing can help us to understand how well an application works and whether it performs its job adequately.
For example, if you’re testing a mobile app that allows users to save to-do lists and shopping lists, functional testing may test functions like creating a new list, saving a list, and making edits to existing lists.
Non-functional testing may assess how well the application works on different mobile devices, how quickly lists load up, and how much app performance is affected when other apps are running in the background.
2. Conclusion: non-functional testing vs functional testing
Both functional and non-functional testing are important types of software testing that can help testers and QA teams to assess whether an application meets its current requirements.
While functional testing tests the functions of the software, non-functional testing tests other aspects that might affect performance, efficiency, and security.
Unit testing, integration testing, and API testing are all forms of functional testing. In each of these stages of software testing, testers assess how well functions and features work either individually or together and identify bugs and defects that prevent functions from operating as expected.
Security testing, usability testing, portability testing, and load testing are all forms of non-functional testing that allow testers to assess how well an app performs its functions and supports users’ needs.
Types of non-functional testing
There are lots of different types of non-functional testing, each of which tests a different non-functional aspect of a software application’s performance or efficiency.
Each of these types of tests will test different parameters, and some tests may test the same parameters in different ways.
1. Performance tests
Performance tests are a type of non-functional test that checks how well different software components work. Rather than testing their functionality, which is what functional testing does, performance tests may test response times, bottlenecks, and failure points. Performance testing helps testers to ensure that software is of high quality and that it is fast, stable, and reliable.
2. Stress tests
Stress testing is a type of non-functional testing that tests how well the software performs when it’s put under abnormal amounts of stress. This could mean testing how the software performs when someone tries to use lots of different features at once, or while running many other applications at the same time.
Stress testing seeks to identify the limit at which the software stops working properly and what happens when the system is under stress. It allows testers to understand whether the system can recover itself and whether it notifies users with appropriate error messaging.
3. Load tests
Load testing is a type of testing that assesses how well software behaves under both normal conditions and when dealing with heavier loads. It’s used to determine how much the software can handle simultaneously without performance being negatively affected.
Load testing can be used to test how applications work when lots of users are using them at once or when users try to download lots of data at the same time.
Load testing is important if you want to check whether your software is scalable.
4. Security tests
Security tests evaluate software applications and look for vulnerabilities in the software’s security. These include potential security risks that could result in data loss or breaches that expose confidential data.
Security testing is important because it ensures that the product is adequately protected from hacking, data breaches, and other external security threats.
Some examples of security tests that testers may perform include security audits, ethical hacking, penetration testing, security scans, and posture assessments.
5. Upgrade and installation tests
Upgrade and installation testing is a type of non-functional software testing that verifies how well the software works on different machines.
The purpose of this type of testing is to make sure that new users can easily install the software on their machines and existing users can upgrade it when new upgrades are released.
Upgrade and installation testing are important because end users must be able to easily install your product provided they’re working with a machine that’s compatible with it.
6. Volume tests
Volume testing is a type of testing that exists to verify what happens when large volumes of data are added to the database at once. This identifies whether the application can handle large volumes of data and what happens to the system if it can’t.
Volume testing is also known as flood testing, and you can use it to assess data loss and error messages that occur when adding significant amounts of data to the system.
Volume testing is the only way to ensure that software can handle the volumes of data that users will expect it to.
7. Recovery tests
Recovery tests involve forcing the software system to fail to test how well the system recovers itself after a crash.
Recovery testing can help testers to understand how software recovers data and prevents losses if the hardware is unplugged during use, if the software is disconnected from the network during a data transfer, or if you restart the system unexpectedly.
This type of testing is important because systems without proper recovery protocols could suffer serious data losses when accidents like this occur.
What you need to start non-functional testing
Before you start non-functional testing, you’ll need to make sure that you have prepared the testing environment and gathered together the tools and data that you require.
1. Test plan
Before you start non-functional testing, make sure that you have a finished test plan that’s been signed off by the appropriate people.
Your test plan should include all the relevant details of what you’re going to test and how. It should explain when you’re going to use manual testing and when you’re going to use automated testing, as well as outline the roles and responsibilities of everyone involved in the testing process.
2. Test cases
Before you can execute non-functional tests, you’ll need to create test cases. Each test case outlines a specific thing that you’re going to test, explains how you’re going to test it, and describes the expected result of the test.
For example, if you’re carrying out load testing, an example test case may be testing how the software behaves when ten users are using the same module at the same time.
3. Functional verification
You can’t perform non-functional testing on software components if they’re not functional.
For example, if you want to test how many users the software can handle being logged in at the same time, it’s first important to verify that individual users can actually log in to the software.
Before you start non-functional testing, make sure that all your functional tests have passed and that your software functions as you expect it to.
This usually means that smoke testing, sanity testing, unit testing, integration, and functional system testing have already been carried out.
4. Testing tools
Before you start non-functional testing, gather together any testing tools that you want to use to conduct your tests.
Whether you’re using automation tools to partially automate some of your tests or documentation tools to help you manage and store test reports for later use, make sure that the tools that you want to use are available and ready to use, and that everyone in the testing team knows how to use each tool properly.
5. Testing environment
Set up the testing environment before you start non-functional testing. You may already have a suitable testing environment, especially if you can use the same environment for non-functional system testing and functional system testing.
The ideal testing environment allows you to test every element that you need on the correct devices.
For example, if you are testing volume handling on smartphone devices, it’s better to test this on an actual smartphone device than to try and emulate a mobile environment on a desktop computer.
The non-functional testing process
Testing non-functional aspects of a software build is a multi-step process that involves readying the test environment, creating test cases, gathering test data, and executing non-functional tests.
It’s possible to break the testing process down into small chunks to make it easier to follow for beginners in non-functional testing.
1. Non-functional testing readiness checks
Before you can start non-functional testing, it’s important to verify that you’re ready for this phase of testing.
This may mean assessing the exit criteria for the last phase of testing to make sure that the software has passed it and making sure that the software has passed all the functional tests required before non-functional testing takes place.
Some teams may create entry criteria for non-functional testing, which comprises all of the conditions that must have been met before non-functional testing begins.
2. Create a test plan
You may have already carried this step out earlier if you’re performing non-functional testing as part of system testing and following your system testing plan. A test plan outlines all the tests that you need to carry out and how you’re intending to carry them out.
Without a clear test plan, it’s easy to lose sight of the scope and aims of the tests that you’re performing.
3. Create test cases
The next stage of non-functional testing is creating test cases designed to test each non-functional parameter of the software.
Each test case should have a test case ID, a test case name, a description, and details of the expected outcome of the test as well as any pass or fail criteria that are determined in advance. This helps testers to understand how to carry out each test and what results to look for.
4. Gather test data
Before you can execute each test case, you’ll need to gather the test data that you’ll use for each test case.
This usually means collecting code and data from different modules and components that make up the functions and areas that you’re going to test. If you’re maximizing test coverage, you should have a lot of test data to work with.
5. Ready the test environment
The next stage of non-functional testing is readying the test environment. The test environment is a testing server that you’ll use to conduct software testing of many different types.
It allows you to create identical conditions in which to test your software and to set your software up with different configurations for configuration testing, security testing, and other types of non-functional testing.
6. Execute non-functional tests
Once the test environment is ready, it’s time to execute non-functional tests. You may decide to execute tests in order of type, for example starting with performance testing before moving on to security tests and other types of non-functional tests.
As you carry out each test, make a note of the results in your testing report. If you’re automating testing, your automation tool will have a standardized way of reporting results clearly and unambiguously as well.
7. Report test results
After executing each test case, compile the results of your non-functional tests into a single report.
This report should be clear about the results of each test and unambiguous about whether each test has passed or failed.
Follow a standardized structure for your test report to make sure that all information that you’ll need to pass on is included.
8. Fix defects
Once the test results are in, hand the software back to developers if tests failed or if you identified any non-functional bugs that need fixing.
For example, if the software doesn’t handle a suitable number of users at once or if performance slows down too much when multiple programs are running at the same time, these issues will likely need to be fixed within the code to ensure that users are happy with the product.
9. Repeat testing cycle
Once developers have repaired any defects found within the initial non-functional testing phase, the testing cycle can begin again.
Developers will sanity test the changes that they make and pass the new build back to QA testers, who will then carry out the full suite of testing starting with smoke testing, unit testing, integration testing, and finally system testing.
The testing cycle is repeated until no bugs or defects occur at any point, after which the build can enter the final stage of testing: user acceptance testing.
Test cases for non-functional testing
Test cases are an important aspect of all software testing, and when you’re carrying out both functional and non-functional testing you’ll use test cases to define what you’re going to test and how you’re going to test it.
Each test case can be seen as a mini-test, and each test case will have its own defined outputs and results.
1. What are test cases for non-functional testing?
A test case is a set of actions performed on a software build to test whether it satisfies the conditions defined in the software plan. Each test case effectively tells testers what to test and how, and is designed to test a specific function or non-functional feature of the software application.
Non-functional test cases might include testing what happens when someone tries to access secure data within the system or testing how quickly the software loads on start-up.
2. How to design non-functional test cases?
When you’re designing test cases for non-functional testing, it’s important to follow standard test case practices while keeping the objectives of your non-functional tests in mind.
Follow the steps below to write test cases for non-functional testing that clearly outline what your testers must do to perform each test.
1. Define the area you want to cover
For each test case, consider what area of your software this test case is going to cover.
For example, if you’re writing test cases for installation and upgrade testing, you might include test cases that assess how easy the application is to install on different devices and how long it takes to upgrade the software using a new patch.
2. Create a unique test case ID
Each test case should have a unique test case ID. This makes it easy to find the test case description and results later and clears up any confusion over which test case you’re referring to if two test cases have similar names or descriptions.
3. Name and describe each test
While the test case ID identifies the test, you’ll also want to provide a name and a description for each test case that you write.
It should be a simple name that summarises what you’re testing, while the description is a single sentence that elaborates on this in slightly more detail.
The description should be clear enough that testers know what to test and how to test it, as well as any particular conditions that need to be met in the test.
4. Specify the expected result
For each test case, outline the result that should occur if the software is working as expected.
In non-functional tests such as performance testing and load testing, this may mean in many cases that the software simply continues to operate normally without slowing down, lagging, or crashing.
In other cases, it may mean that particular error messages occur to notify the user of the problem and recommend a solution.
5. Recommend testing techniques
For each test case, recommend the type of testing techniques and non-functional testing tools that you think the tester should employ during testing.
In non-functional testing, testers may use very different approaches for different types of testing.
For example, load testing and stress testing may require automation because it’s impractical to simulate extremely heavy traffic manually, while other testing types may be easier to carry out without any specific tools or technologies.
6. Get each test case peer reviewed
Before you sign off on each test case, get each case peer-reviewed by someone that you work with. This may be another tester or a QA lead.
Peer-reviewing test cases ensure that they’re clear enough to be followed by a third-party tester and do not include any ambiguities or mistakes that could lead to improper testing.
3. Examples of non-functional test cases
If you’re writing test cases for non-functional testing, they might look something like the non-functional testing examples below.
Scalability testing example
Test case ID: 6671
Test case name: Multiple user login test
Description: Emulate 20+ users logging in to the software at the same time using automation tools.
Expected results: The software should run as normal for each user, allowing each user to log in successfully in under 5 seconds.
Compatibility testing example
Test case ID: 5214
Test case name: Loading the application in the Opera browser
Description: Load the application in the Opera web browser.
Expected results: The application loads as normal in the Opera web browser with standard display resolution and layout.
Manual or automated non-functional tests?
When you’re choosing between different non-functional testing techniques, you’ll have to decide whether you want to carry out manual or automated non-functional tests.
Manual tests are performed by human testers, which means that they’re usually more time-consuming to carry out, but they also offer opportunities for exploratory testing.
Automated non-functional tests are quicker and, in some ways, more reliable, but they also require more resources or tools. Automation and hyperautomation is becoming increasingly popular in testing, especially when it comes to non-functional testing.
Manual non-functional testing: Benefits, challenges, and processes
Manual non-functional testing is carried out solely by testers, who will test each individual non-functional element independently.
When conducting manual non-functional tests, testers must gather information about the software, create individual test cases that match the test plan, and execute those test cases manually.
This takes considerable time, but it also means that QA testers have the freedom to determine what is tested and how.
1. Some of the benefits of manual testing include:
● Manual testing may be cheaper than automated testing because it doesn’t require specific technologies or technical knowledge.
● Manual testing allows testers to offer human insight and subjectivity into how the software functions and whether it functions satisfactorily.
● Manual testing can be used to carry out system testing in scenarios where it’s impossible to automate.
● Manual testing allows testers to assess the visual aspects of the system such as the graphical interface and other factors that might affect the usability.
● Manual testing offers testers a broader perspective of the system as a whole and how different modules and components work together
However, there are also drawbacks to manual testing.
2. Some of the challenges of manual testing include:
● Some types of non-functional testing, including load testing and performance testing, are impractical to carry out manually
● Manual testing takes considerably more time than automated non-functional testing
● Manual testers may get distracted, lose focus, and make mistakes, especially when carrying out very repetitive testing tasks
Automated non-functional testing: Benefits, challenges, and processes
Automated non-functional testing is carried out by automated scripts and testing tools. When using automated testing methods, testers can conduct tests in the background while getting on with other tasks, once the automated tests have been initiated.
1. Some of the benefits of automating non-functional tests are:
1. Save time and resources by reducing the amount of time you spend on lengthy, time-consuming tasks
2. Automation makes it possible to increase test coverage by covering a wider range of components and features
3. It’s more feasible to conduct automated tests frequently because they take less time to perform
4. Automated testing is ideal for time-consuming testing tasks like load testing, volume testing, and stress testing which are very hard to conduct manually
5. There is a lower chance of errors arising when performing automated testing
However, there are also some drawbacks to automated testing which means it’s not always the right approach for all types of non-functional testing.
2. Some of the challenges of automated non-functional testing include:
1. Automated testing is more expensive to set up than manual testing
2. Setting up test automation can take time and technical resources
3. Test automation doesn’t allow space for exploratory testing
4. Automating tests still requires time for the creation of test cases
Conclusion: Manual or automated
non-functional testing?
In most types of software testing, combining manual and automated testing usually offers the best results. This allows testing teams to benefit from the efficiency, reliability, and accuracy of automated testing while also carrying out exploratory testing that enables testers to assess software from a more subjective perspective.
In non-functional testing, both manual and automated testing are virtually necessary for most testing teams.
Manual testing is best used to carry out non-functional testing tasks like usability testing, while automated testing is most often used to carry out tests that would be too time-consuming and difficult to conduct manually, such as stress testing or volume testing.
Non-functional testing is one of the most obvious areas to use test automation techniques because it is a quantitative, measurable type of testing that doesn’t ask for subjective results.
As with other types of testing, non-functional testing is usually carried out using a mixture of manual testing and automated testing.
However, automated testing is practically necessary for many types of non-functional testing, and the parameters and metrics of non-functional testing mean that automation is more suited to this type of testing than functional testing.
Best practices for non-functional testing
When you undertake non-functional testing for the first time, following testing best practices can help you to standardize your testing process and optimize the efficacy of your tests.
Best practices serve as guidelines for software testing teams that want to improve testing processes and align with industry standards.
1. Use automation tools
In non-functional testing more than other types of testing, it’s important to use automation tools to automate certain types of testing, specifically volume testing, stress testing, and load testing.
These types of testing usually verify how well software operates under heavy pressure from users, data, and traffic, which can be very difficult conditions to emulate manually.
Automating these types of non-functional tests will not only be more efficient but also more accurate and allow testers to replicate higher loads and stresses easily.
2. Peer review all documentation
As well as asking peers to review test cases that you create, ask peers within your testing team to review bug reports, test reports, test plans, and other forms of formal documentation created during the testing process.
This reduces the risk of small mistakes that could cause serious delays in the testing and development process.
3. Define measurable requirements
When you define the requirements of your software before non-functional testing begins, make sure that each requirement is objective and measurable.
This makes it easier for testers to ascertain whether the software meets these requirements during testing and leaves no room for interpretation.
What counts as ‘fast’ or ‘efficient’? Use numbers and quantitative values to define what you’re looking for.
4. Carefully consider testing metrics
Before deciding which metrics you’re going to use to measure the performance of your software, consider what the software’s users will want and which metrics actually align with the software plan and requirements.
Most software should be fast and reliable, but what other metrics might your users look for? Are there any software-specific metrics that you should consider during the testing process?
Types of output from a non-functional test
When you’re carrying out non-functional testing, you’ll receive different types of outputs from the tests that you perform.
These are usually quite different from functional testing outputs, which are often clearer cut because functional tests simply test whether a function is working as it should or not.
Like functional testing, testers should set clear expectations for each test case that make it easy to determine whether each test passes or fails.
1. Absolute numbers
When carrying out performance testing, stress testing, and other types of non-functional testing, the outputs that you might be looking at most often are speeds and other absolute numbers.
Performance testing verifies how promptly the system can carry out certain tasks, and this will be measured in seconds or milliseconds.
If you’re carrying out load testing, you may assess how much data the software can handle at once without crashing or lagging.
2. Error messaging
Non-functional testing also verifies how the system works when errors occur, such as security errors, validation errors, and configuration errors.
It’s important that systems display accurate and clear error messages when errors do occur so that users can take steps to correct the problem and continue using the software.
Error messages should also occur during security testing when the system prevents users from breaching the software’s in-built security features.
3. Crashes
Crashing is a sign of system failure, and it usually indicates that the system isn’t able to perform at the level you’re testing and may mean that the test passes.
In some cases, the system might crash and yet still pass the test case you’re working on, for example, if the system withstands the required amount of stress or traffic before crashing.
When performing non-functional testing, testers should expect the system to crash regularly, especially when pushing it to its limits for stress testing and other performance tests.
Examples of non-functional tests
Non-functional test examples are similar to the examples above for non-functional test cases.
You can look at examples of non-functional tests to understand better what non-functional testing is and what it tests within a software application.
1. Performance testing example
If you’re working on a mobile application that connects users to an online database, it’s important that a large number of users can access and download the data from this database at the same time.
This is also a key part of scalability testing, particularly if you want to increase the number of users on the app in the future.
You will then test how the system reacts when, for example, 1000 users try to access the same database at the same time and set requirements for how quickly the application should load under this condition.
2. Compatibility testing
If you’re testing a new document management application, you’ll need to test that it works across all of the devices that it’s intended for.
This means testing that you can install and load the application on all of the most recent versions of Windows, Mac, and any other operating systems (such as Linux) that you want the software to be compatible with.
3. Security testing
When you’re carrying out security testing, you’ll test some of the ways in which people may try to access confidential data or breach the software’s security safeguards to verify that the system behaves as you expect it to in these situations.
For example, you might log in as a user and try to access files that you don’t have security clearance for to make sure that the system doesn’t let you access these files.
Types of errors and bugs detected
through non-functional testing
Non-functional testing can reveal a lot of bugs and defects that aren’t as easy to find as those identified in functional testing. That’s because non-functional testing often requires that testers verify different configurations, setups, and combinations of conditions to assess how well the system performs in a myriad of different settings.
1. Performance defects
Performance defects arise when the system functions, but it doesn’t function as quickly or as efficiently as you expect it to.
For example, you might find that the system doesn’t load quickly enough under certain conditions or even crashes if too many users log in at the same time.
Performance defects don’t completely prevent people from using your software, but they can make your software less usable and less likely to fulfill user requirements.
2. Security defects
Security defects are those defects that impact the security of your software system and the data stored within it.
Security defects may arise if, for example, users can access confidential data that they should not have access to or if certain parts of the application are not correctly password-protected, or if encryption fails.
These could result in security breaches, which can have a serious impact on a software publisher’s reputation.
3. Functional defects
While non-functional testing isn’t designed to test the functions of a software application, in some cases non-functional testing can identify functional defects within the software.
For example, the purpose of reliability testing is not to test whether the app functions but to test whether the app functions reliably on repeated attempts.
This may reveal that some features don’t reliably function properly when an action is repeated, and these may be classed as functional errors.
Common non-functional testing metrics
Non-functional testing metrics describe those metrics by which the performance and efficiency of the system are measured.
Different types of non-functional testing rely on different metrics, and you can choose to use a variety of metrics depending on the final objectives of the project.
1. Time
Metrics of time measure how long it takes to perform certain tasks or how long users have to wait for functions to load up.
Some examples of time metrics are the number of transactions or downloads that an application can make within a given time frame, the response times of different functions, and the time it takes for the application to complete a particular operation.
Different types of tests will measure results in seconds or as a presentation of how many operations per second.
2. Space
Space is another important metric in non-functional testing. Space metrics may test how much CPU space the system requires or how much space on the hard drive the software takes up once it’s fully installed.
Some examples of space metrics include cache memory, main memory, and auxiliary memory.
Software that requires a large amount of space to run smoothly may be suitable for a smaller number of customers.
3. Usability
Some metrics in non-functional testing take into consideration the usability of the system, for example, how long it takes to train users to use the system properly, how many choices users have to navigate through to perform key functions, or how many mouse clicks it takes to carry out certain tasks.
Non-functional tests can measure each of these metrics quantitatively, with lower numbers generally implying higher levels of usability.
4. Reliability
Another important metric in non-functional testing is reliability. Reliability reflects the likeliness that the system behaves the same way time and time again or functions as it should for a long period of time.
Some examples of metrics that are used to measure reliability include mean time to failure, failure rate, availability, and downtime probability.
Each of these metrics helps testers to verify that the system can run for a long time without experiencing failures or crashes.
5. Robustness
Robustness is the measure of how well the system handles failures and how well the system can recover itself in the event of failure.
Some examples of metrics that measure robustness include the time it takes for the system to recover itself after a failure, the percentage of incidents that lead to catastrophic failure, and the probability that data files are corrupted after the system fails.
These are important metrics because users expect that systems may sometimes fail without losing all data or corrupting files.
6. Portability
Portability metrics measure how easily the software can be transferred to different systems or moved to a new location within a network.
Some examples of metrics that measure portability include the percentage of non-portable code and the number of systems on which the software can run.
Ideally, software that can run on lots of different systems is more portable and therefore more convenient for use in settings that might require frequent transfers or relocations.
Strategies for conducting non-functional tests
When you begin non-functional testing, it’s important to approach this phase of testing with a strategy in mind. QA leads and software testing managers should consider the risks of testing, the resources they have available to them, and the purpose of testing before non-functional testing starts.
Developing a strategy can help you to optimize your non-functional tests from the outset.
1. Allocate roles and responsibilities
Before you start non-functional testing, allocate roles and responsibilities to the key members of the testing team. This makes it easier to manage the workload of non-functional testing and to ensure that experienced testers are responsible for maintaining the quality and effectiveness of the tests that you conduct.
Make sure that the people you choose to take on these roles have the knowledge and experience that they need to carry out the tasks you expect them to, particularly if those tasks require technical skills.
2. Gather relevant testing tools
Gather together all of the technologies and tools that you want to use to carry out non-functional testing. Make sure that all of your team know how to use them effectively, and conduct training to fill skill gaps where necessary.
Making sure that everyone knows which testing tools to use and how to use them before non-functional testing starts reduces the risk of having to pause testing or re-do tests because of insufficient knowledge.
3. Prioritize testing
Before you start non-functional testing, make a list of all the aspects of the system that you need to test and prioritize them based on urgency and importance.
You can prioritize non-functional testing based on the level of risk involved in each aspect of the system that you’re testing.
For example, basic security testing might take place because adequate security is considered extremely important in modern software. The earlier you identify high-risk defects, the lower the potential impact of those defects can be other aspects of the system.
7 best non-functional testing tools
Non-functional testing tools can streamline the testing process, make it easier and more cost-effective to automate testing and help QA leads manage the testing and documentation process.
There are a lot of free non-functional testing tools available online, as well as some tools that you can pay a monthly fee to upgrade.
1. ZAPTEST FREE edition
ZAPTEST is a popular software testing tool that allows users to carry out both functional and non-functional software tests quickly and easily. You can use ZAPTEST to automate software tests and utilize RPA technology to emulate various functions and conditions in non-functional testing.
The ZAPTEST FREE edition is just a pared-down version of the enterprise edition, offering many of the same functionalities on a smaller scale. You can seek support on the ZAPTEST forum and carry out performance tests with unlimited virtual users.
2. Appium
Appium is a free software testing tool that’s most suitable for testing mobile applications across a variety of different platforms including both iOS and Android devices. Appium offers users a lot of flexibility to devise their own testing frameworks and strategies while benefitting from the automation abilities that Appium offers.
3. Loadium
Loadium is a non-functional testing tool that’s best used to carry out performance testing and load testing, two types of non-functional testing that are much easier to carry out using automation tools.
Loadium allows users to run large-scale load tests and offers custom solutions so that you can tailor your tests to your software goals.
You can try Loadium for free or pay to download the full version of the application.
4. Obkio
Obkio is a software testing tool that helps QA leads and test managers to prioritize and categorize issues based on how serious they are. Obkio can detect problems before users do, offers users smart notifications, and can help to detect where the problem is.
Obkio isn’t just for non-functional testing, but it’s a highly useful free testing tool that can be employed at all stages of the testing life cycle.
5. SonarQube
SonarQube is an open-source security testing tool that can automatically analyze code to detect bugs and vulnerabilities. Written in Java, you can use SonarQube to analyze code in over twenty different programming languages and the system’s clean interface makes it easy to detect issues that could cause security vulnerabilities in the future.
6. Tsung
Tsung is another non-functional testing tool that’s ideal if you want to automate load and stress testing but don’t get on with the free version of Loadium.
Tsung is an open-source tool that allows users to carry out high-volume load testing across multiple protocols and servers including HTTP and SOAP.
Tsung is completely free and it can help testers to ensure that the software they’re working on offers high-performance levels under a variety of challenging conditions.
7. Sikuli
Sikuli is another application that uses robotic process automation to automate the testing process. The application can automate anything that can be seen on the screen. You can use Sikuli to test non-web-based applications and to quickly reproduce bugs.
Non-functional testing checklist, tips, and tricks
Before you start non-functional testing, check that you’ve got everything you need to carry out thorough non-functional tests in a prepared environment.
Follow the checklist below for tips and tricks before starting non-functional testing.
1. Work to a schedule
Whether you include it in your test plan or create a separate document for it, structure your software tests around a test schedule.
If you find more bugs and defects than you expect to, you might stray from the schedule sometimes, but having a schedule to start with can help to guide testers and motivate them to work efficiently, especially when carrying out time-consuming manual tests.
2. Identify your test team
Delegating responsibilities and assigning members of your test team with official roles and titles can help make sure that the testing process goes smoothly.
Clearly communicate roles and responsibilities within your team before testing starts, and assign different testers with the responsibility for different aspects of non-functional testing so that everyone is held accountable for their own tasks.
3. Select tools and technologies before testing
If you only decide to work with particular tools and technology after you start non-functional testing, this can hold up the testing process and cause confusion among testers.
Instead, do your research in advance and decide whether there are any tools that you want to utilize before testing begins. This makes it easy to incorporate these tools into the test plan and to train your testers to use them before testing starts.
4. Always obtain formal sign-off on tests and documentation
Testing is a quality assurance process, and the best way to maximize the value of the tests that you carry out is to perform basic QA on the tests you plan and run as well.
Introduce simple protocols that require testers to ask QA leads and managers to review and sign off test plans and test reports before moving on to the next stage.
This massively increases the chance that testing mistakes are caught and fixed early.
7 mistakes and pitfalls to avoid when implementing non-functional tests
If you’re new to non-functional testing, it can be easy to make some common mistakes that testers and QA professionals often fall into.
Non-functional testing is a complex job that involves considering a software build from all angles and perspectives.
Below is a list of some of the most common pitfalls that testers make when performing non-functional testing.
1. Not planning
If you’re new to non-functional testing, you may think that you can just dive right into the testing stage without creating a thorough test plan in advance.
Some testing teams may put together incomplete testing documents or superficial summaries of the test plan which don’t adequately outline the actions that testers must take during non-functional testing.
2. Testing mismanagement
Problems can arise if tests are mismanaged at any stage of the testing process. Inadequate management may mean that testers don’t have the proper resources to carry out testing thoroughly or that testers aren’t given sufficient time to test each aspect of the build.
Test managers must be able to learn from the mistakes they make and develop more effective testing plans going forwards.
3. Poor communication
Poor communication can cause a lot of problems during the testing process, specifically within non-functional testing.
This could mean poor communication within the testing team or poor communication between testers, developers, and stakeholders.
This often occurs when testers don’t maintain test documents adequately or don’t communicate regularly with other departments during the testing process.
4. Ignoring developers
Testers and developers usually work quite separately from one another, but testing teams that work closely with developers can benefit from additional knowledge of how the software operates and how different modules interact with one another.
Involving developers in the testing process, or requesting feedback from developers at key times, can help testing teams put together more efficient and thorough test plans.
5. Purpose of testing
Many testers still believe that the purpose of testing is to check that the software works or to demonstrate to stakeholders and investors that the software works.
Instead, testers should approach testing with the attitude that the purpose of testing is to look for defects.
Testers who don’t find defects can be happy that the software they’re testing is free of bugs only if they are satisfied that they’ve looked everywhere that defects might be found.
6. Manual vs automation errors
It’s important to spend time considering whether manual testing or automated testing is better for each type of testing that you carry out.
Automated testing methods are extremely well-suited to almost all forms of non-functional testing, and testing teams used to functional testing may make the mistake of assuming that they can test non-functional features manually just as easily.
7. Using the wrong testing tools
It’s easy to choose the wrong testing tools and technologies before starting non-functional testing, especially if testing teams are used to carrying out manual testing and are not accustomed to using testing tools.
Research the non-functional testing methods that you want to use in advance and choose software tools and automation tools that meet the specific requirements of your project.
Conclusion
Non-functional testing is an essential step in the testing process that enables testers to verify how well a system performs and to what extent it meets non-functional requirements such as loading times, capacity, and security safeguarding.
There are lots of different ways to carry out non-functional testing, but contemporary automation tools make it easier to maximize test coverage and accuracy without compromising the quality of your results.
FAQs and resources
If you want to find out more about non-functional testing, there are lots of FAQs and resources available online.
Browse our favorite online non-functional testing resources below or read the answers to some of the most frequently asked questions about non-functional testing.
1. Best courses on non-functional testing
There are lots of courses available online that could help you to expand your knowledge of non-functional testing methods and approaches.
Some of these courses are available for free, and others may offer a certificate or qualification in return for a fee. If you want to undertake an accredited course, you could ask your employer if they will sponsor you and cover the cost of tuition.
Some of the best courses on non-functional testing include:
- TSG: Non-Functional Training 2-Day Course
- Udemy: The Complete 2023 Software Testing Bootcamp
- Edx: Software Testing Professional Certificate
- Educative: Performance Test Automation 101
2. What are the top 5 interview questions on non-functional testing?
If you’re preparing for a job interview to work in software testing, your interviewer may ask you questions about non-functional testing to make sure that you understand how this essential stage of software testing works. Prepare for your interview by readying effective answers to common interview questions in advance.
● How might the approaches and methods that you use in non-functional testing differ from the approaches you use in functional testing?
● How does non-functional testing differ from functional testing?
● What different types of non-functional testing exist?
● How do you prioritize functional tests and test cases?
● At what stage of software testing is functional testing usually carried out?
3. The best YouTube tutorials on non-functional testing
If you prefer to learn by watching videos, you might find YouTube tutorials on non-functional testing a useful way to learn more about this type of software testing.
Below are some of the best YouTube tutorials on software testing available today.
What is Non-Functional Software Testing? A Software Testing Tutorial
Software Testing Help: Non-Functional Testing
Non-Functional Testing in Software Testing
Visit W3Schools
Functional and Non-Functional Testing
4. How to maintain non-functional tests
Proper test maintenance ensures that software tests can be repeated without compromising on the quality of test results.
By maintaining non-functional tests, you can ensure that testing at each stage of the development process is adequate and that your tests are always updated in line with constantly changing code.
You can maintain non-functional tests by following our tips below.
● Communicate clearly across the testing team when creating test cases and writing documentation
● Always follow best test design practices
● Re-evaluate testing protocols at various stages of the testing process
● Update changes in your test as you go
Consider future projects when making changes to current tests
5. Is non-functional testing black box or white box testing?
Non-functional testing is a type of black box testing, which means that testers aren’t concerned with the internal workings of the system, but only with its external outputs.
This contrasts with white box testing, which tests the way the system works internally. Examples of white box testing include unit testing and integration testing.
Functional and non-functional requirements testing are examples of black box testing. This means that testers do not require advanced technical skills or knowledge of computer programming to carry out black box testing, nor do they need to learn how to implement the systems they’re testing.
