Friday, August 16, 2013

Testing primer

Introduction

I am making this blog post because we have time and again seen that our quality of unit testing, and unit testing coverage has been constantly reducing over the years. This is primarily because people don't see the value in writing unit tests. This post is a primer on testing so the developers understand wht testing is important

Definitions

First to get every one on the same page, these are the definitions of the terms that are commonly used when we talk about testing

Testing

First of all, what is testing? When we say we are testing a particular piece of software, or module or code, what does it mean? Testing simply means that verifying that a particular system is behaving as is expected. This is not limited to software industry. For example, before you buy a car, you take it for a test drive because you expect to be able to drive the car, and you are checking whether you can really drive the car. In terms of software, generally the behavior of software is defined by it's inputs and outputs. For a specified input, you have a specified output. Testing is simply verifying that the real output matches the expected output 

Whitebox testing

What is whitebox testing? Well really, it should be called transparent (or glass) box testing. This kind of testing is done by someone who takes into account the internal structure of the code. You can think of a module being inside a glass box, and the tester can see the entire structure of the module. As the tester provides input, differrent paths through the module will get activated. The goal of the test is to provide enough inputs to make sure all paths are activated

Black-box testing

What is blackbox testing? Here. you assume that the component/module is in a black box. You can't see inside it. All you have are the requirements, which lists all the possible inputs and an output for each input. In blackbox testing, you simply provide the possible inputs and check if the output matches the expected output

Unit testing

Unit testing means that a particular unit of code is tested in isolation. Academically speaking, an unit test for a particular module, should activate pieces of code only within the module being tested, and nowhere else. This means that if one module A is dependent on other modules X, Y and Z, the tester will have to mock the behavior of X, Y and Z and not use the real X, Y and Z. However, practically,  most developers do not mock dependent modules, and use the real modules instead. This causes problems with getting 100% coverage because certain paths through the code (for example validation and exception handling) may not be activated. With the advent of mocking frameworks, and DI frameworks, writing mocks is becoming easier, and developers are encouraged to learn these frameworks
Unit testing can be either whitebox or blackbox. However, most unit tests tend to be whitebox. Usually, unit teswts are written by the developers themselves, but developers tend to not like writing unit tests.

Integration testing

Integration testing is any testing that is done at at a system level. Basically, you put all the modules together and you test all of them together (as opposed to unit testing where you test one module at a time). Generally, integration testing tends to be black box testing because a)it is difficult to activate all paths through the system b) the inputs and outputs are much more well defined at the system level. Integration testing is generally done by QA, but the general shift in the industry is to move more integration testing towards the developers

Functional testing

Basically, functional testing is a kind of integration tests the function of the system. It focuses on the core business requirement, and tests if the application meets the business requirement. Generally functional testers tend to work closely with Business Analysts to understand the system (and might even participate in requirements gathering and definition) At my current employer, this adds an unique flavor to how things are done. The systems that we develop specify business needs in an industry that is very complicated. Finance has it own rules, and jargon, and generally, the developers don't understand the requirements as well as business users do. So, to test the function of the system, we need people who understand the business need very well. This is why our functional testers tend to be less "technical" in software development, but very "technical" in finance. Their core skill set is completely differrent than developers

System testing

System testing is an integeration test that tests the system as a whole. You not only test the software itself but all the various processes that go around using and maintaining the software. You test whether it installs correctly, whether it meets performance requirements, whether it scales well, whether it integrates with third party products well, whether it breaks under load, whether it breaks if user provides weird input and/or behaves like a moron. It basically covers everything that Functional testing doesn't. Good system testers tend to be much more technical than functional testers, and have a deeper understanding of software works. They tend to be people who like to understand how things work. As kids they generally enjoy taking radios and clocks apart just to see what's going on inside (and never put it back again). 

Happy path testing

This is a contentious term. Happy path testing is generally understood to mean that the code is tested enough to make sure the basic functionality is tested. The problem is people don't agree on what is basic functionality. Is validation basic functionality? Is error handling basic functionality? How "wide" is your happy path? Generally developers tend to err on the side of narrower happy paths, and you tend to have "happy balancing wire" (the test only tests the software in this narrow set of conditions)

Smoke testing

No, this is not  kind of testing that you do after your smoke break. This term possibly comes from the plumbing industry but most people believe it comes from electronics. It makes more sense in electronics, where one wrong connection can blow up the whole circuit. Basically "You plug in a new board and turn on the power. If you see smoke coming from the board, turn off the power. You don't have to do any more testing" You are basically testing that you haven't blown up everything. Generally smoke tests tend to be very narrow happy path tests.

Regression

Regression or regression testing is the act of testing existing functionality. Generally in every release, developers add new functionality. Testers go in and test the new functionality. After they are done testing new functionality, they go back and test all the old functionality again to make sure nothing is broken. Testing of old functionality is called regression

Automated Testing

Automated testing is any kind of testing that is automated. The developer/QA implements the test and checks it in. Usually, the continuous integrationon server will run the test every time it runs

Manual testing

Manual testing is testing done by a person. It is not designed to be run everytime the application is built

Ad-hoc testing

Generally testing is done with a goal of checking whether things work as expected. You have a formal set of specs that you test against However, ad-hoc testing is any kind of informal testing that is done with the goal of breaking the application. This is where our friends who break open up clocks and radios come in. One of my favorite testers was testing a kiosk application that was based on a touch screen. This was way back in 2000 before multi-touch was available. She would use multi-touch on single touch screen!. She would try to press ok and cancel at the same time. She would play drums on the screen. It was crazy

User Acceptance Testing

This is a kind of a systems and functional test that keeps a particular user in mind. When you are building a SaaS application, you are defining the behavior of the system. During testing you are checking whether it behaves as expected. However, most times, a particular user may or may not use all the behavior. Or use the system in a way that no one else does, or no one thought of. Also, generally testing is done with test data that is designed to activate many differrtent use cases. UAT is done with the user in mind. You run the exact same scenario that the user will run and you usually do it with "real" data or data that is representative of the kind of data that the user expects
Besides these there are many terms in the testing world, that I'm not focusing on because we don;t use them too often

Why do we test ?

As can be seen above, there are many ways to test the application. However, one thing that developers tend to ask is "Why do we need to build so many types of tests?" After all, won't blackbox testing test whether my code works? So, why do you need to write unit tests? The reverse question is also common:- I built so many unit tests. Why should I spend time doing integration testing
Each kind of test has a differrent kind of focus, and although various kinds of tests overlap with each other, the focus of the testing doesn;t overlap. A lot of developers get hung up in defining the software by it's modules. "Module A is tested and working.  why do we need to test it another way" That is not how users see a system. Users see the system as a collection of "behaviors" not modules. This section talks about each kind of test and why it is important to do that test
Whitebox-unit vs black box-integration testing
White box testing is testing with the main goal of making sure that each line of code is tested, whereas blackbox testing is done with the goal that the behavior of the module is tested. 
Generally, you do whitebox testing at unit level, and black box testing at integration level. It is easier to do the analysis required to determine whitebox tests at the unit level. If you have defined a unit to be small enough, it's easy to figure out the white box tests. No one does integration whitebox testing for systems of even low complexity. You just can;t analyse 3000 lines of code to figure out the unique paths through the code. Integration tests tend to be blackbox. Because, the behavior of high level modules (that are made by integrating low level modules) is rather well defined, it;s easier to start designing black box tests at the integeration level. You normally don;t do blackbox testing at unit level, because  the behavior of low-level modules is not formally defined. Instead, the developer has defined the behavior of low level modules him/herself (and might change it as s/he build code). So, for ease of reading, from now on, whitebox tests means unit test, and blackbox tests means integration tests

Why do whitebox testing?

Well,  the argument that often comes up against whitebox testing is Why go for testing each line? If a system and it;s modules are defined by it's behavior, and you have tested all the possible behaviors (provide all possible inputs and test outputs) haven't you tested it enough. There are 2 problems with that line of thinking
a) A module lives longer than it's behavior
In an agile world, the behavior of application is changing constantly. This means that even if the function of a particular module doesn't change, it does mean that somewhere down the line what kinds of inputs you get will change. If you don't test all the lines, you will never know if there is a latent bug in a line of untested code that will be activated in the future. When a particular module is complete, it should be complete, not just complete now, but complete forever. Which means that I don't want to come back to a module unless the behavior of the module needs to be changed.
b) Golden rule aka Ethic of reprocity
Latent bugs are bad. Let's say you are working on a task that requires you to change a module that is dependent on another module written by someone else. You estimate the effort based on how much work you need to do in this module. You make the changes, test the code, and lo and behold, you find a latent bug in the other module. Now what do you do? First, Get pissed off. Second go talk to the PM and tell him/her "I found a bug in this other code and I need more time to fix it". The PM now has to slip the schedule that gets communicated to management. Imagine how disruptive it is to the project. Did you like being handed this bug that someone else has created? 
 All cultures have this rule that boils down to the saying "One should treat others as one would like others to treat oneself." Different cultures/religions say it in different ways. However, it all boils down to "If you don't like something, don;t do it to others". If you don't like being slapped, don't go around slapping people. If you don't like fixing other people's bugs, don't go around checking in code with untested lines. Practically, it;s not possible to test 100% of the code, but an ethical programmer tries his/her best to get to 100%
b) Contrary to popular belief, you can do whitebox testing using less test cases than blackbox testing
Yes, there is a misconception that doing whitebox testing takes longer than blackbox testing. It should theoritically take less time. If it does take longer, your code is probably bloated. For example, let's say you are writing a module that has 2 if-else conditions. There are 2 branches in the first condition :- A and A' and 2 branches in the second condition B & b'. That means there are 4 possible inputs, one for each distinct path through the system AB, AB', A'B and A'B'. However, you can do a whitebox test using only 2 of those cases(AB and A'B'). Remember the goal is to test every line of code. not every behavior. In any piece of code, it;s always easier to write tests so that each line is activated than to write tests that activates all distinct paths through the system. It's basic graph theory. The proof is left upto the reader.

Why blackbox testing?

So, you say You got me convinced. From now on, I'm going to write whitebox tests for each module so that the module is tested 100%. Why do I need to write blackbox integration tests? After all the code is 100% covered. 100% of the modules are tested 100%. I'm done. Why should I write even 1 more test?
Because, sometimes the devil hides between the doors. Many times bugs appear because one module expects another module to behave a certain way, but it doesn't. In 1999, NASA lost a $125 million Mars orbiter because one team implemented their modules assuming that Imperial measurements will be used, and another used metric. The orbiter was carrying millions of dollars of scientific equipment probably disintegrated 60 kms above the Mars surface. The process of defining modules and the interfaces between modules is done by humans, and unfortunately, humans make mistakes. Someone writing a module might have a differrent idea of what the module's behavior is than the person using the module. Actually, it doesn't even have to be the same person. I have seen instances where people had a wrong understanding of their own modules. Human brains are limited. No matter how smart or educated you are, there is only so many combinations of things that you can hold in your head. When a typical programmer  is working in one module, s/he can picture what that module should look like in his/her head, but the details of other modules fade away. The only way you can check whether all modules are working well together is by testing them together. One way to minimize integration problems is to have simpler interfaces, but there is no guarantee until you try it out
Besides, remember that software is defined by it's behavior not it's modules. An appplication is more than sum of it's modules. Just because that 100% of the modules are tested doesn;t mean that the entire software is tested

Functional testing vs system testing

Why do we distinguish functional tests vs system tests? Why can't we write tests that test everything?
The first reason is simply because the skills of functional testers is different than systems testers. Indeed, in small teams and/or easier to understand domains, there is overlap between systems testers and functional testers (and even smaller startup teams, there are generalists who do  everything from coding to testing). However, as a team grows, and the application becomes complex, a need arises to hire people who specialize. Especially, if your business domain is complicated, you need testers who specialize in the domain. System testers tend to be generalists (even more than developers), and they need to be able to test a lot of different things. They need to have a very caried skill set. For example, if a system tester is doing performance testing, s/he need to be able to identify bottlenecks to make accurate bug reports. Next day, s/he is doign usability testing, and s/he needs to play the role of an end user. Third day s/he is doing secuirty testing, and s/he has to be a hacker. It's very tough to find people who have all these skills, and generally you will find system test teams made up of people who have a very varied skill set. Indeed in really large companies they have entire teams that focus on one kind of testing. 
Also, the kind of insfrastructure required to do functional tests versus system tests can be very different. Functional tests can usually be done on low volume of data, and will probably execute fine on the tester's local machine, whereas performance and security tests requires an infrastructure that mirrors production, whereas breakpoint and scalability tests might require infrastructure that goes beyond production. This is why it's important to distinguish functional tests from system tests. It;s simply not efficient to have the same infrastructure for all kinds of tests 
Lastly, some tests are just more important than others. No one cares if your application runs fast if it doesn't meet the business requirements.  If you have a functional bug in the code, there is no point in testing performance. Your application needs to pass the critical functional tests before it goes to system testing. We will cover this more in Circles of Testing

Happy path and smoke testing

Happy path and smoke testing are both barebones tests. Smoke testing tests if a particular change hasn't blown everything up, and Happy path tests test that the bare minimum functionality pass. Both become important in larger teams. In large teams, you simply don;t want to check in code that brings everyone to a grinding halt. If you introduce a bug that blocks 20 people, It might take you 1 hour to fix it, but it cost the company 20 hours, because 20 people are twiddling their fingers for 1 hour each. It;s better that you don;t check in the code until you are sure that you don't blow everyone up.
Happy path tests and smoke tests both provide a quick and efficient way of checking whether you are not blowing everything up. Everyone has differrent ideas of what "blowing everything up" means, and determining how wide is the happy path can be contentious. There are tradeoffs involved. If you make the happy path too wide, it takes longer to run the test and reduces it's usefulness to the developer checking in the code. If you make the happy path too narrow, a bug introduced in a code change might still block a lot of people.

Automation vs manual 

Manual testing is the kind of testing that used to be done traditionally. With the advent of test frameworks and Continious Integration tools, automated testing is becoming more popular. For automated testing to be effective, it has to be repeated automatically, and it has to be portable; ie; it should work on any machine in the company. If any of your testing steps require intervention by a human, it;s not automated. There might be some setup required by a human (for example installing some tools), but the idea is that once the setup is done, the tests will be able to run by themselves. So, for example, if you write a unit test that needs the user to clean the database before running the test then it's not automated yet. OTH, if all your test requires is an Oracle database, and it setup the schema, loads test data, and runs the test automatically, then it's automated. You can run the test again a 100 times without anyone looking at it
The reason why automated testing is important because it makes the testing effort more efficient. At the end of the day, 20 humans are much more valuable than 20 machines. Also, machines are good at doing the same thing again and again and again. Machines work 24 hours. Humans work 8-10-12 hours. OTH, humans are good at figuring out how to test software, whereas machines are not good at figuring out how to test, but once you tell them how to test, they can run that test without getting tired. So, to have a more efficient testing organization, we need humans to figure out how to test, and machines to actually carry out the test, and that's what automated testing is all about
Another reason why automated testing is important is because of the Law of Software Entropy 
The second law of thermodynamics, in principle, states that a closed system's disorder cannot be reduced, it can only remain unchanged or increase. A measure of this disorder is entropy. This law also seems plausible for software systems; as a system is modified, its disorder, or entropy, always increases. This is known as software entropy
What this means that any application can over time only become more complex, not less complex. As the system becomes more complex, the effort required to test it becomes more. Without automation, the only way testing teams can complete testing in the same amount of time is by adding more testers. So, if you have a team of 5 developers who are constantly adding and modifying the code, which results in increased entropy, you will over time need to add more and more testers. A simple example is that let's say in 3.1, LVP team built functionality for doing cashflow runs. The QA team has 1 person who can test cashflow runs in 3 days. Now, LVP added functionality for FAS5. Now they need 2 people because they have to test cashflow runs as well as FAS5 runs. Then LVP added new business, one more tester. Then Scenario analaysis.. one more tester. Eventually, you will have 100 testers and 5 developers. It doesn't scale. One way for scaling the team is by having automated testing. So, let's say, when LVP added cashflow functionality, testers built automated tests that test cashflow runs. Next release, LVP added FAS5, the testers focused on automating tests for FAS5, while the computer is testing cashflow runs. The same set of testers can keep adding tests as developers are adding functionality. The entropy of the system is well managed
Actually, the law of entropy applies to developer written tests too. The reason why we write JUnit tests is because we can automate the execution of the tests. And the reason why we automate the tests is because we want the same amount of human resources to keep adding functionality to the system. Going back and testing existing functionality is a waste of valuable humans. That's what machines should be doing. The Golden rule applies here too. Do you want to keep testing code written by other people? No. Then make sure you write automated tests for your code.

Ad-hoc testing

Ad-hoc testing is the kind of testing that has be done manually. Basically, during ad-hoc testing, the tester is using all his/her inguenity to find a way to break the application. A lot of developers get ticked off with this testing because It was not in the requirements. The system was not designed to used this way. The tester knows that this is not in the requirements. Why is he doing things to the system that it was not designed to do?. Yes, the tester knows that the test s/he is doing the test that is not in the requirements. The PM's know the test is not in the requirements. Really, everyone knows that the ad-hoc test scenario is not specified in the requirements. Do you know who doesn't know that scenario is not specified in the requirements? The user
For the most part, the end user has no idea what the formal requirements of any given application are. The user doesn't even care. The user wants to do some work on the application, and that is the only thing the user cares about. The formal requirements are a tool designed to make things easier for the developer. Formal requirements are not the end goal. The end goal of the application is to be useful to the user. In reality, the user is apt to use the system in ways that were not specified in the requirements. The user might have expressed a need, but by the time we delivered the app, their needs changed. Or the user just doesn't have the technical skill to use a system properly. Or, once the user gets his/her hands on an actual working system, s/he figures out a way of using the system that makes him/her efficient ways that no one thought about before. The user is a human, not a machine. The user has all the faults and the ingenuity of a human, and s/he will bring all of them down to bear on the application. The user is not going to the thing that s/he was told to do 6 months ago. S/he is going to do something different. S/he is going to use the application in ways that no one thought about.
In this case, we know that the application's behavior is not defined, and it may do things that no one expected. However, we want to make sure that the application doesn't face catastrophic failure or corrupts data. In a SaaS environment, every bug has the potential to bring down the application for all the clients. What the tester is doing is mainly checking for catastrophic failures. If s/he finds a bug while doing ad-hoc testing, then s/he is obviously going to report it

Cost & benefits

Now that we covered what are the differrrent kinds of tests, and why we do these kinds of testing, and important issue is deciding when testing should be done. In an ideal world, all tests will be automated, and it will take no effort to implement the automated tests, and they will execute instantaneously. In that case, we could run all tests whenever each developer changes each line of code. That is not going to happen. So, we need to understand the costs and benefits of testing, and use that to prioritize the tests
The costs assosciated with testing are
a) write the test
b) run the test
c) update old tests as functionality changes
d) continuously run old tests and make sure that they stay green
OTH, there are benefits of testing, or rather, speaking from an economics POV, there are costs associated with not doing testing. If your code is so buggy that the client declines to buy it, there is a lost opportunity cost. Realistically, the client is not going to just up and leave , but they are going to make you fix the bug, and there will be a cost assosciated with fixing the bug that a customer has seen. Typically, the same bug is easier to fix the earlier it is found in the SDLC. A bug introduced in the requirements phase will take 5 times the effort to be fixed in design phase. 10 times during coding phase, 50 times during the testing phase and 100 times after it has been deployed to the client. So, the benefit of testing increases the earlier you find it in SDLC. Also, remember that a bug found and fixed by a developer during the course of his/her development costs will cost 5 times more  if the tester finds it
Note that this is true not only at a macro-level but also at a micro-level. When testers do the tests, they go through testing phases, A bug found in an earlier phase is cheaper to fix than a bug found in a later phase. For example, a bug found during functional test is cheaper to fix than a bug found during performance test simply because it is easier to pinpoint the root cause of the bug found during functional test. This is also true if the developers start using a phased approach to testing. For example, if the developer first writes an unit test for every service, and then write an integration test for the whole module, then it's easier for the developer to fix the bugs found in the unit test than to fix a bug found by the integration test, because testing through unit test cuts down on the debugging time automatically. Actually, if you design your whitebox test the right way and use test first approach, it's very easy to narrow down where the bug is by simply looking at the tests that failed. You don't even need to debug. If you do not write unit tests, and instead do all your testing solely through integration tests, you might save some time writing tests, but you waste time during debugging, which ends up becoming costlier. We will talk more about this when we cover how to do efficinet testing.
One way of thinking about this is in terms of Technical Debt. If testing is not done efficiently in an earlier cycle, you are taking a debt from the later cycle. Once you take the debt, you will have to pay it back with interest. Not all debt is bad. Sometimes, it;s a good idea to take debt. For example, if you introduce a bug that will cause you to delay the release and $100 to fix today, $1,000 if you didn;t delay the release and  fix after you release to the client. But, the client will give you $10,000 to release the software today, then it make sense to release the software with the bug. However

Efficient testing or Circles of testing

No, this is not the Circles of Hell. These are Circles of Testing. To make testing more efficient, we arrange the tests into concentric circles. The idea is that tests in inner circles will be faster to implement and run, and outer circles become more comprehensive. This is based on the idea that 80% of bugs are found by 20% of the tests, and 20% of the bugs that escape cost 80% of the effort to fix. Using a graduated approach to testing allows us to catch more bugs in the inner circles where they are cheaper to fix.
The circles of testing looks like this

Here the circles that are in red and brown are developer activites. The blue circles are automated tests and the green circle is QA run manual test. Please note that the size of the circle doesn't represent effort. The size of the circle represents the cost to fix a bug found in that phase. In reality, the testing effort in the outermost circle is a lot bigger than it appears here
Coding - This phase is where the developer is doing the coding
Unit Tests - In this phase the developer is building and running unit tests only for the modules being built or changed. For example, if you are writing a new service, you write a unit test for the service and execute it. You don't need to run the other unit tests. 
Happy Path- In this phase the developer is building and running a Happy Path Integration test. The tool used to run the Integration test depends on the change being made. For example, in LVP, most changes can be tested using the Excel Test driver. Ideally the first 3 phases should run concurrently using a Test First approach to development
Unit Regression - Once the developer is happy with the changes made, s/he runs all the other unit tests in the module changes, and dependent modules. This is to make sure nothing else is broken. For example, if a developer makes a change to LVP databaseaccess module, the developer should run all the unittests in databaseaccess module. Also, since jobs module is dependent on databaseaccess module, the developer has to run the jobs unit test. Running these tests takes a long time, and if the developer has solved most problems in the first 3 phases, this just has to be run once
Manual Test-  In this phase, the developer is testing the application manually. S/he brings up the UI and executes the scenario. This phase is optional and should really be used more for UI-heavy changes. For changes that are heavy on backend engine changes, more focus should be on automated testing. At this point,l the developer checks in the code and puts the ticket in validation
Smoke Test - After the developer checks in, a nightly build on Team City runs an automated smoke test. The focus of this test is to do some barebones testing to test if everything is blown up. When the smoke test passes, the artifacts generated are pushed to artifactory and they are available to other team members. If the smoke test blows up, the artifacts are not available and other team members continue using older versions of artifacts
Daily Regression - TBD
Weekly regression - TBD
QA tests - TBD

Test first approach

Test first approach or Test driven development is done because when you write a lot of code and then sit down to test it, this is what happens

You spend hours building something, and then you try it out and it doesn't work. What happenned? Now, you have to figure out where the problem is. You have to debug through hundreds of lines of code to find the problem. Also, after you spend time coding the implementation, psychologically, you feel like you should be done. But, you are not. You still have unit tests to write. You might start writing the unit tests, but you won;t put full attention to it because you are done. 
Test First approach tries to avaoid that by making you write the test first and then write the implementation. You don't write all the tests. Instead, you do it iteratively. You take the simplest, most base use case, and implement the test for that first. Then you add the code for that implementation, run the test , fix the code, repeat till it works. Then you go to the next use case. You keep adding code and tests iteratively till you are done. 
When you have to do integration and unit tests both, start with updating the integration test first. This will help you design the interfaces in your classes too! The integration test is usually mimicing scenarios run by real user. The sequence of operations that leads to most effciency is
a) take a use case, put that use case into your integration test. As you are doing this define the interface between the modules
b) Create skeleton implementation of the interfaces
c) Implement/update unit test
d) Fill in the code for an interface
e) RUn unit test for that interface . Fix and run till it passes
f) Repeat d) - e) for all the interfaces
g) run the integration test. Fix and run till integration test passes
h) Repeat a) - g) for all use cases
But, doesn't this take more time? Isn't it better to release code to QA so they can test while I write my test cases?
 No! A study done by NIST shoed that a test first approach actually takes less time to implement that a test last approach. Also, it leads to more consistent quality. Also, remember that bugs found by QA cost 5x than bugs found by developers. Even if it takes the developer 2x the time to write the test, you still save money. Also, it makes no sense to release code to QA unless the developer is confident that his/her code is defect free. If QA finds bugs that the developer already found out by him/herself, you have wasted a testing cycle. QA could have tested something else during that time.

Software development Roles and their impact on quality

A software development organization is made up of many people who play various roles. A common misconception is that quality is the responsibility of QA. QA is Quality Assurance, not Quality Management or the Department of Quality. QA's main job is to assure management that the end product meets acceptable quality standards. It's everyone's responsibility to build quality into the product. Quality has to be integrated into the entire software development process. This section is documenting the roles played by various roles in the organization, and how the participate in defining the quality of the product. 

Product Management /Business Analysts  -  Requirements

Product Management / Business Analysts are the people who define the requirements. They are responsible for understanding the business needs of the clients and translating the needs into implementable requirements. They tend to have very good domain knowledge and are generally the domain experts in the organization. As far as quality is concerned, it is their job to understand what is important for the client and what's not important and translate that into good requirements that express the needs of the users well. Also, it is their responsibility to "design" the software so that it's easy to understand (and explain). Complicated requirements leads to complicated products. Complicated products lead to buggy products. Their expectation from the software is that "it should just work". Generally, when things go wrong, they may or may not understand the root causes of failure, and they usually don't understand the jargon that software development organizations use. When things go wrong, all they need to know is a) how is the user going to be impacted b) when can we fix it

Project Managers - Scheduling

Project managers keep projects on track. They are responsible for making sure everyone knows what they are supposed to do, when it;s going to be done, and how it affects the release date. They keep all the trains running. They are the people who keep asking "When is this going to be done?"
There is a misconception that PMs do not help in quality because the perception is that PMs want everyone to be done. PMs do care about quality because bad quality leads to slippage, and slippage leads to very very bad things. A ticket failing because of bugs is a bad thing for the project schedule because now the PM has to go and rearrange the entire project schedule and figure out how to move tasks around so we cna be done on time. A bug found by the customer is even worse because now the PM has to plan for a patch release which means moving more pieces on the board, and possibly delaying the current release.
When a PM says "When is this task going to be done?" what the PM really means is "When is this task going to be done in a manner that this task will never have to be done again?" 

Architects - Design

Architects are responsible for design. They are responsible for taking implementable requirements and breaking into chunks that developers can get their hands around. They are also responsible for code reviews, and for training the developers to create better quality code. They are also the people who responsible for thinking about the complete impact of a particular change, for example, impact on engineering, impact on scalability, failover, etc. This is why they come up with the weirdest ideas go up to developers and ask them things like "Why are you burning my water?" 
Architects contribute to quality by setting a standard, not only a formal standard, but also are expected to lead by example. They expect the developers to write simple, testable, well documented code and also test the code

Team Leads - Leading

Team Leads lie between the Architect-Project Manager continuum. They have some responsibilties of the architect and some responsibilities of the PM. They are responsible for distributing the work between developers, and make sure people in their team know what they are suppossed to do. They are generally the "best person" on the team and are expected to lead by example. 

Developers - Implementation

Developers are responsible for developing the code. They are the engines of the company. They are also the people who can contribute most to quality by just writing bug free code. They are the first line of defense against bugs. Developers tend to be people who like building stuff. They are also the people (along with QA) who get pushed by the PMs the most. They expect everything else besides their code to work perfectly. Also, developers need to be focused when they write code, so they don't like disruptions. This includes bug reports from QA, especially if QA is doing ad-hoc testing
There is a tendency among devs to be overconfident in their code. Once their code is done, they are pretty sure that it works perfectly, even if they haven't tested one line. And when they try it, and it works on their machine, that piece of code is done and they are ready to move to the next task. When you find a bug, the first thing the developer will say "But it works on my machine"

Functional testers

Functional testers are responsible for testing the application to make sure it meets the business requirements. They also tend to have a good understand of the domain. They tend to be the domain experts on the ground. They are happy when everything works perfectly, because it gives them time to start understanding the requirements for the next cycle

Systems testers

Systems testers are responsible for testing everything that the functional testers don't test. They tend to be generalists; they are good at lots of things. They also like to open stuff up and understand how it works. Sometimes, Developers look down on systems testers, and there is a misconception in the software industry that those who cannot code, test. Actually, the opposite is true because systems testing requires more creativity than development. It;s easy to break something, but breaking something in a way to expose it's weak points requires a lot of creativity.

Golden Rules of Quality

There are a few golden rules of testing (that I came up with.. you won't find it anywhere else)
1) Test early, Test often
Remember that finding a bug earlier in the cycle is cheaper to fix than finding a bug later. I
2) Done should mean done
When a particular task is done, it should be truly done. when devs say that a particular task is done, they should be confident that no one will find a bug in their code. When QA says  particular functionality is working, they should be confident that the users will be able to fulfill the business need. No tasks should come back.
3) Automate Automate Automate
Testing something manually and fixing without automating the test wastes testing effort. If you want to repeat the test, you'll have to run the test manually. Automating the test, and then fixing it makes sure that the test is repeatable forever. Even if you get a bug report from a test that was run manually, it sometimes make sense to reproduce the problem with an automated test case. If it's a sever or complicated bug, you want to make sure it never comes back. Tthe only way you can ensure that it gets caught quickly when it comes back is by checking in a unit test that replicates the problem.

No comments:

Post a Comment