When I was a teenager, I had a friend whose dad owned a rock quarry. They mined for aggregate that is used to make concrete, backfill for roads, and building subdivisions. It was a small operation, but for a teenage boy it was very cool. Large machines with conveyer belts, crushers, screens, and shakers. The engineer inside of me was fascinated with how it all worked. You dump in a truck full of what looked like dirt and rocks into one side of these machines and out on the other side came rocks of several different sizes, dirt and silt. The silt mixed with water made some of the best mud football games you can imagine.
The image of these big machines came to my mind when I was talking to a couple of engineers tell me about our software build, release and test system at work. The source code goes in one side and our product comes out of the other side. And in our case more than one version of the product targeted for different Operating Systems and or configurations. One of the things they were complaining about was the amount of time it took to find out if their changes broke the build or made it past all of the tests in the test farm.
Over the years the amount of tests run with each build has increased and the number of configurations has increased. This has made the turn around time hours instead of minutes as it used to be. The engineers I was talking to were complaining that they wanted instant feedback and could not wait hours to see if what they just typed in was going to bring the whole organization down or but the “magnus opus” that they believe they just created. Our build and test farm becomes overwhelmed when we are close to a release date and build results end up taking days instead of hours.
Again the big machines at the quarry came to mind again. I could see the huge screens, shakers and conveyor belts. I could picture how the first convey belt moved the dirt and rocks up to the top of this tower with conveyor belts coming off in all directions. The dirt then would be dumped into this very large screen big enough for almost everything except for large rocks about the size of large potatoes. These large rocks were sent down a conveyor belt to a crusher that then smashed these rocks into smaller rocks and then dumped that back into the top of the machine. The next set of screens would be smaller and let everything pass through except for rocks the about ¼ the size of the previous screen. In this case the conveyor belt carried these rocks to a big pile next to the tower of machines. This continued on until at the bottom of the stack silt was conveyed out to the silt pond. Did I mention mud football already?
I could see what the engineers were wanting. They wanted to know if their build had big rocks or boulders in it first. They needed a quick feedback smoketest. Well we already had that. But they also wanted the aggregate screen. They wanted product out before everything was screened through the system. So I did what most good engineers do I went to google and looked at the different levels of testing. I found a good powerpoint that shows the canoical view of levels of testing. http://www.cs.swan.ac.uk/~csmarkus/CS339/presentations/20061202_Oladimeji_Levels_of_Testing.pdf. Simply it can be desribed as.
As I kept looking further in my organization I found, Interesting enough, many of the groups started doing placing screens in their build/test systems themselves in an ad hoc manner with CI build systems. Jenkins and Berta seem to be the most popular. As I went around and talked to more teams and more engineers I found one group in particular that had the same vision of the rock quarry as I did but had it in Software build and test.
This group had set up a CI build system that took each check in and pushed the build through several screens of tests to see the quality of the build. They had 5 levels of testing that would occur with each build. The screens became mroe fine grained and would take longer to run at as the source and build would go down the stack. There are a number of screens that can be put in place to give the engineers a quick feedback loop that they are looking for. The key is to time bound the first level of screens. Using a mixture of the levels of testing to give the first screen nice coverage of build and give immediate feedback to the developers. Here is an example of how a nice build/test screen could be set up.
This of course is just an example of what could be. I think it is important to put a plan together on how the build progresses through the screens and in some cases the build is performance more than once on the same code base. Coordination between Validation and Development is key to making this work effectively.
Let me know what you think. Do you have any other good ideas on how to give development feedback quicker without sacrificing the quality of your product.
DWP
The image of these big machines came to my mind when I was talking to a couple of engineers tell me about our software build, release and test system at work. The source code goes in one side and our product comes out of the other side. And in our case more than one version of the product targeted for different Operating Systems and or configurations. One of the things they were complaining about was the amount of time it took to find out if their changes broke the build or made it past all of the tests in the test farm.
Over the years the amount of tests run with each build has increased and the number of configurations has increased. This has made the turn around time hours instead of minutes as it used to be. The engineers I was talking to were complaining that they wanted instant feedback and could not wait hours to see if what they just typed in was going to bring the whole organization down or but the “magnus opus” that they believe they just created. Our build and test farm becomes overwhelmed when we are close to a release date and build results end up taking days instead of hours.
Again the big machines at the quarry came to mind again. I could see the huge screens, shakers and conveyor belts. I could picture how the first convey belt moved the dirt and rocks up to the top of this tower with conveyor belts coming off in all directions. The dirt then would be dumped into this very large screen big enough for almost everything except for large rocks about the size of large potatoes. These large rocks were sent down a conveyor belt to a crusher that then smashed these rocks into smaller rocks and then dumped that back into the top of the machine. The next set of screens would be smaller and let everything pass through except for rocks the about ¼ the size of the previous screen. In this case the conveyor belt carried these rocks to a big pile next to the tower of machines. This continued on until at the bottom of the stack silt was conveyed out to the silt pond. Did I mention mud football already?
I could see what the engineers were wanting. They wanted to know if their build had big rocks or boulders in it first. They needed a quick feedback smoketest. Well we already had that. But they also wanted the aggregate screen. They wanted product out before everything was screened through the system. So I did what most good engineers do I went to google and looked at the different levels of testing. I found a good powerpoint that shows the canoical view of levels of testing. http://www.cs.swan.ac.uk/~csmarkus/CS339/presentations/20061202_Oladimeji_Levels_of_Testing.pdf. Simply it can be desribed as.
- Unit Testing
- Integration Testing
- System Testing
- Acceptance Testing
- Regression Testing
As I kept looking further in my organization I found, Interesting enough, many of the groups started doing placing screens in their build/test systems themselves in an ad hoc manner with CI build systems. Jenkins and Berta seem to be the most popular. As I went around and talked to more teams and more engineers I found one group in particular that had the same vision of the rock quarry as I did but had it in Software build and test.
This group had set up a CI build system that took each check in and pushed the build through several screens of tests to see the quality of the build. They had 5 levels of testing that would occur with each build. The screens became mroe fine grained and would take longer to run at as the source and build would go down the stack. There are a number of screens that can be put in place to give the engineers a quick feedback loop that they are looking for. The key is to time bound the first level of screens. Using a mixture of the levels of testing to give the first screen nice coverage of build and give immediate feedback to the developers. Here is an example of how a nice build/test screen could be set up.
| < Time | Name | Build | Test |
| 10 minutes | Smoke Test | Incremental | High level integration, some unit testing |
| 30 minutes | Check In Regression | Build from Smoke. | Unit Testing for CheckIn |
| 1 hour | Integration | Full Build | Integration Test, Regression Tests |
| 3 hours | Performance | Build from Integration | Performance Tests |
| 6 hours | Nightly | Build from Integration | Integration Tests, System Tests, Unit Tests |
| 36 hours | Weekly/Weekend | Build from Nightly | Full test suites. Performance |
This of course is just an example of what could be. I think it is important to put a plan together on how the build progresses through the screens and in some cases the build is performance more than once on the same code base. Coordination between Validation and Development is key to making this work effectively.
Let me know what you think. Do you have any other good ideas on how to give development feedback quicker without sacrificing the quality of your product.
DWP
