If you are a developer, you probably think it should be FAST;
If you are an automation engineer, you probably think there should be lots of tests;
And if you are a DevOps engineer you ARE worried.
Back in 2016, we faced a problem. Our CI process became madness. Out of a culture where every small feature and bug needed to be automated, the type of culture where automation tests are a MUST and new infrastructure is added on a monthly basis, we found ourselves buried under a heap of tests.
Combining our top 5 automation infrastructures, we had more than 10K tests, that would have taken over 20 hours to run (had they run on a single CI agent). The need to cover so much ground forced us to write a powerful UI/Web test framework to allow us to easily test our full stack. However, despite the many enhancements we made, the execution of each web test still took anywhere between 1 and 5 min.
Typically, when you think of a testing pyramid, you want to have a solid base of unit-tests, reasonable number of integration tests and a few high-quality web tests. In our case, with the power of super-infrastructure, on came the problem — a huge amount of web tests almost preventing our nightly suite from running on our JetBrains TeamCity (TC) instance. Back then, we had about 10 agents, and probably a dozen build configurations in TC, each running a specific type of tests, for a long time. For example, running our integration test assembly took 3.5 hours, while our web test suite took 5 hours.
A minimal number of build configurations sounds like a best practice when having a huge amount of tests, but we faced several problems:
We needed a solution or at least a way to provide the CI consumers a fresh breeze. A solution that would make their life easier, while achieving maximum stability and quick response time.
So we decided to SPLIT. Splitting everything. Class by class. web, integration, unit-tests. Instead of having less than 15 build configurations in TC, we would have more than 200.
Although TC easily handles 200 builds, we couldn’t manually deal with this setup, considering we had 4 branches in development, 3 DevOps engineers, and just 10 agents trying to keep up with the pace. Managing the new build configurations became our day today. As a result, a few days after the initial split we found ourselves in chaos. And as usual, when you don’t have enough, you need to automate what you have.
Splinter - our personal splitter. We realized we needed someone to keep up with the developers, review the changes in the code, observe new classes and new tests, and introduce them to TC while removing previously deleted ones. Time to write some automation.
Here’s the basic concept:
The Splinter utility should support any test assembly from our CI process and do the following:
1) Using reflection, analyze the assembly and find the relevant test classes. Test classes that are ignored, empty or contain only ignored tests wouldn’t be considered.
2) Clone an existing TC build configuration template and set the correct parameters in the new build configuration. If a matching build configuration already exists, just move to the next one in line.
3) Trigger the build configuration.
4) For cleanup, remove any build configurations that don’t have a matching class in the code.
Master Splinter (or Splinter Manager) is triggered every night and initiates all other Splinter, each for a specific test assembly.
In Quali, we write most of our infrastructure code in C#. In the Splinter project, we couldn’t find a library that wrapped TeamCity’s rich RESTfull API for C# for advanced queries, so we decided to develop one — FluentTC.
FluentTC — Easy-to-use, readable and comprehensive library for consuming TeamCity REST API. Written using real scenarios in mind, enables various range of queries and operation on TeamCity
TeamCity’s rich RESTfull API allowed us to implement Splinter easily and add custom logic and enhancements in the following versions of Splinter.
FluentTC in JetBrains community: https://plugins.jetbrains.com/plugin/8980-fluenttc
FluentTC sources: https://github.com/QualiSystems/FluentTc
FluentTC Nuget: https://www.nuget.org/packages/fluenttc
Introducing Splinter has created a new CI culture: