Remaking the lab for DevOps innovation: The Okinawa Open Laboratory example

Posted by admin January 6, 2015
Remaking the lab for DevOps innovation: The Okinawa Open Laboratory example

There has never been more pressure on telecoms and their technology providers to innovate and accelerate time to market for services. The emergence of SDN and cloud computing comes against the backdrop of increasingly capable OTT providers, while service provider organizations R&D labs are often more impediment than aid in achieving DevOps innovation.

The challenge can be thought of around three phases:

  • Innovation: R&D labs can be isolated from test labs, which act as a sort of slow-moving gatekeeping function today due to manual processes and poor, non-collaborative ways of communicating requirements. Innovative new ideas born in the R&D lab have to run a gauntlet of heavy documentation, messy handoffs, and constraint-based thinking when transitioning to the test lab, because test labs are so unwieldy in their operations. Moving to more virtual labs can help, but the fact is that networking is and will remain full of specialized hardware for a good long time. A key here is to allow innovators to do so not in isolation but in live environments that can be handed off to test lab operations without an intervening documentation step.
  • Permutation: Test labs need to be able to rapidly perform live environment permutation of service concepts into real-world test scenarios so that innovations can be exposed, validated and refined as soon as possible in the cycle. Slow, manual test lab operations are the opposite of this concept: A single testbed can often take days or even weeks to assemble. Meanwhile, the innovation spirit dies due to time lag, excessive overhead costs and a permanent urgency of ongoing tactical testing for historical features that don't move the ball forward.
  • Pre-Production Validation: Service providers need to boil down all the testing work done in the QA labs to validation suites that are Operations friendly so that anytime a service is going to be activated it can be easily validated for correct function without needing a high degree of technical knowledge. Today, this is very difficult because the chain of documentation hand offs from R&D to test labs to Operations dilutes utility and quality outcomes.

Internal operations and culture need to change and become more DevOps-esque. The Internet has sparked expectations that services can be self-serve and easy to work with. DevOps, as a cultural movement, builds on those expectations by fostering an innovative collaboration that moves services to real-world environments quickly, which means it must be supported on the technical side by infrastructure-as-a-service.

What shape this process takes can be difficult to envision. A project such as the Okinawa Open Laboratory - created by NTT Communications Corporation, NEC Corporation and IIGA Co. - offers a pretty good blueprint for how labs can evolve to become more agile. The OOL, using QualiSystems CloudShell, has succeeded in demonstrating that innovation and testing stages can happen via automation and live environment sandboxing and testing handoffs.

The OOL example: Enabling SDN, NFV & Open Source innovation via a cloud-like lab
Even with the aid of SDN and NFV technologies that are much more API enabled than traditional networking gear, the challenge facing telecoms and technology manufacturers in moving to DevOps models is significant because it entails integrating a wide range of physical and virtual, commercial,  open source and proprietary components, from bits of OpenStack and SDN controllers to various legacy switches and both commercial and open-source hypervisors. Overcoming the traditional silo-based way of business is much harder.

In the case of OOL, collaboration is essential since most of the lab's stakeholders and users are remote and need easy access to cloud-based test bed resources. A static catalog wouldn't do; dynamic sandbox innovation via a ready-to-go commercial platform was necessary to accommodate DevOps workflows for all the teams involved. Infrastructure had to be agile and responsive to activities such as rapidly rolling out a test environment and then dismantling it and redistributing its resources.

OOL found that DevOps collaboration and automating all infrastructure - physical, legacy, virtual and cloud - go hand in hand. Cultural and technical change are both required. The OOL was able to leverage cloud orchestration tools that allowed for on-demand hardware and software allocation and provisioning. They were able to integrate the overall orchestration in an open fashion with SDN connectivity and converged infrastructure stacks.

The result is that OOL stakeholders save time and money by having access to such a flexible, DevOps-friendly lab cloud. Redundant test beds don't need to be purchased, and hand-offs between collaborators is facilitated via a GUI-driven, live environment basis rather than through laborious and inexact email threads and conference calls.

Ultimately, the R&D and test labs can be regarded as a microcosm of the agility imperative facing telecoms and technology providers right now. That is, they need to be updated on both the cultural and technical ends so that processes are more innovative, collaborative and real-world relevant. DevOps requires a fresh, cloud-like approach to infrastructure, which can be achieved through automation and orchestration akin to what the OOL realized through its use of CloudShell.

The takeaway: Labs can be obstacles on the road toward agility and DevOps. Infrastructure needs to become more agile via automation and orchestration, while internal operations must become DevOps-like in the way that teams collaborate in their hand-offs and in moving service concepts quickly to real-world validation through rapid test permutation. The Okinawa Open Laboratory's use of QualiSystems' CloudShell platform offers an example of what a DevOps lab operation can look like.