post-img

Streamline your CI/CD pipeline with TeamCity and CloudShell Colony’s Environment as a Service

Posted by german lopez April 30, 2019
Streamline your CI/CD pipeline with TeamCity and CloudShell Colony’s Environment as a Service

This article originally appeared on JetBrains on 4/25/19

Posted on April 25, 2019 by Yegor Naumov
This guest post is brought to you by Meni Besso, Product Manager at Quali, the creators of CloudShell Colony

CloudShell Colony logo bugCloudShell Colony is an Environment as a Service platform. It connects to your cloud providers including AWS, Azure, and Kubernetes and automates environment provisioning and deployment throughout your release pipeline in a scalable, maintainable, and smart way.

Creating a scalable CI/CD pipeline is harder than it looks.

Too many times have we witnessed DevOps engineers dive straight into configuring builds and writing scripts, only to find out later that they need to redo a significant part of their work. Any professional engineer will tell you that avoiding this unsavory outcome is easy. You need to treat your pipeline as a product: gather your requirements, design your architecture, develop, test, and improve your pipeline as your product grows.

In order to come up with a good design, you have to take three important aspects into account:

  1. Release workflow: Working out the steps that need to happen from the moment a developer commits code until the code goes live in production.
  2. Test automation: Planning the tests that need to run to validate the quality of the release and the automation tools that will be used to run them.
  3. Test environments: Determining the specifications for each test environment, taking into account test data, debugging, performance, and maintenance.

Much has been said about the first two items, but we at Quali believe that environments are the missing piece on the way to a reliable CI/CD pipeline, because continuous integration only works if you can trust your tests, and running tests on faulty environments is a serious barrier.

So, how do you design environments?

First, let’s distinguish between the two types of environment:

  • Sandbox environments: Environments that you use temporarily to complete a specific task. For example, to run sanity tests or performance tests from your pipeline.
  • Production environments: Environments that are always-on serving users. For example, your live production environment or a pre-production environment that you share with your customers.

A typical release pipeline uses both sandbox environments and production environments. You run various tests on sandboxes to validate your code and then deploy to production.

Sandbox environments can be tricky. Once you have set up your automated pipeline, you may notice the complexities. You need to support different variations of these environments, often in parallel. Some environments need to provide quick feedback while others need to replicate the full production environment. How do you deploy the artifacts, load test data, and set up your test tools? How do you debug failed tests? Are you using cloud infrastructure? How much does it cost?

At some point, you end up realizing that your challenge is also one of scale. Meaning, all these issues grow from bad to worse as your automation gradually consumes more and more environments.

Production environments can be tricky as well, but for different reasons. With production, you need to think about ongoing upgrades, handling downtime, backward compatibility, rollbacks, monitoring, and security.

Integrating your TeamCity CI/CD pipeline with an Environment as a Service platform like CloudShell Colony will ease many of these complexities and provides you with a more stable, reliable, and scalable CI/CD pipeline.

Let’s see how it works.

Integrating CloudShell Colony with your DevOps Toolchain

You start by creating a CloudShell Colony account and connecting it to your AWS or Azure accounts. This is done through a simple process that gives CloudShell Colony permission to create infrastructure in your public cloud accounts.

Then, download the installation instructions.

Once you have your TeamCity server connected to your CloudShell Colony account, you can connect your artifacts’ storage provider and start using CloudShell Colony in your release pipeline: run automated tests on its isolated sandbox environments and update your production using blue-green deployment.
Running automated tests on sandbox environments

Pipeline TeamCity and CloudShell Colony

Running Automated Tests on Sandbox Environments

To use sandbox environments in your TeamCity builds, you need to update your build configuration flow.

The first build step should start a sandbox. Use the ‘Colony’ runner, that comes out of the box with CloudShell Colony’s plugin, and fill in the required parameters. For example, enter the name of the environment blueprint you want to use, the location of your artifacts in the storage provider, and any test data you may wish to load to the sandbox environment.
When the environment is ready, provisioned and deployed, CloudShell Colony saves its details to a TeamCity variable, so you can extract any needed information (like IP addresses or application links) and feed your tests.

Finally, after the tests complete and the sandbox environment is no longer needed, you can terminate it by adding another build step that ends the sandbox and reclaims its cloud resources.

Running automated tests on sandbox environments

Creating Environments from Blueprints

CloudShell Colony’s environments are created from blueprints.

Blueprints define application requirements in a simple YAML format that you store in your source control provider. They include the definitions of artifacts, networking and computing requirements, deployment scripts, variables, and test data. They can also integrate with traditional IaC (Infrastructure-as-Code) tools, such as Terraform.

Once you create your blueprints and store them in your source control repository, they will be available in CloudShell Colony’s self-service catalog, allowing you to launch environments from CloudShell Colony’s user interface or from your TeamCity builds.

Creating Environments from BluePrints

Environment Lifecycle

Every environment goes through several stages including a verification step that ensures that the environment is 100% ready for your automated tests.

Environment lifecycle

And since your environment is composed of applications, CloudShell Colony manages the application dependencies and initialization and provides quick debugging and troubleshooting options.

manage application dependencies in CloudShell Colony

In case of errors, you will know exactly which steps failed and you are able to browse the deployment logs and securely connect to your compute instances.

Clean up your environments in a snap with CloudShell Colony UI environment

When the environment is no longer needed, CloudShell Colony cleans up the environment’s infrastructure from your cloud account, ensuring that you won’t have to pay for unused infrastructure.

Environments have a crucial part in building a stable application pipeline, and in DevOps processes in general. Using Platforms like CloudShell Colony enables teams to get fully provisioned and deployed environments quickly while keeping cloud expenses under control.

Like what you’ve just read? Start your free trial of CloudShell Colony.

CTA Banner

Learn more about Quali

Watch our solutions overview video
post-img

Augmented Intelligent Environments

Posted by german lopez November 19, 2018
Augmented Intelligent Environments

You have the data, analytic algorithms and the cloud platform to conduct the computations necessary to garner augmented insights.  These insights provide the information necessary to make business, cybersecurity and technology decisions.  Your organization seems poised to enable strategies that harness your proprietary data with external data.

So, what’s the problem you ask?  Well, my answer is that things don’t always go according to plan:

  • Data streams from IoT devices get disconnected that result in partial data aggregation.
  • Cybersecurity introduces overhead which results in application anomalies.
  • Application microservices are distributed across cloud providers due to compliance requirements.
  • Artificial Intelligence functionality is constantly evolving and requires updates to the Machine & Deep Learning algorithms.
  • Network traffic policies impede data queues and application response times

Daunting would be an understatement if you did not have the appropriate capabilities in place to address the aforementioned challenges.  Well…let’s take a look at how augmented intelligent environments can contribute to addressing these challenges.  This blog highlights an approach in a few steps that can get you started.

  1. Identify the Boundary Functional Blocks

Identifying the boundaries will help to focus on the specific components that you want to address.  In the following example, the functional blocks are simplified into foundational infrastructure and data analytics functions.  The analytics sub-components can entail a combination of cloud provided intelligence or your own enterprise proprietary software.  Data sources can be any combination of IoT devices and the output viewed on any supported interfaces.

 

  1. Establish your Environments

Environments can be established to segment the functionality required within each functional block.  A variety of test tools, custom scripts, and AI components can be introduced without impacting other functional blocks.  The following example segments the underlying cloud Platform Service environment from the Intelligent Analytics environment.  The benefit is that these environments can be self-service and automated for the authorized personnel.

 

  1. Introduce an Augmented Intelligent Environment

The opportunity to introduce augmented intelligence into the end to end workflow can have significant implications for an organization.  Disconnected workflows, security gaps, and inefficient processes can be identified and remediated before hindering business transactions and customer experience.  Blueprints can be orchestrated to model the required functional blocks.  Quali CloudShell shells can be introduced to integrate with augmented intelligence plug-ins.  Organizations would introduce their AI software elements to enable augmented intelligence workflows.

The following is an example environment concept illustration.  It depicts an architecture that combines multiple analytics and platform components.

 

Summary

The opportunity to orchestrate augmented intelligence environments has now become a reality.  Organizations are now able to leverage insights from these environments which result in better decisions regarding business, security and technology investments.  The insights derived from these environments provide an augmentation to traditional knowledge bases within the organization.  Coupled with the advancement in artificial intelligence software, augmented intelligence environments can be applied to any number of use cases across all markets.  Additional information and resources can be found at Quali.com

CTA Banner

Learn more about Quali

Watch our solutions overview video