Managing Multiple Environments in a Hybrid Cloud

With the shift from traditional Client Server Application Software to Cloud Aware Application many Software Engineers have found themselves dusting off old System Administration Books from college. With multiple services running on multiple machine or containers software engineers have to be able to manage their applications across more and more complex environments. As I have been talking to some of my customers I have found common pain points in managing these complex applications: 

• Consistency between environments
• Single point of failure services
• Differing environment requirements  (Not all environments are created equal)
• Managing multiple environments across multiple clouds

All of these factors and many more can lead to time wasted, applications being released into production before their time, or worst of all unhappy software engineers.

DevOps to the rescue?

Wouldn't it be nice if the software engineer just worried about their application and its code, instead of all of the environments that it has to run on? In some places that is exactly what happens. Developers develop on their local laptops or in a development cloud and then check in their code and it moves to production. DevOps cleans up any problems with applications using single instance bottle-necked services, out of sync versions of centralized services, or adding load balancing services to the front end or back end of the application. The App developers have no clue what mess they have caused with their code changes, or a new version of service that they are using. Somehow we need to make sure that the application developer is still connected to the application architecture but disconnected  from the complexity of managing multiple environments.

Single Definition Multiple Environments

Working on my Local machine

One approach that I have been looking at is having the ability to define my application as a set of service templates. In this simple example I have a simple Node JS application that uses Redis and MongoDB. If I use a yaml format. It might look something like this.

MyApp:     
  Services:  
    web: NodeJS  
      ports: 80  
      links: mqueue, database  
    mqueue: Redis  
      ports: 6789  
    database: MongoDB  
      ports: 25678, 31502  

So with this definition I would like to deploy my application on my local box, using Virtual Box. I put this yaml file in my home directory of the application. This should be very familiar to those of you that have used docker-compose. Now I should be able to launch my application on my local machine using a command similar to docker-compose.

$ caade up

After a couple of minutes my multi-service application is running on my local laptop.

I can change the application code and even make changes to the services that I need to work with.

Working in a Development Cloud

Now that I have it running on my laptop I want to make sure that I can run it in a cloud. Most organizations work with development clouds. Typically development clouds are not as big as production and test clouds but give the developer a good place to try out new code and debug problems found in production and test environments. Ideally the developer should use the same application definition and just point to another environment to launch the application.

$ caade up --env=Dev

This launches the same application in the development environment. Which could be a OpenStack, VMWare or Kubernetes based SDI solution. The developer really does not care about how the infrastructure gets provisioned, just that it is done quickly and reliably. On quick inspection we see a slight difference in the services that are running in the development cloud. There is another instance of the NodeJS service running. This comes from the service definition of the NodeJS service. The NodeJS service is defined to have multiple instance in the development cloud and only one instance in the local environment.

NodeJS.yml - Service Definition

1. NodeJS:  
2.   Local:  
3.    web:  
4.       image: node-3.0.2  
5.       port: 1337  
6.   Dev:  
7.     web:  
8.      image: node-3.0.2  
9.      port: 1337  
10.     worker:  
11.      image: node-3.0.2  
12.      port:1338  
13.      cardinality: 3  
14.   Test: …  
15.   Prod: …  

This definition is produced by the service and stack developer not the application developer. So the service can be reused by several developers and can be defined for different environments (Local, Dev, Test, & Production). This ensures that services are defined for the different requirements of the environments. For example Production NodeJS might have a NGNX load balancer on the front end of it for serving up NodeJS web services for each user logged in. The key is that this is defined for the Service

that is reused. This increases reusability and quality at the same time.

Working in the Test Cloud

Now that I have tried my application in the development cloud. It is time to run it through a series of tests before it gets pushed to production. This is just as easy for the developer as working in the development cloud.

$ caade up --env=Test

$ caade run --env=Test --exec runTestSuites

We launched the environment and then run the test suites in that environment. When the environment launches you can see additional instances of the same services we have seen before in the development cloud. Additionally, there is a new service running in the environment. The Perf Monitor Service is also running. It is monitoring the performance of the services while the tests are running. Where did the definition of this service come from? It came from the application stack definition. This definition just like the service definition can show that the application can have a different service landscape for each environment. But the software developer still sees them as the same. That is to say, code should not change based on the environment that is running the application. This decouples the application from the environment and frees up the software developer to focus on code and not environments.

What about Production 

The ultimate goal of course is to get the application into production. Some organizations, the smart ones, don't let developers publish directly into production without some gates to pass thru. So instead of just calling "Caade up --env=Prod" we have a publish mechanism that versions the application, its configurations and supporting services.

$ caade publish --version=1.0.2

In this case the application is published and tagged with version 1.0.2. Once the application is published, it will then launch the environment if it is not currently running. If it is running then it will "upgrade the service" to the new version. The upgrade process will be covered in another blog. Needless to say it allows for rolling updates with minimum or no downtime. As you can see additional services have been added and some taken away from the test environment.

Happy "Coder" Happy Company

The software engineer in this story focuses on writing software not on the environment. Services are being reused from application to application. Environment requirements are being met with service and application definitions. Stack and service developers are focusing on writing services for reuse instead of fixing application developers code. Now your company can run fast and deploy quality products into production,

What's in a name - Utopian Development Ecosystem

We are in the process of changing our way of working in our group. Many times when you are trying to effect large change you need a new perspective. You need something new to get people excited about the change; a new name. Companies do this all of the time. They spend millions of dollars coming up with new names for new product launches. So in an effort to give my team a fresh look at things, we came up with a new name for our development environment. The Utopian Development Ecosystem, UDE for short. Of course, we are engineers and need three letter acronyms for everything. This is a change from the standard “Development Environment” that we constantly talk about.

Let’s breakdown the new name, first Utopia. Thanks to Sir Thomas Moore we have the word Utopia from his book, “Utopia” (1516). He describes a perfect idealized society possessing highly desirable or perfect qualities. Everything in Utopia is in balance. Sounds like exactly what we were aiming for, perfection. If we are going to name something let’s pick the best.

Next let’s look at the word ecosystem. Wikipedia defines ecosystem as:

A community of living organisms (plants, animals and microbes) in conjunction with the nonliving components of their environment (things like air, water and mineral soil), interacting as a system. These components are regarded as linked together through nutrient cycles and energy flows.

What does this have to do with developing and delivering a product? It sounds more like a description of the ecosystem that is growing in my refrigerator, than a team of software engineers trying to deliver a product. If we are focusing on delivering a product let’s make some changes to some words that fit our situation.

• living organisms → People
• non-living components → Tools
• nutrient cycles → Guiding Principles
• energy flows → Processes

With these substitutions we get a new definition of ecosystem that fits our situation much better.

An Ecosystem is a community of people, (Developers, Validators, Architects, and Managers), in conjunction with the tools of their environment, (things like computers, storage, labs and cubes), interacting as a system. These components are regarded as linked together through guiding principles and processes.

Therefore, our perfect development ecosystem is all about people and tools and how they interact and are linked together through guiding principles and processes. This fits well with what we want to accomplish with our new development environment. It broadens the scope of the typical development environment to include guiding principles and people. Things that easily get overlooked.

Our Utopian Development Ecosystem defines everything we want to help our team deliver products. This has deep meaning and invokes strong feelings, but is simple enough for us to get the team to rally behind it.

DWP

Strawberries, Chocolate and Tomatoes

Recently my wife and I decided to try an experiment with 7 of our children. Each child got to pick out their favorite ingredient for dinner. It could be anything they wanted, but it had to be a single ingredient. They were all told their favorite ingredient would be put into dinner that evening. They all wrote their ideas on a piece of paper. No one knew what the other person wrote down. The challenge for my wife and I was to use all the ingredients to make something for dinner.

So here's a list of ingredients, starting from youngest to oldest child.

• strawberries - very good choice
• maple syrup - another great choice
• chocolate chips - sounds like a yummy dessert
• coconut milk - again a great combination
• olive oil - this is no longer dessert, it’s something else
• tomatoes -fits well with the olive oil
• parmesan cheese - the finishing touch

We had bow tie pasta planned for that night so we made 2 batches. One batch with chicken alfredo, the other with the concoction of favorite ingredients. The results were amazing....amazingly gross. All of these individual ingredients were great, but together a gastronomical disaster.

At work I have noticed the same sort of thing with development tools. If we don't spend time looking at the big picture, we can sometimes end up with great point solutions that combined together make a less than desirable system. A great build tool combined with a great configuration management tool adding in a requirements management tool can be great as long as they can all talk to each other and use a common systematic data model. If not, then you end up with a Frankenstein system that is hard to manage and not too pretty to look at. Some simple planning upfront with goals and objectives for each of the ingredients, can quickly change disaster into a utopian system.

First, focus on a common data model. This does not mean a single database, just a common data model. That means that when I say the word Defect, it means the same in all of the tools, has common fields and if possible resides in one place.

Second, focus on integration of tools not through replication of data. Replicating and translation of data always leads to problems. It is far better to link data across tools with as little data manipulation as possible. If a tool already has a notion of defect that does not match your primary tool for defects, then you can have a problem. Where is the real defect?

Third, remember the “best of breed” tool that does not fit into the system well, does not belong in the system. No matter how good the chocolate was that went into our family’s dinner, it tasted horrible with tomatoes and olive oil. The same is true with point tools that cannot effectively talk with other tools.

Last, look for the low touch integration. Just like in a well designed system you do not want to have a highly coupled tool integration suite. It causes problems when tools are upgraded. Look for loose coupling and high cohesion in the tools that you integrate together.

Good luck with the tool soup you are creating, remember that not all great things thrown together are necessarily a good combination.

DWP

Change Escape Velocity for an Organization

I was recently measuring how long it took for an idea to move through our organization. I compared several ideas over the last 3 years. I Looked at why some seemed to hit some critical mass and then just take off, while others required constant care and feeding and took a long time to be adopted. Reviewing the changes with my boss, he said there seems to be some escape velocity for ideas in the organization. Of course this made me want to investigate this more. At the suggestion of my boss Bradley Mitchell, I opened up wikipedia and took a look at the definition of escape velocity. This is what I found out:

1. Escape velocity changes depending how far away from the surface you are.
2. Escape velocity decreases with altitude.
3. It is the speed at which an object will not fall back to earth.
4. An object at any speed as long as it has propulsion, can escape the earth’s gravity.
5. If propulsion is removed on an object that is below escape velocity it will fall back to earth.
6. If an object is at or above escape velocity it has enough kinetic energy to “escape.”

There is also a formula that allows us to measure the escape velocity of any object.


G = gravitational constant
M = the mass of the planet or star
r = the distance from center of gravity.

So if we apply this to organizational change we get some interesting observations that seem to be true.
G = Change Constant
M = the size of the organization
r = how close to core values that change are.

So if the organization is large (large M) it will take more velocity to get the organization to change. If the change is far from the changing fundamentally core values of the organization then the escape velocity is smaller than if it profoundly affects core values. This seems to make some sense. I also think we can play with the Change Constant and say that it is not constant but corresponds to how old the organization is. The older an organization the harder it is to change and therefore the escape change velocity is higher.

Let’s look at these principles and see how they might apply to change in an organization. So how can I use escape velocity observations for escape change velocity?

1. The further you are from a profound change to core values the easier the change.
2. Change is easier for smaller changes.
3. To make a change permanent you must achieve the escape change velocity or it will fall back to earth.
4. Adding constant pressure to a change, will eventually make it stick.
5. If a change is launched without enough velocity to escape, it will revert back to its current situation.
6. If a change has enough velocity at launch, it will not need constant pressure to succeed.

As you can see the engineering side of me wants to put an equation to describe how I can help an organization change. But it is never that easy. Human beings are unpredictable and in a large organization with hundreds of individuals it is closer to chaos. The key is to try and determine what things can help control the chaos during the change. Finding the perfect velocity of change for the organization is a combination of the amount of change and the size of the organization.

DWP

Someone to Watch Over Me

In my work as well as at home it is always nice to have someone help me get my todo list done. Whether it is a honey do list or a work breakdown structured list from work, it is nice when I have the chance to have another set of eyes to make sure I did everything. Of course I didn’t think this applied to my honey do list, but my 6 year old son taught me something interesting this last week. I have been trying to fix our trash compactor for the last month or so. The problem is it is just hard to open the door. First thing I did was pull all of the garbage that fell behind the trash bin. That seemed to help some, but it was still really hard to open the door. Next I took the bin out and made sure all of the tracks were working fine. Then I started taking apart the wheel assembly. Putting it back together and tightening all of the screws. It was better but still hard. I was very frustrated and out of patience with this appliance from the underworld. Then my 6 year old son flipped the red switch that said,  “locked” to “unlocked,” and everything worked fine.

So, like a typical engineer I was looking for a hard problem to solve instead of looking for something obvious. This tends to be the case not just at home with household appliances, but at work as well. This could not just be for us engineers either. In the software engineering world there is a big movement to perform something called Pair Programming. Basically two people sit down to one computer and write code together. One person typing the other making sure the program is working, typed in correctly ,etc...

Can you imagine someone looking over your shoulder making sure you are typing correctly? Sounds annoying and it can be at first. But the results are huge according to some research that there can be up to 50% decrease in the number of mistakes (bugs) made in the code. Two people tend to consider more design alternatives, find bugs faster, and fix problems right the first time. I have used pair programming a couple of times in my career and in every case I have seen great improvement in hitting timelines, with higher quality.

However, I have also seen engineers not wanting to work nicely together. Maybe it goes back to elementary school when we are graded individually. Not as a group or team often. It is kind of funny. We teach kids in kindergarten to play together. By 3rd grade you are not allowed to share answers with your classmates, that is cheating. In fact one of the largest complaints about pair programming is who gets the credit. Especially when annual reviews come around. Somehow we have to overcome the social barriers that have been ingrained in us to not share or work together.

As I learned with my trash compactor, pair programming helps find solutions that would be otherwise overlooked. Even a 6 year old can see the obvious things that years of my domestic appliance repair was overlooking.

Patience and Patents

About 7 years ago I went through a patent harvesting exercise with my employer. It was 2005 and there was a big push in the software industry to patent everything they could. So they could be leveraged in licensing technologies between companies. The lawyer and my team submitted 14 patent applications and the waiting process began. About a year later I left the company and pretty much forgot about the patents. One day about 2 years later I got an email from one of the members of my former team. To my surprise,  we had been award 4 patents. Over the next year 2 more patents were approved. Not notice from my former employer. I found out thanks to “Google Scholar” monitor  (very cool tool that monitors your name and publications you have). Then silence; Nothing happen for 4 years and all of a sudden in the last 3 months 3 more patents were awarded.

I have lots of questions about the patent process. Maybe someone that has experience in this can give me some ideas.

• Why has it taken so long?
• Why does there seem to be some renewed interest in 7 year old patents.
• Do patent applications expire sometime?
• Is the high tech sue happy atmosphere today affecting number of patents filed and the time it takes to be reviewed? (Samsung and Apple for example)

With all of the renewed interest in my old patent applications, it reminds me of the old team I worked with, the cool technology we created, and the fun we had. So I guess the benefit of a long patent process might be nostalgia. I have lots of questions

DWP

Water Glasses and the Cost of Reuse

This summer my wife and I had a huge dilemma in our house. With kids home from school and the hot weather, our family of 9 ( 7 kids at home) was going through all of our glasses in our cupboards. We actually have more than 9 glasses, in fact we have plenty of glasses. But for some reason our glass cupboard was empty and our sink was full of glasses all summer long. Something about teenagers not being able to reuse the same cup more than once. In some respects I don’t blame the older kids. The younger kids, especially our 5 year old, just grabs any cup they can reach and gets himself a glass of water. If my 5 year old was not a walking magnet dirt, grime and everything sticky, I am sure the older kids wouldn’t mind to reusing the glasses sitting by the sink. So there is the dilemma, I have a limited number of glasses (supply), a fixed number of kids, at least right now (demand), and an unlimited demand for drinks of water. What a better reason to show the benefits of reuse.

Warning software geekiness alert!!!!

The costs of not reusing the glasses is very measurable. If I have to buy more glasses that costs money, more glasses probably means  more cupboard space as well. Not to mention more cycles of the dishwasher running. Another option is to limit the number of drinks of water per child per day. So reuse most definitely has a benefit. A measurable benefit. Money in my pocket. This is great. More money for my vacation fund or more than likely kids college fund.

Being the engineer that I am I started looking to see if there was a cost to re-use as well. It didn’t take long to find one of the costs of re-using glasses on the counter. All I had to do is open a bill from the doctor’s office.  Little kids tend to be petri dishes for germs and virus and anything else they pick up at school, the playground or a friend’s house.  And many of these little bugs spread like wildfire when we have reuse model of sharing glasses. Ok a bit of a stretch, but sharing is not always a good thing.

So what does that have to do with software engineering? Code reuse has been touted as the holy grail of software engineering. We always look at the benefit of reusing a component or code in a product. The benefits are great. Development team reduction, maintaining one component instead of multiple components, validation cycle reduction, etc... But what is the cost of re-use? Is there a cost? Just like the water glass situation in my house it depends on the design and architecture of the solution. The costs could be high or it could be low based on the what you value most.
There are different levels of re-use that should be explored. An article by Amber shows the different areas of reuse.

Architecture-Driven Reuse	·       The reuse and/or development of reusable technical and business components and services described by your enterprise architecture models ·       Sometimes referred to as domain-component reuse
Artifact Reuse	·       The reuse of previously created development artifacts: use cases; standards documents; domain-specific models procedures and guidelines; and other applications
Code Reuse	·       The reuse of source code within sections of an application and potentially across multiple applications ·       Includes use of Web services
Component Reuse	·       The reuse of pre-built, fully-encapsulated components in the development of your application
Framework Reuse	·       The reuse of collections of classes that, together, implement the basic functionality of a common technical or business domain
Inheritance Reuse	·       The use of inheritance in your application to take advantage of behavior implemented in existing classes
Pattern Reuse	·       The reuse of publicly documented approaches, called patterns, to solving common problems
Template Reuse	·       The reuse of a common set of layouts for key development artifacts – documents, models and source code – within your organization

Many times we just focus on the code reuse without considering the other areas of reuse. Ignoring these other areas can lead to increased costs. It may not always be the best thing to re-use product artifacts. I think we need to spend more time analyzing the benefits and costs before just jumping into something.

By the way, my wise parents (6 kids)  had a solution to the water glass problem. No reuse at all. They put a water faucet in the kitchen. No more water glasses in the sink. Reuse was just too costly and messy.


DWP