Killing Your Long Lived Dev/Test Cloud Environments

One of the key roles of cloud network managers is containing costs. It is so easy to add to networks and scale resources in the never ending quest for lower latency and better application performance.

Even if you are working for or consulting to large organisations with seemingly unlimited budgets, sooner or later once your application or project is in production, the sights will be turned on the ongoing cloud spend.

During the initial development stage it’s not uncommon to get multiple developers and/or teams involved in order to meet project deadlines and in doing so you can often spin up multiple development, test and staging environments so the various branches of development activity can be isolated but tested thoroughly before being merged into production.

The problem is, when the additional development teams and consultants go away, very often their development and testing environments do not. After several rotations of developers, it’s extremely easy to lose sight of exactly what environments are related to production and ongoing development and which ones are unused development or test environments that nobody is using or sometimes even aware of.

While a complete replication of your production environment can be obvious, sometimes other resources can be lurking in the cloud account that aren’t so obvious, like a copy of a database that was taken at some point in history for analysis of a problem and is now sitting in isolation, in it’s own VPC but nobody really knows why.

One of the side benefits of using Hava to automatically generate your cloud network topology diagrams is these outliers are brought to the surface.

When you connect a cloud account to Hava, be that an Amazon AWS, Google Cloud or Microsoft Azure account, Hava will scan the console settings and generate an infrastructure diagram for each VPC or virtual network detected. This is the first opportunity to spot unexpected environments.

When you log into Hava you will start at the environments dashboard.

If there are additional VPCs or virtual networks appearing on the environments dashboard, you will be able to dig into these unexpected environments and delete them if they are no longer required. Maybe they are residue from long gone developers, or test environments used in the frenzy of bringing your application to market that are no longer required.

Typically these environments aren’t something you go looking for and can be buried deep in the console and not immediately obvious on the cloud account bill.

The second opportunity to spot rogue resources are on your production VPC / Virtual Network diagrams. Here you might encounter a stand alone subnet with a single resource link a database instance, or a handful of resources that aren’t connected to your production resources.

If you manually draw diagrams, these outliers will never appear or be included on diagrams so avoid any scrutiny for months or maybe years until someone digs into the console settings looking at services that you may not be aware are in use and stumbles across a running instance.

If this sounds unlikely, rest assured it’s more common than you think. In one of the more extreme examples, a customer connected their AWS account to Hava and uncovered a stand alone database instance that was running but not in use. A stand alone VPC with a single RDS instance with no association to production. It was costing the company close to $2k per month. It had been running for over 3 years untouched. You do the math.

What resources or entire dev/test environments do you have running undetected in your AWS, GCP or Azure accounts?

How much are they costing you?

By connecting to Hava, you’ll know the answer.

So what is hava.io?

Hava is an application that is a SaaS that can be accessed online or run in isolation self-hosted on your own cloud infrastructure. Hava automatically generates cloud network topology diagrams, security diagrams, container workload diagrams and AWS best practice compliance reports.

You simply connect your AWS, GCP and Azure accounts or connect your Kubernetes Clusters to Hava and your diagrams will be generated hands free and equally as important, they are automatically kept up to date. Hava continuously polls your network config and as changes are detected, new diagrams are generated even when you are not logged in.

Superseded diagrams are moved into version history, so you always have an audit trail of changes.

AWS NETWORK TOPOLOGY DIAGRAMS

There are a number of cloud visualization diagrams that can prove useful to your engineering and operations teams. These include an AWS Infrastructure diagram like this:

The diagram shown above logically lays out all the resources discovered when an AWS account is connected to Hava’s cloud visualization tool. The main VPC is depicted by the green outside border and is surrounded by associated resources like internet and VPN gateways, S3 Buckets, VPC endpoints, VPC peering connections, Elastic Beanstalk environments and so forth.

Inside the VPC, your configured availability zones are set out in columns. These columns contain the individual subnets set up in those AZs. All the resources contained in each subnet are visualized as are any load balancers that are routing traffic to the various subnets.

Having your diagrams automatically created from your AWS console configuration data enables interrogation of each of the resources to see the settings and associations related to them. By selecting any of the individually visualised resources, the attribute pane to the right hand side of the diagram displays all the known metadata related to the resource. This is one aspect of Hava’s cloud visualization diagrams that isn’t possible with manually created drag-and-drop diagram makers.

Automating the diagram build using Hava also reveals resources you may not be aware of. Long forgotten database instances and sometimes entire development, staging or test environments are regularly discovered by this automated cloud visualization process.

GCP CLOUD ARCHITECTURE DIAGRAMS

Hava’s cloud visualization diagram tool will connect to your GCP accounts and visualize your GCP networks, zones and subnets.

AZURE CLOUD ARCHITECTURE DIAGRAMS

In addition to visualizing your AWS and GCP cloud environments, you can also import your Azure environments into Hava. Hava will create diagrams for each resource group discovered in Azure. The resource group will detail any virtual networks found in the resource group which in turn can contain multiple subnets and other resources like virtual machines, load balancers, peering connections and storage accounts.

CUSTOM CLOUD VISUALIZATION

There is always a compromise when it comes to automatically generated documentation. The way that Hava discovers and generates network topology diagrams is based at a VPC or resource group level. If more than one VPC is detected in your Amazon cloud account for instance, then one diagram set per VPC is created.

You may however want to combine two or more VPCs onto a single diagram, or just diagram a subset of your resources. This can be achieved using the custom query function built into Hava.

CUSTOM CLOUD DIAGRAM GENERATOR

This custom query would create a new custom diagram containing the two nominated AWS VPCs specified. You would then have the ability to save this diagram so it is always present in your dashboard until you choose to delete it. As with all other diagrams, your custom infrastructure diagram would keep itself up to date and retain a version history every time a resource change is detected. The filter and deep search allows you to quickly find resources across hundreds of cloud accounts using resource names, tags, IP addresses and a range of other search criteria.

CLOUD RESOURCES LIST

In reality it is just not practical to include every single component detected in your cloud config onto a network topology diagram. Take for instance network interfaces, volumes or WAF rules. In a large network with hundreds or maybe thousands of these second tier components, trying to visualize every single component would make the diagram almost unreadable, or certainly too busy to easily recognise the core components.

Hava’scloud visualization toolsolves this issue by providing detailed components list that details every single resource detected. This is known as the “List View”

This is a comprehensive list of resources that can be sorted and exported along with estimated monthly costs so you can see at a glance what resources are costing you the most money. This detailed list view is also interactive. Selecting a resource on the diagram will reveal all the known settings and associations that resource has.

AWS SECURITY GROUP VISUALIZATION

Another major benefit of using Hava to automate the discovery of your AWS environment diagrams is the ability to capture and visualize security group information. While only available for AWS currently, security views for Azure and GCP are on our development roadmap.

AWS SECURITY ARCHITECTURE DIAGRAM

With an AWS security group diagram you can view all of your configured security groups. All your open ports are overlaid on the visualization to enable an instant understanding of the traffic flow, ingress and egress points. The security view diagram is also interactive which ensures you can select any security group on the diagram and important information relating to the security group, like the connected resources, ingress and egress IP addresses and associated tags are revealed without having to exit out to your AWS console to investigate.

Hava also provides an Azure security diagram.

Kubernetes Cluster Diagrams

Hava supports the direct connection of Kubernetes Clusters and will generate diagrams. You can connect any K8s cluster to Hava using a generated configuration file. Hava will then scan the cluster and visualise the cluster and running pods giving you a visualisation of your containerised workloads.

Hava will scan your cluster and auto generate a complete visualisation of the resources detected, cluster, nodes, pods and the relationships between them.

In the above example all the pods are green indicating a healthy state. As with all Hava interactive diagrams you can select a pod to see all the containers running in it, you could hover over a node in the attributes pane to highlight all the pods running on that node. The power of the visualisation is when things aren’t running as expected.

The colour of each pod indicates the run status of the workload. The green indicates healthy and running, but the red indicates a sub-optimal health status like stopped. An amber pod denotes the status is changing, so it could be starting or stopping.

By selecting a suspect pod, you are able to see the meta data in the attribute pane including the connected containers and associated nodes in the cluster so you can quickly identify what needs to be addressed in order to return the workloads to optimal health.

As well as stand alone Kubernetes Clusters, Hava also generates container diagrams for AWS EKS and ECS containers, Azure AKS and Google Cloud Platform GKE managed container environments.

AWS WELL ARCHITECTED COMPLIANCE

Hava includes a reporting module that features an AWS Compliance Report.

This report steps you through all the resources and regions you have configured as well as users and roles in use and will draw your attention to adverse well-architected findings graded as high, medium or low and the reasoning behind the finding.

The report is presentation quality and gives you insights on your AWS configuration and highlights opportunities for improvement.

AUTOMATED CLOUD VISUALIZATION UPDATES

Your documentation is only useful if it is up to date. It is quite possible to spend weeks manually constructing network topology diagrams only to have them rendered useless by a minor change to your network configuration. Given the nature of services provided by the major cloud vendors like autoscaling that provision or remove resources in response to traffic demand, the chances are your cloud infrastructure is changing more often than you would first imagine.

Built into Hava is an automatic synchronisation function that polls connected cloud account data sources and automatically updates cloud visualization diagram sets when changes are detected. This means your diagrams are always current and up to date. The superseded diagram sets aren’t deleted or overwritten however. They are placed into Hava’s version history. This audit trail of changes stores superseded diagrams which enables you to pull up and investigate older diagrams in the same fully interactive format as the live current diagram set.

This allows you to quickly identify changes to your cloud network topology either visually by placing diagrams side-by-side or by exporting them for programmatic comparison.

Versioning also allows you to demonstrate the status of your network to key stakeholders at any point in history. This can be invaluable during a PCI compliance audit, insurance or security audit should your network design ever be called into question.

EXPORTING CLOUD DIAGRAMS

The native cloud architecture diagrams created by Hava are the nearest we’ve seen to the examples and recommendations provided by the major cloud platforms. These are great to view and interrogate via the Hava application dashboard, however sometimes you might need to pull a set of diagrams for audit purposes or to incorporate into management or sales presentations.

The built-in diagram export function allows you to do this by providing these five export options

CSV, VSDX, JSON, PDF and PNG

When viewing any of your Hava diagrams, you can select the export function, tell Hava what format you would like and Hava will generate the export file for download.

EDITING YOUR CLOUD VISUALIZATION DIAGRAMS

Should you want to manipulate or embellish your auto generated Hava diagrams, exporting to VSDX format and using Visio, draw.io or any VSDX compatible drawing package will allow you to edit your diagrams as required.

Should you not have access to Visio but would like to try this out, try opening one of your exported VSDX files in draw.io (diagrams.net)

Getting Hava to do the bulk of the heavy lifting by initially generating accurate diagrams based on what is actually configured and running in your cloud environments enables you to access a base diagram ready to edit which will save you hours or possibly days preparing management reports.

If you are not already using Hava to auto generate cloud network topology diagrams and documentation from your AWS, Azure, GCP or Kubernetes deployments, you can take a free trial at https://www.hava.io

Who knows what you will discover.

Originally published at https://www.hava.io.