The vCurve

Less is not always more. We can agree that virtualization and server admins need less data to process and analyze. In a recent post, I mentioned some brilliant advances that are serving that end, including data compression technology, data crunching speeds and algorithm sophistication. We also agree that a virtualization management solution can and should minimize the need for root cause analysis. But the steps between a) the information about your virtual infrastructure and b) the data you must process and deal with to effectively manage the environment are important and varied.

Processing and storing the vast amounts of data about both the physical and virtual network and all areas of interoperability is hard—really hard. That’s why some vendors want to convince you it’s unnecessary. Their thought process goes something like this: You don’t really need all that data. It’s just for troubleshooting. Wouldn’t you rather have a solution that just solves your problems outright? And I have to admit, it sounds enticing, dare I say, too good to be true.

In truth, a virtualization management solution that strips out useful data required for effective problem-solving and performance management is doing you a disservice. That doesn’t mean the right solution leaves you overwhelmed underneath a mountain of data. Rather it’s what data you’re presented with—in the context of your unique challenges.

When a virtualization management tool skimps on data and analysis, you could end up wasting time, effort and money—in direct contrast to the promised benefits. Many of these tools base results on the most recent hour of data only—and probably can’t correlate that data with historical maximum resource usage. In consequence, the results and recommendations don’t tell the whole story about your virtual environment. You could end up moving resource workloads into some unnecessary places. And if the tool constantly makes recommendations from hour to hour, you’ll end up thrashing your virtual environment around for no real performance benefit. The most effective virtualization performance management solutions are aware of historical resource usage, and may go even further to place that usage into context. Following this example, the ideal solution remembers that a particular VM spikes every Tuesday at 2 p.m. without negative consequences, and shapes its recommendations accordingly. The knowledge is based on more comprehensive data collection, aggregation and analysis, but it means less work and wasted effort for you.

The problem isn’t too much data. The real problem is wrapping context around the data. We have devoted a lot of intellectual property to solving this challenge in order to put performance, capacity, right sizing, workload, configuration management, monitoring and security data into the proper perspective for our customers. It’s context, not simply less data, that enables more efficient and effective virtualization management.

written by Aaron Bawcom \\ tags: Performance, Predictive Analysis, Virtualization Management, Visualization

Recently Gartner Group announced what they consider the top five server virtualization trends for 2012.   In the brief analysis which we are sharing below, they emphasize that while server virtualization is maturing it is still pretty dynamic and ever changing, so much so that it is actively impacting their own decisions and guidance to their clients.  As price and selection has become varied, it is important for Reflex to be diligent in providing the information and education our current and future customers need to make the best possible decisions on virtualization solutions for their environment.  We thought it was an informative and encouraging piece to share with our own followers, along with our thoughts on the 2012 trends that Gartner has identified:

1)     Competitive Choices Mature: VMware’s competition has greatly improved in the past few years, and price is becoming a big differentiator. Enterprises that have not yet started to virtualize (and they exist, but tend to be small) have real choices today.

We believe this growth in competition is great for customers and vendors. Not only are prices coming down, but customers now have more choice than ever, and are not beholden to VMware’s architecture or pricing model. We see customers making much more informed decisions, selecting solutions that deliver the breadth of functionality they need not just today, but for their future plans to grow and scale their infrastructure and develop private and hybrid cloud solutions at a reasonable price.

2)     Second Sourcing Grows: Existing VMware users may not be migrating away from VMware, but they’re concerned with costs and potential lock-in. A growing number of enterprises are pursuing a strategy of “second sourcing” – deploying a different virtualization technology in a separate part of the organization. Heterogeneous virtualization management is mostly aspirational, although there is interest.

We agree that many users are currently looking for additional solutions for virtualization technology and the lack of management is a current obstacle for these end users.  Specifically, management solutions from the individual hypervisor vendors can be problematic for the growth of second sourcing.  Cross-hypervisor management will become essential within the next 12-18 months as we see more diversification of basic virtualization technology. The integrated management platform strategy vs. a multi-point solution strategy becomes key when expanding consistent management capabilities across multiple hypervisors like VMware, Red Hat KVM, Microsoft Hyper-V, etc. to enable holistic management of the virtual and cloud infrastructure. We believe that the management layer will be the great equalizer as this market progresses.  Red Hat and Microsoft are beginning to embrace the ecosystem in a way that makes holistic management a real possibility.  We expect to be able to provide significant parity for VMware, Red Hat, and Microsoft as we enter 2013.  These advancements in the management market will help accelerate the adoption of other hypervisors and bring true flexibility to the market place because the management technology will provide the features that may be lacking in some of the hypervisors.

3)     Pricing Models in Flux: From expensive hypervisors to free hypervisors to core-based pricing and now memory-based entitlements – virtualization pricing has always been in flux, and trends toward private and hybrid cloud will ensure that virtualization pricing will continue to morph and challenge existing enterprise IT funding models.

We believe this is also a positive for customers, who now have alternatives to VMware’s pricing.  Customers do not like to be negatively impacted by a pricing model that penalizes customers for gaining the efficiency and benefits from virtualization.  Customers will be very mindful of pricing scenarios and choice of vendor as more continue to build out private and hybrid clouds.  These pricing practices serve to artificially stunt the growth of virtualization as customers pause to understand the financial impact.  As the management technologies become a larger portion of the budget allocation for virtualization, it is very important to make sure that the pricing model allows the customer to achieve the scale in the infrastructure as well as benefit from its efficiency and agility.  Consumption based pricing models are difficult for most enterprises to plan and execute today.  This may change over time, but the key is to find a way to allow the customer to grow its usage of the solution as it grows value in their enterprise without financial penalty.

4)     Penetration and Saturation: Virtualization hitting 50% penetration. Competition and new, small customers driving down prices. The market is growing, but not like it used to, and vendor behavior will change significantly because of it. And don’t forget the impact on server vendors – the next few years will prove to be a challenge until virtualization slows down.

We actually see the market for virtualization management growing just as fast, if not faster, than it has in the past. While straight server virtualization purchases may be slowing, customers are wising up to the fact that they need to manage these environments as well, if not better, than they have managed their physical environments in the past, if they want to really get the benefits virtualization promises. We believe the management market has tremendous growth ahead, and will provide most of the value added features that deliver on the promise of agile and elastic datacenters.

5)     Cloud Service Providers Are Placing Bets: IaaS vendors can’t ignore the virtualization that is taking place in enterprises. Creating an on-ramp to their offerings is critical, which means placing bets – should they create their own standards (perhaps limited their appeal), buy into the virtualization software used by enterprises (perhaps commoditizing themselves), or build/buy software that improves interoperability (which may or may not work well)? Not an easy choice, and winners and losers will being determined.

Many of our customers, who are cloud service providers themselves, realize that they must provide solutions that 1) customers are familiar with, and 2) can be integrated into a broader data center vision that includes both private and public cloud, leveraged for different needs of the business. Developing a platform that enables these two things is key to their success.  Service Providers have struggled to get enterprises to buy into the promise of a more efficient cost model using the public cloud.  This is primarily because they do not want to let go of the true business critical applications.  The private cloud is growing in popularity, and that is being driven by the technologies being delivered by a new generation of software companies that spend every day trying to solve these problems.  The service providers are going to have to embrace these technologies/vendors and work with them in a meaningful way in order to get true enterprise buy in for use of cloud services.

As virtualization continues to mature and shape how IT functions, organizations should become educated on the virtualization options available and look for a strong management solution that offers flexibility, scalability and comprehensive capabilities that evolve with the dynamic nature of the virtualized data center.

written by Preston Futrell \\ tags: Cloud, Gartner, Private Cloud, Trends, Virtualization Management, vmware

Now, a few words on looking for things. When you go looking for something specific, your chances of finding it are very bad. Because of all the things in the world, you’re only looking for one of them. When you go looking for anything at all, your chances of finding it are very good. Because of all the things in the world, you’re sure to find some of them. – Daryl Zero

This sage advice from the greatest private detective in the world isn’t just applicable to figuring out who is blackmailing you; it’s also useful for general problem solving. We’ve had some great discussions lately with some people who have some really tough data problems. Now mind you these problems are all over the map. Some problems are completely different and some others overlap. One challenge in problem solving is imposing arbitrary restrictions on how a problem is solved. People sometimes latch onto a particular way of thinking and as my Uncle Olaf used to say “When all you have is a hammer, everything looks like a nail…”. Take the following data for example. The first image below with the single green line looks pretty simple. It looks like it is telling you all you need to know without a lot of complexity. The problem is that the next image with the blue line is the actual data that the first image was created from. After you look at both, it is clear that the view of the single straight line could potentially be misleading as the trend is actually going down at the end of the detailed graph.

What is far more useful, offers simplicity, and provides detail for validation is the following image that includes both types of data in relation to each other.

Continue reading »

written by Aaron Bawcom \\ tags: Big Data, Predictive Analysis, Sampling, Trends

Lots of buzz around big data and cloud these days.  Thinking about how the technologies of big data, virtualization and cloud intersect is also being trumpeted by many of the big IT vendors.

Reflex has always seen virtualization as an opportunity to do things differently.  The scale of large enterprise virtualization implementations and the trajectory of virtualization, not to mention cloud, creates an interesting big data question.

“Can the tools of yesterday’s data center be adapted to operate efficiently in this new environment?”

We think the answer is no. At best, they will “function”, but what they wont be able to do is take advantage of the wealth of information available to extend the value of virtualization and eventually cloud.

I recently wrote an article for Enterprise Systems Journal that discusses the general concept.  If you’re familiar with Reflex it will be obvious that we are serious about the intersection and have developed some really cool technologies to leverage the big data of virtualization in our VMC product. Much of what I discuss in the article is materializing in our products.

The two primary technologies are the evolution of the VQL language and our more recent introduction of Complex Event Processing technology with a VQL/Virtualization specific implementation.  It is through this focus that we have proven our common platform for virtualization management at enterprise scale.

written by Mike Wronski \\ tags: Cloud, Data, Scale

The first Virtualization Query Language was officially born around February of 2008 and was first released in the summer of 2009. VQL provides data awareness of the IT environment by easily surfacing information from different data sources such as the VI Java library produced by Steve Jin. Since VQL was first developed it has been getting lots of new objects added to the library but the grammar has not changed in any major way…until now. Today we are announcing some pretty major changes to VQL that we are very excited about. The new capabilities include functions, new objects, and real-time query processing.

Functions!

As an object pipeline VQL was great as a classifier but we realized that objects can be somewhat difficult when you want to perform analytics on data so we introduced a generic object type that makes the production of analytical data possible (think mapping objects to spreadsheets). Some of the new built-in VQL functions are the usual suspects like aggregates (sum,count,avg,min,max) but some of the other functions are also nice like top(), math(), density(), and select() which suddenly allows the VQL query engine to produce partial objects which makes transferring VQL objects over a network rather zippy.

New Objects

• Some of the new objects joining the team include performance metrics, datastore mounts, vCPU, and VQL queries themselves are now VQL objects. Why make VQL queries objects? Well one of the new functions introduced is called QueryOutput() which takes the name of a VQL query as a parameter. This capability allows VQL queries to be chained together in a stream by reference so changing the output of one query does not effect the definition of another query. The payoff here is performance. The ability to chain trees of queries together at the application level eliminates complexity and adds a lot of performance gains.

Real-Time Query Processing

One of the most disruptive new capabilities in VQL 2.0 is the addition of soft real-time processing of VQL queries. Processing VQL data in real-time means you can instantaneously see extremely sophisticated information about your virtual environment. One of the most straightforward applications of this technology is reporting. Everyone is used to running a report and waiting some time before viewing the report. Usually the more useful the report is the longer it takes to run. And if any of the data that makes up that report changes, the entire report has to be computed again. What the Reflex real time processing engine offers is the ability to recompute only the portion of that report that may have changed based on the new data. This type of data computation provides a double-whammy of utility. The segmented/streamed processing can produce sophisticated data instantly and the computation of that data actually takes less overall CPU cycles than computing it using standard database techniques.

This type of technology can be applied to not only reports but any type of complex data computation. Some examples of how real-time data computation can be used:

• Instantly understanding complex forecasting of resource usage and supply
• Instantly reacting to new environmental data and instituting a modified security policy based on the new information
• Alerting when the performance of an application might be suffering due to resource constraints or new load placed on the application
• Instantly adapting the load balancing of resource demand across resource surplus that may exist for a forecasted amount of time

Real-Time Intelligence

The previous examples provide some insight into the possibilities of real-time data processing but another great example is using real-time processing as a component itself to produce new intelligence that then higher level decisions can be based off of. Today if you wanted to find out which virtual machines in your environment are oversized or undersized you would buy a product that produces that type of report or you could write some scripting logic to produce that information yourself. The time needed to compute that type of information can be anywhere from a few seconds to a few minutes depending on the size of your environment. Now imagine being able to constantly compute that data within milliseconds no matter how large your environment is. You then could produce new metrics that record that data so now you can know at any point of time in the past how undersized or oversized a VM was and even graph those trends over time. That being said, you could envision the software you would need to produce that type of intelligence. The real innovation that VQL 2.0 provides is making that type of incredibly complex processing to occur with a single VQL query specification.

Another way to understand the new real-time processing capabilities of VQL is to think of constantly computing the output of a Powershell script so that if any of the data that the Powershell script queries for changes that the output of the Powershell script instantly changes.

You might be asking how this technology is different from some other software that exists out in the world. A lot of real-time processing engines that exist can only process data in a very specific form and can only produce data for a very specific output. Since VQL is a graph based language the new Reflex real-time processing engine is one of the first graph based complex event processing systems that can analyze any data which means if you have Key/Value based data with relationships then it can probably be packaged into a VQL object, historically persisted, and analyzed in real time.

For more information please visit the VQL section of the Reflex Website.

written by Aaron Bawcom

On April 12th McAfee and Reflex announced a new product integration. Since then I have received many requests for clarification on what the relationship means and how it differs from other offerings and previous Reflex partnerships.

Let me first start with a little about Reflex’s philosophy on integration and partnerships. Our goal is to be the go-to company for integrated virtualization management and security. But that does not mean that we believe that all the technology that goes into the solution will be home grown.  In some cases there are subject matter experts that are far  more knowledgeable than Reflex and thus are better equipped to solve specific problems.  Intrusion detection and prevention (IDP) is one of these areas.  IDP is more than just building software to inspect network packets, it also needs to be backed up by a team of security researchers that provide the content, or signatures, for the scanning software that make it effective.  It is for this reason that Reflex prefers to integrate with the top vendors in the IDP space and thus the McAfee relationship.
Continue reading »

written by Mike Wronski

If your virtualization environment has snapshots growing like weeds in the yard then you are not alone. The more snapshots that exist for longer periods of time degrades the performance of the virtual machine the snapshots are on. To further illustrate the flexibility of the automation engine in the Reflex VMC we will illustrate a real world example of addressing this problem. This example dives deeper into the concept of an Action. An Action is at minimum a script and can also include a VQL query. An action can either run a script either on a periodic basis or whenever the output of a VQL query changes. We will illustrate an Action that uses the output of a VQL query in a subsequent post. For now, we will take a look at Actions that are run on a periodic basis. First lets describe what we want to accomplish with this policy:

1. Send an e-mail to the owner of a VM and the IT Admin when they have reached X snapshots and the image is not marked as an exception
2. Send an e-mail to the owner, IT Admin and the Group Admin when they have reached X snapshots, the image is not marked as an exception and the X snapshot condition has lasted more than Y days
3. To make sure virtual machines have the proper data on them, Query for all machines that have no owner set and tell the Virtualization Architect the name of the machine and what functional group the VM is in if any

Continue reading »

written by Aaron Bawcom \\ tags: automation, python, scripting, sprawl, virtualization, VQL

A very big feature that was added to the Reflex VMC in version 2.9 that was released in the fall was the addition of VMC PlugIns. This feature allows for users of Reflex to dynamically add features to the product without the necessity of a new installation from Reflex. The new PlugIn capabilities offer the ability to:

1. 1. Add new Right Click capabilities to any object
   2. Add new strongly typed SOAP Services
   3. Add new Policies that are constantly enforced

 
Continue reading »

written by Aaron Bawcom \\ tags: python, scripting, vijava

Many people ask me, “How do I ensure that the network segmentation is happening and how can I prove that my systems are in compliance?”. This is a difficult thing for people to understand, in the virtual environment, but it is also an easy thing to answer. With Reflex’s vTrust, part of the VMC, you can quickly and easily see which policies are associated with a particular VM, or set of VM’s, as well as view all ACL’s for that VM(s). This not only shows what is allowed or denied from a high level policy view, but also with the low level details on the physical access to and from the systems.

After you have defined what policies and rules are desired for the VM(s) the VMC and vTrust automatically create the access rules and also update them if and when they might need to be updated.

There are a few major components in the VMC that help to create, manage, and ensure network segmentation for VM’s in the virtual infrastructure. These components are:
Continue reading »

written by Tommy Speigner

The scenario is this, you want to create a Host profile with vProfile and then have that profile applied to all new servers as they come online. We will use the VMware Host Hardening Guidelines as our example vProfile. We will then schedule this for all the existing hosts and then use this for any new hosts that come online.

Let’s start by using the default ESX_DISA_SECURITY Profile that comes with vProfile.

1. Launch Configuration Management, in the VMC.

   

2. Select Host > Profiles
   1. Bind a Profile to a Host > Click Profiles

      

   2. Select the desired Profile.

      

   3. Now create the Host Binding:

      

   Continue reading »

written by Tommy Speigner \\ tags: Configuration management, Policy, policy enforcement, Reflex VMC, vProfile