Prior to vCenter Server 4, customers wishing to prune their VC database down needed to do so manually.  VMware provides an SQL script via KB 1000125 that allows administrators to delete old tasks, events, and performance stats.  By the way, I am the author of that SQL Server purge script — fear my mad T-SQL skillz!

What a difference a vSphere makes…  Now in vCenter Server 4, there is a simple administrative option to configure the retention period for tasks and events.  By default, the data is retained indefinitely, but as you can see, it’s quite easy to set it up according to your needs:

Related posts:

1. VMware vCenter really needs an agent
2. VMware vCenter Server performance stats levels
3. PowerShell Prevents Datastore Emergencies
4. Installing vCenter on Linux Technical Preview

The statistics levels are easily — perhaps too easily — configured by using the VI Client:

You can read in more depth about the performance statistics in the Basic System Administration manual and the VirtualCenter Monitoring and Performance Statistics Tech Note, which has this to say about stats level selection:

Level 1 has very low overhead on both the VirtualCenter server as well as the ESX Server hosts. Levels 2 – 4 have slightly greater overhead on ESX, but can adversely impact VirtualCenter performance if there are more than 10 ESX Server hosts. In particular Level 4 can quickly fill the VirtualCenter database, so it should only be turned on for limited periods of time.

When it says Level 4 can quickly fill the VirtualCenter database, that’s not hyperbole.  Take a look at this chart to see what happens when the “5 minute” interval is changed from level 1 to level 4:

The VPX_HIST_STAT1 table — the one that corresponds with the “5 minute” interval — immediately begins consuming many times its previous size.  After a day, the number of rows in that table increased about 20 times!  (The tick marks across the x-axis of this chart each represent an hour.)

Recovery is simple, but not immediate.  After resetting to level 1, the additional rows disappear through attrition as the rollup process does its job.

The statistics levels are configurable in order to collect more detailed performance data on your VMware infrastructure.  As with most things, there is a tradeoff — making it critical for administrators to know that a seemingly simple change via the UI can put a huge hit on your database — not only in storage requirements but performance, too.  Do yourself a favor and talk to your DBA before changing your vCenter Server performance stats levels.

Related posts:

1. VMware vCenter Server 4 task and event retention
2. VMware vCenter really needs an agent
3. vSphere Thin-Provisioned Disk Performance
4. Upgrading your management platform from the entry-level edition

Managing the data behind these charts is actually a very demanding responsibility of vCenter: receiving, storing, and later aggregating to keep the database size under control.  Starting with VC 2.5, a couple of big changes were made in this area to help with scalability: the schema was modified to divide stats data into multiple tables, and the rollup duty (data aggregation) was delegated to the database server’s native job scheduling mechanism.  This article focuses on SQL Server and the SQL Server Agent; there is an equivalent capability in Oracle as you would suspect.

The VC 2.5 installer creates a set of three jobs (for past day, month, and week) and throws up a notice reminding you to make sure the SQL Server Agent is running on your database server — for good reason, ignore it at your own peril.  You can see the jobs in the SQL Server Management Studio.

You’ll notice in this example there are actually three instances of each of the three jobs because there are multiple vCenter databases hosted on this SQL Server.  While not typical, it makes perfect sense in a test environment.

Backstory In the initial VC 2.5 release, the jobs always had a static name, thus making it impossible to run multiple vCenter Servers against one SQL Server instance.  I knew that was going to annoy a lot of people, so I filed a bug to have it changed — finally got it in 2.5 Update 2.

What happens if the SQL Server Agent is stopped and these rollup jobs quit running?  A picture is worth a thousand words:

In this chart, the abrupt increase in database rows corresponds to the point in time that the rollup jobs were disabled.  vCenter continues receiving performance stats from hosts and dutifully depositing them in VPX_HIST_STAT1.

Essentially, the most granular data (VPX_HIST_STAT1) starts piling up, never aggregating into the summarized tables for longer-term storage.  In a large environment, it is not uncommon to see millions of rows in these tables.  If something goes wrong with the rollup jobs, those millions start multiplying in a hurry.

One nice thing about having the rollups controlled by the SQL Agent is that VC does not even need to be running for the process to take place, which also takes some load off of the VC server.  After resuming the jobs, the data flood quickly subsides and things go back on track:

One note about SQL Server Express, the bundled database included with VC: There is no SQL Agent in that product.  Before you panic, please realize that VMware engineers know this.  For those deployments, the rollup stored procedures are simply kicked off by vCenter Server (vpxd), just like VC 2.0.x.  I could not believe the creative workarounds some well-meaning folks have attempted to implement because of this situation — it had something to do with the Windows task scheduler.  Please do not do that!

Hopefully now you have a little better understanding of the relationship between your database job scheduler, vCenter Server, and the size of your vCenter database.

Related posts:

1. VMware vCenter Server 4 task and event retention
2. VMware vCenter Server performance stats levels
3. Installing vCenter on Linux Technical Preview
4. New VMware ESXi Management Kit