Krystaltek

Blog Moved

2010-10-25T11:25:00.003+11:00

Welcome. I have moved my blog to blog.krystaltek.com. Please head on over there.

Kicking the Netapp Tyres

2010-10-23T20:52:00.004+11:00

This past week for me was spent in the Melbourne Netapp office on the NCDA training. Having had no real exposure to their arrays prior to me starting my role with Empired about 6 months ago I generally approached everything from what I had seen and knew about HP and EMC SAN’s. Let me tell you by the second day I knew that was probably the worst thing I could have done.

I am very fortunate that my employer is a very strong Netapp partner and being the beast that I am I love to learn new tech. In a few weeks time I am sub-contracting (for the want of a better term) to Netapp to work on a project so it was probably an opportune time to get my feet wet.

Netapp run some fantastic courses:
NCDA - http://www.netapp.com/us/services/university/certifications/ncda.html
NCIE - http://www.netapp.com/us/services/university/certifications/nso-502.html
ASAP - http://www.netapp.com/us/forms/education-emea-nasap.html

My instructor was great, which made all the difference. I think he managed to give everyone in the class what they wanted to take away. For me personally, I wanted not so much to learn how to read the command line help to find the right commands and switches (I can do that any day), but to understand the fundamentals of ontap and wafl. I Have had the ontap simulator VM installed at home and on my laptop for some months and have used it to do various bits and pieces.

One of the main things I picked up on was the lack of competitor bashing. Frankly, I expected quite a bit. I am a pretty open minded guy and I know that EMC, HP and all the others don't employ idiots. They engineer their technology their way because they believe in it and good luck to them.

Having said that, some of the Netapp technology like Flexcache, Flexshare, Snapmirror, Snapvault, Metro Clusters along with their snapshot and dedupe methodology impressed me quite a bit. The fact is all their products are relatively easy to understand when you grasp the 3 cornerstones of wafl:

1. WAFL is heavily write optimised and can very efficiently place 4k blocks
2. WAFL always knows the difference between 0 and non 0’d blocks
3. WAFL only ever writes to a non 0’d block.

Over the course of the 5 days the answer to nearly every question on how ‘x’ feature can work so well was because of one or all of these rules. Some of the way they do with dedupe, CRC checking, write chaining and minimising disk seek times as well as how they implement read cache is honestly, to me, quite amazing. In addition I liked that regardless of the smallest 270 right up to the 6080’s you can run the same software and license pretty much the same features!

I know also that they have some additional technology being released by the end of the year which is game changing and I look forward to seeing it in action or in fact reading more about it. I also recently saw a video using VMware Orchestrator in conjunction with Netapp Storage Service Catalogue and VMware vCloud Director to provision compute and storage workloads in the one process which was totally cool.

All in all, I’m pretty taken by the technology the have created and are still creating and I also quite like the go-to-market message of not driving sales dollars, but helping customers and partners implement the best solution for them, hence the popularity of their V-Series I suppose. I havent closed the door to learning as much as I can about other storage arrays, but for now my focus will be Netapp and I’m more than happy to play on that swing-set at this point in time. Rarely have I got so much out of a week as I did last week.

VMware vCloud Director - My Notes

2010-10-23T10:11:00.005+11:00

Over the last little while I have been reviewing some documentation to help me with an upcoming project. In my typical style I used Evernote to generate a dot-point list of tips, thoughts, considerations for implementing vCloud Director.

Over time I added to these until I thought I had a good list. No doubt there will be more that I haven't quite covered here but these are my own notes and I urge you to review the documentation for yourself.

On a side note, I have always found the vCloud gurus like Duncan Epping, Frank Denneman and the like helpful for bouncing ideas off and you can check out their blog's for more information on the topic, I'm even sure they'd answer a question or too if you were nice. Point is, there is plenty of information and resources available now.

Thanks to Duncan for picking up some errors in my notes. I have rolled his comments into my notes here and updated my working notes.

For ease of reference if broken the lists down into a series of headings, in no particular order

Network

Separate network traffic for management, VM
Seperate External Networks from Network Pools
Full network and switch path redundancy
Use vDS in each cluster for vApp
Increase the MTU to 1524 to cater for vCD-NI
Increase the vDS switch ports from 128 to 4096
Port Groups are created dynamically with a vCD-NI or VLAN backed Network Pool
Use trunk ports for hosts
Make sure you've got a pool of IP's available to dish out to orgs for the required networks

Storage

Redundant storage paths, host, switch, controller
RDM's are not supported, must be VMFS
Avergae vApp size x number of vApps + any overhead/spare capacity
Don't load more than 15 or so VM's per LUN (general rule of thumb, but should always be based on workload pattern)
vCD places VM's on org Datastores automatically based on free space
Consider creating LUN's with different RAID profiles to meet org requirements
How big are the vmdks
How many per vApp
How many VM's
What is the workload pattern
RTO required etc
vCD transfer volume - Mounted by all cells, required to allow customers to upload vApps. Initial size recommendation 250gb. Must be NFS!

Cells

vCD cluster - One RP per cluster
2 provider vDC's
Gold and Silver
Provider vDC can only map to one RP
Datastores should not be shared across vDC's although can be

The following image helped me conceptualise a typical gold/silver type arrangement

Management/Host

Single management cluster of 3 hosts running vCenter, vCD, vSM, Syslog/vMA, Chargeback, Oracle, Active Directory/DNS/DHCP, vDR, VUM.
Use FT to protect if necessary
Management cluster workload is failry predictable/static
Consider the sizing of the management VM's i.e how much grunt for vCenter/Database server with such a workload
Org workloads are generally very unpredictable and should be as scalable as possible
2 resource clusters presented as separate logical resource groups
Use vSS for Management/vMotion on resource hosts. Use vDS for external networks/pools
Place management and provider cells are in the same site to guarantee the same network/site performance
Catalog of vApp templates which can be shared with all orgs, called 'master catalog'. Internal catalog used for staging or dveloping vapps
Can specify the max time that a vapp runs for or gets stored on disk, storage lease/runtime lease.Poweroff and Delete options after expiration
VM's provisioned by vCD cannot be FT enabled or SRM. The additional vCD layer causes problems with traditional vCenter techniques
Keep in mind the scalability of a single vCenter server and the max cluster sizing to allow the maximum amount of hosts/vm's per server
Keep host config standardised by using Greg's special sauce pxe boot and post config esxi scripts and vCenter config script. Or you could use Host Profiles!

Moniroting/Logging

Use the Manage and Monitor tab in the UI
Prefer a syslog server (vMA) for everything other than vShield Manager (doesnt do syslog)

Tennant Security

Enable host lockdown mode
Set all vSwitch settings to Reject
Use AD auth for vCenter, remove local admins group access
Use LDAP/AD authentication to lock down organisational access
vShield Edge devices protects Layer 2 traffic being seen across orgs
Firewall configuration available in the vCD UI

There may be more added in the future. If you have any I could add/or correct let me know.

HA Maintenance Mode – Just do it!

2010-10-13T21:30:00.001+11:00

After seeing Duncan Epping’s (Yellow-Bricks.com) VMworld presentation today about VMware HA I was reminded by something he said and an event that happened recently.

In Duncan’s presentation he used the term ‘HA Maintenance Mode’ which was not overly familiar to me, however I immediately knew what he was referring to.

When enabling VMware HA you ‘should’ as a best practise, enable Host Monitoring. Host Monitoring allows HA to monitor the heartbeats of the hosts within the cluster. If a heartbeat has not been received form any particular host for 15 seconds that host is declared dead and the virtual machines running on that host are restarted on the other hosts within the cluster. Likewise if the hosts stops receiving heartbeats from the other hosts in the cluster for a period of 12 or more seconds and it cannot ping the isolation address the same result occurs (if the network is not restored within 15 seconds.)

The main reason to disable Host Monitoring is when performing any form of network maintenance on the network, thus potentially increasing the chances of heartbeat loss an a HA occurrence. You could of course disable HA altogether at this stage, however in the customers case they still needed to protect the cluster from other real server-based outages whilst still providing dedicated restart priority and admission control.

Why do I mention this? Well partly because Duncan reminded me, but also I saw this happen last week when network maintenance was performed by a third party and it caused some painful moments for the service desk team. Thirdly because it’s been in the availability guide since 4.0 so people should know about it! Make life easy on yourself and your service desk guys.

VMware vSphere 4.1 Feature Comparisons

2010-10-13T13:25:00.003+11:00

For a customer project recently I had to do a comparison of vSphere and other virtualisation options for them to satisfy their due diligence requirements.

The particular comparison they were interested in was between vSphere and Hyper-V. I found a couple of links that were of great help and saved me a huge amount of time. I thought i’d share them in the event they may help you one day too.

VMware vSphere 4.1 Feature Comparison
http://www.vmware.com/files/pdf/vmware-vsphere-features-comparison-ch-en.pdf

Compare vSphere Kits http://www.vmware.com/products/vsphere/buy/small_business_editions_comparison.html

Compare vSphere Editions
http://www.vmware.com/products/vsphere/buy/editions_comparison.html

Monitoring your vSphere environment with System Center Operations Manager – Part 1

2010-10-09T14:47:00.004+11:00

A big part of my customers requirements when designing or implementing a VMware environment is what happens after it’s installed. What sort of alerting, monitoring and reporting is available to both the service desk, managed services team, key business people etc.

There are many options which we could consider for each of these areas however for many of my customers they already have monitoring systems for just about everything else. In most cases as well they are also Microsoft customers with Enterprise licensing agreements, meaning they have easy access to most if not all of the Microsoft suite. System Center Operations Manager (SCOM for short) is a monitoring product that allows them the ability to improve their availability and performance metrics through enhanced service level monitoring.

I recently spent some time setting up SCOM and seeing what how I could integrate this as an existing monitoring system into a VMware environment. For this purpose I choose to use the Veeam nworks solution and not use the System Center Virtual Machine Manager (SCVMM).

Operation Manager Pre-requisites
I won’t run through the ins and outs of installing SCOM as its pretty much like any other Microsoft product, get all the pre-reqs and dependencies installed and the process is pretty straight forward. In my case I was using a fairly simple setup on a single box Windows 2008 R2 + SQL 2008 R2 + SCOM R2. The only issue I encountered was that because I was using my chosen version of SQL, Ops Mgr was unable to create the required database during install. I had to use the DBCreateWizard tool which is found in the extracted installs folder. It’s a pretty easy task to accomplish, the gui is fairly self-explanatory. Subsequently, Microsoft released this KB article which mentions the need to use the DBCReateWizard tool along with some more details. The Veeam documentation has full list of requirements for all Windows versions and also a list of tcp port requirements and I would recommend reading it before attempting an install.

You will require a couple of service accounts for the Veeam servers, I only used one account for both the EMS and VIC and continued to use it through the install. The Ops Mgr agent will need an account with local administrator permissions on the SCOM server, the same is true for the VIC and EMS servers. In addition, you will also need an account with Read-Only permission in vCenter. If you want to perform vCenter tasks from Ops Mgr then you will need to set the appropriate permissions on this account, although I personally wouldn’t recommend using Microsoft management tools against vCenter. Once I had created my appropriate service accounts, databases and installed the Ops Mgr features I required all was in readiness.

Getting the Software
The wonderful folks at Veeam have produced a Management Pack for Ops Mgr, known as nworks. The nworks landing page can be found at http://www.veeam.com/vmware-microsoft-esx-monitoring.html.

You can register/login and download a free trial of the management pack. The system requirements can be found in this link (requires login)
http://www.veeam.com/files/guide/nworks_mp_for_vmware_ops_mgr_2007_install_guide.pdf

The nworks file will extract to three separate installers and the management pack itself

-    Enterprise Management Server
-    Enterprise Management UI
-    Virtual Infrastructure Collector
-    nworks management pack

Installing the Veeam software
Begin by installing the Enterprise Management Server. You will need the license key that was mailed to you when you downloaded the package and pre-created service accounts. Attach the license file, accept the default port, select you service account, that’s about it.

Next install the Enterprise Management UI which is a web based configuration tool. It will require IIS installed on the server. Again, very straight forward. The defaults should be fine.

Finally install the Virtual Infrastructure Collector service. Use the same or a new service account.

Note: You can install these services on any server in your environment but for our purposes using a single server was sufficient.

Configure EMS
Open the EMS desktop icon or from the Start menu. Configure the connection details as per the screenshot below using the Connected Servers tab. I used all the defaults and entered my vCenter Server details.

Install the Ops Mgr agent
In order for all this to work together we need to install an Ops Mgr agent that will take the data collected by Virtual Infrastructure Collector and deliver it to Ops Mgr. This combination is referred to as a Virtual Enterprise Monitor System. It is possible to install the agent on either the vCenter Server or the Ops Mgr server itself. In larger deployments a dedicated VEM system is preferred.

From the Administration view select Configure computers and devices to manage. Select Windows computers then search for the system you want to use.

Select all the defaults and your service account (should have local admin rights on the server). You can verify that all is well by opening the Agent Managed view and verifying that the server is listed.

The next important step is to ensure that you enable proxying on the VEM server. Proxying enables the VEM to be able to update Ops Mgr with the discovered VMware topology and info as it is collected. To do this in the Administration pane open the Device Management-Agent Managed node. Right click the VEM server and open the properties. Go to the Security tab enable the tick box.

Importing nworks Management Pack
Management packs generally come with a set of dependencies. In this case the only dependency for nworks is the Windows OS core library pack. If you are monitoring any kind of Windows environment with SCOM then you should already have this installed. If not, select Import management packs and search for it from the catalog.

Repeat this step whilst locating the nworks management pack from disk (where you extracted the installer to)

Optimising VEM performance
In large VMware environments or where you are using multiple VEM systems it may be important to make a few registry changes to ensure that dealing with large amounts of collected is not an issue. To do this navigate to the nworks VMware node on the Monitoring tab, select _VIC State and right click your VEM server(s). Run the Configure Ops Mgr Agent task. For a list of the registry changes it makes see the install guide.

Verifying and Configuring Data
Once the installation is completed and we have left sufficient time to ensure data has been collected and passed to Ops Mgr we should be able to configure the view within the Monitoring Console.
To do this navigate nworks VMware node and select any of the views. You should see something that resembles your VMware environment. I have selected the _Datacenter Topology view below to illustrate.

The final step I did was to go through all views and select the metrics I wanted to display in the graphs. See below how I have selected both LUN to graph. (I had stopped the server for a while so that’s why you don’t see any data for previous hours/days).

There are quite a few views and metrics in the nworks management pack and hopefully we will be able to get some great info from them as well as other servers in our environment.

In Part 2, I will talk about creating workspaces to show only the information you really care about as well as how to create notifications for alerts.

Some additional reading:
Operations Manager Page
http://www.microsoft.com/systemcenter/en/us/operations-manager.aspx

Operations Manager R2 – Supported Configurations
http://technet.microsoft.com/en-us/library/bb309428.aspx

vCenter Task Lifetime and Timeout Settings

2010-10-05T21:04:00.003+11:00

One of my most used tabs in vCenter is the Tasks and Events tab as well as the Recent Tasks pane. Something I have been playing with is the ability to increase the number of tasks that can be displayed and for how long.

In order to explain this let’s revisit how some of it works.

As we know (based on our reading of this article) vCenter 4.1 can now handle up to 500 concurrent tasks. When a task is initiated it appears in the Recent Tasks pane for a default of 10 minutes. After which time it is only available in the Tasks and Events tab. This Recent Tasks list is capped at 200 events. Although the defaults should suit, you can modify both numbers.

Caveat: Always backup your vpxd.cfg file before making any changes, formatting or spelling errors will render your vCenter Server Service 'unstartable'.

Edit the vpxd.cfg file on your vCenter server (C:\ProgramData\VMware\VMware VirtualCenter\vpxd.cfg). You must restart the vCenter Server Service for the change to be recognised. I added the following two options to do some inital testing.

Note: Default = completedLifetime - 600
Note: Default = completedMaxEntries - 200
Also Note: The official documentation has a formatting error. It omits the '/' in the first value. See caveat!

Task Limits

After setting the completedMaxEntries to 3 to see the result and then invoking what I called 'death by folder creation/deletion' I never retained more than 3 entries as seen below.

Simon Long made the observation here that you can in fact control the number of Events displayed in the VI client. The smallest number you can set is 10 and the largest 1000. I set mine to 1000 for kicks. Notice the scroll bar on the extreme right of the second image below.

Task Timeout
vCenter tasks timeout when there is no activity between the host and the vCenter server within a certain timeframe. By default these options are set to 60 secs for normal operations and 120 secs for long operations (such as maintenance mode). You can change these settings by editing the Timeout Settings in vCenter server Settings.

In addition after setting the completedLifetime entry in the vpxd.cfg file to 10 for testing all tasks, a completed task only appeared in the Recent Tasks pane for a period of 10 secs, before it dissapeared up into the Events tab.

Summary

Although I would encourage precaution, these settings should be sufficient enough for you to tailor your task and event views for any particular time and situation.

vCenter 4.1 Performance and Scalability Improvements

2010-10-03T15:49:00.001+11:00

Recently I had to document some of the improvements in vSphere 4.1 to version 4.0 for a project requirement. As you could imagine it was a substantial document as there were many enhancements in version 4.1. What I wasn’t aware of was that there were a number of scalability and particularly performance gains in vCenter 4.1 over 4.0.

As you probably know VMware installations are becoming larger and larger these days especially with the advent of cloud deployments. VMware has driven the product to cater for these large installations by increasing the scalability of its product and in particular vCenter. You should be aware that vCenter server 4.1 is now 64bit only. I personally heard some wailing and gnashing of teeth when this was announced, but in reality I wasn’t surprised. 32bit has been a hindrance even at a desktop level for a number of years (hell even my mum uses 64bit) 64bit is essential in the year 2010. How many of your core infrastructure servers run a 32bit OS? Not many I bet, if they do, then you should plan an upgrade!

Evidence of this scalability increase is in the configuration maximums guide

Note 1: The data for vSphere 4.0 is based on a 64bit vCenter. 32bit installations have less capability in many instances.

Note 2: In doing some research I found conflicting documents that mentioned one or two different maximums. For the sake of consistency I referenced the config maximums guides, as I believe them to be correct.

Scalablity and Performance
Such scalability improvements were in some cases ultimately provided by some notable performance enhancements, such as:

Faster vCenter server startup – around 5 mins for maximum vCenter inventory size
Faster vSphere client login
Better vSphere client responsiveness (see note)
Increased number of concurrent operations with lower latencies and increased throughput
Better integration between HA/DRS/DPM and FT
Faster VM recovery with HA
Better load balancing due to improved DRS/DPM algorithm
Lower resource usage by vCenter agents by up to 40%
Improved performance at higher vCenter inventory limits
Improved performance at higher cluster inventory limits
Faster host operations
Faster VM operations on standalone hosts – up to 60% reduction in latency
Reduced VM group power-on latency by up to 25%

Note 3: VMware recently released a KB article detailing how to speed up the vSphere client when running on Windows 7 systems. http://bit.ly/95pwRM

vMotions
vCenter 4.1 also increased the number of simultaneous vMotions

I would recommend for anyone who requires more detail to read the following document as i have not covered all the relevant information here.

http://www.vmware.com/files/pdf/.../VMW-Whats-New-vSphere41-Networking.pdf

JVM
Finally the last feature I liked was the ability at install or otherwise, to configure the amount of RAM given to the JVM for Tomcat. This feature will help you control the memory utilization of Tomcat even when you scale your environment.

At Install

Post Install

Articles referenced:

http://www.vmware.com/pdf/vsphere4/r40/vsp_40_config_max.pdf
http://www.vmware.com/pdf/vsphere4/r41/vsp_41_config_max.pdf
http://www.vmware.com/files/pdf/techpaper/VMW-Whats-New-vSphere41-vCenter41.pdf

Storage DRS in our lifetime - You better believe it!

2010-09-01T12:24:00.001+10:00

For quite a while now, and I'm sure I'm not alone, I have wanted the ability to automatically balance or even re-tier my Virtual Machines without needing the input from those funny (looking and smelling) storage guys. Furthermore I've always dreamed about being able to trigger Storage vMotion when a Datastore reaches a configurable threshold. Someone more highly skilled with Powershell could probably whip that up, but I digress.

Whilst I am fortunate to work with the storage teams for most of my clients, in some cases the VMware guys have little knowledge of the LUN configuration on the backend. I have seen some creative Datastore labeling at times which indicate usage definition or even tier labeling which can help in management. I have always wanted to be able to logically group Datastores into something that would be more useful than using a folder and would represent the Datastore characteristics a bit better.

You will all agree DRS is one of the outstanding features of vSphere and has been for quite some time. What if we could extend this ability to not only migrate VM's to different hosts but also to different Datastores!

Well the word on the street is that in the near future you will be able to do just this. What's more, I understand you will be able to create Datastore Groups in order for Storage DRS to load balance VM's across Datastores in the group. What's cooler, you will even have the added functionality of affinity rules to ensure VM's never (or always) reside on the same Datastore Have you ever wanted to put a Datastore into maintenance mode in order to make some backend changes, and not have to manually or script Storage vMotion? It appears it will become a reality.

Storage DRS will be configured for capacity and also performance requirements and will not be anywhere near as aggressive as regular vMotion, nor would you want it to be! I can only imagine there will be some nice advanced tweaks for us to be able to fine tune the parameters.

There is little information around at present but I'm sure In future more technical details will come to light, i'll definitely be keeping my ears and eyes open. I also don't have any wonderful inside info but let's hope that this feature can make it's way into the next version of the vSphere product and life could be even easier!

Exchange 2010 and vSphere 4.1

2010-09-01T00:19:00.002+10:00

I have been working with some colleagues on some Exchange 2010 designs in the past month or so and one of the questions that has come up is what options do we have around VMware. I decided to do some digging and have noted some thoughts.

Exchange 2010 is a little different to its predecessors and especially Exchange 2007 in its High Availability model. For those who are not familiar Exchange 2007 introduced single-copy clusters, local continuous replication, standby continuous replication and cluster continuous replication. I wouldn't delve into each of these here but would simply point you to this document on Technet http://technet.microsoft.com/en-us/library/bb124721(EXCHG.80).aspx

Exchange 2010 simplified (in my view) this with the introduction of Database Availability Groups. A simplistic explanation is a DAG is a group of mailbox servers which provide a working copy of a database and other replicas of other working databases on each server. Essentially DAG's use a combination of CCR and SCR to replicate copies of the databases to different mailbox servers in the DAG. In the event the a mailbox server goes down one of the replica's becomes the primary copy. Figure 1 is the most well known image of how DAG's are set out.

Figure 1

Microsoft's implementation of DAG's or even CAS server arrays use Microsoft Clustering Service or MCSC to provide a truly Highly Available scenario also had implications for VMware and indeed Hyper-V environments.

As far back as ESX 3.5 there were some wonderful constraints like:
- Guest boot disks had to reside on local storage
- MSCS guests could not be part of VMware clusters
- vMotion was not possible
- No support for iSCSI
In general, forget it :)

In vSphere 4.0 the game changed a bit:
- MSCS and Fault Tolerance were not supported
- vMotion still not supported
- Hardware version 7 was required
- Boot from FC SAN possible where volume = VMFS
- MSCS guests could not be part of a HA/DRS cluster still

With the recent release of vSphere 4.1 we now have some more options:
- HA clusters now fully support MSCS
- DRS can be enabled but you must use affinity or anti affinity rules
- Use Advanced DRS rule 'ForceAffinePoweron' to enforce affinity rules
- Set the automation level for MCSC guests to partial
- Use DRS Groups and vm-host affinity rules
- vMotion is not supported
- Still only supported on FC disks
- All the hosts that run MSCS guests MUST be managed by vCenter
- Still no support for FT
- Host versions must still match

So we are good to go right, well not quite. That's only half the story. Microsoft in a Technet article http://technet.microsoft.com/en-us/library/aa996719.aspx details the following:

Microsoft doesn't support combining Exchange high availability solutions (database availability groups (DAGs)) with hypervisor-based clustering, high availability, or migration solutions that will move or automatically failover mailbox servers that are members of a DAG between clustered root servers. DAGs are supported in hardware virtualization environments provided that the virtualization environment doesn't employ clustered root servers, or the clustered root servers have been configured to never failover or automatically move mailbox servers that are members of a DAG to another root server.

As ambiguous as that statement seems to me it appears that Microsoft will still not support any VMware clustering features in conjunction with Exchange DAG configuration. As far as my information goes, simply disabling these features on the guests is also not supported.

Unfortunately the design we were working on were quite large, one was 80k users so we had nowhere to turn but to ensure that they were completely supported on this one.

On a side note, this is actually no different than running Exchange DAG's in a clustered Hyper-V server or clustered guest configuration. Now if you're a thinking man, you've probably worked out a few other ways you could set up High Availability using VMware alone without DAG's however the main customer requirement was Exchange best practise.

I've gathered some VMware links on MSCS for any futher reading:
http://www.vmware.com/pdf/vi3_35/esx_3/vi3_35_25_u1_mscs.pdf
http://www.vmware.com/pdf/vsphere4/r40/vsp_40_mscs.pdf
http://www.vmware.com/pdf/vsphere4/r41/vsp_41_mscs.pdf

Overcommit Much?

2010-08-27T20:43:00.001+10:00

Most days I just love VMware, today was esepcially one of those days. I had a good discussion at work and we got a considerable piece of work lined up for the next 12 months which our whole team can be involved in.

Debates have raged over the use of memory overcommit in the datacenter over the last few years, generally I found it’s non VMware people telling me its a rubbish feature and you would never use it. I’ll leave that debate for others, its been done to death and quite frankly you’ll covince yourself when other products actually can do it.

Along those lines, a funny thing happened to me tonight which provided me some humour at the end of my week and I hope it will to you.

I was building another test ESXi 4.1 server, virtualised on my 4.1 lab (of course) when I accidentally hit the memory size config wrong and it defaulted to 255gb (as it does)

Although vCenter kicked me out I had no problem connecting to the host directly to find out what the matter was. I dont know about you but I think configuring 260+ gb of memory on a host with only 8gb and still being able to ping, rdp etc to the other 2 guests running on the box ws pretty cool!!

I took some screenshots so I could continue to laugh at the crazyness :)

Dude where's my LUN

2010-08-24T13:55:00.001+10:00

I encountered an issue today which I’d never seen before. A few weeks of tinkering in the 4.1 lab had left it in God knows what state. I needed to connect up the storage and get everything going again.

I had a look at one host and noticed no iSCSI luns were present. Strange, networking was all configured and the lun was present on other hosts. Checking the iSCSI adapter and it was enabled. Looking at the adapter properties I couldn’t see any properties configured, what’s more I couldn’t configure them, everything was greayed out.

The next two screen shots were what I was looking at.

Note how the ADD targets button was disabled.

The solution was not highly technical and it was probably the first thing I thought of after checking the configuration and log files, short of rebooting the host.

I connected to the host via Putty and queried the iSCSI initiator using:

esxcfg-swiscsi –q

Software iSCSI is enabled

OK all good there, lets disable and re-enable

esxcfg-swiscsi –d

Disabling software iSCSCI for next boot

esxcfg-swiscsi –e

Enabling software iSCSI

Next I’ll do a rescan

Esxcfg-swiscsi –s

Scanning vmhba38

Flicking back to the GUI I noticed that now my lun was present, all the iSCSI options were enabled.

Weird, but at least an easy fix.

What ESX Admins Group? – A Tale of RTSM and AD Authentication

2010-08-16T16:33:00.001+10:00

What ESX Admins Group? – A Tale of RTSM and AD Authentication

A few weeks back a friend mentioned to me he was configuring Remote Tech Support Mode with AD authentication and that he had some issues with it in the beginning and I should check it out for myself.

Initially it appeared he was indeed right. When the ESXi host is joined to the domain and you have enabled Remote Tech Support Mode and you try to log on to the host via SSH client with a domain account. At first authentication was failing. At this point I had to dig a little further to find out what is going on..

The hostd.log file was showing error messages about not having an ESX Admins group. Now I had no such group so I was a little confused. I decided to go digging.

The configuration guide mentions the following:

vCenter Server registers any selected Windows domain user or group through the process of assigning permissions. By default, all users who are members of the local Windows Administrators group on vCenter Server are granted the same access rights as any user assigned to the Administrator role. Users who are members of the Administrators group can log in as individuals and have full access. Users who are in the Active Directory group ESX Admins are automatically assigned the Administrator role.

Now as I mentioned I didn’t have an ESX Admins AD group and nor did I want one! So in order to test my theory and get authentication working I created one and added myself to the group. Bingo I was in!

pam_per_user: create_subrequest_handle(): doing map lookup for user "INTERNAL\gmulholland"

pam_per_user: create_subrequest_handle(): creating new subrequest (user="INTERNAL\gmulholland", service="system-auth-generic")

PAM password auth succeeded for 'INTERNAL\gmulholland' from 192.168.100.6:49841

Interactive shell session started

This model of adding everyone to this group doesn’t really work for me so I decided to proceed further.

At this point I tried to login into a host which has authentication configured for a group called ‘lab_cluster_au’ of which I am also a member of and could login fine (authentication for this group was configured on the host). If I remove the ESX Admins group from AD and wait for any replication I can no longer login to any host again with the same account. I won’t bore you with all the log traces however I have kept them should I need them in the future.

But that is not where the story ends. I put the problem down for a while and came back to it today. The ESX Admins AD group had long since (days) been deleted and permissions were granted only to my ’lab_cluster_au’ group (see below)

I fired up an SSH session and logged in as root. Tailed the /var/log/messages file and tried to login again with my username (which was a member only of the lab_cluster_au group). Again it worked!

Aug 16 05:16:27 dropbear[883204]: pam_per_user: create_subrequest_handle(): doing map lookup for user "gmulholland"

Aug 16 05:16:27 dropbear[883204]: pam_per_user: create_subrequest_handle(): creating new subrequest (user="gmulholland", service="system-auth-generic")

Aug 16 05:16:27 dropbear[883204]: PAM password auth succeeded for 'gmulholland' from 192.168.100.20:51701

At this point I was left scratching my head. I had on completely separate occasion’s verified very different behaviour. I had been meticulous in my configuration and testing and I was still getting varying results. In fact the latter is how I would expect it to work but I have seen on different days and tests that it in fact doesn’t.

Initially I had seen what my friend had picked up and we were both under the impression that RTSM AD authentication in vSphere 4.1 was completely dependent on the existence of a Security Group called ESX Admins and that without it no other authenticated group would work at all. Additional testing showed me that this actually was not the case. Furthermore the difference in the results were leading me to believe there was an inconsistency in the way it worked. I have little doubt I will be able to get different results in a day or so when I can come back to it again.

I’d love to hear from anyone else who has experienced such symptoms or has tried it. At this point, with the tests I have so far carried out, I’m not sure I could rely on it.

Problem adding AD Permissions in vCenter 4.1

2010-08-07T14:01:00.001+10:00

This morning I was playing with Active Directory Authentication in my vSphere 4.1 lab and came across a weird problem which wouldn’t allow me to search or list any accounts in the domain.

The answer was dead simple and although most people won’t generally run into but I thought I’d put my info down somewhere.

The error message I was getting was:

I did find this KB article and tried setting the timeout and query limit as it suggested but didn’t help

For some reason (probably laziness at install) my vCenter services were running under the Local Administrator account.

After creating a service account within my domain and configuring the services to use this account I was off to test the outcome. Yes, I probably should have done this before and in this instance I could have also used Local System account.

I was then able to select the users and groups from my Active Directory

Adding AD group permissions a cluster now works, which was my original goal, yay! I should have known better :)

Greg

vSphere 4.1 FT and Update Manager Anomaly

2010-08-06T20:02:00.000+10:00

vSphere 4.1 introduced interoperation with FT and DRS. Recently Frank Denneman wrote a great article about this feature here. Without re-hashing all the details here is the important info:

• DRS can now balance and place FT virtual machines in a cluster.
• EVC must be enabled on the cluster for this to occur. If EVC is not enabled vCenter disables DRS on the FT guests as it did in vSphere 4.0

A few weeks back I looked at the new VMware Update Manager remediation options and recently something occurred to me. If DRS is in fact able to place FT guests then why is there an option that requires you to disable FT before remediating hosts? It seems to me that if this was the case, wouldn’t it make sense to allow the DRS and vMotion to do these steps for you?

There are a couple of reasons I tossed around for this requirement, possibly if EVC is not enabled on the cluster you should disable FT when updating hosts, however in this instance DRS is disabled for the primary and secondary guests and a manual vMotion needs to be performed anyway prior to remediation, so does disabling FT actually buy you anything?

Maybe another reason could be when using VUM to disable FT it is VUM’s recommendation that you remediate all hosts in the cluster to ensure compatibility. This seems a bit ambiguous to me considering the Availability Guide, What's New in 4.1 (Availability and Resource) Guide and this KB article mentions there is no requirement for the host build numbers to be the same, only the FT versions, however page 119 of the VUM admin guide details:

‘FT requires that the hosts, on which the Primary and Secondary virtual machines run, are of the same version and have the same patches installed. If you apply different patches to these hosts, FT cannot be re-enabled’

The only other reason I can see would be to ensure that FT version numbers are the same across all hosts in the cluster, however if DRS is capable of determining this if it can place the vm’s then why not use it??

To me it appears that the Update Manager options or at least documentation are out of sync with the product information. I am still yet to confirm which of these statements is true as in my case all cluster hosts are the same patch level but personally I am believing the Availability Guide here.

I am also unclear as to the logic behind disabling FT during remediation, if anyone knows any different I’m happy to listen. I would prefer Update Manager to be fully automated and hoped that the new FT features would have extended to VUM as well so FT guests could in fact be remediated the same way they are balanced/migrated in the cluster. There is a case in point here for requiring DPM to be disabled during update, but that’s for another day.

For more information on Fault Tolerance I’d encourage anyone to refer to the following:

vSphere Availability Guide 4.1 (specifically Chapter 3)
http://www.vmware.com/pdf/vsphere4/r41/vsp_41_availability.pdf

Changes to Fault Tolerance in vSphere 4.1 KB article
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1022844

VMware Update Manager Admin Guide
http://www.vmware.com/pdf/vsp_vum_41_admin_guide.pdf

DRS/Fault Tolerance Placement Restrictions

2010-08-02T14:33:00.000+10:00

Today I came across something in doing some tests of the VMware Fault Tolerance feature in vSphere 4.1. I was attempting to migrate some of the Primary FT virtual machines to another host in the cluster when I received the following error message:

Host already has the recommended number of 4 Fault Tolerance VMs running on it

In vSphere 4.1 DRS and FT are the best of friends however this error message relates to a default scenario where DRS will not initially place or load balance more than a certain number of FT guests on any given host. The default number is 4 FT guests.

Configuring the Advanced HA option ‘das.maxftvmsperhost’ to a higher number such as ‘6’ will fix the problem. Setting this option to ‘0’ will instruct DRS to ignore the restriction completely.

Probelmo solved!

Greg

CPU Scheduler Enhancements in vSphere 4.1

2010-07-20T19:54:00.001+10:00

The CPU Scheduler is one of the core features on any ESX host. Its role is to balance and provide resources in an ESX environment. In modern times when servers are equipped with multiple CPU’s and multiple cores per CPU the allocation of such resources and ultimately the host performance depends greatly on it.

In 2009 vSphere 4.0 introduced some changes into the way the CPU scheduler worked in 3.5. These features included Multicore-Aware Load Balancing, further relaxed co-scheduling to what was in ESX 3.5 and the removal of the CPU scheduler cell.

Before diving in, let's refresh our understanding of NUMA nodes. NUMA nodes are a group of processors or usually processor cores that share a common system bus, I/O channel within a system. For my favourite article on NUMA nodes and ESX have a read of Frank Denneman’s article here http://frankdenneman.nl/2010/02/sizing-vms-and-numa-nodes/

One common mis-conception is that NUMA is purely a scheduler feature when in actual fact it is a combination of memory management and CPU scheduling. In ESX 4.0 virtual machines with multiple vCPU’s were (where possible) placed inside a NUMA node. Each NUMA node in the system has its own memory. vCPU’s within a NUMA node can access the local and also remote node memory (when required). Obviously there is a greater latency accessing the remote memory.

This was generally not an issue when the virtual machine has a 4 or less vCPU’s, as with quad cores CPU’s for host servers a virtual machine would fit snugly into a NUMA node. So what happens when we have a 6 or 8 vCPU virtual machine? In ESX 4.0 the virtual machine would be unable to fit inside a single NUMA node (defined as a Wide-VM) and the virtual machines will be placed into round robin mode whereby memory can be supplied by local or non-local memory. This ultimately means a performance hit on such virtual machines.

Wide-VM NUMA Support

Wide-VM NUMA support is a new feature in ESX 4.1. It allows the ability to split the virtual machine into what is refered to as a ‘NUMA client’. The advantage of this method is that the NUMA client size does not exceed the number of cores in the NUMA node. Furthermore, it will assign a ‘home node’ to each ‘NUMA client’. The benefit this allows is the ability for each client to schedule within its home node and hence within the correct NUMA node. In doing so we remove the need for cross socket scheduling and remove the performance penalty of a virtual machine requiring access to non-local memory.

It is important to note that Hyperthreading is not taken into account in this case, it is purely a matter of CPU cores.

The diagram below illustrates how with ESX 4.1 VM1 would reside within the NUMA Client on Node 1 and hence would be able to access memory local to the node without crossing to the other node to non-local memory. In the case of VM1 which has 6 vCPU’s this is a benefit as there is no requirement to utilise memory from Node 2.

Figure 1 – Wide-VM NUMA

Now compare that to the following diagram which illustrates how VM1 would be scheduled without Wide-VM NUMA in ESX 4.0. In this instance the operating system will have to use non-local memory for some of its operations as all vCPU’s are not able to fit within the same NUMA node.

Figure 2 – Normal NUMA

You can use esxtop to verify the memory allocation and current usage of each NUMA node realtive to the host or even guest. See Figure 3 below.

Figure 3 – NUMA statistics in ESXTOP

In this illustration the VM MDCDBS03 which is only a 4 vcPU is running on node 0 and is using the majority of local memory. In a 6vCPU virtual machine (without Wide-VM NUMA) I would expect to see a larger amount of NRMEM being used. Unfortunately I am unable to test the exact impact of this on this environment as it would require 4.1 upgrade which isn’t on the radar at the moment, however when I can I most certainly will.

In 2010 most servers will now ship with 4, 6 or 8 cores per pCPU and as such the benefit of Wide-VM NUMA support will only truly be felt for virtual machines that have 4-8 vCPU’s. The reason for this is that in general a 2 or even 4 vCPU virtual machine will most likely fit into a single NUMA node, depending on your hardware. Having dealt with some 4+ vCPU virtual machines in the last few weeks I can see how this feature would have been of benefit.

In my case the client was virtualising SQL Server and requested to run an 8 vCPU virtual machine with minimum 16gb Memory on a 4 socket (quad core) machine. In this case the workload for the server was highly taxed by this one machine, however they had some special requirements for what we were doing at the time. The problem with Wide-VM NUMA support in this instance was that the virtual machine will split into as few nodes as possible, in this case 2, which also left 2 nodes un-utilised (see Figure 4).

Figure 4 – 8vCPU on a 4 socket quad core host with Wide-VM NUMA

There is a good and bad to this. Consider the event in spreading the vCPU’s across all 4 nodes, granted there is now the ability to utilise all sockets however this also increases the tendency to access non-local or shared memory and hence we have a performance trade-off. Generally speaking, the ability to access localised memory is of greater benefit particularly in the case of systems that are not highly utilising each vCPU equally. Perhaps this is also why it’s the default setting. I would encourage anyone to test the outcome of both configurations under a typical or expected workload before drawing any conclusions.

Of course this would be a rare scenario but the good news is there is a way around it.

MaxPerMachineNode

Should you need to, you can limit the number of vCPU’s per NUMA client from within the VI Client.

- Power off the virtual machine

- Select the virtual machine and select Edit Settings.

- Select the Options tab.

- Under Advanced, click General, on the right, click the Configuration Parameters button.

- Click Add Row.

- Add numa.vcpu.maxPerMachineNode set to new value (i.e. 2).

Figure 5 – Setting the maximum vCPU’s per node for a virtual machine

In Figure 5 we are setting the maxPerMachineNode value to 2, in order to force our 8 vCPU virtual machine to utilise all four NUMA nodes within the host. Again the trade-off here is that although we can utilise all nodes there is still an overhead.

Conclusion

Wide-VM NUMA support is a new feature introduced in ESX 4.1. It allows a virtual machine with a large number of vCPU’s assigned to be split into smaller chunks called clients within a node. There is an associated performance benefit as the scheduling and memory management can be done within the home node and will never cross the NUMA node boundary (socket). The need for non-local memory access has been reduced with this technique and as such allows for a performance increase in virtual machines that generally have 4+ vCPU’s. Truly a cool feature!!!

Greg

VMware Update Manager 4.1 - Cluster Remediation Options

2010-07-15T20:30:00.006+10:00

A discussion a few days ago prompted my interest in the new VMware Update Manager remediation features. The discussion centred around how to guarantee a specific HA failover level with admission control enabled and whilst also having the ability to patch hosts at the same time.

In essence the problem we faced was if we were in the process of patching a host in that cluster and an isolation even occurred and HA invoked, how would we ensure that we still had enough resources available in the cluster to ensure all machines were restarted promptly. This would be a very rare occurrence and there are ways around it like, simply disabling admission control before patching, sizing the cluster for greater HA failover capacity, standby hosts etc..

In vSphere 4.1 this no longer becomes a problem. The Cluster Remediation Options enable the disabling of some features when patching hosts in a cluster. There are two ways this can be achieved; either at the time of actual remediation see Figure 1 below or from within the VUM plugin configuration tab. See Figure 2

Figure 1

Figure 2

Cluster Remediation Options Explained

Disable DPM – If DPM is enabled on any cluster remediation of any hosts in that cluster will not proceed. In order to remediate these hosts you will need to enable this option so DPM will be disabled at remediation time.

Disable HA admission control – If HA is enabled on any cluster, remediation of any hosts in that cluster will not proceed. Again you may need to enable this option in order to remediate such hosts.

Disable FT – If any primary or secondary virtual machines that reside on the hosts you wish to remediate, FT will need to be disabled prior. The reason for this is that FT requires that the hosts where the primary and secondary live be the same patch level.

It should be noted that once remediation has completed and the hosts are in a connected state again these features will be re-enabled on the cluster.

You will also notice the tick box to temporarily disable removable media devices which prevent the host from entering maintenance mode. This is a nice feature and circumvents the need to run any scripts to find out which virtual machines have attached devices. I am yet to fully test this with all kinds of devices I can think of however it should work with floppy drives and cdroms.

Remediation Report

The other handy addition in 4.1 is the ability to generate a report before remediation is actually performed. Run the report generation to see what issues were found within the cluster and the steps to take to ensure a successful remediation.

Figure 3

The things that the report will identify:

Connected floppy drives and CDROM drives – These items can prevent the host from entering maintenance mode. See the maintenance mode settings in the previous section.

HA admission control enabled – Prevents the migration of virtual machines where capacity may not be available or guaranteed by remaining hosts.

FT enabled – FT will not remediate a host where a secondary or primary reside.

DRS not enabled on the cluster – DRS should be enabled to facilitate best placement of virtual machines and cluster balance.

EVC is disabled on the cluster – EVC may be required for maximum vMotion capability.

DPM is enabled – DPM should be disabled to ensure hosts that are being remediated are not put into standby mode.

Figure 4

All in all some very nice additions and definitely going to life easier for all of us.

vSphere 4.1 - My Favourite Features

2010-07-13T16:24:00.001+10:00

With the release of vSphere 4.1 I have managed to identify some of my favourite new features. This is not a complete list but I think that they will have a large impact overall.

Networking Enhancements

• NIOC - Quality of Service prioritisation for Networking (Requires vDS)
• Increased vMotion throughput to 8gbps
• Improvements to VM-VM and VM-Native throughput
• NFS Performance enhancements
• 15% reduction in CPU cost (read and write)
• 15% reduction in throughput (read and write)
• Load Based Teaming (Requires vDS)
• Dynamic load balancing on physical nics in a team when traffic exceeds 75% mean utilisation over 30 secs
• Cisco Nexus enhancements
• Scalability will increase to 300+ hosts per VSM in line with the scalability of vDS.
• No dependency on VEM and ESX build numbers

Storage Enhancements

• SIOC – Quality of Service prioritisation for Storage (Block storage only)
• vStorage API’s for Array Integration - VAAI (Block storage only)
• Full Copy – Xcopy like function to offload work to the array
• Write Same -Speeds up zeroing out of blocks or writing repeated content
• Atomic Test and Set – Alternate means to locking the entire LUN
• Faster Storage vMotion with VAAI
• Server CPU and Memory, network bandwidth and storage front end controller I/O benefits
• New Performance Monitoring stats
• NFS stats in esxtop
• DS activity per host
• Storage adapter and storage path graphs
• DS activity per VM
• Virtual Disk activity per VM

ESXi Improvements

• Boot from SAN in ESXi (FC, ISCSI, FCoE)
• Tech Support Mode fully supported (local and remote)
• Active Directory authentication integration
• Lockdown mode – Total lockdown to force all host management through vCenter

Cluster Improvements

• 32 hosts/cluster
• 320 VMs/host (regardless of # of hosts/cluster)
• 3000 VMs/cluster
• HA operational and healthcheck status windows
• HA is now more DRS aware – Checks rules when HA occurs to provide better placement
• DRS interoperability with FT (requires EVC)
• DRS can load balance the primary’s and secondary’s
• FT VMs can run on different patch level hosts
• Network and logging enhancements in FT allow for better throughput and less CPU penalty
• Migrating Hosts to different clusters doesn’t break FT
• Increased maximum vMotions to 4/8 (1gb/10gb) up to 8x improvement for individual machines. Helps speed up VUM evacuation.
• EVC Enhancements
• DPM can now be scheduled for certain times
• DRS Host Affinity – required and preferential (required for licensing in some cases, multi tenancy)
• DRS Host groups and Virtual Machine groups

Management

• 64bit only version of vCenter allows for increased scalability
• VUM – Patch recall support, Third party modules, ability to disable features that prevent a host from entering maintenance mode
• EVC mode, HA, FT, DPM features, CD ROM attached, disabled at a cluster level, re-enabled when remediation done
• VUM is now DPM aware. If a host evacuates and the load is not enough, DPM can bring hosts out of standby
• Host Profiles integration with ISCSI support, N1k, setting the root password.
• AD authentication support for vMA
• UBS Passthrough with vMotion support

OK I couldn't pick one! What's your favourite??

APAC Virtualisation Roundtable – Do you know what you are missing?

2010-07-12T13:34:00.002+10:00

The APAC Virtualisation Roundtable takes place every second Wednesday 9pm Sydney time. We are now up to episode 17 and are going strong. For those who are yet to get involved I’d say ‘Where the bloody hell are you?'

If you haven’t been around or have only just heard about it here is some of the previous guests

• Cisco UCS – Stevie Chambers

• Microsoft Hyper-V- Nathan Mercer

• Vizioncore – Simon Poulton

• VCDX certification path – Duncan Epping, Andrew Mitchell

• Backup and Replication with Veeam – Rich Brambley, Doug Hazelman

• Powershell – Alan Renouf, Luc Dekens, Hal Rottenberg

• VMware View – David Wakemen

• Cloud Strategy – Rodney Haywood

• 3Par – Steve Kelly

This week – We turn the tables on Mike Laverick as he becomes our ‘chinwagee’

Please join us every second Wednesday for more insight and discussion into virtualisation in APAC and abroad.

You can download previous versions of the podcast or use itunes to subscribe.

Virtual Machine Management and Resource Pools

2010-07-10T09:34:00.000+10:00

A discussion during the week with a colleague prompted me to put down some notes. The question was asked how should I re-design the cluster resources. Of course he got my ‘how long is a piece of string’ response but after some discussion it appeared we were on the right track.

I won’t bore you with a running commentary on the discussion we had but these were some of the points or outcomes.

Resource Pool planning can be one of the trickiest things to get right in an ESX environment. There all kinds of connotations and unforeseen consequences I have come across. In this instance they wanted to breakdown the VM’s into a nice manageable view in Hosts and Clusters view, possibly by department.

Proposed Resource Pool design

The problem I had with that design was there was no real reason to do so other than it would be nice aesthetically. There was no business reason to do this, no chargeback, no allocation of resources on a departmental basis. Furthermore how would we differentiate between guests within an RP which may require higher shares than its neighbours, without setting per VM shares (which can be a pain to manage in any sized environment really)

VM’s and Templates View

In my opinion Resource Pools where never meant to be used as a logical view of the VM ownership, that’s why it is called the Hosts and Clusters view. If you want to view a logical breakdown of your VM’s then I would encourage you to use the VM’s and Templates view. Would you use the Hosts and Clusters view to manage your datastores? No, you wouldn’t, you would use the Datastores View.

The VM’s and Templates view provides the ability to create a logical view of your VM’s using folders and possibly, depending on your design an overview of VM management or ownership. It also allows you to set permission levels at any folder level thus allowing staff who need to provision their own environments to do so in certain circumstances. Likewise it allows for templates to be provided for departments to provision from rather than from global templates.

Typical folder design

Resource Pools

Given we could and should use the VM’s and Templates view for its intended purpose we were left to come up with a Resource Pool design that would work in this instance. There are many schools of thought when it comes to Resource Pools such as:

- Single RP

- High, Medium, Low

- Nested RP’s

- Per VM shares

It was suggested that a High, Medium, Low model was best and in general I lean towards this model, however there is one, often overlooked problem with this method. To illustrate with an example if you have a Resource Pool with 2000 shares and contains 4 VM’s then 2000/4 = 500 shares per VM. Imagine you have a High Resource Pool with 8000 shares and 10 VM’s then 8000/16 = 500 shares per VM. This indicates that all the virtual machines would actually receive the same amount of resource shares in the cluster. Take that one step further and increase the number of VM’s to 20, then 8000/20 = 400 in fact less shares that those in the Low Resource Pool. Where your Resource Pools would be evenly balanced as far as the number or VM’s in each then this method makes good sense.

Whilst initially we could have used a departmental Resource Pool view and nested High, Medium, Low share Resource Pools within them, take for example 6 departments, times 3 nested Resource Pools each would mean a minimum of 18 Resource Pools. For a design such as this, 8 hosts in the cluster it wasn’t necessary. In any case my recommendation would be Resource Pools require thought and planning, don’t make life hard for yourself. Think of all the time taken to troubleshoot performance issues on a VM with a complex design.

Per VM shares are another story as well. They can be time consuming to manage but perhaps the greatest problem with them I see is that per VM shares will only be shared within the Resource Pool in which they exist.

In this scenario it was decided that the best method was to use Reservations at the Resource Pool level. We had some constraints where we had to guarantee cluster resources to particular VM’s which meant that shares were not going to work. In saying that, Reservations worked in this case by no means indicates it is the best solution overall. Reservations can be just as tricky if your calculations are not correct and lead to more pain in the long run, but it seems to be working well for this environment now.

In short:

- Don’t use Resource Pools as a means to create VM views, that’s what the VM’s and Templates view is for.

- Always give careful consideration to how you design your Resource Pools. Just because a method worked in your last design, doesn’t mean it will in the next one.

There is plenty of reading out there on Resource Pool design and I would encourage anyone who struggles with it to hit as many websites and blogs as possible, but remember it's your design and you have to deal witht the consequences.

I’d welcome any comments or feedback

Greg

The Importance of VMware Cluster Design

2010-06-23T20:35:00.003+10:00

A few days ago, whilst doing my morning scan I stumbled across an article posted by Duncan Epping which referenced HA failover capacity and the % of reserved resources model and it got me thinking. Before is start I should mention I won’t try to rehash Duncan’s HA info as I couldn’t do it any justice anyway, suffice it to say you should start your reading here

http://www.yellow-bricks.com/tag/ha/

http://www.yellow-bricks.com/vmware-high-availability-deepdiv/

I started contemplating a design close to my heart and how it may work in this particular way. To provide some background, as you may know when a HA cluster is configured the first 5 hosts that are added to the cluster are known as ‘primary nodes’. Primary nodes are responsible for replicating information about the cluster between themselves hence they are a crucial part of HA. Now let’s suppose that you have purchased a fully populated (8 blade) enclosure all running vSphere. The first 5 hosts (Host1-Host5) would generally be the first hosts added to the cluster, hence the primary nodes. Now suppose that the decision was made to purchase 4 additional blades and another enclosure to fit them in, joining them to the same cluster some time later. I think you see where I’m going? All your primary nodes will all reside in the one enclosure.

Example blade configuration

So let’s expand that out a bit, regardless of whether this is 12 node cluster is a great design or not at this point. Regardless if you are using failover capacity or % resources method for HA, if that blade enclosure fails then all primary nodes go with it and when all primary nodes are down no HA restart is possible. You could even extend this further and say if special attention to enclosure power cabling wasn’t taken you may have 2 blade enclosures or even 10+ rack mounted servers lost in an entire hit, granted you should avoid this design.

Clearly this is a worst case scenario, but I’m afraid it could and therefore probably will happen, so how do we work around it. Essentially we need spread our primary nodes across whatever physical barrier we might encounter, in this case enclosures. Although it is technically possible to reconfigure the primary nodes in your cluster by typing the following /opt/vmware/aam/Cli then ln

You can even demote or promote additional nodes by using promotenode ‘hostname’ or demotenode ‘hostname’ however this is not a VMware supported method and I wouldn’t recommend it in a production environment.

In the event that we remove a host from the cluster manually one of the secondary nodes in the cluster gets automatically promoted randomly to a primary node. Now as good as this may sound, imagine this case where we have 5 primary and 3 secondary in the same cluster. It would take considerable time to remove all the hosts and by process of elimination promote one in the second enclosure to a primary node. Obviously the best practise would be not to populate your enclosures past 4 blades (hereby ensuring you have 1 primary in another enclosure) or to buy multiple enclosures at the start and add hosts to the cluster in an alternating fashion, but business and monetary decisions might not always make that possible. For my recommendation, you should be able to shutdown a blade server and move it to the second enclosure with little problem if the cabling config is the same (which is should be if they are the same cluster). This should do the trick quite nicely, but it’s certainly something to think about as cluster design has great implications on your overall design and is often overlooked. Ultimately you always need to design with the impact of failure in mind.

There is of course one more cluster setting which I haven’t addressed yet and that is a dedicated failover host. I generally steer away from this choice and here’s why. By dedicating a failover host it presumes that you have enough capacity on that host to deal with the workload that would be moved to it when HA invokes. To this end you will probably find that admission control will prevent VM’s from starting when resources cannot be allocated on this host so it may not be great use anyway, unless you have a very low consolidation ratio. The other reason is by using this method it prevents the host participating in DRS. Now if you have enough money to dedicate to a super specced failover host and you don’t mind it not making you any money then fine, I on the other hand generally don’t like hampering DRS in any way, shape or form and I would prefer to harness the full computer power of my cluster. In any event you still need to protect your primary nodes from a simultaneous failure.

So I guess the bottom line is, when designing VMware datacenters and/or clusters there are many variations, there is no one size fits all. Be prepared to do some good analysis of your current and potentially future requirements and spend some time on a design that works for you.

Greg

Data ONTAP PowerShell Toolkit version 1.0.1 released

2010-06-23T19:50:00.002+10:00

A few weeks ago Netapp released version 1.0.0 of their Powershell Toolkit. The good news is there has been some more development on this and Version 1.0.1 is already available.

Although version 1.0.1 is a minor revision it does add some extra features such as:

Added Invoke-NaSnapmirrorThrottle.
Fixed bug to allow domain credentials with HTTP/HTTPS connections.
Added ConvertTo-DateTime.
Fixed synopsis in Set-NaVfilerNis.

For more information check out the community site here http://communities.netapp.com/community/interfaces_and_tools/data_ontap_powershell_toolkit

If you would like to contribute your feature requests for the toolkit feel free to do so here http://communities.netapp.com/thread/8970

Happy scripting.

Greg

Standard or Distributed vSwitches? My take

2010-06-15T19:18:00.000+10:00

Personally I read with interest some articles recently on vDS. To be quite frank although it is a nice feature I cannot see why the attraction or at least eagerness to enable them in many cases.

With automated deployments and the like, matching switch names between hosts with sundry configuration parameters is not terribly difficult. Don’t get me wrong I’m not having a go at VMware or anyone for this. I am sure there are a lot of people out there who use them exclusively, and ‘bravo’ I say!

For my 2c I can see some downsides to them which have been discovered in the past. One which Jason Boche uncovered here vDS Catch-22. In general the idea of losing vCenter and not being able to manage your vm’s could be a problem. My other issue is that I generally always push for vCenter to be virtualised read more and VMware themselves will not support vCenter running on the vDS. So what’s the fix?

Many people are keen to separate their storage and management networks physically and logically. To this end I would continue to use standard vSwitches for this purpose. I would go one step further and suggest that possibly vCenter (and database VM’s) could be on a separate vSwitch alone. Of course there could be a number of design constraints around this. Perhaps you would employ VLAN’s or other nics for this purpose if this works for you. By doing this you would ensure that you could, all things being equal get access to vCenter and thereby be able to manage any your vDS and hopefully avoid an outage causing you more problems.

Now you might so if you are going to use the hybrid model then some would say why bother at all. Well that’s an issue I’ve dealt with, personally and again I emphasise that, I don’t see the a huge advantage of managing a switch in a centralised point. I mean once the ESX networking is configured across a cluster or set of hosts (which I can automate) I very rarely would do any management of those switches on a daily basis, so a central point of management doesn’t do me many favours. Granted there are some features that are only available when using vDS like inbound traffic shaping, private VLAN’s etc but in all honesty I’m not using them in most designs. If they become a design goal then perhaps I will look at vDS but for now I’ll stick to standard ones.

Now again, this is only my opinion and it very well could conflict with yours but that’s what makes the world go round, so feel free to disagree :)

Greg

Introducing the Netapp DataONTAP Powershell Toolkit

2010-06-09T18:58:00.004+10:00

In the last few weeks I’ve been introduced to Netapp storage and let me say that from what I’ve seen so far there is allot to like, especially for someone from a VMware background. Netapp are doing some really cool stuff in the virtualisation space right now. All that went to another level today when I stumbled across the DataONTAP PowersShell Toolkit.

The toolkit enables a total of 357 native cmdlets for configuring and managing your Netapp storage array. Although not all API’s are currently supported a vast amount of them are and certainly enough to get you started. I’m sure that it won’t be long before there are more on available.

The PowerShell Toolkit is free however a Netapp NOW login is required to gain access to the download, which quite frankly , if you are reading this article you should already have.. You can find the link to the community and a download link here:

http://communities.netapp.com/community/interfaces_and_tools/data_ontap_powershell_toolkit

In order to get going with the toolkit grab the download from the above link and follow the instructions below.

Installing the toolkit

· Download the data.zip file and the install.ps1 scrit into the same dir.

· Run the install.ps1

· Verify the DataONTAP module has been installed – Get-Module -ListAvailable

(notice I use powercli as my powershell command line tool)

· Import the module – Import Module DataONTAP

Playing with the toolkit

Use Get-NaHelp cmdlet to show all available commands.

and now for my next trick...something I whipped up earlier.. the limits are endless.

#####################################################

#Creating a new NFS volume on Netapp SAN for ESX use#

#####################################################

$sancontroller = “san01”

#Connect to SAN controller

Connect-NaController $sancontroller

#Create new aggregate with 10 SAS disks and a raid type of raid_dp (Netapp's custom RAID)

New-NaAggr aggr1 –DiskCount 10 –DiskType SAS –RaidType raid_dp

#Create a new 500gb volume called esxvol in aggr1

New-NaVol esxvol aggr1 500gb

#Enable deduplication on the volume

Enable-NaSiS /vol/esxvol

#Set dedupe schedule

Set-NaSiS /vol/esxvol “sat@@,12”

#Add NFS License if needed

#Add-NALicense XXXXXXX-YYYYYYYYYY

#Start NFS Service

Enable-NaNFS

#Creat an NFS export for the volume

Set-NaNFSexport –path /vol/esxvol –readwrite all-hosts -root -persistent

#My NFS volume is now ready for my ESX servers.

######################################################

Stay tuned to the Netapp community and here for future scripts.

Greg

Storage Views

2010-06-08T20:25:00.003+10:00

A nice feature in vSphere is the ability to gain better insight into your storage via the GUI/VI Client. Nothing illustrated this better to me than the introduction of Storage Views. It must be said that Storage Views is a feature of vCenter and not ESX itself.

The Storage View tab is split into two distinct but equally important areas. Storage Reports and Storage Maps.

Storage Reports

Displays information about how an object is associated with the particular storage component. They also offer summarized storage usage data for the object’s virtual and physical storage resources. You should use Storage Reports to analyze storage space utilization and availability, multipathing status, and other storage properties of objects in your vSphere environment. Some of the viewing options are shown below:

Although the default view might appear somewhat underwhelming, if you are like me and are willing to dig a little deeper you will notice that you can add fields to the view. Some of these fields can be quite powerful in what they can reveal. One of my favourites is being able to see the snapshot space a Virtual Machine is using. Another is the ability to see a list of provisioned storage for each datastore. In the current climate with thin provisioning being the buzz word this can be an invaluable insight into the real size of your datastores. Of course if you are doing thin provisioning at the array level then you should also setup reporting there as well.

Whilst I won’t go into all details here, I would simply suggest that if you haven’t poked around with Storage Reports as yet, that you do so to find the information that may be pertinent to your own environment.

By default reports are updated every 30 minutes however using the Update button, you can recreate the report on the go.

Storage Maps

The other cool storage view built into the client is Storage Maps. Storage Maps are not unlike the Maps feature in the vCenter GUI but does not include the mapping to networks, virtual machines etc. It deals primarily with the relationship of storage to hosts, clusters and datacenters.

If providing a graphical representation of these relationships is what you require than Storage Maps is your weapon of choice. You can select whichever controls you wish to add to you map using the legend on the right hand side of the map.

Also notice that you can change the size of the generated map by selecting and dragging the zoom control.

In more complex designs it may be necessary to resize youre map to ensure it fits on a page or similar. The zoom tool is good for this however one nicer feature is the ability to select and drag nodes to an area that suits best. In my simple design here I have moved my cluster node out to the left to separate it. I could continue this with all other nodes until my map looks presentable. Again, in a far more complex environment this would almost be the rule and not the exception in my experience.

With both Storage Reports and Storage Maps it is possible to export the results to an external file. Of particular note you are able to export the Storage Map to and image file or my personal favourite, an EMF (Visio compatible) file, which can come in very handing when it comes time for documentation.

Storage Views has added the ability to gain a wonderful insight into our vSphere storage environment. I would like to see this evolved in coming versions and hopefully with the integration with storage vendors we are seeing these days there are some good things to come.

Greg

vEcoShell 1.2.6 Released

2010-06-07T15:11:00.001+10:00

The new version of the vEcoShell has been released. Version 1.2.6 is the first stable (non-beta) release of the software. It also still it remains free.

Probably the most significant features in this release is that all code has been re-written to support PowerCLI 4.0 Update 1 and that you can now run scripts externally.

For a full list of features check
http://vcommunity.vizioncore.com/administration/vecoshell/m/filelibrary/21.aspx

Greg

Troubleshooting vCenter not starting - service-specific error 2 (0x2)

2010-04-27T20:30:00.000+10:00

A few days ago I had to troubleshoot an problem whereby all vCenter 4 services would not start immediately after a reboot. The vCenter server was virtual and fully patched.

Essentially the two services that wouldn't start were the vCenter Server Service itself and the VMware Web ..... In this case the server was running only with a local SQL Express 2005 instance. The logs showed the following really helpful error.

Event ID 7024 -The VMware virtual center server service terminated with service-specific error 2 (0x2) - A very generic message.

9 times out of 10 I have seen this error it turns out to be an issue with dependant services. The first bit of advice I would give is to always check the logs. You always need the full story. The logs in the default log location 'C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs\vpxd.log' should give you some indication. Filter your search for the text 'error' and you will no doubt see somelines like

error-- Failed to intialize VMware VirtualCenter. Shutting down...

The devil is in the details above this line. There should be some info regarding what specific problem you are dealing with. You should look for more obvious problems like licensing, database connectivity and authentication issues. Whilst the info in the logs is by no means overly specific it will certainly give you a starting point on what to look for.

As I mentioned earlier, in most times I have seen this particular issue it has been a service dependancy issue. Sometime restarting IIS Admin service has worked as well as other VMware related services. Generally a fairly trivial exercise.

In this instance my problem was that the vCenter service was trying to start before the SQL instance was ready. vCenter should try and start the service every few minutes but this can still be annoying. I was able to use Regedit to make the vCenter Server Service dependant on the SQL Instance.

Run Regedit
Navigate to HKLM\System\CurrentControlSet\Services and locate the SQL Instance name. In my case it was MSSQL$SQLEXP_VIM
Navigate to HKLM\System\CurrentControlSet\Services\vpxd
Locate the DependOnService value and edit it
Add the SQL Instance name 'MSSQL$SQLEXP_VIM'

You can verify the dependency by viewing the service properties in Services.msc

Now my vCenter service should always run when SQL is up and running.

I should mention here that whilst this may help in some cases there can be many reasons why vCenter services will not start. The logs should always provide more info and should be step 1 info gathering when something like this occurs.

An additional point. One thing that became evident when I was troubleshooting this problem is the need to consider your log requirements. Keep in mind that vCenter will by default only keep 10 logs at any point in time. If you have a large environment this might mean half a day or even less worth of logs. My suggestion would be to give some thought to how many log files you keep, their maximum size and where you keep them. I would suggest storing logs on a drive that is not the OS disk and even looking into if there is a need for backup.

For more information on configuring logging for your vCenter environment read Duncan Epping's article here http://www.yellow-bricks.com/2010/03/10/changing-the-directory-of-your-vsphere-vcenter-log-files/

Greg

IPv6 causing vCenter slowness

2010-04-27T16:24:00.005+10:00

If you are living in the present day you are probably using a mix of Windows 2000-2008 R2 server operating systems. As you know IPv6 is now enabled by default in Windows 2008 and Windows 2008 R2. You may be experiencing something I did. I have a number of Windows 2008 (R2) guests and have also tried to embrace IPv6 in my network. However I found that vCenter was not as responsive as I would otherwise expect. To satisfy my curiosity I decided to have a look around in the logs. For the record My vCenter 4 U1 server is a Windows 2003 SP2 guest.

Filtering my logs for text containing ‘error’ I came across a number of entries similar to the following.

27 12:00:33.847 02872 error 'App'] [VpxdMoVm] Invalid guest IP address for VM dc-krystaltek.net: fe80::c81b:921b:e21e:f3d2%11

Doing some digging it appears that this occurs when Vcenter uses IPv4 to handle IPv6 addresses and protocols.

To resolve this issue you need to install version 6 of TCP/IP protocol on the vCenter server.

Open the Network Connections and Install the Microsoft TCP/IP Version 6 protocol

You should see the v6 protocol installed

Reboot the server and you are done.

No more errors in the logs

Greg

Virtualising vCenter - My Take

2010-04-22T10:50:00.001+10:00

This topic is not new by any stretch but during a design discussion the other day it raised it's ugly head again and I thought I should put my ideas down.

Quite often I hear the case against the virtual vCenter summarised into some of the following reasons:

* We cant afford a VM or host outage to render management of our Datacenter helpless.
* Why would we put the management tool on the management infrastructure?
* It makes recovering from an outage more time consuming.

Lets touch on these points a little.

Firstly, it is true that a VM outage will cause a loss of vCenter access however a host loss does not, in most cases. In a cluster configured for HA, HA will still invoke when the vCenter guest becomes isolated, in fact vCenter is only required to enable HA on a cluster, not to run it. In the event of a guest outage for whatever reason, yes the guest might be down but how would that be any different if the vCenter server was physical? In some cases businesses may even utilise FT for vCenter if appropriate. If you think about it, a hardware failure on the vCenter server will most likely incur a much greater downtime.

The root of the problem as I see it is poor planning. Recovery plans are just as important in a virtualised environment as they are in a physical. Documentation is king! Fortunately I have had the luxury (if you can call it that) to see what a complete power outage would do to our environment and I was able to learn enough from that to make some critical changes to our recovery plan.

As it turns out DNS is much overlooked in such a situation. The first thing on our list is to recover the DNS and Domain Controllers within our environment, of course we could have this debate about them as well, but that's another story, albiet much the same. Without DNS, AD, SQL AD authentication will not work. Pay special attention to where these services are hosted and get them up and running quick smart, even before vCenter.

One of the misconceptions is that if the vCenter server is down, then you're toast. It couldn't be further from the truth. In a well designed setup, your current VM's will remain and you will be able to connect directly to the host where it is running and rectify the issue, how do I know where it was running? Easy. VMware recommends setting the DRS affinity rules for the vCenter guest to Disabled. This will effectively pin the guest to a certain host, I generally like the first host in the cluster, it's easy to remember and document. You may want to do this with other guests like DNS servers, Database server etc etc.

It is my opinion that running the vCenter guest as a VM will actually provide you all the benefits that are afforded the rest of your environment, i.e Performance, HA, snapshots, Image level backups etc. rather than leave you at the mercy of an irrecoverable disaster as some may infer.

Some other considerations for running vCenter as a VM:

* Try to install the vCenter server separately from the database server where possible, especially in large environments. SQL Express has limitations for larger sites.
* Pin vCenter and database server to first host, possibly even other core networking VM's
* Run the database server on the same host as the vCenter guest. In some cases you may have to use an in-situ physical server.
* Set the startup priority on critical VM's to high to enable faster recovery
* If using DPM, you may need to disable it on the host
* For larger environments, run a separate cluster for management VM's

HTH

Greg

Enabling Powershell Auto-Completion For Notepad++

2010-03-12T20:43:00.002+11:00

Notepad++ is one of the handiest text/code editors known to man and woman for that matter. For many years I've used it as a fulltime replacement for Windows Notepad. Even when I first saw it the idea of tabs was a winner. In the latest release I believe Notepad++ provides native support for Powershell highlighting.

That means that I can now continue to use Notepad++ for viewing and editing Powershell scripts as the good lord intended. However taking that a step further, one of the best features of other Powershell editors I've used is the tab autocompletion.

With the help of the Koen Vermoesen over at http://blog.koenvermoesen.be/2008/05/08/powershell-auto-completion-for-notepad/ I was able to grab a copy of the powershell.xml autocompletion file.

To enable autocompletion download this file and copy it into you %APPDATA%\Notepad++\plugins\APIs folder. Open Notepad++ and go to the Settings menu. Choose Preferences, Backup/Auto-Completion. Depeding on your needs you can'Enable auto-completion on each input or simply 'Function parameters hint on input'. My advice is to play around until you find something that suits your needs.

On the flip side...

I find that Notepad++ has its purpose. I generally don't like it as a Powershell editor strictly, although the ability to use it in this way certainly gives it a big advantage as far as I am concerned.

If you are looking for Powershell script editors and interactive environments i cant go past Powershell Plus by Idera http://www.idera.com/Products/PowerShell/PowerShell-Plus/. Not only does it have a great script editor and debugger it also has probably the best interactive console I am yet to see. It comes with plenty of pre-loaded learning and demo scripts. It's my number 1 console.

Other Powershell editors and consoles I have used and like are:
PowerGui
The standard Windows Powershell console
Powercli (If you are into VMware)
Powershell Analyzer

Greg

Fixing Sender Authentication Required in Exchange 2007 Distribution Groups

2010-03-11T16:44:00.010+11:00

You may already know the default for creating new Distributions Groups in exchange 2007 is that Sender Authentication is required. This can cause problems when the group is to be emailed outside the organisation. External users/clients will receive a bounce message when emailing an internal Distribution group.

On occasions when groups are created this property is missed or the group was created so long ago in Exchange 2007 that we forgot about it. In which case I developed a quick powershell one-liner to help you fix this.

The following command will show you all the groups that require sender auth. I like to sort by the WhenCreated property however you could sort by Name or other if needed.

Get-DistributionGroup | Where {$_.RequireSenderAuthenticationEnabled -eq "True"} | select Name, WhenCreated, RequireSenderAuthenticationEnabled | sort Whencreated

As for fixing it, you could simply take this a step further and pipe that command to a Set-DistributionGroup using a “false” however there may be occasions when you actually want sender authentication to be enabled or even prevent external users mailing your internal groups this way. In thise case setting this property to false on all groups would be bad.

Fortunately of the hundreds of groups I have, only a few were needing a change. If you really needed to you could generate a txt file of groups that need changing and run your Set cmdlet against that, or alternatively individually against the name of each group you need to change, or even use the GUI.

As far as I know there is no way to change the default behaviour.

HTH

Deploy VM From Template with PowerCLI

2010-03-10T20:03:00.002+11:00

Following my previous post on granting users permissions to deploy VM's from a template I decided it would be much easier if users could run a script which did the same thing for them. Here is the example. There are further mod's but due to time constraints this is all I could put togetether in a few minutes.

I'll add some more changes when I can. I've already got some better ideas.

Granting your users Deploy From Template permissions in vSphere

2010-03-10T17:27:00.004+11:00

Need to grant users the ability to provision their own dev or test VM’s in some circumstances?

Here’s how with vSphere .

Initially you will need to create a user or more likely a group in Active Directory, unless a suitable one exists. In my case I used a user to illustrate the method.

1. Create an AD account called ‘DeployTemp’ (your choice) and remember the password J

2. Open vCenter with VI Client as a privileged user and switch to Administration/Roles using Ctrl-Shift-R. Create a Role called VMDeploy (or whatever you choose)

3. Add the following privileges to the Role:

Data store – Allocate Space

Network – Assign Network

Resource – Assign virtual machine to resource pool

Virtual Machine – Configuration – Add new disk

- Interaction – Power On

- Inventory – Create From Existing

- Provisioning – Customize, Deploy template, Modify customization specification, Read customization specifications

4. Assign the VMDeploy Role and Read Only Role in the following places:

Datacenter – Add Read Only Role to user

Datastore – Select LUN or LUNs where the user can provision their VM to. Choose permissions tab and add the VMDeploy Role to the user – no propagation

Cluster – Add VMDeploy Role to user – no propagation

Hosts – Add Read Only Role to user on host(s) you wish the user to place their VM. You may have a specific host dedicated for this task alternatively you could set this on all hosts in the cluster. If it’s a DRS cluster than perhaps one would suffice, it’s certainly easier on permissions when there are a number of hosts.

Resource Pool – Add VMDeploy Role to RP where their VM will reside. Depending on the usage case it could be a ‘Low’ share RP or even a department RP

5. Set permissions on the VMs and Templates folders using the following method:

Ctrl Shift V to switch to the VMs and Templates view. In my case I have departmental folders so I simply set VMDeploy Role to the user at that level and propagate permissions to the Templates folder below. This will also allow the user to create VM’s in their own Blue folder, where other access permissions are already granted.

6. Login to vCenter with VI client using the credentials of the user created in step 1.

The user should see something like the following image when in VM’s and Templates view. From here they can successfully right click the appropriate template and deploy the VM. NOTE: They will not have permission to create new VM’s on the host or datastores.

The user should have sufficient privileges to choose all appropriate options in the deployment wizard, including customising the template as needed.

Finally the task completes and we see the powered on VM in the list of Virtual Machines.

There are some more options which you may need to enable in your own environment, however this is the bare minimum I had to configure in order to get it working in mine. Hope you have success.

Cheers!

Problems installing WSUS SP2

2010-02-26T12:12:00.001+11:00

I ran into a problem installing WSUS 3.0 SP2 on Windows 2003 R2

I did everything like I would normally do, installing from a local source etc etc (WSUS 3.0 SP1 running fine) however Installation fails with a generic ‘read the log file for more info’.

Review the setup logs by opening the log located \documents and settings\%username%\Local Settings\Temp\1\WSUSSSetup.log

Corresponding to the setup log were errors in application log on the server.

2007 - Cannot repair performance counters for WSUS: API Remoting Web Service service. Please re-install manually using LODCTR tool.

3009 Installing the performance counter strings for service WSUS: API Remoting Web Service (%2) failed. The Error code is the first DWORD in Data section.

At this stage I needed to rebuild the perf counter library. To do this I used the following command c:\windows\system32>lodctr /r More info can be found here http://support.microsoft.com/default.aspx/kb/300956

I then re-installed the update as per normal and was prompted to reboot the server. All worked fine, no event log errors.

Greg

SVMotion Error: Cannot connect to host

2010-02-19T11:56:00.008+11:00

I spent some time today with new LUN's for our ESX setup at work. After configuring the LUN's as normal I was about ready to pull out smvotion to move some guests to the newly created LUN.

Thankfully now vShere has the GUI which allows you to do this..goodbye to old script, at least for non automated moves.

I ran into some trouble with guests on a particular host in the cluster. These blades are all configured identically so I had to wonder why it was the case. The error message was as helpful as 'Cannot connect to host'.

After doing a little checking and troubleshooting i came across this article which help explained the issue. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010837

Apparently this problem occurs if the IP of the hosts has been changed whilst the host is being managed by vCenter.

Repeated the steps in the article and all systems go!!

Greg

Offline Desktop with VMware View 4

2010-02-10T16:25:00.003+11:00

So after a week or so tuning and learning about VMware View 4 and doing some PCoIP tests I decided to give the whole Offline Desktop stuff a go.

It should be said here that at this time OD is considered an expirmental feature in View 4

That being said I still wanted to have a shot at using OD

I had problems intially even with OD sessions enabled in the Manager for my dekstop pool and using the full View client I was not seeing anywhere the Offline Desktop option

At this stage I had to go back to the View Manager Administration Guide which informed me that there are a number of limitations. The main ones I ran into were that OD wont work on a PC running the View client that is a) Windows 7 and b) has VMware Workstation installed.

When using Offline Desktop you must be aware of the following considerations:

View Client with Offline Desktop cannot be run on a virtual machine.
View Client with Offline Desktop does not support the use of smart cards.
You cannot download a desktop to a system where the guest exceeds the capabilities of the host
You cannot download a desktop if another user is currently logged in to that desktop.„
ESX supports two simultaneous desktop checkouts. ESXi supports five simultaneous desktop checkouts.
Host CD‐ROM redirection is not supported.
When a desktop is checked out, NAT is used for network communications.
TheMAC address of the offline system remains the same as its online equivalent.
As with RDP, you can copy and paste text between host and guest systems.
However, you cannot copy and paste system objects such as folders and files between systems.
Local drives are automatically mounted on the guest system.
Once a desktop is checked out on a client system, any changes made within View Administrator to the desktop or desktop pool settings will only be applied after the desktop has been checked in again.

The full document can be viewed here http://www.vmware.com/pdf/viewmanager3_admin_guide.pdf

I have heard of the ability to virtualise the View client with Thinapp on and run it on a workstation with VMware already installed, not sure about the Windows 7 though, although I assume that it would work.

Greg

View 4 Manager unable to connect to vCenter

2010-01-13T17:17:00.000+11:00

So I spent some time upgrading hosts, vCenter and all to U1. Finally downloaded and installed View 4 for a bit of a trial.

After initially installing the connection server and trying to connect it to my vCenter I got the following error:

No matter what local account I used, which incidentally had full priveledges to log into VC I couldn't make it work. It seems the error message is a little misleading. The cause of the issue was I had to use a domain account that had VC priveleges.

I ended up creating a service account in the domain, granting priveleges in VC and connecting without a hitch and i'm happy to say View is running like a charm.

G

Virtual Center 2.5 Update 3 Released

2008-10-06T16:13:00.002+11:00

Vmware has released Update 3 of Virtual Center 2.5

The update addresses issues from previous builds and also adds some functionaility missing in previous versions relating to HA and DRS clustering.

The release notes can be found here:

http://www.vmware.com/support/vi3/doc/vi3_vc25u3_rel_notes.html#resolvedissues

Grab the download from the Vmware website.

G

Process Monitor 2.0 Released

2008-10-02T10:24:00.002+10:00

Process Monitor 2.0 has been released.

Mark Russinovich and Bryce Cogswell have done some great work at Sysinternals and their tools are a must have for any serious System Admin.

The new version adds real-time TCP and UDP monitoring to its existing process, thread, DLL, file system and registry monitoring. You can now see the TCP and UDP activity processes performed, including theoperation (e.g. connect, send, receive), local and remote IP addresses andDNS names, and operation transfer lengths.

On Windows Vista, Process Monitor also collects thread stacks for network operations.

Link
http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx

G

Moving VMware ESX guests with Powershell

2008-10-02T10:12:00.002+10:00

I cam across this in my travels this morning and i have to say its raised my coolness alert a fair bit. Mike DiPetrilli has created a powershell script that will whenr un prompts you for Virtual Center address and creds and the VM that you wish to move. It will suspend the disk, move the VM and un-suspend the disk.

Link
http://mikedatl.typepad.com/mikedvirtualization/2008/10/quick-migration.html

Obviously there is a little downtime for the guest but this in the right situation what does that matter. Microsoft's Quick Migration for Hyper-V is a similar process.

I'm very keen to give this a try in my test lab.

G

Hyper-V server has been released

2008-10-02T09:53:00.003+10:00

Microsoft's standalone version of their hypervisor has been released. not server core, not full blown windows, just the hypervisor and a few bits and pieces added to make it usable.

For more info and a download go here
http://www.microsoft.com/servers/hyper-v-server/default.mspx

and for a lighter side of things
http://landofsilly.mypodcast.com/2008/08/Stand_and_Shout_HyperVs_Out_Now-134895.html

G

Guest VM sysprep issue

2008-09-30T20:44:00.004+10:00

So i found something weird the other day when i was trying to use sysprep to customize a template. Everything looked liked it worked but the customizations weren't taking hold. I was running ESX 3.5 U2 and VC 2.5 U2 fwiw

So i did some digging and apparently it is a known issue. VMwarewolf's blog pointed me to the right location. http://www.vmwarewolf.com/guest-customization-fails-on-virtualcenter-25-update-2/

There is an official VMware KB article that references the issue and has workaround details. It is only an issue post U2 so it seems.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006848

In short, I am NOT crazy!

G

Vmware Workstation 6.5, Server 2.0 Released

2008-09-30T20:33:00.003+10:00

So the new versions of VMware Workstation and Server have been released. I'm already using them and am very impressed. Some of the cool new features I like are Unity, Record/Replay, Streaming and Easy Install.

The server version also lifts the restriction on 3.6GB RAM per VM which is very important for the SQL and Oracle vm's we use in our test and dev environments.

http://vmware.com/support/ws65/doc/releasenotes_ws65.html
http://vmware.com/support/server2/doc/releasenotes_vmserver2.html

Cheers
Greg

Vmware ESX 3.5 Update 2 Date Issue

2008-08-12T16:18:00.004+10:00

I just ran into this one 3.5 Update 2 (ESX, ESXi)

I had creted new SQL VM this morning and couldn't power it on, not matter what i did i still got the message “A general system error occurred: Internal Error”.

As this was a non production host and or only update 2 hosts so far there were only a few guests running. After nornmal troubleshooting methods and a shutdown of all guests and rebooted the box and still the same behaviour.

It sent me looking in log files and google and i wound up with a licensing bug which stops you powering on any machine once the date has reached 12th august. We had to disable ntp and set the date back and all is fine.

Vmware have acknowledged this issue and say that he only work around is to set the host date back before August 12 and wait until the patch is available.

I can only imagine if we or others had large ESX farms and if you managed to update all your 3.5 servers to U2 and hit this issue. Given that it also effects Vmotion, you are in for a world of pain!!

They seem to be ramming updates down our throats these days and i can only put it down to the fact that they are pushing hard becuase of the Hyper-V release. Updates are good but not when the impacts far outway the benefits and this isnt the first issue ive come across!

It seems that they have now pulled the download from their website too!

G

Introducing HTTL protocol

2008-08-07T18:08:00.003+10:00

This came up on a mailing list a couple of days. It seems no one ,or at least no one had reported a typo in the GUI.

For those who have an ISA 2006 server, open the System Policies, go to the General tab of CRL Download and check out the last sentence of the explenation.

It gave me a laugh! i know.. i know..

G

Welcome

2008-08-06T21:07:00.001+10:00

Well its true! I finally gave in. I thought it about time that I got going on this whole blog thing. Maybe its my own mis-guided way of thinking what I do is actually useful or remotely interesting, but it may one day provide some information, relief or maybe even just a laugh for someone... so here goes.