Unable to GPT format the disk "da3": gpart: geom \'da3\': Operation not permitted on FreeNAS 11

While configuring my FreeNAS virtual appliance for iSCSI I encountered a problem with re-creating device volumes [1]. A pop-up appeared in the FreeNAS GUI with the message:

[MiddlewareError: b'Unable to GPT format the disk "da3": gpart: geom \'da3\': Operation not permitted\n']

As there was no data I decided to try wiping the device (Storage > Volumes > View Disks, "Wipe"). Even a full wipe of the device didn't make a difference. I then tried removing the device and replacing it with a new (HDD) device and that made the symptom go away. I then encountered the same symptom on a different device so I tried a different approach. After doing some research [2][3][4][5] I came up with this:

topolhnas01mg# diskdevice=da8
topolhnas01mg# disksizeblocks=$( dmesg | grep -i "byte sectors" | grep -i "^$diskdevice[:].*" | head -1 | cut -d"(" -f 2 | cut -d" " -f 1 )
topolhnas01mg# partitionsizeblocks=34
topolhnas01mg# seekblocks=$( echo $disksizeblocks - $partitionsizeblocks | bc )
topolhnas01mg# sysctl kern.geom.debugflags=16
kern.geom.debugflags: 0 -> 16
topolhnas01mg# dd if=/dev/zero of=/dev/$diskdevice bs=512 count=$partitionsizeblocks
34+0 records in
34+0 records out
17408 bytes transferred in 0.002115 secs (8232582 bytes/sec)
topolhnas01mg# dd if=/dev/zero of=/dev/$diskdevice bs=512 seek=$seekblocks
dd: /dev/da8: end of device
35+0 records in
34+0 records out
17408 bytes transferred in 0.002137 secs (8145309 bytes/sec)
topolhnas01mg#

After running this the FreeNAS virtual appliance was rebooted and I was able to create the volumes. Later I was able to successfully format the iSCSI devices with VMFS.

This is a super quick way of overwriting the GPT partition table at the beginning and and of the disk device. The key to the speed is that only 34 blocks at the beginning and end of the HDD are overwritten. I suspect that this would work equally well for other operating systems (FreeBSD, Linux) that implement 'dmesg' but haven't tried. This method in no way attempts to address issues of bad blocks or media issues.

I suspect that the reason for this happening was due to upgrading and then down grading my physical ESXi host -- trying to get the "ESXi MAC Learn DvFilter"[5] working. That is another story.



[1] Ungraceful error if volume create fails if disk needs wipe
[2] This is where I got the idea of overwriting the GPT partition tables. (Unable to GPT format the disk ada0; unable to wipe ada0)
[3] Comment from DBronx highlighted the importance of "sysctl kern.geom.debugflags=16" for this to work. (Error: [MiddlewareError: Unable to GPT format the disk "ada0"])
[4] Another page that suggests overwriting the GPT partition table and clue as to size of partition table (GPT Rejected - how to wipe for ZFS?)
[5] First hint at how to overwrite GPT partition table at and of HDD device (How to completely wipe a hard drive?)
[5] ESXi Learnswitch – Enhancement to the ESXi MAC Learn DvFilter

vSphere/vCloud home lab - new direction

About that Dell C6100. It's on hold permanently. There were a number of problems that didn't have easy solutions and I needed to get on with my VCP6-DCV studies.

First problem was noise. Because the C6100 is four servers crammed into one 2U package and you can stuff it with 28 HDDs there is a great need for cooling which means fans which means noise. I mean a lot of noise. Like freakishly loud. Not your normal 1U or 2U rack server noise. Worse. It was bad enough that I put it in a separate room and ran cables through the wall. Even on the other side of the wall the C6100 is very loud. I live in an high-rise apartment which means there is no basement option.

Second problem is lack of native support for ESXi 6.0. I did find an article on adding the necessary drivers ... blah, blah, blah. Even though it might make for a good exercise I didn't want to spend a lot of time on this.

Third problem was capacity. RAM and CPU were fine, but I made the mistake of cheaping out on the HDDs. My original idea was to use VSAN as the C6100 does not have RAID, only JBOD (just a bunch of disks). I didn't feel like buying more disks.

My solution was to to purchase a couple of HP ProLiant DL360e Gen8 servers, each with 2 x Intel E5-2430L, 96 GB RAM, 3 x 1.2 GB SAS 10kRPM HDDs. This included RAID and iLO Advanced (allows graphics mode via out of band management interface). I was fortunate that a local reseller of used computer equipment had a shipment of these units. A special shout-out to Micropeer.

The result is that I have been able to get further faster in setting up my vSphere lab. Because the DL360e Gen 8 supports ESXi 6.0 the install was easy with the bonus that ESXi 6.0 includes ESXi 6 as a guest OS which makes nesting ESXi easier.

So now I have a vSphere lab that will not make me deaf or go insane. More details in another post.

vSphere/vCloud home lab -- layer 1

Starting from the bottom and working my way up I decided on the following cabling scheme:

This is possible because the system board on my PC has two Ethernet NICs. One NIC is connected directly to my ADSL router/modem/switch for direct Internet access. The second NIC is connected directly to the SG300 switch for my private network. This is necessary as the default subnet for the ADSL router/modem/switch is 192.168.2.0/24 and the default subnet for the SG300 switch is 192.168.1.0/24. I'm in too much of a hurry change the configuration so that both switches are on the same subnet, and I'm not sure I would want to.

The obvious question is what am I doing for routing? Even though the SG300 switch is capable of being configured for static routes I'm opting to use virtual routers on NODE 1. I would have like to have the version of the switch which includes RIP routing, but that more than doubled the price. My choice of virtual routers at this point are: (1) Zebra on Red Hat Linux or the current equivalent, or (2) GNS3 or something equivalent. I have already have prior experience with Zebra on Linux for an earlier lab and it worked exactly the way I wanted. GNS3 is compelling due to the opportunity to get my hands dirty with Cisco routing to complement the Cisco switch work.

BTW, those aren't the real cable colours.

vSphere/vCloud home lab -- Cisco SG300-20 switch

If I'm building a home lab then I'm going to need a network. The DSL router/modem/switch thingy that I have falls a bit short of requirements. My requirements were:

• Must have enough ports for the Dell C6100. A fully populated C6100 will need a total of twelve ports: four nodes each with three NICs (two for OS, one for BMC).
• Gigabit. Require for things like vMotion.
• IEEE 802.1q VLAN tagging.
• Remotely manageable.
• Some sort of layer 3 routing would be good.

I settled on the Cisco SG300-20 (SRW2016-K9-NA) as it has a total of twenty ports: 16 for LAN, plus four for connecting to other switches. The C6100 occupies twelve ports and two more are occupied for connectivity to my PC and DSL router/modem/switch thingy. This model includes a layer 3 mode for static routes. It would have been nice to have the model with RIP I/II but this more than doubled the price. Besides, my plan is to use virtual routing either in the form of a standard OS with routing added, or GNS3, or Cisco IOS XRv Software.

One thing I was a bit fuzzy on were the MGBT1 SFP ports: were they standard RJ45 copper or not. They look like a standard RJ45 ports. Experimentation with these will have to wait for another time.

In contrast to the Dell C6100 this switch is fanless and therefore quiet. The web GUI is quite handy as its been 10 years since taking CCNA and I really haven't done anything with Cisco since. My immediate focus is on vSphere and vCloud, the networking will have to wait until later.

vSphere/vCloud home lab -- Dell C6100

I have been working with VMware ESX since version 2.5 and Virtual Center/vCenter since version 1 where I work. I tried creating a lab at home a few years ago but my PC just didn't have enough juice. There is only so much you can do with 8 GB of RAM. Last year (2014) while on a VMware course I was talking to one of the instructors about home labs and he clued me in to the Dell C-series cloud servers. He had a C6100 that he bought from eBay and was happy with it.

The benefits of the C6100 are that there are four independent nodes within a single 2U chassis. Each node contains its own processors, RAM, NICs and one PCI slot. Each node is hot-pluggable in the back of the chassis. The HDDs are also hot-pluggable and inserted to the front of the chassis. Each slot is allocated to a specific node: HDD slots 1 to 6 to node 1, HDD slots 7 to 12 to node 2, and so on. It looks like this allocation is hardwired. There are a total of three NICs per node. Two of the NICs are useable by the OS and one is dedicated to the BMC (baseboard management controller) for OOBM (out of band management). The BMC is not the same as iDRAC (Integrated Dell Remote Access Controller), but it does provide the means to act as a IP KVM for remote console access without needing an OS, and control the power (power up, graceful power off, forced power off, reset). Finally, the power requirement is a NEMA 5-15 grounded (Type B) plug which is your normal everyday household plug in North America.

If there is one disadvantage to the C6100 it would be noise. This thing is freakishly loud. I'm used to 1U/2U server noise. This is louder than that. The fans, four in total, are rated at 70 dB. After doing research on what other people are doing about this I came to the conclusion to not to replace the fans. The concern I have is that with so much horse power concentrated into a small space I didn't want to run it hot. I ended up moving the server to my dining room and running a couple of Ethernet cables through the wall. Even then the noise is noticeable, but at least I no longer need to wear ear plugs. Like I said, freakishly loud.

My criteria in choosing this configuration were:

• Something that was server class and was on the ESXi 5.x HCL. Research indicated that this hardware is on the ESXi HCL for version 5.1 Update 1 (need to double-check), and that other people have had success getting ESXi 5.5 working with some additional effort (more later when I get to that).
• SAS HDDs, not SATA, as I wanted to reduce HCL compatibility problems with ESXi.
• SSD HDDs, one per node, for two reasons: (1) there is no cache on the RAID controller, and (2) as I wanted to experiment with VMware vSAN. SSD is only available in SAS which fits nicely with the previous requirement.
• A generous amount of memory, without being prohibitively expensive. I know from experience that this will be the first resource exhausted.
• A selection of HDD speeds to experiment with SDRS. I can always add more storage internally (subject to the power envelope) or externally.
• Low-ish power. Yes, this is a contradiction as this is four server class machines stuffed into a small space which is why I picked low power processors, a reasonable amount of RAM and didn't go HDD crazy.

After doing lots of research I bought the following from NES INTERNATIONAL:

• Quantity = 1
  SYSTEM : DELL POWEREDGE C6100 XS23-TY3 w/ 4 x HOT PLUG NODES 24 x 2.5" BAYS
  PROCESSOR : 8 x INTEL XEON QUAD CORE L5630 2.13GHz 12MB CACHE LOW POWER
  MEMORY : 192GB DDR3 ECC REGD MEMORY (48GB PER NODE)
  HARD DRIVE : 4 x 160GB 2.5" SOLID STATE SSD HARD DRIVE
  NETWROKING : DUAL GIGABIT ETHERNET NIC CONTROLLER
  REMOTE ACCESS : IPMI 2.0 REMOTE MANAGEMENT PORT
  RAID CONTROLLER : LSI 1068E MEZZANINE RAID CONTROLLER Y8Y69
  POWER SUPPLY : DUAL REDUNDANT HOT PLUG POWER SUPPLY
• Quantity = 2
  SYSTEM : DELL POWEREDGE 2950 1950 2970 R900
  HARD DRIVE : 600GB 10K 2.5" 6Gb/s SAS HARD DRIVE
• Quantity = 2
  SYSTEM : DELL POWEREDGE 2950 1950 2970 R900
  HARD DRIVE : 500GB 7.2K 2.5" 6Gb/s SAS HARD DRIVE
• Quantity = 2
  SYSTEM : DELL POWEREDGE 2950 1950 2970 R900
  HARD DRIVE : 146GB 15K 2.5" 6Gb/s SAS HARD DRIVE

vCenter database health

We have recently encountered a number of problems with vCenter databases in our production and non-production (dev/test) environments. Some problems and fixes are common to both and one issue has a different fix for version 2.5 vs. 4.0. The problems are related to the size & growth of the database and the size & growth of the database logs.

The production vCenter is version 2.5 U5 and the two non-production vCenter is version 4.0.0 U1. In all cases we are using Microsoft SQL Server 2005 Enterprise Edition x64. The prodution databases are on clustered bare-metal Windows servers and the non-production databases are on non-MSCS virtual machines. In all cases we are using named instances of SQL Server (http://msdn.microsoft.com/en-us/library/ms143531(SQL.90).aspx), rather than depending on the default instance. All instances are patched with SP2.

First the database growth problem. Based on the internal database size calculator in vCenter 2.5 & 4.0 (Administration menu > vCenter Settings ... > Statistics) the the factors that determine size are: number of virtual machines, detail, sample rate, retention and number of hosts. Of these the number of virtual machines, detail and sample rate seem to have the biggest impact. Additionally there is the record of all events & tasks which impacted more by changes to the environment (creating, changing, deleting virtual machines and hosts, etc) which the calculator obviously can not predict. If you have a relatively static environment then the growth of the events & tasks talbles will be smaller than for an environment with constant changes.

But it seems that there is more to the size of the database than just the number of records stored in each of its tables. Our default configuration for databases is to not configure a hard size limit, but let it grow indefinately. More on that later. Obviously the more records that are stored then the bigger the size of the database and more of the file system is consumed. What was not obvious to me was what happens when record(s) are deleted: the database does not automatically shrink.

If you look at the calculator window you will see that statistical data passes through four intervals. The level of detail in the statistics data can be selected so long as the following intervals have the same or lower level of detail. There is also the real-time statistical data collected and displayed in the Performance tab for each host and virtual machine. Using deductive reasoning, some meandering through the database  and some information from VMware support the life cycle of the statistical data is like this:

1) Statistical data is gathered by the host for both ESX/ESXi and all virtual machines running on that host. This is what you see in the Performance tab for each host or virtual machine. These data points are 20 seconds apart based on the resolution of the charts in vCenter and vSphere Client. I will call this interval 0. I'm not sure where this is stored.

2) The interval 0 data points are summarized into data at a definable number of minutes (1, 2, 3 or 5) and definable level of detail and definable retention period (1 to 5 days). I will call this interval 1. This appears to be stored in table VPX_HIST_STAT1.

3) The interval 1 data points (1, 2, 3 or 5 minutes) are summarized into data points that are 30 minutes apart with a definable level of detail. I will call this interval 2. This appears to be stored in table VPX_HIST_STAT2.

4) The interval 2 data points (30 minutes) are summarized into data points that are 2 hours apart with a definable level of detail. I will call this interval 3. This appears to be stored in table VPX_HIST_STAT3.

5) The interval 3 data points (2 hours) are summarized into data points that are 24 hours apart with a definable level of detail and definable retention (1 to 5 years). This appears to be stored in table VPX_HIST_STAT4.

This begs the question: what is doing all of the data summarization at each interval? It is not vCenter, its the database. The rest of this database explanation will be focused on SQL Server 2005 as we do not use Oracle for vCenter. All of our databases have "SQL Server Management Studio" installed, or "SQL Server Management Studio Express". In SSMS, navigate to the "SQL Server Agent" and then "Jobs" there you *should* see a number of jobs: "Past Day stats rollup", "Past Month stats rollup" and "Past Week stats rollup" for vCenter 2.5 and 4.0, and "Event Task Cleanup" for vCenter 4.0. BTW, if these jobs do no exist or are unable to run then you have a problem and should look at http://kb.vmware.com/kb/1004382 on how to re-create these jobs for vCenter 2.5 and 4.0, or http://kb.vmware.com/kb/1000125 to purge old data, or http://kb.vmware.com/kb/1007453 to completely delete old data.

These jobs will read data from one interval table, summarize the data, write it to the next interval table and delete from the original interval table. This constant creating and deleting of records fragments the database occupies more space in the file system than there is data. But that's not all, in addition to the statistical data there is historical data in the form of events and tasks which are stored in tables VPX_EVENT and VPX_EVENT_ARG. In the previous paragraph I mentioned that vCenter 4.0 has the additional job "Event Task Cleanup". These jobs are configurable in the menu Administration > vCenter Settings ... > Database Retention Policy. In vCenter 2.5 and previous, the VPX_EVENT and VPX_EVENT_ARG tables can grow indefinately.

What I learned from all of this is that the database size in the file system can be no where near what the vCenter calculator would tell you. Consequently you can encounter problems with the database filling up its file system (drive letter) or it becomes sluggish.

So, how to fix this? Here is a list of KB articles that VMware support sent me and that I forwarded to my DBA who found them very helpful.

How to increase the performance of the VirtualCenter Database
http://kb.vmware.com/kb/1002825

Investigating the health of a VirtualCenter database server
http://kb.vmware.com/kb/1003979

Troubleshooting transaction logs on a Microsoft SQL database server
http://kb.vmware.com/kb/1003980

Defragmenting VirtualCenter performance data indexes on a Microsoft SQL database
http://kb.vmware.com/kb/1003990

Reducing the size of the VirtualCenter database when the rollup scripts take a long time to run
http://kb.vmware.com/kb/1007453

Purging old data from the database used by vCenter Server
http://kb.vmware.com/kb/1000125
 
My next post will talk about the size and growth of the database logs. Once we have modified our SQL Server maintenance plans for the production and non-production vCenter databases I will post details.

VMworld 2009 - specifics

Now to the meat and potatoes (what does that really mean?).

VCO (vCenter Orchestrator) and LCM (Life Cycle Management). This information comes from the LCM Instructor led Lab and the VM2120 session.

• Orchestrator is (one of) the product(s) that VMware got when they acquired Dunes a couple of years ago.
• It would seem that Orchestrator could be used as a general purpose workflow engine, but VMware has bundled it with vCenter 4.0. It includes the 'plug-ins' (I'm not sure if that's the correct terminology) for Active Directory and SSH (Secure SHell) with the potential to add plug-ins for other applications such as EMC Navisphere, HP Remedy, Microsoft Sharepoint, etc.
• The current 1.0.x version are written for the VI3 APIs and that version 1.1 will include the vSphere 4 APIs.
• LCM is a pre-assembled 'package' of Orchestrator logic to handle the provisioning, change management, and deprovisioning of virtual machines. Depending on how much LCM deviates from your current business requirements will determine if it is easier to modify LCM or create a new workflow in Orchestrator. The VM2120 session had a detailed breakdown of this analysis.
• Products such as SRM (Site Recovery Manager) do not leverage Orchestrator, but conceivably SRM could be (re-)written to be an Orchestrator package.

Performance. Most of this is from the Performance and Troubleshooting Instructor led Lab and the Performance Best Practices XenApp breakout sessions.

• Unfortunately, all of the participant spots for the Performance and Troubleshooting lab were full so I was relegated to spectator status.This was not such a bad thing as I got a chance to talk to Irfan Ahmad who is the author of the vscsiStats tool. He was able to confirm that vscsiStatis is *not* available for any version of ESXi, but that he has suggested that the functionality be added directly to vCenter.
• Based on what I saw of the Lab the vscsiStats tool is critical in determining storage bottle-necks, whether its between the virtual machine and ESX, or between ESX and the storage system.
• Lots of other good stuff was covered, some of which I knew already, but most of it new. I got a printout of the presentation.
• It seems that a lot of performance issues start with storage. I suspect this is due to the fact that a lot of VI3/vSphere administrators such as myself are new to dealing with massive amounts of shared storage such as what you find in a SAN.
• Designing, testing and verifying are crucial to implementing a proper vSphere environment, but one thing that was emphasized in the Citrix XenApp (formerly Presentation Manager) presentation is to test using your actual applications under actual workloads. The example repeated during the presentation was that the usual Terminal Services/XenApp benchmarks use Office. What if the application you are deploying is not Office? Then the results you get are going to be different.
• From the sessions I attended it seems that the mantra from VMware when virtualizing applications is to scale-out rather than scale-up. Even though ESX/ESXi 4.0 is capable of scaling up virtual machines, the preference is to create more smaller virtual machines. This is due in great part to the fact that the scheduling of multiple threads/process does not scale linearly in the x86 world for both Windows and Linux. By contrast, I have seen comparisons to Solaris on SPARC hardware with near linear scaling for up to 64 processors. The XenApp presenation made the point that the recommended virtual machine size could be smaller than the bare metal solution it is replacing, but that utlimately you will end up with more users per physical server. This is similar in logic to IBM's configuration of one thier x3850M2 systems with and without ESX 3.5. The result is that ESX allows the same physical server to support more users.

Upgrading. This is from the Things to Consider When Upgrading and Monitoring Hardware with ESXi breakout sessions.

• My biggest concern(s) with moving from ESX to ESXi is the lose of functionality due to the lack of HMA (hardware management agent) such as Dell OpenManage or IBM Director. I have seen the hardware monitoring capability built in to ESXi 3.5 and ESXi 4.0 first hand to know that simple things like a failed (pulled) HDD or failed (pulled) power supply are detected and reported to vCenter. I always wonder how complete the CIM monitoring capabilites are from the hardware vendors -- is it as good as thier native HMA? The bigger issue is the inability to remotely update system firmware such as BIOS, BMC, FC HBA, etc. See http://communities.vmware.com/thread/228037. I have heard from hardware vendors that they are still waiting on VMware to add the necessary capabilities to ESXi to allow them to do remote firmware updates. After the Kroger presentation on thier roll-out of ESX 3.5 I asked them if lack of the ability to perform remote firmware updates would impact thier decision to stay with ESX vs. ESXi and the answer was yes. But it seems that this issue has not escaped VMware as one of thier people hinted that this capability would be added *soon*.
• Rolling upgrade of ESX 3.x to ESX 4.0 was not a problem. My concern here was that in my early ESX 3.0.x days we had a cluster with both 3.0.0 and 3.0.1 and had some problems that cleared up once everything was at the same version. Ever since I always configure all hosts in the same cluster to have the same minor version and patch level for both ESX and firmware. This has not prevented all problems, but I'm sure it has reduced them.
• I had a discussion with another attendee who had a mixed 3.0.2 and 3.5 environment and was debating on whether to upgrade the 3.0.2 hosts to 3.0.3 (EOS in 2011) or upgrade to 3.5. I asked why not upgrade to ESX 4.0? His response was that they did not feel ready. So I suggested that they at least upgrade vCenter to version 4.0 and upgrade to ESX 3.0.3 as this would give them exposure to the new version of vCenter and allow them to use Update Manager to patch all of thier hosts.

Powershell. This is from the PowerShell instructor led lab and the Managing VMware with Powershell breakout session.

• First and foremost, Powershell is wickedly cool. With all of the plugins that vendors, not just Microsoft and VMware, are creating its obvious that Powershell will become the automation tool of choice to replace CMD or VBscript scripts.
• Learn Powershell.
• In the Managing VMware with Powershell session I learned about a couple of tools that should come in handy. The first one is VESI Virtualization EcoShell (http://www.thevesi.org/) which gives you the ability to perform administrative activities and then allows you to copy-and-paste the necessary Powershell code to script those same activies. The other tool (I don't remember the name) acts as a proxy between a vSphere Client and an ESX/ESXi host or vCenter management Server to also generate Powershell code for your administrative tasks. The difference is that this tool will generate the necessary code for the API calls even if the appropriate Powershell plugin is not available. The VESI tool by contract requires a couple of Powershell plug-ins such as the Active Directory plugin from Quest and PowerCLI  from VMware.
• One of the coolest demonstrations of the entire VMworld conference was watching VESI generate a Visio diagram of a simple vSphere environment in real time. You could actually follow the adding and moving of objects in the Visio application. You can see a video of this on the download page under "Video Library".
• I really need to get on that Powershell thing.