Awesome resources at http://www.virtuallyghetto.com/ Community scripts: http://communities.vmware.com/docs/DOC-9852 Company acquired by EMC but operates separately. Acquired SpringSource. DESKTOP SOFTWARE. Type 2 "hosted" hypervisors. Run on Windows, Linux, OS X. + VMware Workstation commercial + VMware Fusion similar, but ONLY FOR MACs + VMware Server freeware. DISCONINUED 2010. Formerly "GSX Server". Runs on win & Linux. + VMware Player freeware Free "Workstation". Can now create VMs. For commercial use supposed to pay $100 to make it "Plus". Require installation of drivers in host OS. VMWARE SERVER Looks like management only through web browser Chrome works except for Console plugin. Recent FF completely does not work. Console seems to require its independent window. Release focus with CTRL+ALT just like VI Client's console. ENTERPRISE SOFTWARE HYPERVISORS FOR SERVERS. Type 1 direct hardware. incl. vCenter + VMware ESX. Formerly "ESX Server". Has Service Console. ARCHITECTURE { VMkernel servers and supports VM COS runs agents and supports all administration tools/cmds. } Admin/config CLIs: COS, vCLI Troubleshooting: COS COS = Console OS. Red Hat RHEL. Not in ESXi. Some people recommend vmkfstools and depr. vmkpcidivy + VMware ESXi Seems preferred now? New version is 5.0. ARCHITECTURE { VMkernel only. See separate section 'ESXi VMkernel'. } Admin/config CLIs: COS, vCLI Troubleshooting: COS COS = Console OS. Red Hat. Not in ESXi. + VMware ESXi Seems preferred now? New version is 5.0. Support local and Active Directory authentication. See separate section 'ESXi VMkernel' FS CLI: vifs. Manages both ESXi system files and files within datastores. I guess you fetch files, modify, send back? SNMP support. CIM system is the VMware-host/center side of an agent framework. (for vCenter too) VI API. Obviously. (for vCenter too) vSphere Hypervisor is the entry level edition of VMware vSphere. There are many paid editions with different features. Admin/config CLIs: PowerCLI, vCLI Command-line recommendation is esxcli commands for dev and vcli for prod environments. esxcfg needed but deprecated. (These commands require the hostd. Use "localcli" only when hostd not available). CAN USE SHELL DIRECTLY. Recommend only for troubleshooting. DCUI = Direct Console UI = ESXi Shell = Support Console. = all ESXCLI commands + (probably esxtop too) Disabled by default. Can enable at console or with VI Client. Can then either * ALT+F1 (reg. console is ALT+F2) * Serial console * Ssh in (before v. 4.1 had to enable in DCUI) PowerCLI. WINDOWS-SPECIFIC! Windows PowerShell snapin. > 150 PowerShell cmdlets, docs, samples. inc. ImageBuilder ImageBuilder creates and maintains custom ESXi Image Profiles. Get-EsxImageProfile Displayes ImageBuilder profiles. Get-DeployRuleset AutoDeploy server rules Get-VMHost ads17a.vmworl.d.com | Get0MHostAttributes vCLI. Available for Win + Linux. Previously RCLI. At least for "vmware-cmd.pl", target VMs specified with absolute path to the .vmx file. (Run with -l to get canonical paths, + may use the sym-links) They are just Perl scripts. Win distro includes ActivePerl and I have to invoke with .pl extension, though maybe something is set up wrong. On Linux there are no .pl extensions. Linux includes .so's, so I wonder how portable. Hm, though Linux ships as a tar distro, it comes with Make.pl and installer scripts, so it looks pretty invasive. Commands based on esxcfg-* cmds of ESX service console. Many new names are vicfg-* w/ compatibility esxcfg-* aliases. See vcli-cmd.list EXAMPLE TO ESXi HOST: vicfg-nics --server=esx-host1 --username=joe -list (will prompt for password) EXAMPLE FOR HOST COMMAND THROUGH VCENTER: vicfg-nics --server=vc-host1 --username=joe \ --vihost=esx-host1 -list Authentication info cached on client for 30 min dflt. Fine-grained privileges per vCenter or local ESXi host (only if you have a vCenter though). ESXi host, ESX host, vCenter server may be targeted by any remote APIs or clients. Troubleshooting: Tech Support Mode or ESXi Shell (dep. on ver.) (Can ssh in to Tech Support Mode). Serial port connectivity just added for v.5.0. freeware since 2008. Resource requirements much lower. Intended to run from server flash disks but can run do normal. Managed by vCenter or [Virtual Infrastructure Client] vSphere. Can not manage with console. Has minimal BusyBox installation instead of Service Console. Bare metal embedded hypervisors needing no underlying OS. CLOUD OS. Just marketing phrase for Cloud-relevant vSphere-related products. vCenter. Previously VMware Infrastructure and VirtualCenter. = vMotion + DRS + HA + Storage vMotion ONLY! ESXi as of v. 5.0. RUNS ONLY ON WINDOWS! "Cloud computing virtualization operating system". "Flagship virtualization suite". Cloud of products to manage ESX/ESXi environment. If CLI or VI Clients connect to the vCenter, it authenticates to a vCenter account and then internal protocols go to the hosts. VSA = Vsphere Storage Appliance. STORAGE clustering! Enable VSphere plugin VSA Manager installer gives VI Client a VSA Manager tab. Most basic objects are a VSA Cluster + vMotion network. Give new IPs to each host which share NFS dir. Sets up vMotion + storage clusters + HS clusters. vMotion: Allows migration of operational guest VMs which share storage to another ESX host. Happens transparently. Storage vMotion: Move running VM to different storage device. DRS: Distributed Resource Scheduler. Automatic load balacing using vMotion. Migrates VMs and resources among hosts in the DRS cluster. HA: Automatic VM failover to another ESX host in the cluster. VMware vSphere Web Client (nw) to manage it. Has Portlets. OR vCenter Web Access portal. VI Client = Virtual Infrastructure Client = vSphere Client Can target either ESXi hosts or vCenter, just like RCLIs. vCenter Converter. Converts physical machines into VM (P2V) and VM categorization. + automated config tasks. VMware VMFS is a native FS specifically for holding and clusting the big server-side images and snapshots. "An alternative is NFS". Guest OS == VM is the client OS. VMware Tools. Guest OS drivers and utilities. However, OpenSUSE automatically installs a package with kernel drivers and that seems to work great! HGFS are shared directories made available by Vsphere. Not necessarily persisted on a VM-related host. open-vm-tools project supported on Sourceforge.net (alt. to Tools?) THESE TOTALLY SUCK. Nothing works with OpenLinux 11.4. VMware has an apparently commercial bundle VMwareTools-*.tar.gz (which comes in an .iso). No real bin/'s in it, so I guess it's source code. :( Apparently when working requires its daemon running. I think the normal way to install VMware Tools is not with open-vm-tools, but by using the VMware Tools area of the VM manager, which provides pseudo-CD device to the VM. VM must them mount /dev/cdrom and install the rpm from there. (There's also a tar.gz, but who the hell wants to deal with that?) This sets the Gui hooks up to automatically work upon Gui logins. Reports that you can run "vmware-toolbox" during an X session. I don't know what that toolbox is. DETAILS Due to hardware detail dependencies, stopping VM solves vMotion problems. PXI (pronounced Pixie) Boot = DHCP + TFTP booting. Auto-deploying Hosts (not VMs). Basically do full install then apply vCenter host profile + autoDeploy server registered w/ vCenter. Each time ESXi hosts boot they Pixie boot with rules profiles. There is also OVF = Open Virtualization Format. Used with ovftool and directly with VIClient. VIClient doesn't give as many deployment options. Example of ovftool deployment to ESXi host: ovftool --skipManifestCheck --name=ITSM8 --diskMode=thin --datastore=green3tb ITSM8-Single.ovf vi://root:ROOTPWD@esxi1 VM Setup Automation ------------------- Awesome scripting book. Get this or newer: http://searchsystemschannel.techtarget.com/feature/Creation-of-virtual-machines-utilizing-command-line-tools Apparently several ways, some or all of which clone an existing VM. Source VMs should be prepped. Windows with Sysprep. Either in DCUI or vCLI: vmkfstools -i pathtoTemplateVM/template.vmdk pathtoDestVm/newvm.vmdk vmware-cmd -s register /path/to/DestVm/newvm.vmx Should also be able to use 'vmcp' and 'ovftools'. ESXi VMkernel ------------- Shell is funky in a couple respects. With "set -o vi", some basic vi functionality is missing. (e.g. can't search with "/"). Sometimes filename patterns do not work. For example "x > *.txt" will write to new file "*.txt" instead of an existing file matching pattern *.txt. I think highly stripped Linux. I think BusyBox. sysctl -w vm.swapiness= # (percentage) set paging policy No Bash. /bin/sh is ash. Ash reads only .profile at login, and $ENV for each new shell. Ash PS variables don't do dynamic execution. man pages /home 'ifconfig' 'nano' 'mount' 'man' 'curl' (but has wget) 'file' 'rsync' 'shutdown' (has 'halt' and 'reboot' for which you must use -f switch) Should shut down VMs before doing this? 'vim' but does have 'vi' 'ps' is a VMware replacement 'parted' but does have partedUtil (may have nothing to do with...) (also has 'fdisk'). See section below about 'partedUtil' regular /dev/sd*. Does have USB devices, and /dev/disks/*, /dev/lvm*. outgoing ssh connections,even with nc, hang badly. But outgoing 'wget' and pings work fine. useradd, etc., are present but deprecated. Now dual-image to support safer upgrades. Since kernel, fiddling is discouraged: Agents must be "authorized" Remote administrative tools and shells encouraged FS The only things mounted, by default (as shown nicely by "df -h'), are some /vmfs/volumes/* links and dirs. /etc/vmware/... Config files like "hostd.conf" for vmware-hostd daemon. /var/log/vmware/... /usr/bin -> /bin Sym link /usr/sbin -> /sbin Sym link /vmfs/devices -> /dev Sym link /vmfs/volumes/* -> Sym links here are just datastore names Sym links here are store names and get mounted as type NFS or VMFS-5. The sym-links point to hex*-hex* dirs that are not directly mounted. The non-symlink-target hex*-hex* dirs are mounted as type vfat. I don't know the purposes. /vmfs/volumes/STORE_NAME_SYMLINK/ dirs contain vm dirs and .*.sf files. /vmfs/volumes/STORE_NAME_SYMLINK/VM_NAME/ The sym link target gets mounted with FS VMFS-5. Within these are: *flat*.vmdk The raw FS image. Abbreviates VMwareDisK OTHER*.vmdk Metadata like *.vmx (refs to the *flat* raw above). *.nvram, *.log, *.vmsd (empty) *.vmxf is XML metadata about *.vmx file *.vmx CRITICAL VM DEF!!! Executable. When allocating a "virtual disk" to a VM, you select a .vmdk, .dsk, or .raw file. You choose either "Steor with the VM" or "Specify a datastore or datastore cluster". Former results in *.vmdk file-pair with base name suffix like _1, in the main VM dir. IMPORTANT: The .vmx references a wrapper .vmdk. The path to wrapper .vmdk path may be abs or rel .vmdk is for a single virtual disk ovftool works with them /sbin Includes most VM commands. See list below. Local VMXi accounts (created with vSphere Client) really are created in the VMkernel, so get shell and VI Client access. Apparently could also edit /etc/passwd, etc., but no command-line utilities for it. (One doc says that there's a esxcfg-auth for this purpose, but it's not on my ESXi host). But even though have full VI Client with these non-root accounts, they can't run /sbin/esxcli, /sbin/esxcfg-* successfully (at least without extra auth). Can write SSH authorized_keys files in directories /etc/ssh/keys-. Account vpxuser is used for, at least, vCenter and ESXi host comms. /usr/bin is just a link to /bin. /bin: Most standard UNIX stull like kill, dd, tar, /sbin: ethtool esxcfg-* esxtop esxupdate powerOffVms vmkbacktrace vmkchdev vmkdevmgr vmkdump_extract vmkerrcode vmkeventd vmkfstools vmkiscsi-tool vmkiscsid vmkload_mod vmkmicrocode vmkmkdev vmkperf vmkping vmkramdisk vmksystemswap vmkvsitools Lots of great subcommands like lspci, ps, lsof, uptime vmkpcidivy was for server-side device allocation but now deprecated except for hba allocations. I THINK COMPLETELY GONE NOW. esxcli (PREFERRED! Depends on agent 'hostd' process) localcli (use only if can't use the regular commands.) vmfs-support vmdumper vmcp vmtar vmware-cmd To create a new (empty) virtual disk: [vmkfstools -c 4g filename.vmdk -a lsilogic # chk an existing VM for -a vals. -a IS DEPRECATED} vmkfstools -c 4g [-d zeroedthick] newName.vmdk vmkfstools -E oldName.vmdk newName.vmdk # to rename. How specify new name? vmkfstools -X extendDisk 8g [-d zeroedthick] name.vmdk But web console lets you extend without shutting down the VM or detaching the disk from the VM. vmkfstools -U name.vmdk # to Delete See 'automation' section above for 'vmware-cmd' example. VMware Installation vSphere Hypervisor 5 License: 04612-62L9M-J8R41-0H9KH-1M94M vSphere Hypervisor 6.5 License: NN6AN-FUL05-68589-0H08K-2NFM5 (Up to 8-way virtual SMP). Boot installation media, CD or USB (complicated instructions for writing USB media, but I suspect can just copy the CD image to USB drive). Server clock must be set to UTC in BIOS. VMware Server licenses for Windows: A0N8W-PFCAF-11M2M-4JJ0H Server http port 8222; server https port 8333. for Linux: AAM0D-P4V8C-00P73-4T5L8 TIME Need to keep clock accurate on VM server host too, pref. via ntp. VMware actually recommends NTP over their own tools. Many details. Just follow the VMware article @ http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006427 But... vmware-toolbox-cmd command not available. /etc/ntp dir discussed is present on SLES but not OpenSUSE. Choose UTC hardware clock whenever given choices. According to my notes, "ntpq -p" verifies and should not just give a "LOCAL" line, but sometimes does not work. VDI = Virtual Desktop Infrastructure (other vendors implement too) VMware 5.1.0 Provided Linux VMware tools has the non-rpm VI Client says that storage units are "GB" but they are actually GiB. I don't know if same with RAM settings, and I don't know if same with CLI commands. Beware that after you add a new resource, the Edit Settings... UI does not update after it says "...Adding...". You must close the Edit Settings window and re-open to see updated status. VM Server on Linux At least on OpenSUSE, needs a C complier "to buld the vmmon" module. Need kernel-devel and kernel-source too, to get /usr/src/linux/include. And ncurses-devel That may be all that's necessary since I didn't try to run vmware-config.pl after installing kernel-source but before running the makes. kernel to generate some derived header files. cd /usr/src/linux make cloneconfig. I set a custom suffix and default hostname, but don't know if that necessary make if necessary. I don't know what this is. VMWare Console operation ctrl+alt+del in VMware console = ctrl-alt-ins ctrl+alt+ENTER toggles fullscreen partedUtil Seems to have no interactive mode. Generally run: partedUtil CMD /path/to/dev Can not "add" partitions. Can resize and delete partitions, but to add you must rewrite the entire partition table. partedUtil getptbl DEV output like: partition-table-type [total-disk-geom] SECTORS SEC_PER_TRK HEADS CYLS [each-part] PT# START_SEC END_SEC TYPE ATTR Only common ATTR is 128 = 0x80 = Bootable Since only reports in sector units, can use "fdisk -l DEV" to get Logical sector size (in bytes). TYPEs are in decimal and there's a nice table of them at bottom of page below. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1036609 ESXi 5.1 OVF Bug. Doesn't happen if OVF created before put OS on there. [Fix per http://lukebarklimore.wordpress.com/2012/10/25/esxi-5-1-fixing-failed-to-deploy-ovf-package-the-task-was-canceled-by-a-user/ ] Writes the wrong vmware.cdrom.* value to the .ovf (.ova in some cases) file. Change vmware.cdrom.iso to vmware.cdrom.atapi And update or remove the *.mf checksum file. Has SHA checksum for the other 2 files. (Just run "shasum "). Extending VM disks. Can definitely just enlarge VM disks, even thin prov. ones, then resize2fs. This requires available space immediately after the partition-to-enlarge, so it usually only works if it's last partition on the disk. Other way is to use LVM and add new VM disks and add as pv's. Use 'partprobe' for OS to see resized partition table without rebooting, then recreate the partitions with moved end using 'parted'. To determine ESXi RAM: cim-diagnostic.sh > mem.out | grep Capacity My exsi1 host has 4x8 GiB DDR3 1600 (PC3 12800) 10-10-10-30 G.SKILL SDRAM 1.5V 240 pin Can't change disks between thick and thin. Would have to re-create them. Only disk resizing possible is to enlarge Thin disks. Can't reduce size of any disk. Can't resize Thick disks at all. Can assign new disks any available SCSI number. When "Deploy OVF Template..." you have opportunity to change Thin vs. Disk. I guess must be for just the root disk because there is only one value to set. Boot options and BIOS settings are not saved with OVF. OVF Template installation On the Details screen it shows two "size on disk" sizes. They are either/or sizes depending on whetheryou choose Thick or Thin. The default for thick/thin is not based on the OVF, but on last choice in VI Client. OVF does not support instantiation of mixed disk types. All disks will be the one importer-specified type. OVF size doesn't changed based on Thick or Thin disk. Same stuff in there. Thick vs. Thin distinction all occurs in Import time. esxi1 RAM usage. 32 GB total available. 4 addm_demo-9.0 4 addm-9.0 3 bdadb01 (Centos) 16 centos 3 centos-oracle12.1 4 freyr REDUCE TO 2!! 8 suse VM Backups Somebody says at http://hardforum.com/showthread.php?t=1599986 can take a snapshot and just back up those files. I had to use: vmkfstools -i centos-base/centos-base.vmdk rhToCentos/centos-base.vmdk to copy just the virtual disk for an offline VM directly. VM file sizes Thick disk ESXi server *-flat.vmdk sizes are exactly equal to allocated size. FS usage irrelevant. Thin disks ESXi size is about same as FS used space, but *space for file removals does not get reclaimed*. OVF *.vmdk's are proportional to VM Think disk file sizes, about 55-75%. OVF *.vmdk's for thick ESXi server *vmdk's seem to be compressed the same as if the EXSi files were thin. Reducing thin disk physical size Get source code from http://intgat.tigress.co.uk/rmy/git/zerofree.gituild and compile program "zerofree" Requires libext2fs-devel rpm. gcc -o zerofree -lext2fs zerofree.c gcc -o sparsify -lext2fs sparsify.c zerofree -v /dev/sdb1 "vmkfstools -K pathToX.vmdk" Do this to the NON-*flat* *.vmdk file (and it will shrink the corresponding *flat.vmdk file). Benchmarking state 1G disk *flat.vmdk OVF 1G disk OVM *.vmdk -------- ------------------ --- ------------------ orig 1.0 GiB 684M 70K 1tinyfile 1.0 GiB 684M 100K 662MibFile 1.0 GiB 1361M 667M rmed 1.0 GiB 1353M 667M preK 1.0 GiB 676M 116K VMware Windows/Linux desktop products as of Sat Apr 19 10:06:24 EDT 2014: $249 Workstation, VMware Player ($100 license makes same thing "Player Plus") Get Linux to see resized partition. cd /sys/class/scsi_device/*/device # For desired device * echo 1 > rescan Then use parted to "fix" and then resize using 'resizepart ' Let prompt you for size and enter "100%". For unknown reason it not accepting end location if given in the command. If get bootup message in vmware console like "nsc-ircc. Wrong chip version 00" then change grub vid res. grub_gfxmode=1280x1024x24 (in most large modern monitors) OR grub_gfxmode=1024x768x16 (1024x768 is safer in older or smaller systems) Standalone console client: https://my.vmware.com/group/vmware/details?downloadGroup=VMRC1001&productId=614&download=true&fileId=645c0739ca41f95be3ec4aa67821a81b&secureParam=3d32e4a40a93fec5a938346be73c2317&uuId=854b3758-069a-4752-8e43-fa7b300e4039&downloadType= Damned thing wants to install something to /etc/init.d/. Why would a client need an init script! It doesn't work on archLinux anyway. CTRL+[ does not get passed though with the Chrome VMware console. Use ESC key. Shared/multiwriter disks require some clustering software to coordinate writes by multiple VMs. Therefore, I guess not useful generally. Generally use NFS to share virtual disks. Remember when adding independent disks to a VM to tell VM UI: Disk mode = Independent - persist* Only Nic adapter type E1000 works with archLinux. This despite archLinux kernel driver explicitly for the faster, new vmxnet3. ESXi local account admin. Forget the web UI. Just use esxcli like at https://pubs.vmware.com/vsphere-6-0/index.jsp?topic=%2Fcom.vmware.vcli.examples.doc%2Fcli_manage_users.9.5.html