ghettoVCB-restore.sh Compatiable with: ESX/ESXi 3.5u2+ and 4.x+
Please find more details located at: http://communities.vmware.com/docs/DOC-10595
Ghetto Tech Preview - Restoring VM's backed up from ghettoVCB to ESX(i) 3.5 and 4.0+
ghettoVCBg2.pl Compatiable with: ESX/ESXi 3.5u2+ and 4.x+
Please find more details located at: http://communities.vmware.com/docs/DOC-9843
This tool is the follow up release of the ghettoVCB backup utility which allows users to perform backups of virtual machines residing on ESX(i) 3.5+ servers using methodology similar to VMware's VCB tool. By incorporating highly constructive feedback from the VMware community and utilizing the existing VI API, ghettoVCB’s framework was completely rewritten to be harder, better, faster, stronger.
vGhettoLinkedClone.pl Compatiable with: ESX/ESXi 4.x+ and vCenter 4.x+
Please find more details located at: vGhettoLinkedClone
vmwarevSphereHealthCheck.pl Compatiable with: ESX/ESXi 4.x+ and vCenter 4.x+
Please find more details located at: http://communities.vmware.com/docs/DOC-9842
This script generates a health check report similiar to that of vmwareHealthCheckScript but for the new vSphere release of VMware ESX(i) 4.x and VMware vCenter 4.x and it's managed entities.
ghettoShutdown.pl Compatiable with: ESX/ESXi 3.5u2+
Please find more details located at: http://communities.vmware.com/docs/DOC-9531
This script is similar to Joseph’s work How to configure ESXi to shutdown using an APC SmartUPS although it will not necessitate enabling the unsupported SSH console on the ESXi hosts. This is made possible by utilizing the VI Perl Toolkit to communicate with the ESXi hosts. The only possible drawback however is that next revision of ESXi may disallow write operations and require that the user purchase ESXi license(s) to make full use of the VI API (read/write).
vmwareHealthCheck.pl Compatiable with: ESX/ESXi 3.5+ and vCenter 2.5+
Please find more details located at: http://communities.vmware.com/docs/DOC-9420
This script generates a health check report of all attributes pertaining to a single ESX/ESXi 3.x+ host or VMware vCenter and its managed entities (i.e. clusters and individual ESX/ESXi 3.x+ hosts). The generated report is based off of some earlier Powershell script work created by Ivo Beerens/Duncan Epping.
ghettoQuickMigrate.sh Compatiable with: ESX/ESXi 3.5u2+
Please find more details located at: http://communities.vmware.com/docs/DOC-9400
This script provides administrators running the free version of ESXi on several hosts with the capability to perform Hyper-V-like "quick" migrations of virtual machines residing on shared storage between the hosts. VMware's VIMA virtual appliance was chosen as the central launch point for the quick migration process. ghettoQuickMigration.sh is executed from within VIMA and is compatible with ESXi 3.5u2+.
This script follows similarly to the Hyper-V Quick Migration process with the exception of replacing the disk resource swap with a virtual machine unregister/register command on the source and target hosts respectively. The state (powered ON/Suspended/OFF) of the virtual machine designated for migration is preserved through the entire process. Online virtual machines undergoing this migration will experience a temporary pause due to the suspend and resume steps.
ghettoClone.sh Compatiable with: ESX/ESXi 3.5u2+
Please find more details located at: http://communities.vmware.com/docs/DOC-9321
This script provides administrators with a means to automatically clone and deploy a large subset of virtual machines into a virtual environment without requiring tedious Service Console work or Virtual Center. Furthermore, for ease of accessibility, the free VMware VIMA appliance was chosen as the central launch point for the cloning process. ghettoClone.sh is executed from within VIMA and is compatible with both ESX and ESXi 3.5u2+. It accepts a single source virtual machine along with inputted configuration parameters to tailor cloned virtual machines.
ghetto-esx-linked-clones.sh Compatiable with: ESX 3.x
Please find more details located at: http://communities.vmware.com/docs/DOC-9020
This script allows users to create linked cloned virtual machine(s) from a master (or golden) virtual machine. Clones may be stored on any type of datastore (LOCAL, SAN, NFS) that is presented to the ESX host. The script is very capable in deploying a large VDI environment in a relatively short amount of time.
There are currently 3 supported use cases:
1) Default
--Description: Generate linked clones located in the same datastore that houses the master (golden) virtual machine. Writes will be directed to their respective linked virtual machine delta file with reads originating from the master virtual machine.
2) Distributed Write(s) I/O
--Description: Generate linked clones evenly across available datastores of choice. Reads will originate from the specified master virtual machine while writes are directed towards their respective virtual machine delta files that have been dispersed across the specified datastores.
3) Distributed Read(s) & Write(s) I/O
--Description: Duplicate the master virtual machine on available datastores of choice and evenly generate linked clones off of each duplicated master virtual machine. Reads and writes will be isolated onto the datastore of the linked clone(s) in question. This process is similar to running multiple instances of the default behavior of the script.
Note: This linked clones script is virtual machine OS independent (i.e. it is not restricted to just Windows OS's for VDI environment). For example, one can utilize this script to their advantage in development environments where new VMs need to be (mass) cloned quickly for testing.
ghetto-esxi-linked-clones.sh Compatiable with: ESXi
Please find more details located at: http://communities.vmware.com/docs/DOC-9202
This script allows users to create linked cloned virtual machine(s) from a master (or golden) virtual machine on ESXi. The only supported use case is that of the default behavior of the "ghetto-esx-linked-clones.sh" script which is due to some of the limitations in the ESXi Busybox Service Console like environment.
my-vmware-cmd.sh Supported On: ESX 3.5+ and ESXi
Please find more details located at: http://communities.vmware.com/docs/DOC-9061
This is a custom management script to help administer large virtual machine environments. This script is used to run administrative commands across a large set of virtual machines specified in a file. In particular, this tool complements the developed linked clones. scripts. For example, to save unique attributes (Active Directory personalities, etc.) for each linked VM, a final snapshot is executed on all linked VMs after joining each to the domain. After a specified duration, this script is then used to revert each VM back to a clean state. The same may be applied on other (non-linked) virtual machines currently residing on the ESX(i) server(s).
Usage:
Local execution-
my-vmware-cmd.sh [operation] [vm_input_file]
Remote execution-
my-vmware-cmd.sh remote [ESX_ESXI_IP_ADDRESS] [operation] [vm_input_file]
Operations:
start [vm_input_file]
-- Start all VMs in the input file
stop [vm_input_file]
-- Stop all VMs in the input file
suspend [vm_input_file]
-- Suspend all VMs in the input file
resume [vm_input_file]
-- Resume all suspended VMs in the input file
reset [vm_input_file]
-- Hard reset all VMs in the input file
shutdown [vm_input_file]
-- Shutdown all VMs in the input file (VMware Tools required)
reboot [vm_input_file]
-- Reboot all VMs in the input file (VMware Tools required)
snap [vm_input_file]
-- Create administrative pristine snapshot of all VMs in the input file
revert [vm_input_file]
-- Revert all VMs in the input file back to pristine state
purge [vm_input_file]
-- Removes from local inventory and purges all VMs in the input file
mac [vm_input_file] [generic|nixdhcp] [NETWORK (172.30.0)] [HOST_COUNT_START (200)] default=0
-- Extracts MAC addresses and generates either a generic file or one compatible with *nix dhcpd
( e.g. my-vmware-cmd.sh mac [vm_input_file] generic )
( e.g. my-vmware-cmd.sh mac [vm_input_file] nixdhcp 172.30.0 200 )
vnic [vm_input_file] default=0
-- Change vNic portgroup for all VMs in the input file
( e.g. my-vmware-cmd.sh vnic [vm_input_file] 3 )
Download:npiv_wwn_discovery-esx.sh - 09/20/08 Compatiable with: ESX 3.5+
This script is used to locate all virtual machines that have the NPIV option enabled and provide the VMs Name, Port WWN, Node WWN and the WWN TYPE on an ESX host. This can possibly list virtual machines with the NPIV option in a cluster if the ESX host is connected to all datastores present to the cluster.
Download:npiv_wwn_discovery-esxi.sh - 09/20/08 Compatiable with: ESXi
This script is used to locate all virtual machines that have NPIV option enabled and provide the VMs Name, Port WWN, Node WWN and the WWN TYPE on an ESXi host.
rdm.sh
Please find more details located at: http://communities.vmware.com/docs/DOC-10188 Compatiable with: ESX 3.5+ and ESXi
This script is used to locate all virtual machines that have an RDM mapping and provides the VMs Name, Hard Disk label shown on the VIC/VC, Datastore, LUN UUID, HBA/LUN, Compatibility Mode (Phys/Virt), DiskMode and Capacity.
ghettoVCB.sh Please find more details located at: http://communities.vmware.com/docs/DOC-8760 Compatiable with: ESX(i) 3.5 and 4.0+
This is a non-interactive script that backs up a specified list of VMs to a chosen datastore. Supported datastores are: local storage, SAN and NFS. The nature of the script allows backups to be scheduled via crontab. The script also provides an option to backup virtual machines to a non-persistent NFS share. The NFS connection is created on the ESX(i) server and is disconnected upon completion of backups.
1. Download VMware ESXi 3.5 Installable Update X from VMware's website (e.g. VMware-VMvisor-InstallerCD-3.5.0_Update_3-123629.i386.iso) and save the ISO to your desktop.
2. If you're on a Windows machine, you can use a free utility called WinImage to extract the contents from the ISO, if you're on a UNIX/LINUX system, you can just use "mount" and do mount loop to extract the contents.
3. Extract the following files from the ISO file:
install.tgz
--This contains the files you'll need for PXE Boot
mboot.c32
--Boot file that will gets loaded via PXE Boot
4. Use another free utility called Winrar for Windows and users on UNIX/LINUX can use gunzip/tar/bunzip to extract the files.
--This is another zip file for the non-installer for ESXi that'll be used for the stateless boots.
6. Open up VMware-VMvisor-big-3.5.0_Update_3-123629.i386.dd.bz2 using Winrar to extract yet another file:
VMware-VMvisor-big-3.5.0_Update_3-123629.i386.dd
7. Finally, this is the image that contains the necessary files for your stateless ESXi boot, open this using WinImage and extract the following files:
binmod.tgz
-- Not sure what this is, I haven't had a chance to poke around in this tar ball
boot.cfg
-- Default boot method (this will be modified with our own version)
cim.tgz
-- I assume related to CIM (Common Interface Management) for hardware monitoring
environ.tgz
-- This is the actual File System structure for ESXi (more on this later)
license.tgz
-- I assume license related information
oem.tgz
-- I believe this is for hardware drivers and customization
vmkernel.tgz
-- As the name states, this is the VMkernel for ESXi
8. Okay so recap, we should have a total of 8 files that'll be used for the ESXi stateless boot
mboot.c32
binmod.tgz
boot.cfg
cim.tgz
environ.tgz
license.tgz
oem.tgz
vmkernel.tgz
9. I'll assume user's have setup a TFTP server and PXE Boot before, if not, there are many articles online. In additon to these 8 files, you'll need to make sure you have a very well known PXE Boot file called "pxelinux.0", if you don't you can search for "syslinux" and obtain the files.
10. The nice thing about PXE Boot is you can control what IP Address a system will have based on it's MAC address and whether it's going to be pure DHCP or Static DHCP.
11. In our labs, within the PXE Boot directory, we create boot configuration files based on a specific MAC Address, and that'll be the example will be showing
12. Create the following structure under your TFTP root folder (our example is under /var/tftpboot):
[root@esxiPXE esxi]# pwd
/var/tftpboot/esxi/pxelinux.cfg
13. Put all 10 files into the /esxi directory
14. Create a file which will be named the MAC Address of the system that you would like to bootup, you can find this informaton within the iLO/DRAC and it should look like:
Now this is cool and all, but what if you want your ESXi Server to have stateful information when the host boots up? Well the next steps will show you how to customize your ESXi images to have state and also backup the configurations
1. You'll need the RCLI Toolkit for this and I would actually suggest downloading VIMA (VMware Infrastructure Management Assistant ) which is a consolidated RHEL Virtual Appliance that has both the RCLI + VI Perl Toolkit set intergrated and ready to go and its FREE!
2. If you need help setting up either RCLI or VIMA, I would suggest looking at the documentation on VMware's site, it's pretty straight forward.
3. On VIMA, to be able to work on a specific ESX or ESXi Host, you'll need to first add it into the managment center and then you can call to the VI Fastpass which allows you to pass your credentials in once and store that as a session key.
4. Once your ESXi has booted up, you'll want create/set a password, then you'll login to VIMA to do the rest.
5. On the VIMA Server you'll do the following commands:
sudo vifp adserver [ESXI_IP_ADRESS]
-- This will prompt for your local account password (vi-admin) and then the password to the ESXi Host which you've configured
vifpinit [ESXI_IP_ADRESS]
-- This enables fastpass if you've done the step above correctly
-- You'll now backup the current configuration of the ESXi host to the local VIMA system (the last param is only needed if you have more than 1 managed ESX/ESXi host on VIMA)
The backup is really just a collection of files stored in this tarball and specifically containing the contents of /etc/vmware/ which also has esx.conf which is the the main host configuration file.
6. With this file, all you have to do is upload it to your TFTP Server, you can rename this file however you want, I've just done hostname.tgz.
7. On your MAC Address file in the TFTP Server, you'll now want to append an additional param that says "--- hostname.tgz" what this will do is read this in during the PXE Boot and set the configurations you've saved.
8. With this, you can setup a host once with all the configurations such as vSwitches/Portgroups/Firewall/NTP/DNS/Users/Passwords and then run the backup process which will allow you to quickly deplaoy ESXi hosts on the fly and pre-loaded and ready to go.
9. Now, there are a few caveats to this method:
#1 any new changes made to the host will require a manual backup of the file else on the next reboot the changes will not persist
#2 the initial backup process must always be done because they contain unique UUID to the local storage, System UUID, etc. so you'll need to pull down the intial copy and then modify and upload it back
10. The workaround is to create a script that'll backup your current configuration and then upload to your TFTP Server, this can be done easily with SSH keys and VMware actually does this on the local system running /sbin/auto-backup.sh which calls to /sbin/backup.sh to create a backup file to /tmp/local.tgz (which is the contents of your backups).
11. Other customizations such as file system related changes like ..oh enabling SSH Console so you don't have to do this each time, you can extrac the contents of "environ.tgz" and modify the /etc/inetd.conf to uncomment SSH and I've gone ahead with that and added my SSH keys to the system so on bootup I just need to know the ESXi's IP Address and I can SSH in.
12. You can see with all these methods, it would not be hard to seutp a cluster of configured ESXi hosts without requiring them to be installed on disk and hold state.
13. Additional note on working on "environ.tgz", make sure when you package it back up, you're doing that within the directory as the folders are laid back down on "/" on the ESXi host, so if you want to include hidden directories like .ssh you'll want to run the following command inside of the directoy of files
tar cvf * ../environ.tgz .ssh
Also I would like to thank Jim Mccann & Mike Dipetrillo for their initial findings on stateless ESXi which helped me to create/test this setup!
Update: 11/25/2008
Talking with Dave Mishchenko on the VMTN forums, I learned that it's best to update the oem.tgz since this is where you would create your customizations of your ESXi Host and more information can be found on Dave's site
Update: 11/21/2008
ESXi crontab is located at /var/spool/cron/crontabs/root
VMware's undocumented "vimsh"
Vimsh
is an interactive shell buffer. It allows execution of shell commands in a vim
buffer, without having to suspend the vim session. It provides an interactive command
line session.
In the context of VMware ESX it has been implemented as an API for the ESX host
agent. VI Client via the ESX host and/or VirtualCenter, and can report and configure
numerous items. Changes made by using this command appear in the VI Client console
at normal refresh schedules.
What is vmware-vim-cmd?
Introduced in ESX 3.5 this command is a 'wrapper' for the vimsh shell. It simplifies the
command and means that vimsh can be run without having to enter and exit the shell
and use previously documented command switches and "quotes".
Why use it?
There is some overlap from current Service Console tools, such as vmware-cmd and
esxcfg-* but this API also provides some excellent reporting options and offers new ways
to tailor and automate scripting tasks.
The above definitions are the work/findings of Xtravirt and their initial publishing's on VMware vimsh.
To see the available commandset for a given "vimsh" release, you just execute the command normally on the shell.
You'll be be brought into the interactive shell, you can hit "tab" to see the various options and flags.
When running specific commands on Virtual Machine(s), vimsh requires the internal Virtual Machine Id called "VmId" which
needs to be passed as a parameter, the VmId for a given VM can be located when running vmsvc/getallvms as shown below.
Unlike the traditional vmware-cmd, where you pass in the .vmx path of a VM, you need to extract the VmId if you're looking
to automate/script any tasks using vimsh
[vimsh|vmware-vimsh|vmware-vim-cmd] hostsvc/datastore/listsummary | grep -E '(name|url|capacity|type)' | sed 's/,//g'
Create NFS Datastore as you would from VIC or VC (different from traditional NFS mount, so you can run vmkfstools w/o having to break up VMDK(s) into 2GB Sparse files)
This is the new name for the VIPT which are the Perl bindings to the vSphere SDK, all utilities are now under the following path: /usr/lib/vmware-vcli/apps
One of the new utilties that is packaged with the toolkit is /usr/lib/vmware-vcli/apps/general/credstore_admin.pl
add -s|--server -u|--username -p|--password
Add new entry into the credential store.
get -s|--server -u|username
Retrieve password from credential store
remove -s|--server -u|--username
Remove existing entry from credential store.
list [-s|--server ] [--showpw]
List existing entries.
clear
Delete all entries from credential store.
help
Display this menu.
VMware vCLI
VMware vCLI is the next version of the popular VMware RCLI toolset, all vCLI commands no longer contain the .pl extension, be sure to update your scripts if you're planning on testing/porting over to vCLI. There have been numerous bug fixes and new features, including some new commands. The following will document some of the changes and formal documentation will be available from VMware upon vSphere's GA.
System Management
vicfg-advcfg / esxcfg-advcfg
--set-message
-m
Set DCUI welcome message.
vicfg-module / esxcfg-module
--list
-l
List the set of modules on the host.
vihostupdate [for ESX(i) 4.x+] and vihostupdate35 [for ESX(i) 3.5]
--bulletin
-B
Parameter to specify the selective bulletin(s) to install. Use
comma to specify multiple bulletins (eg. bulletin1,bulletin2).
All bulletins will be installed if this option is not specified.
--list
-l
List the bulletins in the bundle or in the depot.
--nodeps
-d
Ignore dependencies during install operation.
--nosigcheck
-c
Ignore integrity checking during install operation.
--remove
-r
Remove selective bulletins from the host.
vmkuptime
This was never implemented in VIMA 1.0 and looks like it's been removed in vMA 4.0
Network Management
vicfg-route / esxcfg-route
--add
-a
Add route to the VMkernel,
requires <network> (described below)
<network> can be specified in 3 ways:
* As a single argument in <IP>/<Mask> format
* Or as a <IP> <Netmask> pair.
* Or as 'default'
--del
-d
Delete route from the VMkernel,
requires <network> (described below)
<network> can be specified in 3 ways:
* As a single argument in <IP>/<Mask> format
* Or as a <IP> <Netmask> pair.
* Or as 'default'
--family (default 'v4')
-f
Address family to work on ('v4' or 'v6'). Default to 'v4'.
--list
-l
List configured routes for the VMkernel
vicfg-vmknic / esxcfg-vmknic
--add
-a
Add a VMkernel NIC to the system. Requires ip, netmask, portgroup.
--delete
-d
Delete the VMkernel NIC on the given portgroup.
--disable-vmotion
-u
Disable VMotion for the VMkernel NIC on a specified
portgroup.
--dvport-id
-v
Setting DVPort ID of the connection point. Requires --dvs-name parameter.
--dvs-name
-s
Setting DVSwitch name of the connection point. Requires --dvport-id parameter.
--enable-ipv6
-6
Enable (true) or disable (false) IPv6 for next boot.
--enable-vmotion
-E
Enable VMotion for the VMkernel NIC on a specified portgroup.
--ip
-i
The IP address for this VMkernel NIC where IP address can be:
<X.X.X.X> - to use static IPv4 address specified
DHCP - to enable IPv4 DHCP for address
<X:X:X::/X> - to use static IPv6 address
DHCPV6 - to enable IPv6 DHCP for address
AUTOCONF - to enable IPv6 Router advertised address
--list
-l
List VMkernel NICs on the system.
--mtu
-m
MTU for the interface being created.
--netmask
-n
The IP netmask for this VMkernel NIC
--portgroup
-p
Setting portgroup as connection point.
--tso
-t
Disable TSO for the vmknic being created.
--unset-ip
-U
The IP address for this VMkernel NIC where IP address can be:
<X:X:X::/X> - to remove the specified IPv6 address
DHCPV6 - to disable IPv6 DHCP address
AUTOCONF - to disable IPv6 Router advertised address
vicfg-vswitch / esxcfg-vswitch
--add-dvp-uplink
-P
Add an uplink adapter (pnic) to a DVPort
--del-dvp-uplink
-Q
Delete an uplink adapter from a DVPort
--dvp
-V
The name of the DVPort
Storage Management
esxcli (Take a look at Duncan Epping's post for more information)
esxcli is a framework for CLI commands
vicfg-iscsi / esxcfg-iscsi (We finally have a way of managing iSCSI configurations using the vCLI and there are quite a few options)
--adapter
-H
List iSCSI adapter(s).
--add
-a
Add operation. Used with --discovery or --static option.
--alias
-k
iSCSI initiator alias name. Used with --iscsiname option.
--auth_password
-w
Authentication password. Used with --authentication option.
--auth_username
-u
Authentication username. Used with --authentication option.
--authentication
-A
Authentication properties and configuration.
--detail
-f
Details of iSCSI parameters. Used with --parameter option.
--disable
-q
Disable operation. Used with --swiscsi option.
--discovery
-D
Discovery addresses properties and configuration.
--enable
-e
Enable operation. Used with --swiscsi option.
--gateway
-g
Default gateway. Used with --network option.
--ip
-i
Specify IP address or DNS recognized domain name. Used with
--discovery, --static, --authentication, --network, or
--parameter option.
--iscsiname
-I
List or configure iSCSI initiator name or alias.
--level
-c
Authentication level. Used with --authentication option.
--list
-l
List operation. Used with --discovery, --static, --target, --lun,
--authentication, --phba, --network, --pnp, --iscsiname, --parameter,
--swiscsi or --adapter options.
--lun
-L
List active LUNs information.
--method
-m
Authentication method, allows 'CHAP'. Used with --authentication
option.
--mtu
-M
MTU size. Used with --pnp option.
--mutual
-b
If set, indicates mutual CHAP. Used with --authentication option.
--name
-n
Initiator or target iSCSI name. Used with --static,
--authentication, --iscsiname, or --parameter option.
--network
-N
Network properties and configuration.
--parameter
-W
For iSCSI parameters operations.
--phba
-P
List Phba and node information.
--pnp
-p
List Physical Network Portal properties.
--remove
-r
Remove operation. Used with --discovery or --static option.
--reset
-o
Reset target level specified iSCSI parameter to be inherited from
adapter level. Provide <name>. The <name> can be one of the
parameter names listed in the --parameter --list option. Used with
--parameter option.
--reset_auth
-z
Reset target level authentication properties to be inherited from
adapter level. Used with --authentication option.
--set
-j
Set iSCSI parameter specified by <name> to the value specified by
<value>. Provide <name>=<value> pair to this option. The <name>
can be one of the parameter names listed in the --parameter --list
option plus 'dataDigestType', or 'headerDigestType' when used with
--parameter option. The <name> can be 'ARP' when used with
--network option.
--static
-S
Static discovery targets properties and configuration.
--subnetmask
-s
Subnet mask. Used with --network option.
--swiscsi
-E
Software iSCSI enabling configuration or information.
--target
-T
List active targets information.
--target_id
-t
Target ID. Used with --lun option.
vicfg-mpath / esxcfg-mpath
for ESX(i) 3.5 use vicfg-mpath35 / esxcfg-mpath35
for ESX(i) 4.x+ use vicfg-mpath / esxcfg-mpath
vicfg-nas / esxcfg-nas
--readonly
-y
Add the new NAS filesystem with readonly access
vicfg-vmhbadevs / esxcfg-vmhbadevs
is now vicfg-scsidevs/ esxcfg-scsidevs
vicfg-volume / esxcfg-volume (Take a look at Duncan Epping's post for more information
--list
-l
List all volumes which have been detected as snapshots/replicas.
--persistent-mount
-M
Mount a snapshot/replica volume persistently, if its original
copy is not online.
--resignature
-r
Resignature a snapshot/replica volume.
--umount
-u
Umount a snapshot/replica volume.
VMware RCLI
What is VMware RCLI?
The VMware Infrastructure Remote CLI provides a command-line interface for datacenter management from a remote server. This interface is fully supported on ESX(i) 3.5u2+. Download it here.
You can also download the RCLI command documentation here
Here are some awesome examples and usage of the RCLI commands by Dave Mischenko
VMware vMA is the next release of the VMware VIMA vApp (Virtual Appliance) which will be available as part of the vSphere platform launch, there have been numerous fixes and enhancements from the last release and more information will be available upon vSphere's GA.
Interesting tools
There will be no direct upgrade paths from VIMA 1.0 to vMA 4.0, you'll need to re-deploy the vApp and reconfigure and re-add all your ESX(i) 3.5u2+ and ESX(i) 4.x hosts manually using vifp. Here are two tools that I are worth mentioning when deploying vMA 4.0.
The first tool is called bulkAddServers.pl which allows a user to specify a list of host(s) and providing a username/password once and the script will automatically import the entire list, without having to add each host manually and providing the credentials each time. [this works very if you a common account across all your host(s))].
The script is stored in: /opt/vmware/vima/samples/perl/bulkAddServers.pl
[vi-admin@rafaeli ~]$ sudo /opt/vmware/vima/samples/perl/bulkAddServers.pl --filename hosts_to_add
Do you want to use the same password for all of the hosts (yes/no) : yes
Enter common root password:
All Targets Added Successfully.
The next tool is primarly to help user's execute a command or series of command across a set of ESX(i) host(s), in the past you would need to manually do this for each of the manage host(s) with the vCLI/RCLI tools, say to add/update portgroup on a vSwitch for example. This can be pretty redudant and you could write a quck for loop to do this but VMware wrote a script called mcli which allows a user to execute a command and it's argument against a set of host(s) listed in a file.
The script is stored in: /opt/vmware/vima/samples/perl/mcli.pl
Sample execution to check for syslog configuration
[vi-admin@rafaeli ~]$ /opt/vmware/vima/samples/perl/mcli.pl hosts_to_execute_cmd_on esxcfg-syslog --show
everest.primp-industries.com:
No remote syslog server configured.
himalaya.primp-industries.com:
No remote syslog server configured.
What's new with vMA
vma-help
For help on the vSphere Authentication Component (vi-fastpass)
run 'vma-help ' Possible commands are:
vifp vifpinit vilogger
For help on vSphere CLI commands run 'vma-help '. Possible commands are:
esxcfg-advcfg esxcfg-cfgbackup esxcfg-dns esxcfg-dumppart
esxcfg-iscsi esxcfg-module esxcfg-mpath esxcfg-mpath35
esxcfg-nas esxcfg-nics esxcfg-ntp esxcfg-rescan esxcfg-route
esxcfg-scsidevs esxcfg-snmp esxcfg-syslog esxcfg-user
esxcfg-vmknic esxcfg-volume esxcfg-vswitch esxcli resxtop
svmotion vicfg-advcfg vicfg-cfgbackup vicfg-dns vicfg-dumppart
vicfg-iscsi vicfg-module vicfg-mpath vicfg-mpath35 vicfg-nas
vicfg-nics vicfg-ntp vicfg-rescan vicfg-route vicfg-scsidevs
vicfg-snmp vicfg-syslog vicfg-user vicfg-vmknic vicfg-volume
vicfg-vswitch vifs vihostupdate vihostupdate35 viperl-support
vmkfstools vmware-cmd
vima-update scan
Lists applicable bulletins with updates
vima-update -b update
Updates to a specific bulletin
vima-update update
Applies all available updates
Add a vCenter 4.0 as a target
[vi-admin@vma ~]$ sudo vifp addserver mauna-loa.primp-industries.com --username primp
primp@mauna-loa.primp-industries.com's password:
This will store username and password in credential store which is a security risk. Do you want to continue?(yes/no): yes
[vi-admin@vma ~]$ sudo vifp listservers
himalaya.primp-industries.com ESX
everest.primp-industries.com ESX
mauna-loa.primp-industries.com vCenter
Querying specific ESX/ESXi host through vCenter target (use --vihost to specify ESX/ESXi host)
Locate all Virtual Machines with snapshots on all ESX/ESXi servers being managed by VIMA
snapshotmanager.pl --operation list | grep -v "No Snapshot" | sed '/^$/d'
Locate all VMs with snapshots on a specific ESX host
snapshotmanager.pl --operation list --server [ESX_HOSTNAME] | grep -v "No Snapshot" | sed '/^$/d'
[vi-admin@vima-primp-industries vm]$ ./snapshotmanager.pl --operation list --server himalaya.primp-industries.com | grep -v "No Snapshot" | sed '/^$/d'
Snapshots for Virtual Machine Tuan-test under host himalaya.primp-industries.com
Name Date State Quiesced
s 2008-12-14T00:38 poweredOff N
Snapshots for Virtual Machine Thinapp under host himalaya.primp-industries.com
Name Date State Quiesced
Before Thinapp 2008-12-16T20:57 poweredOff N
Pristine-After Thinapp Install 2008-12-16T21:17 poweredOff N
Snapshots for Virtual Machine Labclient-2 under host himalaya.primp-industries.com
Name Date State Quiesced
Cloned 2008-12-13T14:41 poweredOff N
pristine 2008-12-13T14:52 poweredOff N
Snapshots for Virtual Machine Labclient-3 under host himalaya.primp-industries.com
Name Date State Quiesced
Cloned 2008-12-13T14:41 poweredOff N
pristine 2008-12-13T14:52 poweredOff N
Snapshots for Virtual Machine Labclient-1 under host himalaya.primp-industries.com
Name Date State Quiesced
Cloned 2008-12-13T14:41 poweredOff N
pristine 2008-12-13T14:52 poweredOff N
List all information regarding datastore(s)/filesystem/vm/etc... for specific ESX/ESXi server
dsbrowse.pl --server [ESX_HOSTNAME]
dsbrowse.pl olga.resnet.ucsb.edu
Information about datastore : 'olga-local-SAS.Storage'
---------------------------
Summary
Name : olga-local-SAS.Storage
Location : /vmfs/volumes/49167992-ac31f0a5-3f9f-001f29c7a04c
File system : VMFS
Maximum Capacity : 255.25 GB
Available space : 244.658203125 GB
Hosts associated with this datastore.
olga.resnet.ucsb.edu
Virtual machines on this datastore.
None
Templates on this datastore.
None
Datastore Folder Structure.
Folder Path: '[olga-local-SAS.Storage]'
Files present
UCSB-GAUCHOS
........
.......
......
List specific attributes regarding datastore(s)/filesystem/vm/etc... for specific ESX/ESXi server (this will include VM information by default)
./dsbrowse.pl --server olga.resnet.ucsb.edu --attributes name,location,capacity,freespace
Information about datastore : 'olga-local-SAS.Storage'
---------------------------
Summary
Maximum Capacity : 255.25 GB
Available space : 244.658203125 GB
Location : /vmfs/volumes/49167992-ac31f0a5-3f9f-001f29c7a04c
Name : olga-local-SAS.Storage
Datastore Folder Structure.
Folder Path: '[olga-local-SAS.Storage]'
Files present
UCSB-GAUCHOS
If you would like to exclude VM information, use the following and extract what you need:
dsbrowse.pl --server olga.resnet.ucsb.edu --attributes name,location,capacity,freespace | grep -E '(Name|Location|File system|Maximum Capacity|Available space)'
Maximum Capacity : 124.75 GB
Available space : 34.1806640625 GB
Location : /vmfs/volumes/48e1cd81-81223593-6cd8-00215acaa2b2
Name : dlgCore-FC-LUN200.Templates
Maximum Capacity : 255.25 GB
Available space : 244.658203125 GB
Location : /vmfs/volumes/49167992-ac31f0a5-3f9f-001f29c7a04c
Name : olga-local-SAS.Storage
Maximum Capacity : 151.75 GB
Available space : 30.5517578125 GB
Location : /vmfs/volumes/48e1a4a6-3b55e69b-02a9-001f29c948e2
Name : dlgCore-FC-LUN202.VMstorage2
Maximum Capacity : 152.75 GB
Available space : 55.947265625 GB
Location : /vmfs/volumes/48e16ed0-420158d5-5601-00215acaa2b2
Name : dlgCore-FC-LUN203.VMstorage3
Maximum Capacity : 150.75 GB
Available space : 23.7802734375 GB
Location : /vmfs/volumes/48e1b819-3443974c-03c4-001f29c948e2
Name : dlgCore-FC-LUN201.VMstorage1
Quick and dirty script of pulling information about all Virtual Machines residing on a given ESX/ESXi 3.5+ Server
#!/bin/bash
if [ ! $# -eq 1 ]; then
echo "Usage: `basename $0` [ESX_ESXi_HOSTNAME_OR_IP]"
exit 1
fi
ESX_HOST=$1
for VM in `/usr/lib/vmware-viperl/apps/vm/vminfo.pl --server "${ESX_HOST}" | grep -E '(Name:|vmPathName:)' | sed -e '$!N' -e 's/\n//;s/vmPath//g' | sed 's/[[:blank:]]$//g' | sed 's/Name://g' | sed 's/[[:blank:]]\{3,\}/ /g' | awk -F' ' '{print "\""$2"\";\""$3"\""}' | sed 's/\] /\]\";\"/g'`;
do
VM_NAME=$(echo "${VM}" | awk -F ";" '{print $1}' | sed 's/"//g')
/usr/lib/vmware-viperl/apps/vm/guestinfo.pl --vmname ${VM_NAME} --operation display --server "${ESX_HOST}"
echo -e "###################################################################################\n"
done
[vi-admin@vima-primp-industries ~]$ ./getallvms.sh
Usage: getallvms.sh [ESX_ESXi_HOSTNAME_OR_IP]
[vi-admin@vima-primp-industries ~]$ ./getallvms.sh olga.resnet.ucsb.edu
Guest Info for the Virtual Machine 'UCSB-RESNET' under host olga.resnet.ucsb.edu
UCSB-RESNET guestFamily: windowsGuest
UCSB-RESNET guestFullName: Microsoft Windows XP Professional (32-bit)
UCSB-RESNET guestId: winXPProGuest
UCSB-RESNET guestState: running
UCSB-RESNET hostName: WinXPeDev.
UCSB-RESNET ipAddress: 169.254.74.15
UCSB-RESNET toolsStatus: VMware Tools is running and the version is current.
UCSB-RESNET toolsVersion: 103908
UCSB-RESNET Screen - Height: 0
UCSB-RESNET Screen - Width: 0
UCSB-RESNET Disk[0]: Capacity 10725732352
UCSB-RESNET Disk[0]: Path : C:\
UCSB-RESNET Disk[0]: freespace : 5716586496
UCSB-RESNET net[0] - connected : 1
UCSB-RESNET net[0] - deviceConfigId : 4000
UCSB-RESNET net[0] - macAddress : 00:50:56:b4:69:0b
UCSB-RESNET net[0] - network : IsolatedNetwork
UCSB-RESNET net[0] - ipAddress : 169.254.74.15
###################################################################################
You can see that you can easily modify the output to extract certain pieces of information from this output, including substituting other RCLI commands
VMware VIX 1.6.2 "vmrun" can be installed on either a Windows or Linux/UNIX host (examples will be showing off both Windows XP and VMware VIMA)
To use "vmrun", you need to be running VMware ESX/ESXi 3.5u3 and have VMware Tools installed on your guestOS
You should always double quote your vmrun parameters or arugments to a program if you're executing something within the guestOS that has spaces in the input
List running VMs on specific ESX/ESXi host
[vmrun.exe|vmrun] -T esx -h https://ESX_IP_OR_HOSTNAME/sdk -u ESX_USER_NAME -p ESX_PASSSWORD list
# Windows example of running vmrun on Windows XP
C:\Program Files\VMware\VMware VIX>vmrun.exe -T esx -h https://himalaya.primp-industries.com/sdk -u root -p ********* list
Total running VMs: 14
[himalaya-local-SAS.VMStorage] Fullerene/Fullerene.vmx
[himalaya-local-SATA.Storage] Primp-DevAD/Primp-DevAD.vmx
[himalaya-local-SAS.VMStorage] Synapse/Synapse.vmx
[himalaya-local-SAS.VMStorage] STA202I/STA202I.vmx
[himalaya-local-SAS.VMStorage] Bitsy/Bitsy.vmx
[himalaya-local-SAS.VMStorage] William-XP/William-XP.vmx
[himalaya-local-SAS.VMStorage] VIMA/VIMA.vmx
.......
.......
# Linux example of running vmrun on VMware VIMA
[vi-admin@vima-primp-industries ~]$ vmrun -T esx -h https://himalaya.primp-industries.com/sdk -u root -p ********* list
Total running VMs: 14
[himalaya-local-SAS.VMStorage] Fullerene/Fullerene.vmx
[himalaya-local-SATA.Storage] Primp-DevAD/Primp-DevAD.vmx
[himalaya-local-SAS.VMStorage] Synapse/Synapse.vmx
[himalaya-local-SAS.VMStorage] STA202I/STA202I.vmx
[himalaya-local-SAS.VMStorage] Bitsy/Bitsy.vmx
[himalaya-local-SAS.VMStorage] William-XP/William-XP.vmx
[himalaya-local-SAS.VMStorage] VIMA/VIMA.vmx
.......
.......
vmrun commands that may have a bug or may not work properly:
revertToSnapshot Path to vmx file Set VM state to a snapshot
Snapshot name
clone Path to vmx file Create a copy of the VM
Path to destination vmx file
full|linked
[Snapshot name]
Automate vmrun using batch script (save as .bat file)
setlocal
@echo off
::set up custom user variables
:: ######## ESX Variables ##############::
set ESX_IP=172.30.0.61
set ESX_USER_NAME=root
set ESX_PASSSWORD=FILL_IN_PASSWORD
::#####################################::
::##### GUEST VM Variables ############::
set VM_USER_NAME=administrator
set VM_PASSWORD=FILL_IN_PASSWORD
::#####################################::
::set path to VIX vmrun binary
set Path=%PROGRAMFILES%\VMware\VMware VIX
::some test VM
::this can be automated by passing in a list of VMs and comparing with the registered list
set VM_NAME=WILLIAM-LC-22
set VM_PATH=[himalaya-local-SAS.VMStorage] %VM_NAME%/%VM_NAME%.vmx
::vmrun command
REM vmrun.exe -T esx -h https://%ESX_IP%/sdk -u %ESX_USER_NAME% -p %ESX_PASSSWORD% -gu "%VM_USER_NAME%" -gp "%VM_PASSWORD%" start "%VM_PATH"
vmrun.exe -T esx -h https://%ESX_IP%/sdk -u %ESX_USER_NAME% -p %ESX_PASSSWORD% list
endlocal
C:\Documents and Settings\Administrator\Desktop\VIX_samples>vix_sample.bat
Automate vmrun using shell script (save as .sh file)
#!/bin/bash
######## ESX Variables ##############::
ESX_IP=172.30.0.61
ESX_USER_NAME=root
ESX_PASSSWORD=FILL_IN_PASSWORD
#####################################::
##### GUEST VM Variables ############::
VM_USER_NAME=administrator
VM_PASSWORD=FILL_IN_PASSWORD
#####################################::
# some test VM
# this can be automated by passing in a list of VMs and comparing with the registered list
VM_NAME=WILLIAM-LC-22
VM_PATH="[himalaya-local-SAS.VMStorage] ${VM_NAME}/${VM_NAME}.vmx"
# vmrun command
# vmrun -T esx -h https://${ESX_IP}/sdk -u ${ESX_USER_NAME} -p ${ESX_PASSSWORD} -gu "${VM_USER_NAME}" -gp "${VM_PASSWORD}" start "${VM_PATH}"
vmrun -T esx -h https://${ESX_IP}/sdk -u ${ESX_USER_NAME} -p ${ESX_PASSSWORD} list
Multiple Displays:
This (option found on: sanbarrow.com however unsupported it may be by VMware and/or HP) is by far the most useful feature we've discovered while testing HP's RGS display protocol in our VDI environment. It works in versions pre-ESX3.5 however we don't remember how far back the compatibility stretches. From what we can recall, this does not work in ESX 3.0. This option has been tested with HP Remote Graphics Software Sender version 5.1.5 running on Windows XP SP2 and SP3 and is being used daily with dual 1280x1024 screens on an HP thinclient.
To enable multiple displays for virtual machines in ESX 3.5, insert the following into the .vmx file or use the "Configuration Parameters" in the VI client that corresponds to the virtual machine:
svga.numDisplays = "x"
where x is the number of displays including the primary display. In the case of this example, it should be set to 2.
The PC version of HP's Remote Graphics Software only supports 2 screens so please set it accordingly. Afterwards, connect to the virtual desktop using the RGS Receiver and enable the second screen as if on a physical workstation. This option will allow you to fully expand windows in their respective screens without using special display spanning software (Go RGS!!).
Issues: - When connecting to the virtual desktop using the RGS Receiver, the video will be somewhat corrupted if any of the displays on the virtual desktop is set to 32bit. This can be fixed by setting the display depth to 16bit for both screens. The problem seems to be related to the VMware SVGA driver as the corruption can be seen to occur inside the VI client console.
- The second display can not sit directly above the first display. To fix this, set the second (secondary) display to be below the first (primary) display and align the entire virtual display to the top of your physical display layout.
- Some little icons don't fully display. It is pretty annoying when it happens but we've seen this seldomly occur when using the VI client (or the Device Manager as another example) inside the virtual desktop on the primary screen. Moving the VI client (or affected application) over to the secondary screen or clicking on the icons (if possible) will sometimes fix it...
These settings are documented in the HP RGS documentation and are needed to increase the virtual machine's display resolution capability (to 1920x1200) and video memory size (to 24MByte). Please be aware that this option imposes additional memory overhead for the virtual machine being modified. Please size the parameters accordingly to your VDI setup.
RGS on Windows 7 on vSphere
After the Windows 7 RC came out, we were naturally inclined to throw it into our ESX3.5 environment. The main aim was to test it using RDP 7 (the VI console is a tad bit unresponsive because the SVGA driver isn't compatible) but we realized that only Windows 7 has the RDP 7 client so in turn, you would need another Windows 7 box (physical) to connect to the Windows 7 VM with RDP 7 (inefficient and no fun to say the least). So instead of installing 7 on a physical box, we decided to go the unconventional (or more affectionately, ghetto) route: get RGS running on 7.
The first issue in our quest to get a "rich remote Windows 7 desktop user experience" is the SVGA II video driver. RGS relies on the underlying "virtual" video card to display information. Video performance without the SVGA driver, as it stands, is quite simply crap and would continue to be so through RGS. To get around this problem, this virtual machine was pushed over to vSphere (with motivation from Eric Sloof's ntpro.nl article) and it's virtual hardware/VMware tools upgraded.
- Make sure the SMBIOS.reflectHost = "true" parameter or licenses are installed where appropriate
- Connect to the console of the virtual machine
- Installing the RGS sender (we used v5.1.5) on 7 can be accomplished by setting the compatibility mode of the installer to Windows XP SP2 (or SP3). Don't enable single sign-on. Upon reboot, the service will not start because of missing dependencies.
- Go to the installation directory and set the rgsender.exe compatibility mode to Windows XP SP2 (for performance reasons, oddly, SP3 gives laggy remote desktop response).
- Run rgsender.exe
- The screen will then lock, so unlock it in the VI console (if you try to connect using the RGS Receiver at this point, you'll get a blank screen)
- Log in using the RGS Receiver
- Enjoy the somewhat rich remote Windows 7 desktop user experience without RDP 7 (somewhat because advance graphics features needing acceleration won't work due to SVGA II limitations)
Issues encountered so far:
While usable, this ghetto rig isn't practical and should be used for demo or testing purposes only.
- Collaboration sessions with another user will not work because the dialogue does not pop up
- UAC notifications pause the screen because RGS can not display it, you need to go to the console to click out of it. A quick fix is to disable the UAC notification all together
- If at any point 7 locks the screen while you are away from it, you won't be able to log in. Doing the console walk of shame is the only way to unlock it so far or one can turn off the "lock when away from computer" feature all together.
- The sound may be distorted, turn the volume down to fix this. The output audio is oddly using the microphone input...
- Disconnecting from the virtual desktop will leave it unlocked. While this is good because you can log back in (until 7 locks itself), please be advised that the virtual desktop is vulnerable to others that have VI client console access.
- Although vSphere was used for this, RGS may run on Windows 7 in ESX 3.5 but without the SVGA II driver (not tested).
- We pulled this RGS setup all the way up to 1920x1200 with good results. We haven't tried dual screens yet although it may take some effort to do so because the second display is not coming up in the virtual machine.
And for fun, this post was created using Windows 7 with RGS through a VPN connection over wireless. But of course an image does not do it justice.
HP RGS (v5.2.5) on Windows 7 on vSphere (Addendum)
Thanks to a tip from Greg Hughes, he has noted that the latest version of HP RGS can be installed onto Windows 7 without using the workarounds noted above. Due to the licensing changes, the parameter:
SMBIOS.reflectHost = "true"
is no longer needed in the .vmx file.
Some issues encountered during this run:
- Screen blanking does not work. RGS does not even give a warning that the screen is visible at the console.
- The sound may be distorted, turn the volume down to fix this.
- RGS icon does not show up on task bar.
- If the system is locked on the console side, the RGS session will lock of course. Entering the password on the RGS session will unlock the session however mouse control is cut from the RGS session. Enabling remote keyboard/mouse may solve this however the RGS icon is not available on the taskbar. There may be another way to enable it...
We have included a quick demo of RGS 5.2.5 running on Windows 7. The recording program that we used made the video a little sluggish (as can be seen from the not-so-smooth mouse movements which should otherwise be smooth when using RGS regardless of connection quality).
Provided is a quick recording of how RGS (v5.1.5) performs on a "cloud" hosted machine when playing Flash and Quicktime videos. The capture software that we used made the video a little sluggish (as can be seen from the not-so-smooth mouse movements which should otherwise be smooth when using RGS regardless of connection quality). User interface performance was crisp and interaction was very near to the all important “local feel” (which is uncharacteristic of RDP connections).
A Windows AMI configured as a "High-CPU Medium (c1.medium)" instance type was selected for this session. The HP RGS image quality was arbitrarily set to 50% to conserve bandwidth and processing load. As this is an initial proof of concept, we did not optimize the image quality parameter and as so it will be left as an additional exercise in the future. Due to time constraints, common office productivity software was not tested and demonstrated on the cloudafied desktop however we strongly believe that RGS would perform well in this area as most common office productivity software is static and thus not bound by the CPU and connection bandwidth requirements necessary for dynamic multimedia content.
We believe that performance can be improved because the EC2 hosted Windows machine is not running an optimized video driver, that is, it is using the standard Windows VGA adapter driver. This drawback limits the maximum desktop size to 1280x1024 (without explicit hacks to increase the resolution capability if at all possible). Additionally, an older version of RGS was used during testing on an operating system not intended for desktop use (although changing processor scheduling and memory usage parameters may improve apparent desktop performance); newer versions of RGS running on the intended platforms may offer improved performance. Despite these drawbacks, multimedia performance was still surprisingly satisfactory.
We recently discovered an anomaly while backing up one of our development VMs using ghettoVCB.sh. When attempting to back up this powered on VM, the backup was successful however oddly, we were left with a powered off VM immediately following the first VMDK clone operation. After some investigation, we found that the problematic VM contained virtual disks spread across two datastores with dissimilar blocksizes (1MB and 2MB).
The VM configuration alongside its main OS disk was stored on the datastore with a 1MB blocksize while it’s data disk (>256 GB) resided on the other datastore which was initialized with a 2MB blocksize. We came to the conclusion that this might have had something to do with the VM configuration residing on a datastore with a blocksize that was smaller than what is needed for the larger VMDK (which was on a datastore with an ample blocksize). Manually snapshotting this VM apparently fails however different behavior was experienced when the commands are executed from a script.
Believing that this was a corner case, we decided that it was best practice to keep all VMFS volume block sizes consistent. This was to be remediated at a later time.
If you have a similar configuration from above, it is guaranteed that the VM will crash if you run a script that tries to take a snapshot of the described VM and then subsequently exports the VMDK using vmkfstools.
[root@himalaya ~]# ./crashVM.sh Quentin /vmfs/volumes/dlgCore-NFS-bigboi.VM-Backups/Quentin-clone.vmdk
thinking, give me a few ...
The power state of VM: "Quentin" is On
Extracted VmId and locating VM configured datastore (which should live on a smaller VMFS block size)
Located VMDK: "/vmfs/volumes/himalaya-local-SAS.Savvio/Quentin/Quentin.vmdk"
Trying to create snapshot ... (this should fail)
Create Snapshot:
Trying to vmkfstools copy "/vmfs/volumes/himalaya-local-SAS.Savvio/Quentin/Quentin.vmdk" (this should REALLY fail! right?)
Destination disk format: VMFS zeroedthick
Cloning disk '/vmfs/volumes/himalaya-local-SAS.Savvio/Quentin/Quentin.vmdk'...
Clone: 9% done.
