In case of contention, ESXi will use up to 10% (by default) of the VM's provisioned memory as compression cache. The memory required for the cache will automatically be freed up by the compression itself, i.e. by reducing memory pages from 4kB -> 2kB.
André