Welcome to the Community - To troubleshoot will need more information about your environment -
How are your ESXi hosts configured - memory, CPU, number of VMs they are hosting and network configutration.
How are you VMs that are acting as the terminal servers configured - memory, number of vCPUs, reservations or limits set?