On the docs, there is not such limitation. http://pubs.vmware.com/vsphere-51/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-511-upgrade-guide.pdf
Page 87.
But really don't know why this is not listed on the download section. Never tried. If the upgrade does not work, try upgrading the GAs to 5.1.0a and then to U1.