1. ESXi SNMP is constrained by resource pools both for memory and cpu.
2. ESXi SNMP is a read only agent and does not support SNMPv1 SETs operations
So having access to it is quite unlikely to do anything untoward the system just because the udp port is reachable
itself though once having access to the data might be used in a further attack.
3. UDP source IP addresses can be spoofed meaning packets may still get to the agent regardless of configuration.
In any case typical network deployment limits the availability of the mgmt interface (typically vmk0) IPv4 address to a network/vlan that is already secured for mgmt to/from the systems that will access it and all other traffic to the host goes to vlans/networks that carry traffic for the VMs. I believe that using SNMPv3 (in ESXi 5.1) is a better solution than SNMPv1 plus firewall since packets benefit from SHA1 authenticity and/or AES128 privacy akin to TLS/SSL.