I'm fairly certain Google requires you enter a confirmation code when setting up a phone number (the code is sent via SMS to the phone) for specifically this reason. It's not as though this is the only piece of software using the Calendar API; Google has already had to think of these potential exploits.