Social engineering will almost always work. I don't really fault Sendgrid for this (though I could see this not working as well if you were using Amazon SES...no support to even talk to!). It sucks that they got caught with their pants down but I bet a good social engineering attempt on ChunkHost might have yielded similar results.
The lesson here is to have multiple defenses. 2 factor auth is a great start and it worked in this case.
You should fault Sendgrid as they specifically have a policy NOT to perform this change of email request (from the article).
Sendgrid can also change their systems so that phone support personnel can NOT perform this change or perform this change with approval from a supervisor.
Sendgrid being in the business they are in should also know that they are susceptible to these types of attacks and what they can lead to (many, many systems which can have password requests sent to email addresses).
I don't know... Mistakes happen. There seems to be little to gain from faulting SendGrid, but faulting SendGrid would force them to take some kind of action such as terminating the rep. I think I'd prefer the rep remain employed, because I trust they'd never make this mistake again. Also, now all the other reps know to avoid it.
EDIT: May I ask what can be gained from faulting SendGrid in this case?
I agree with you that terminating the rep is not interesting, and I think you're mistaken if you feel like that's what anyone thinks will solve this problem.
Actions SendGrid could take:
* Make it impossible for their front-line support staff to change the email address on file. If you want that -- which should be extremely rare! -- you talk to a high-level manager who is competent at authenticating you.
* Send the email that says "hey, we're going to change your email address now" with a lead time to allow for the possibility that, even after your authentication, you've been conned.
* Make a phone call to the phone number on record, too.
You ask what's gained by faulting SendGrid, because you take it as a given that they will make these changes. But that's not how blame works. The blame serves a function of ensuring those changes by holding them accountable for their current problems.
In a nutshell, social engineering can be countered by proper software & processes engineering.
So if social engineering is possible, blame the software architects.
Maybe in extreme cases changing the email of an account might be needed, but there's no excuse a first level rep was able to do it. Least thing, he/she should've been forced by the system to escalate to someone above her, who has a much lesser chance to screw up.
They didn't say it was first-level rep. Maybe the rep passed it up the chain.
This was a pretty weak request, though. This one should have been pretty easy to at least make some attempt to verify.
But as a human being I can verify how hard it is to not help the person crying on the other end of the phone because they need help RIGHT NOW. This didn't get to that level, but if you are designing a recovery process, you really need to think about how you handle that situation, and make sure the people making the call have the guts to say "I'm sorry but we need to do this one by the book."
The lesson here is to have multiple defenses. 2 factor auth is a great start and it worked in this case.