The problem is not that the specific tasks are too heady for most programmers, per se, but that it is easy to introduce a weakness into the system and almost impossible for most programmers to know when they have done so. Implementing an algorithm correctly is one thing; ensuring that your whole system does nothing to undermine the security you hope to get from that algorithm (even through side channels) is a lot iffier.

Remember, people with years of training and experience in the field have had their work defeated time and again. Do you really feel certain Joe Coder will make fewer mistakes than they did?