It just comes down to entrenchment and popular opinion. Much in the same vein, alcohol and tobacco "aren't drugs," but pot and 'shrooms are.