Even worse is that others can retroactively opt out on another's behalf. If someone owned the domain with no exclusion, then their content seemingly disappeared in 2003 because they lost ownership/stopped caring about the domain, and the new owner puts up a robots.txt, the original owner's content will not be publicly accessible.

At least that was the case a few years ago. I know the Wayback Machine has seen some changes in the last year.