User agent catching is rather effective. You can serve different responses based on UA.

So generally people will use a robots.txt to catch the bots that play nice and then use useragents to manage abusers.