The most annoying fan bases will always be your rivals and perhaps some pesky program's fans that occasional manages to beat you, but really is inferior.
I knew quite a few Ohio State fans in Cincy of course, we'd talk football a fair bit. The ones I knew were good fans. I still know two of them and carry on about CFB.