I believe that one of the reasons that many people actively dislike PvP is not the mechanics of the game (which are often rather fun, inventive and reward skill), but the superior, childish and aggressive attitude of many of the people who play it. The assertions that it is somehow "better" than PvE, or that it is not possible to post intelligently on the subject unless you have a full set of Gladiator epics, really seems to pervade that side of the game.
I think arena has made the problem much worse, because of the very personal head to head nature of the fights. Frankly the entire arena experience is incredibly intimidating and is made much worse by the macho posturing of many of the players, both in game and on the forums. I know several very good PvE players who don't arena at all for this very reason.
A personal tale (and the catalyst for this post). I "stepped into the arena" for the first time at 80 on Tuesday night (I used to play a little on my mage at 70). I was specced PvP arms, in my PvE gear. Zero resilience. I'd just remapped my action bars to put all the buttons I'd need for PvP in one small place, and made some macros. As it turned out, I was dreadful - with no muscle memory I needed to look before pressing a button (otherwise I ended up spamming rend...) which is fatal in PvP, and we lost nine of our first ten matches. One match was against a priest-warrior team from my server, which ended particularly badly. I then started receiving charmingly aggressive and insulting whispers from the warrior (who was, of course, a gnome). I reported him and introduced him to my ignore list, but that's not the point - it's this sort of behaviour which gives PvP its current awful reputation, and in my view it is part of the reason that arena participation is in decline.
I used to do old-school PvP on my mage at level 60 (before resilience, before arena and before the honour point system) which was a very different experience. I used to love the three hour AVs, and really regret the fact that the battleground and world PvP system has been overtaken by the arena.