This is what broadly happens when you come up against a boss that challenges your execution, rather than being a gear check. I am thinking of Rotface or Professor P here, where the raid is required to learn new tactics.
Stage one - Utter fail
At this stage you don't really understand the mechanics of the fight. There are uncontrolled adds everywhere, people are standing in the slime, no-one really knows what to do. This stage is scary because the fight seems incomprehensible to you. You may have read up on the strategy or watched tankspot videos, but nothing can really prepare you for the first couple of tries.
Stage two - Regular failure
You now understand most of the boss mechanics and are trying different strategies to help minimise their effect. You still have regular failures of execution, tank swaps being missed for example. You are wiping because you haven't hit on the optimum strategy and/or because your raid team can't reliably execute the fight mechanics.
Stage three - 40% wipes
At this stage you have a settled strategy but it still isn't working. This stage can be quite depressing, because people are still messing up - not that often, but often enough to mean that you are wiping. The raid leader is starting to get annoyed with people for screwing up.
The key to turning this stage into a kill is usually improving dps. Where dps classes have to run around, kite mobs, switch targets etc then they lose dps time. Low dps means that the fight goes on longer, which means more opportunities for people to mess up.
What dpsers have to learn is to execute the fight mechanics properly, while at the same time maintaining high dps. Once they do this you will progess to the last stage (or even bypass it).
It is possible to get past this stage with flawless execution and poor dps, but that strategy is very sensitive to people messing up. Long fights mean more chances for things to go wrong and generally once things go wrong they tend to get out of control very fast indeed. Improving dps is usually better than improving execution at this stage.
Stage four - 10% wipes
Failures at this stage are often due to "tunnel vision" - people focussing on the kill and their execution of fight mechanics. Tell your team to stay calm, concentrate on the mechanics, and keep doing what they are supposed to be doing. The kill will come.