Let's start running with no tanks. I mean no tanks, not even overgeared DPS who can tank as well as a tank in blues (or close enough).
Madness? Oh yes. Madness. Ha! Ah ha ha. Heh. *sigh*
Trash is simple enough: crowd control and kiting. Imagine a group with a druid healer, rogue, mage, hunter, and ret paladin. Roots on a melee is effectively a CC. Sap one, blind when that fades. Sheep of course. Careful trapping mixed with kiting can handle another. The ret can repentance another or use it as a supplement to the rogue's CC. That means a group of four mobs only has one 'free', which of course means the mage is going to be snaring, with the hunter using distracting shot to keep aggro, leaving the melee free to tear it apart. The group makeup is a bit idealized, but a warrior could take the paladin's place while a skilled warlock can substitute fear for sheep, and even has howl of terror as a last ditch time-buyer.
It would be slow. Mobs would have to be marked. DPS would be limited to single targets and distracted by their CC duties. But what if it was proportionally more fun? Or beyond! Imagine if trash wasn't just an annoyance before the boss, but something to actually challenge a group, something worth having there. "Equally interesting yet non-epic-dropping non-bosses" as Tigole referred to it.
Bosses, now those could be tricky. They generally cannot be snared and terrain rarely allows for uh... 'interesting tactics'. They're going to hit people, so the group needs a way to deal with that. The tank and spank model clearly won't work, so we need something else. Time to live still matters, since it won't do to have a boss just two-shot everyone, one by one. Fortunately DPS aren't perfectly helpless.
Warlocks have metamorphosis. Hunters have deterrence and possibly another second with disengage. Paladins have divine protection. Rogues have evasion. And so on.
Aggro management would be critical, to get mobs to attack the right player. Some classes can wipe their aggro, hand of salvation offers a slight reduction, and two classes can increase the aggro of other classes. Carefully balancing aggro would allow players to keep bosses focused on the player which can survive the attacks.
I doubt this would buy enough time for a group with blue DPS. All these abilities only last so long. While trash can be CCed to buy rest moments, bosses do not allow that luxury. So possibly most important is high DPS, to ensure that bosses aren't outliving the players' life-saving cooldowns.
As fun as it sounds, I doubt this would ever catch on. It wouldn't have to be refused, but it would naturally die. As we gear up, gain new talents, a higher tier of gems, new glyphs, and so on, we gain so much DPS that bosses may not have time to hurt anyone. I've seen this when soloing some raids, particularly in ZG, where at 70 some bosses seemed impossible, but at 80 my damage is so high that the challenging mechanics don't come into play. Beyond just higher damage, players would gain tanking stats, not intentionally, but as the natural course of their gear. Rogues is particular would gain significant amounts of agility, meaning more dodge and eventually a non-trivial amount of armor, in addition to the stamina increases, so that eventually they too are as good as a tank. Without meaning to, players would again trivialize the challenge. The only counter is standards which would not allow damage, avoidance, or health values beyond a certain point, but that creates arbitrary restrictions and those aren't always fun.