It's all the animations that slows down PvP and the timer is misleading.
The moment you press space bar doesn't mean your opponent can play immediately. All the actions/animations go off first to show what you have done that turn (draw card, play card(s), use card abilities, etc.) in addition to all the other animations (damage, healing, pillar/mark), before your opponent can play, and this all happens while your opponent's timer is ticking.
So even though it shows that your opponent is taking over 20+ seconds a turn, a good 5-10 seconds are all animations that show what happened during your turn for your opponent.