Making it affect both players would turn it into a rather unused card. Sure, it would still help slightly, but unless you really have an advantage, it could actually hurt you some. Such as if the opponent has lots of damage out.
My immediate instinct is to say this card is OP, but the more I think about it the less I think that. Like Daxx said, rushing and stalling with this isn't all that easy to pull off. As you chain silences together, you aren't really drawing other cards. You are drawing those silences.
The best use of this card would probably to hold 2-3 of these in your hand, and then chain them together when you've got several other cards in your hand to make the advantage yours.
All In all, Silence seems like more of a sundail type of card, just stalling for a turn, or a one trick pony, waiting until you have the cards you need to create your bigger set up and then use a few in a row.
The card is powerful, but I don't see it being used too much after maybe a month or two. It will be similar to Nightmare, used a lot at first and then slowly get weened out and not used as much.