Although I am not religious, I don't see a building as a church. A group of people gathered together worshiping the lord is what is a church. I believe that's actually stated somewhere, be it definition or in the bible somewhere. I don't remember where I saw that. So, going to a building called a 'Church' or not doesn't matter, in my mind, but gathering with people to worship the lord is what matters.
But going or not going doesn't make you this religion or that religion. I say it's more about just what you believe in.