Well the stuff you are referring to as coming from the imagination is the main point of the debate. We don't care if some guy said, "Good day, sir," or that someone drowned because they couldn't breathe underwater. When I say you're saying religious texts are false, I'm talking about the important stuff that applies to this debate.
Your assumption may be true, but you can't claim it as absolutely true. That would be like saying, "There is no evidence of any gods, so none exist." Instead, it should be, "There is no evidence of any gods, so it is possible none exist, so I choose to believe none exist." It is possible that no one has had any contact with any gods, but it is also possible that they have.