Sounds a lot like my life, actually :p (except with less anime and wanking) I have a lot of acquaintances, but I'd say I have one actual friend left, not because I'm an enormous loser (well, maybe that...) but because of the reasons you've mentioned. He was one of my two best friends growing up, and we still do stuff together maybe once or twice a month. We try to make time, even though work is busy and he has a wife and four kids.
I've never felt lonely in the sense of being friendless, as I can always find things to do, but I do find myself wishing I could find a nice girl and settle down, but at my age (33) they're all taken. Unless I want to find a girl just out of high school or some old widow, there's nobody left that isn't hooked up. It also doesn't help that the city I live in has a lot of men who come here for work in the oilfield, and consequently the male/female ratio is out of whack, as well as the fact that they have more money than I do to impress girls with.
So yeah, you aren't alone, but I don't know what the solution is.