Should i sacrificed myself and forget about sex or should i kick him out?

I met him in college, we move in after a months of dating. We were in love till i got pregnant only after 4 months of being together. He was jerk and selfish, he would say things like he don't want me or baby...he wouldn't have sex with me ..only ones in 3 months. I am very cutie so it wasn't that. I bought a condo so he move in with me. The day after i had my son he told me that no one likes me and i am a bitc! and that is why i don't have friends...i do have 3 best friends...So..he never helped with the baby and never gave me a day off...Later i had enough so i packed him and didn't let him in. He came back next day and said he love us and that he will change. He said that he has been taking vicodin for 5 years like 1 a day to help him go true work day faster...plus he is a legal pothead lol i don't smoke pot. well know he has been clean for 5 months but a doctor said he has anxiety disorder so he gave him zanax lol...He became a good father but shitty bf..he is not in to sex at all. Should i sacrificed myself and forget about sex or should i kick him out?