Should i break up with him because hes a pothead?

i love him,but hes 25 smokes pot 24/7 and doesn't work. i feel like its not hes fault and i shouldn't judge him,i don't mind my friends smoking. and yet he does this for months sometimes he stops and works a bit then it all over again..im scared of him when hes really high because he just wants sex and makes jokes and gets angry if i say no. but then he never does any serious drugs and he can be very nice and sweet,but he ones was very disrespectful and almost physically violent or when high he laughed when i fell and hurt myself...thing like that