My boyfriend sex teases?

My boyfriend and I will be lying in bed, or couch and he will grab my breast really hard. I have told him to not do it so hard, but he does it again. I am confused because when he first started doing this I was ouch please softer, and he just kept doing it. After grabbing my breasts I think he will take it further, but he never does. It is really frustrating because he never initiates sex. I am wondering if he is sexually attracted to me. He says he loves me, and shows me, but this sex issue is making me feel un desirable. Should I be conserned. Oh and he is a total pot head, but I have been with a guy that was a pothead and he would initiate sex with me.