Should I keep my boyfriend?

It sounds like you should dump him. He won't come and see you and that's a telltale sign of lack of interest. He also is taking you for granted. He knows you will come and see him because he thinks you "love him." With everything you said and the grabbing part, he sounds immature and not someone to build a relationship with.