Help with my boyfriend?

well when we were first going out we kinda rushed things because we really liked each other.We went on a trip together and I said something about my ex like about with him and it kinda slipped out... I felt terrible like I ruined everything. He told me he was upset about me saying it and he said that he forgave me but he always keeps bring it up and saying why would you say something like that I don't understand. I was baling my eyes out saying how much I was so sorry and I didn't mean it and it wont happen again.But when he went on his trip and came back he said he didn't want to talk to me for 4 days and that he needed time to forgive me . He also said he was thinking about breaking up with me but he wanted to give me an other chance......There is more that when he went on that trip it was with a friend that was a girl and they went to go pick up her future husband Ive meet them both.He went to California with her on a road trip I have never felt so left out in a relationship before like what the hell why would you not bring me. I never said anything to him though I am really quiet and hold things in.He also said when he got back that he know believes in after marriage and that we had to much anyways. I don't mind the after marriage thing but to do that after you had with someone for 2 months. So after the not talking thing we went on one date and it as alright and stuff but it was like there was no affection or anything.If we don't have i still need some affection its like i have a friend that's a guy. He wont even kiss me till I get out of his truck.I feel affection starved. He was also mad at me before for not calling him as much he said that he talked to his friends more then he talked to me.He never gives me that look anymore, its like there's no pion of any kind. But we both like the same things and enjoy the same things. Its just I like it when I feel special to my boyfriend and like I mean a lot to him and I just don't feel it.He's the type of guy that says how it is and he told me if he didn't want to be with me he would have dumped me and he says he does care about me.He says he had a girlfriend for 3 years and never had sex with her and the one he had for 6 months and he said they were the best relationships when there was no sex.He says he really likes sex though he says its harder for him then it is for me.I don't know hes also always hanging out with his friends having fun and working a lot I see him like 2 a week sometimes 3