My Bf's daughter is turning 1?

My Bf has a daughter from a previous relationship. Her bday is coming up in april. i have only met the baby's mom once, and it was kind of awkward at first, but then we all started to get along. But my problem is, the baby and babys mom are both living at the babys grandmas house (moms mom) and i would feel like an intruder if i came over for the bday thing, plus that would be really weird. i want my bf to go (duh) but when i tried telling him i wasnt comfortable going he got all hurt. her bday is in like 2 weeks and i just dont know what to do. anyone else been in my position. and how do i put my foot down and let him know that im serious about my feelings? (im 20, my bf is 19 and the babys mama is 18, me and my bf are living together, please no rude comments about the age thing i realize we are young!) thanks :)