Issues with my baby's dads mother? (Sorry kind of long)?

First off her father and I aren't together. We haven't been since a month after I found out I was pregnant. He wasn't very involved in the pregnancy and now that she is here he spends more time playing my brothers xbox than spending time with his daughter. The two times I stayed over to help take care of her he fell alseep and didn't help at all. He hasn't contributed anything nor has his family. My family and I have paid for everything. I realize he is her dad but he needs to start acting like it. His mother is a total witch and told me that SHE had the right to see her granddaughter whenever she felt like it and I had to accept the fact that I couldn't keep my daughter to myself...whcih I have never done. They are always welcome to come see her at my home. I just don't believe they have the right to expect me to drop everything and do what they say because they have a right. Am I wrong to think that maybe they should be more supportive and help out a little more before I just hand my daughter over to visit with them whenever they make time to see her? I'm not trying to be spiteful or immature but I think they can't call themselves her family until they start acting like it. Am I wrong in this?