If I'm worried about being selfish when taking the time for myself, I sometimes try to think about it from a different perspective. If I don't focus on what I love for part of the time, I'll get cranky and tired. That will then affect how I interact with other people, which will change the dynamic of the interaction. 

Taking the time to be a little selfish sometimes means that when it's time to be selfless, I'm more likely to put my whole heart in, knowing that I have that balance in my life. 

It sounds a little like spinning the story, I know, but I do think there's some truth in it.