"Instead of saying "I don't have time" try saying "it's not a priority," and see how that feels. Often, that's a perfectly adequate explanation. I have time to iron my sheets, I just don't want to. But other things are harder.
ALWAYS be confident in your decisions. Live life for you. If someone doesn't like it i guess they don't have your best interest in mind. only you can make the decisions that will change your life into what you want it to be.