I'm the same way, but I don't think it's distraction, for me. I simply haven't dedicated myself to anything. I mean, I have a career that I work on every day, obviously, but it's not the same it seems as it is for others, where the creativity or the result of their work comes out of them ALL THE TIME. I used to draw a lot more, I used to write a lot more, but I've never put myself full force into something that was just part of who I am. I'm a drummer, always have been, and I suppose for a time that was my life. But even that, while still a huge part of who I am and a skill I'm never not using, has moved into the background.
I don't have journals of written words. I don't have sketchbooks loaded with drawings the way my wife does. She draws all the time, seemingly never out of ideas of what to draw and seemingly capable of just doing things without getting too frustrated or giving up. She gets ideas for paintings and paints them. I get an idea and if I even start it, it never becomes what I see in my mind and most of the time I never finish it.
I wouldn't say I get distracted. I just lose interest, focus, and motivation.