Childrens books change lives. Stories pour into the hearts of children and help make them what they become.
Just write. If you have to make a choice, if you say, 'Oh well, I'm going to put the writing away until my children are grown,' then you don't really want to be a writer. If you want to be a writer, you do your writing... If you don't do it, you probably don't want to be a writer, you just want to have written and be famous—which is very different.
A child who can love the oddities of a fantasy book cannot possibly be xenophobic as an adult. What is a different color, a different culture, a different tongue for a child who has already mastered Elvish, respected Puddleglums, or fallen under the spell of dark-skinned Ged?