The only thing to worry about is the damage that can be done by worrying. Why are some scientists worried? Perhaps because they feel that to stop worrying may mean to stop being paid.
I feel we're on pretty solid ground in interpreting orbit around the sun as the primary driving force behind ice-age glaciation. The relationship is just too clear and consistent to allow reasonable doubt.
What I think is this: Man is responsible for a PART of global warming. MOST of it is still natural.