Honestly, it's the luck of the draw. If you are comfortable with the actor that you're opposite of - it just breaks down a lot of those insecurities and you can just say, "Okay, I trust this person, and I respect them and know they respect me," and then you can just go with it. When that doesn't exist, it's a lot harder to let go.
