Off with you" is a phrase used by people who lack the curtesy to say something more polite, such as, "if there's nothing else you require I must be going" or "I'm sorry but I'm going to have to ask you to leave, please" or even "excuse me but I believe you have mistaken my home for your own and my valuable belongings for yours and I must ask you to return the items in question to me and leave my home after untying me from this chair, as I'm unable to do it myself, if it's not too much trouble.