The horse responds to comfort, they respond to peace better than about anything else you could do. All he wants you to do is just leave him alone. So if the horse responds to you and you give him a little peace and comfort that means more to him than anything.
When riding, ask yourself what will my horse get out of it if I get what I want? Many times, human nature is to take and to not give anything back.
All your horse looks for is the thing that matters most, peace and contentment. Give it to him.
