We should hold abusers — and no one else — responsible for the damage they inflict.