I believe the writer... should always be the final judge. I have always held to that position and have sometimes seen books hurt thereby, but at least as often helped. The book belongs to the author.
Maxwell Evarts Perkins (2004). “The Sons of Maxwell Perkins: Letters of F. Scott Fitzgerald, Ernest Hemingway, Thomas Wolfe, and Their Editor”, p.251, Univ of South Carolina Press