If we think of the field as being removed, there is no 'space' which remains, since space does not have an independent existence.