How compact can a neutron star get before collapsing into a black hole?