US congressman refuses to resign after allegations of affair with staffer