'People aren't dumb': Republicans worry they're not doing enough on affordability