Prominent pastor notified books loaned to African American museum may be returned amid review