Several Congressional Black Caucus members are trying to convince Rep. John Conyers (D-MI) to resign, according to a report Tuesday.

A few Democratic sources told CNN that these members are attempting to ensure that Conyers leaves Congress without tarnishing his legacy as the longest currently-serving member of the House.

The sources say that House Minority Leader Nancy Pelosi is not spearheading the effort, but is providing support for it behind-the-scenes.

“There is a feeling among some of our members that we need to protect his legacy,” one CBC staffer said, stressing Conyers’ status as one of the group’s founding members and civil rights movement leader.

Some of Conyers’ Democratic colleagues have publicly called for his resignation. Rep. Kathleen Rice (D-NY), Rep. Debbie Dingell (D-MI), and Rep. Gregory Meeks (D-NY) have all stated publicly that the Michigan Democrat should step aside.

The House Ethics Committee announced that it would investigate Conyers after a BuzzFeed News report revealed that Conyers’ congressional office paid a former female staffer $27,000 of taxpayer funds to settle a sexual harassment claim.

Conyers released a statement through his attorney, Arnold Reed, stating that he would not resign despite the allegations made against him.

Despite his reluctance to resign, the Michigan lawmaker seems to be scaling back on his committee assignments. Conyers announced Sunday that he would step down as ranking member of the House Judiciary Committee.