WASHINGTON (CNN) — Top House Republicans are moving to quiet dissent over Speaker's Paul Ryan's rocky relationship with Donald Trump as some on the far right of the GOP conference are threatening to stir up trouble in leadership elections next month.

Regardless of what happens on Election Day, Ryan is in a difficult situation given the deep divisions in the Republican Party. He has become a target of Trump's grassroots supporters, and there is some concern among members that if Trump loses outside groups could try to put pressure on GOP offices to dump Ryan as speaker, especially if they don't like the outcome of the year-end spending bill Congress will take up.

And if, as most members told CNN, there is a smaller Republican majority — potentially 230 seats or less compared to the 247 they have now, the speaker's room to maneuver will shrink with fewer moderates in his party, boosting the clout of his biggest critics: members of the conservative House Freedom Caucus.

"Paul is in a tough spot," Rep. Peter King, R-New York, a Trump supporter, told CNN. King said he will vote for the speaker again and says he's the best person to lead the party on Capitol Hill after the internal warfare inside the GOP in recent months.

"I'm convinced the last thing we need to be doing is dividing the party especially when on any ballot the overwhelming majority will be for Paul Ryan," he said.

Ryan is sticking with his strategy to stay singularly focused on keeping the majority in the House and ignore Trump. After telling members earlier this month on a conference call that he would no longer defend the GOP nominee following the release of the "Access Hollywood" tapes showing Trump using graphic language about women, the speaker has kept a grueling political travel schedule and avoided press interviews. He continues to raise money and appear with candidates across the country.

On Sunday, Ryan will headline a rally in Billings, Montana, where GOP Rep. Ryan Zinke is facing a tougher than expected challenge from Denise Juneau. Democrats have not won a race for a House seat in the state in more than 20 years. Before that trip, the speaker has visits planned in the final days of October in eight cities, appearing at 14 campaign events over a three-day swing in California, where Trump's decline in the polls could pull down veteran House Republicans.

Related:

Ryan's break from Trump prompts talk of GOP rebellion

House Speaker Paul Ryan's abandonment of Donald Trump is aimed at protecting Republican control of the House. But it may test his hold on his own job, and his long-term ambitions.

Multiple House Republicans say if someone wants to challenge the speaker they are entitled to make their case, and they should use the use the current system to appeal to colleagues as to why they would be a better candidate. But some are frustrated that those who aren't happy with Ryan won't actually run but instead could derail him by casting protest votes in January that would deny him the 218 votes he needs to be sworn in again as speaker on the House floor.

Republicans close to Ryan are pushing back hard on the far-right of the conference, saying the party needs to come together behind the House speaker after a divisive and difficult election, arguing he has raised loads of cash to save his majority.

"If someone were to challenge him they would certainly get some support but wouldn't come anywhere close enough to win," Oklahoma GOP Rep. Tom Cole, a strong ally of Ryan's, told CNN, insisting the Wisconsin Republican is "the odds-on favorite as long as he runs again."

Ryan's spokeswoman AshLee Strong tells CNN, "He is running for speaker."

Ryan primary foe mounts quixotic bid for speaker

So far no House GOP member is floating their name as an alternate candidate, and multiple House Republicans tell CNN there aren't any moves behind the scenes to mount a challenge to Ryan. But that could change after the election results are in, and House GOP members are bracing for double-digit losses.

Few believe that Democrats will actually get the 30 seats they need to retake the House majority, and allies of Ryan argue even if the worst-case scenario happens and they lose the House, Ryan will still be in position to remain the top GOP leader.

But one man is putting his hat into the ring: Paul Nehlen, a Wisconsin businessman who lost resoundingly to Ryan in their August primary fight.

Nehlen told CNN Tuesday he plans to run against Ryan for speaker given that the Constitution allows people who are not elected representatives to serve as House speaker. (Non-members like Colin Powell have often received protest votes during speakership elections.)

"It's going to be nationalized," Nehlen said of his quixotic bid to become speaker, accusing Ryan of "backstabbing our nominee."

Nehlen said that Ryan has "damaged his brand," saying he would pressure the House Freedom Caucus in particular to revolt against the GOP speaker.

"We are going to make it uncomfortable for House members who say we have no confidence in Speaker Ryan" but ultimately decide to vote for him.

Nehlen, however, is the longest of long shots, and it's almost certain many House Republicans won't take his bid seriously.

But Ryan's supporters say if critics can't defeat him in a vote of House GOP members they abide by the results and not keep pressing to undermine someone who can get the votes of the majority of the conference.

"To me that's unacceptable, you are either a member of the conference or you're not," Cole told CNN, saying he wasn't always supportive of former Speaker John Boehner, but once he prevailed in a leadership race he backed him, and remained supportive of him when he ran for the top slot.

Rocky relationship with Trump

Cole downplayed any bad blood between Trump and Ryan, predicting if the billionaire wins the White House Ryan will "be Trump's best friend. Trump will sign just about anything he passes."

But even those who enthusiastically back Ryan to keep his gavel tell CNN he has mishandled his relationship with Trump. As the chair of the Republican Convention, they recognize that he had to stay out of the primary race. But once Trump became the nominee these GOP members say the speaker should have kept a disciplined message in support of the top of the ticket and avoided weighing in with opinions that kept up a perpetual public appearance that the two were constantly at odds. The other top congressional Republican leader, Senate Majority Leader Mitch McConnell, pronounced his support for the GOP nominee but has kept mum throughout the tumultuous weeks following the convention and the allegations from a string of women saying Trump sexually assaulted them.

Four House Republicans told CNN that the speaker's decision to hold a conference call after the release of the tapes from Access Hollywood showed Trump making vulgar remarks about women was a mistake. While they say members were reeling and fearing fallout in their districts, the speaker's announcement on that call that he wouldn't defend Trump — something these members say he wasn't doing anyway — became a flashpoint, and unnecessarily rattled the Trump campaign.

House Republicans, both supportive and critical of Ryan, agree that while Ryan could prevail and get the simple majority he needs to win another term in the GOP conference vote slated for November. But those opponents who don't want him to remain speaker could easily upend the process when the official vote happens on the House floor in January.

Some conservatives are finalizing a letter to House Republican leaders seeking a delay for the vote for speaker. These members say publicly they want to see how a final spending deal plays out and instead of voting in mid-November vote sometime in December. But those close to Ryan say this is more about critics throwing a wrench into the leadership election even if they don't have a candidate who can beat the speaker.

"We shouldn't be holding things as a blackmail," King told CNN, saying the vote for speaker should take place at the same time it always happens and even those who vote against Ryan in the secret ballot contest should be bound by the will of the broader conference which he confidently predicted would re-elect Ryan. "We can't allow it to become regular order that 30-40 people are going to decide who is speaker of the House."

As for those who will work to blame Ryan for any losses, Cole brushes that off as "amazingly disingenuous," arguing Ryan deserves credit for keeping things afloat and it's expected that the majority contract some because it's the largest one House Republicans have had in 88 years. Cole also pointed out that the Ryan broke spending records set by Boehner and has traveled to campaign for any House Republican who asked, including members of the conservative House Freedom Caucus.

"He developed a platform for us to run on, and he's given people something to talk about besides the presidential race," Cole said.

The-CNN-Wire ™ & © 2016 Cable News Network, Inc., a Time Warner Company. All rights reserved.

Related topics