The N.C.A.A. is investigating the University of Tennessee’s football program for recruiting violations involving a group of outside donors, signaling an escalation of efforts to rein in the growing influence of money flooding into college sports, according to documents and people familiar with the case.

The investigation is focused on Tennessee’s high-profile donor collective, a group of alumni and wealthy boosters who support the team by channeling payments and other benefits to players. The inquiry is looking at, among other things, the group’s role in flying a high-profile recruit to campus on a private jet while the football team was wooing him, one person familiar with the case said.

Having the booster group pay for the trip by the recruit, Nico Iamaleava, now Tennessee’s starting quarterback, would be a violation of N.C.A.A. rules. The inquiry comes after the N.C.A.A. penalized Tennessee for earlier recruiting violations and signals the organization’s growing concern about the huge sums being injected into the nominally amateur world of college sports by donor collectives.

The case could have profound implications for the direction of high-profile programs across the country, especially in football, where outside money raised and disbursed to players by collectives has reshaped the economics of the game. News of the investigation into Tennessee’s athletic program was first reported by Sports Illustrated.

Officials at Tennessee are concerned that the investigation could result in a devastating blow to its football program, according to a person briefed on the matter. The program is already on probation for the earlier recruiting violations, and school officials are worried about the potential for the N.C.A.A. to take drastic action, like banning the team from postseason play and disqualifying players.

Facing that possibility, the school has hired several law firms and is considering a range of legal options to stave off any consequences.

At the heart of the investigation are donor collectives, which are organized groups of alumni and other boosters who donate money to support teams. They have become a major force in college sports over the past several years by exploiting a new system set up to allow players to benefit from endorsements, known as name-image-likeness deals, or N.I.L.

Collectives increasingly arrange for athletes to be paid sums that rival what professionals make. Mr. Iamaleava, Tennessee’s quarterback, has a deal with the school’s collective that may be worth $8 million. After playing a limited role for most of this past season, he became the team’s starter in the Citrus Bowl on New Year’s Day, leading Tennessee to a 35-0 victory over Iowa.

The precise violations being pursued by the N.C.A.A. in the Tennessee case, and the penalties it might seek, remain unclear. As of Tuesday, Tennessee had yet to receive a formal notice of allegations, in which the findings of the inquiry would be laid out.

On Monday, after N.C.A.A. enforcement officers met with Tennessee officials to discuss the investigation, the university’s chancellor, Donde Plowman, fired off a letter to the N.C.A.A.’s president, Charlie Baker, criticizing the allegations as “factually untrue and procedurally flawed.”

In the letter, obtained through a public records request, Ms. Plowman assailed the N.C.A.A. for “two and a half years of vague and contradictory” memos about how schools should deal with N.I.L. issues and donor collectives. She said the N.C.A.A. had “created extraordinary chaos that student-athletes and institutions are struggling to navigate.”

At many Division I schools, collectives, while technically not affiliated with the universities they support, have become closely integrated into recruitment of high school students. And in an era when athletes can easily transfer from one school to another in search of better opportunities, they have become entwined in providing lucrative deals to retain star players.

The New York Times has counted at least 140 collectives operating at schools with big-time football and basketball programs. Collectives now account for about 80 percent of all name, image and likeness payments to athletes, far more than all the commercial brands that the system was devised for.

The N.C.A.A. has set rules for these groups, including prohibiting them from explicitly offering cash to entice recruits, saying any deals can be struck only after an athlete commits to a school.

But the N.C.A.A. has also been hobbled by court losses, eroding its power to regulate collectives. Until recently, there was little evidence that it was policing them at all.

As a result, top-tier college sports programs, especially in football and basketball, have become a nearly unfettered marketplace, with coaches openly exhorting alumni and other backers to keep them competitive by donating money.

Some schools have become increasingly emboldened, enlisting their state’s lawmakers to fight back against the N.C.A.A. when it does try to lay down rules.

The latest example came in December, when attorneys general in seven states — including Tennessee — filed an antitrust lawsuit against the N.C.A.A., calling any eligibility restrictions on transfers a restraint of trade. The suit was joined this month by the Justice Department.

Amid the barrage of litigation, Mr. Baker of the N.C.A.A. has asked Congress for an antitrust exemption. He testified on Capitol Hill that these lawsuits — along with recently enacted state laws that target N.I.L. rules — made it all but impossible for the organization to govern its members.

In examining Tennessee’s football program, the N.C.A.A. is investigating a team backed by one of the country’s richest and most outspoken collectives, a booster-funded group called the Volunteer Club. That group is closely linked to a marketing agency called Spyre Sports Group: The two entities share the same top officers and the same address in Knoxville, Tenn.

Last year, the website, which tracks collectives, called the Volunteer Club the “leading collective in the country” after the group said it had raised $13.5 million for Tennessee athletes.

The biggest prize was Mr. Iamaleava, a 6-foot-6 quarterback from Long Beach, Calif., who had been a highly ranked recruit nationally in his high school class.

“The nice word that’s used is ‘collective.’ But make no mistake: This is a war chest,” Hunter Baddour, a top officer of both Spyre Sports and the Volunteer Club, said on a podcast in 2022. “We are out fund-raising, creating an N.I.L. war chest, where Tennessee is going to be as competitive as anybody in the country.”

As its collective grew, Tennessee improved on the field. After a long dismal period, the Volunteers posted a 9-4 record this past year, and the team finished the season ranked in the top 20.

Mr. Baddour also organized a lobbying group for this new industry, the Collective Association, which has reportedly called for the N.C.A.A. to share some of its extensive television revenue with collectives.

Mr. Baddour and James Clawson, the other top officer at the Volunteer Club, did not respond to requests for comment on Tuesday.

The N.I.L. rules that took effect in 2021 allowed players to be paid for endorsements but continued to bar students from being paid to play. But the collectives that sprang up at many schools effectively found a way around that constraint.

They signed athletes to huge contracts for tiny amounts of work — sometimes as little as one social-media post a month — to keep them happy and playing at their chosen school. Some of the groups, though not the Volunteer Club, were set up as nonprofits that allowed donors to claim a tax deduction for their contributions.

Last July, the N.C.A.A. fined Tennessee $8 million and placed its athletic program on probation for five years, after finding “repeated and egregious violations” of the ban on coaches’ using cash to recruit players. Those violations took place before the N.I.L. system. Instead, coaches paid football players the old-fashioned way, in cash.

Since collectives popped up in late 2021, the N.C.A.A. has announced two cases where it punished schools because of name, image and likeness payments from boosters. Last year, it imposed mild penalties on the University of Miami after a booster posted photos of himself wooing potential transfer students for the women’s basketball team.

This month, however, the N.C.A.A. imposed more stringent penalties — including a fine and two years’ probation — against Florida State, after a football coach there drove a potential transfer student to a meeting with a collective representative. The representative then offered the player $15,000 a month to sign with Florida State, the N.C.A.A. said. The player declined the offer and stayed at his original school.

The N.C.A.A. is also investigating the University of Florida over its recruitment of Jaden Rashada, a quarterback who signed a deal with the now-defunct Gator Collective worth $13.85 million over four years. Mr. Rashada, who had originally committed to the University of Miami, eventually landed at Arizona State.