Without the often self-overblown hysteria over the "new" X-Men, people would remember The Doom Patrol was the better "misfit" superhero team?

Then maybe Incredible Hulk 181 would be affordable in high grade....Wolverine's debut would be as popular as Wendigo.

How many of the characters would be members of the Avengers or Defenders during this time?

By the 80s, the X-Men and Mutants were the premiere books at Marvel. Without the new X-Men, the Bronze Age would have been completely different, with super heroes perhaps not surviving. It was the Xbooks that lead DC to revamp their titles.
I doubt it would just be some other title that did what the X-Men did.

Here are my predictions by character:

-Jean would have split up with Scott because he wasn't cool enough to keep their title popular. She joins Beast in the Avengers, the Phoenix and Dark Phoenix storylines still happen but in that title.
-Angel is a Defender.
-Iceman sometimes teams up with Spider-Man and a new lady called Firestar.
-Scott goes off and wallows.

The difficulty of a hypothetical is figuring out what's the sliding door moment that would allow the counterfactual to exist. There's cause and effect; it's not just like flipping a switch. I've been browsing through The X-Men Companion (a couple of books of interviews of creators published in the early 80s) as something informing my views.

One of the ingredients for the creation of the All-New, All-Different X-Men was that the series had been revived as a reprint series for several years and there were also several guest appearances by X-Men during the reprint era. Both were apparently received well enough to create interest in a revival, so does the history get massaged enough for the just a 60s relic scenario? "Just a relic" is an unusual assessment, because the X-Men were an existing thing in the early 70s, albeit in the reduced reprint and guest appearance capacity.

Also, Wolverine's creation according to Roy Thomas may partially have created because his pitch for an international team of X-Men was out there, so creating new characters who could might be used in that new series was encouraged.

So if relic means that X-Men never even had a period of reprints, than it would probably mean the team/characters would go the guest appearance route, but those appearances would be less frequent as there would be less interest among readers. If all past events happen and it's just at the end they don't decide to go with the pitch of a new series, then I think the X-Men continue to make guest appearances. They either get revived at a later date with the original members (but probably not Beast), Havok, and Polaris as potential members, with the possibility that Sunfire and/or Banshee get folder in. Or if were not allowed a revival of the series under the thought exercise, then after a while some of the characters may end up as Defenders or Avengers.

Also, keep in mind that the new X-Men weren't created out of whole cloth when the pitch started. A lot of it came from sketches that Dave Cockrum had been working on potentially to use for Legion of Super-Heroes or a LOSH spin-off. While Storm was a combination of elements from different characters, the visual for Nightcrawler was largely unchanged from work Cockrum did before he was assigned to X-Men. So at some point we'd probably get characters that had elements of those sketches in whatever series Cockrum is working on instead of X-Men.

I can see them giving the characters solo stories in anthology books or mini-series to "test" if they could carry a series on their own.