My guess is that the events of 1204 were not fatal if the Empire of Nicaea, or arguably the Epirot regime led by the Angeli that had closer personal links to the old regime in Constantinople that had fallen, had regained the capital and the heartlands around it some time between 1230 and 1261 and not then faced both:
1. the evicted Latins (with their opportunist allies in Naples once Charles of Anjou, a much more dangerous opponent that his predecessor Manfred of Hohenstaufen as he had French resources and the Papacy behind him, had taken over in 1266). This threat then turned into annexation of the Latin state of Achaia in the Peloponnesse and then to invasion of Epirus, heading for Constantinople, in 1280-1 as a repeat of Robert Guiscard's ambitions of 1081-5 and Bohemund's of 1107-8 - just as the Turks were turning into a major menace to Bithynia. Also, the threat of Latin invasion backed by the Pope caused Michael VIII to force the pro-Latin reunion of the Churches on his reluctant Empire, infuriating nationalist and culturally / religiously autonomist opinion and breaking up the consensus among the elite - some rival Byz successor states from after 1204 like Thessaly then backed the anti-Unionists. Michael fought the Latins off but died before he could tackle the Turks, and though his son Andronicus then cancelled the Union and reunified public opinion the Empire had been too financially exhausted by the war to keep up its full armed forces and so keep its enemies at bay. No Latin invasion threat (as a result of Manfred beating Charles of Anjou?), no Reunion of Churches, no early death of Michael (aged 58, his son lived to 73), and Michael has the time and resources to conduct a holding operation in W Antatolia.
2. The Ottomans or any other expanding Turkish state in N or W Anatolia which was bringing the disparate small emirates together into one larger power with a potential for a navy. The naval-based emirate of Aydin further S, based on Smyrna, also had the potential to unite Asia Minor and then cross to Europe as it was dynamic and expanding, and had charismatic leaders; its 1330s ruler Unur was a Byzantine ally but this need not have lasted under his successors. If Aydin , of when it fell to the Western Crusaders sent in by the Pope in 1334 a Turkish revanchist successor, had turned on the Empire and taken Gallipoli it could have succeeded as the Ottomans did in OTL; the other main expansionist Turkish emirate, Karaman which had the old Saljuk capital of Konya so more legitimacy, was in SE Anatolia so if it had emerged as the leading reunifier of Anatolia it would probably not be interested in Europe and would turn on Syria as the Mongol Ilkhanate declined and then fight the Mamlukes over Syria.
There is also a third factor - Michael was a usurper to the throne of Nicaea in 1258, forcing his co-rule on his 10-year-old colleague John IV Vatatzes Lascaris and murdering the then regent George Muzalon - and later having John blinded and deposed.In effect this is a coup by the aristocrats at court against the court bureaucracy and palace elite of Nicaea, and it deeply divided the state and led to dissension and some abortive revolts after 1261 in the old Lascarid heartlands of W Anatolia - precisely where the Turks were to strike after c. 1280. A universally accepted Emperor taking back Constantinople, e.g. John IV's father the capable if paranoid warrior emperor Theodore II (died 1258 aged 36 of epileptic attack) or later an adult John IV, would not have faced this anger and constant fear of subversion. Also, if Theodore's father John III, a canny and financially astute man unusual for Byz emperors, had retaken Constantinople when he secured the last Latin lands in Anatolia and crossed to Thrace in 1235 , or soon after, the Empire would have had far longer to recover before the Ottomans emerged and probably no Western invasion - in the 1240s the Pope was too busy trying to depose Frederick II from Germany and Sicily.
After Michael VIII the Paleologi were mostly hopeless as rulers, except for the virtually powerless Manuel II, John VIII, and Constantine XI, and arguably the weak and infuriating Andronicus II started this chain of disasters by cutting the navy to save money in the 1280s and not forcing more taxes out of the nobles to pay for a stronger army - though doing this would have needed an entirely different and tougher personality who would be accused of dictatorial behaviour and face noble plots. The Catalan troops' mutiny and rampage then weakened the empire further, plus the 1320s civil war - both just before the Turks started to think of Europe as a target. But the vigorous Andronicus III pulled an army together and retook Macedonia and Epirus by 1341, making him the strongest power in the Balkans assuming he and Serbia did not have a head-on clash. I see the death of A. III in 1341, aged ?44, and the next civil war to 1347, followed by the Black Death, as the 'crunch' - so if Andronicus had lived to his 60s and passed on a secure throne to his (incompetent and divisive) son John V or John had been quickly deposed by the capable general John VI Cantacuzene the Empire would have had a chance as a second-ranking power - unless it got embroiled in a ruinous fight over Greece with Serbia. Instead, we get the minor successes by John VI in 1347-54 disrupted by another civil war started by John V, and then John winning but at the cost of losing Thrace (the last local source of men and taxes) to the Ottomans. I agree that the sheer greed and stupidity of the Byzantine leadership in the period 1341-91 is infuriating! But if Andronicus III or John VI had had long reigns... ?
i