Make an ice bath by setting a medium bowl or large measuring cup into a large bowl of ice.
Strain the milk mixture through a fine mesh sieve into a clean, small saucepan; discard the solids. Bring to a simmer over medium heat. In a medium bowl, whisk the egg yolks and sugar until very pale and thick. Very gradually whisk the hot milk mixture into the eggs. When full incorporated, transfer back to the saucepan. Reduce the heat to low and add the brandy and orange zest. Switch to a wooden spoon and cook, stirring, until the mixture coats the back of the spoon, about 5 minutes.
Strain into the bowl set in the ice bath. Add water to the ice to cool the sauce down faster. Stir the sauce until cool. Cover and refrigerate until cold, at least 1 hour.
For the beignets: Mash the bananas in a large bowl using a potato masher or large fork. Add the banana liqueur, vanilla extract, granulated sugar and muscovado sugar, and mix well. In another bowl, mix the flour, baking powder, cinnamon and nutmeg. Add the banana mixture to the flour mixture and pour in the milk. Mix until smooth and thick, like waffle batter, adding a little more milk as needed. Set aside to rest for at least 10 minutes or up to 1 hour.
In a heavy pot set over medium heat, heat 2 to 3 inches of vegetable oil to 350 degrees F. Drop the batter by heaping teaspoonfuls into the oil and fry until golden brown on both sides, turning once when the edges are firm. Drain on paper towels. Serve immediately, or keep warm in a 200-degree oven for up to 2 hours.