For the pudding base: In a heavy-bottomed saucepan, bring milk to a gentle simmer over medium heat. In a separate bowl, combine egg yolks, sugar, brown sugar, cornstarch, and salt. Beat with an electric mixer until light and fluffy, about 2 minutes.
Stir a small amount of the warmed milk into the yolk mixture to temper the eggs. Whisk the yolk mixture back into the saucepan with the remaining milk. Reduce heat to medium-low and cook until the custard starts to thicken, 3 to 5 minutes. Stir in the bourbon and vanilla and continue cooking until glossy and quite thick, another 1 to 2 minutes. Remove from heat and whisk in the butter, stirring until it is completely melted.
Push pudding through a fine mesh sieve into a large bowl. Press plastic wrap directly over the surface of the pudding and refrigerate for a minimum of four hours, or overnight.
To assemble: In a medium bowl, beat the heavy cream and vanilla with an electric mixer until stiff peaks form. Fold the whipped cream into the bourbon pudding until no visible streaks remain. In individual serving glasses, layer pudding, gingersnaps, and bananas, finishing with a layer of pudding on top. Chill for 4 hours (and no longer than eight) before serving. Crush remaining gingersnaps and sprinkle crumbs over the top.
Note: While most recipes for banana pudding call for a big trifle dish, I thought these would be best piled into my favorite cocktail glasses; the individual servings are perfect for your next adults-only dinner party, no I.D. required.