Preheat the oven to 350 degrees F. Grease a 9-by-13-inch baking dish with cooking spray. Add the cubed bread to the baking dish.
Place the walnuts in a dry skillet over medium-low heat; cook and stir until the walnuts are golden brown and fragrant, about 2 minutes. Remove from the heat and set aside to cool.
In a large mixing bowl, whisk together the heavy cream, egg yolks, brown sugar, cinnamon, salt, vanilla and nutmeg. Once combined, stir in the mashed bananas and 3/4 cup of the walnuts.
Pour the custard over the bread cubes. Carefully toss to coat, making sure that the bread cubes are submerged in the custard, and let stand to allow the bread to soak up the custard, 10 to 15 minutes. Top with the remaining walnuts.
Bake until the center is set, about 45 minutes. Let cool for 5 minutes. Serve with vanilla ice cream.