Heat the half-and-half, pumpkin, brown sugar, salt, cinnamon, and nutmeg in a heat-proof bowl set over a pan of simmering water until hot, about 5 minutes. Whisk the egg yolks in another bowl, stir some of the hot pumpkin into the egg yolks to heat them, then pour the egg-pumpkin mixture back into the double boiler and stir well. Heat the mixture over the simmering water for another 4 to 5 minutes, until it begins to thicken, stirring constantly. You don't want the eggs to scramble. Remove from the heat.
Dissolve the gelatin in 1/2 cup cold water. Add the dissolved gelatin, banana, and orange zest to the pumpkin mixture and mix well. Set aside to cool.
Whip the heavy cream in the bowl of an electric mixer fitted with the whisk attachment until soft peaks form. Add the granulated sugar and vanilla and continue to whisk until you have firm peaks. Fold half the whipped cream into the cooled pumpkin mixture.
Pour the mousse into a 7 by 4-inch souffle dish. Pipe or spoon the remaining whipped cream decoratively on the mousse and sprinkle, if desired, with orange zest. Serve chilled.