Preheat the oven to 350 degrees F.
Lightly grease 2 (9-inch) cake pans. Cut 2 (9-inch) parchment paper rounds and line the pan bottoms. Grease and flour the parchment rounds.
In a large mixing bowl with an electric mixer, cream the brown sugar and butter. With the mixer running, add the oil in a steady stream. Add the egg yolks, 1 at a time, beating well after each addition. Sift the flour, baking soda, baking powder, spices and salt into a medium-size mixing bowl. Alternately add the flour mixture and the buttermilk to the batter, mixing well. With the electric mixer, in another large mixing bowl, beat the egg whites until stiff peaks form, then fold them into the cake batter. Pour the batter evenly into the prepared pans. Bake until the center springs back when touched, about 25 minutes. Remove from the oven and cool on wire racks. After the cakes have cooled, invert them onto sheets of parchment paper. Slice each cake in half and set aside.
Spread a layer of the frosting over 3 layers of the cake. Place the layers of cake on top of each other and top with the fourth layer of cake. Frost the top and sides of the cake with the remaining frosting. Slice the cake into individual servings.
In a large mixing bowl, with an electric mixer, cream the cheese and butter. Add the sugar, 1/2 cup at a time, mixing after each addition. Add the vanilla and milk, if needed to thin consistency, and mix well. Fold in the walnut pieces.