Preheat oven to 375 degrees. In a large mixing bowl, place flour, sugar and walnuts. Add 1/2 cup cold milk and slowly add the rest of the milk, stirring constantly to avoid lumps forming in the batter. Grease a 9-inch, non-stick baking pan with 1 tablespoon oil and pour in the batter. Drizzle the top with the remaining oil and distribute 2 tablespoons rosemary evenly over the top. Bake for 45 minutes, or until a toothpick inserted in the center comes out clean.
While castagnaccio is cooking, combine the lardo, the remaining 1 tablespoon rosemary, prosciutto, garlic and freshly ground black pepper to taste. Stir well to combine and set aside. Remove castagnaccio and serve warm with a thin layer of lardo spread on top.