Preheat the oven to 375 degrees F.
In a large mixing bowl, place the flour, sugar and walnuts. Add 1/2 cup milk and slowly add the rest of the milk, stirring constantly to avoid forming lumps in the batter.
Grease the cake pan with 1 tablespoon of the oil and pour in the batter. Drizzle the top of the batter with the remaining olive oil and distribute the rosemary evenly over the top. Bake for 45 minutes, or until a toothpick pierced in the center comes out clean.
While the castagnaccio is cooking, in a small saucepan, combine the lemons, sugar, and 1 cup water over medium heat. Cook until quite soft and bubbly, about 20 minutes. Remove the castagnaccio and serve either warm or cool with lemon sauce.