Preheat oven to 375 degrees F.
In a large mixing bowl, place flour, sugar and walnuts. Add 1/2 cup cold milk and slowly add rest of milk, stirring constantly to avoid lumps to form in batter. Grease pan with 1 tablespoon oil and pour in batter. Drizzle top with remaining oil and distribute rosemary evenly over top. Bake for 45 minutes, or until a toothpick pierced in the center comes out clean.
While castagnaccio is cooking, place grapes and sugar in a small saucepan over medium heat. Cook until quite soft and bubbly, about 10 minutes.
Remove castagnaccio and serve either warm or cool with grape sauce.