Preheat oven to 400 degrees.
For the filling: Roast the sweet potatoes on an aluminum foil-lined baking sheet for approximately one hour, or until tender. Cut the sweet potatoes in half and cool until able to touch. Peel and mash the sweet potatoes with a fork or potato masher. (This should yield approximately 4 cups.)
In a large bowl, combine the mashed sweet potatoes, sugar, milk, butter, eggs, vanilla, cinnamon, and nutmeg. Beat with an electric mixer until smooth. Spoon the sweet potatoes into a lightly greased 1-1/2 to 2-quart casserole dish.
For the streusel topping: In a medium bowl combine the brown sugar and flour. Using a pastry blender or your fingertips, cut the butter into the brown sugar mixture until it resembles coarse crumbs. Stir in the walnuts. Sprinkle the streusel over the sweet potato mixture. Bake the casserole until the streusel is golden brown and bubbling, 35 to 40 minutes.
The sweet potato casserole can be assembled the day before and kept in the refrigerator until ready to cook.