Preheat the oven to 350 degrees F. Rub a 9-inch cake pan with 1 tablespoon olive oil. (This cake can also be made in a 10-inch cast-iron skillet.)
Add 1/4 cup olive oil to the cake pan and place over medium heat. Once the oil is warm, sprinkle in the brown sugar and stir. Remove from the heat. Add the apricots in a spiral pattern on top of the brown sugar. Dot the pan with the blackberries. Chop any remaining apricots (only if you have leftovers that don't fit) and reserve to add to the cake batter.
Whisk together the flour, cornmeal, baking powder, salt and lemon zest.
Add the eggs, granulated sugar and the remaining 3/4 cup olive oil to the bowl of a stand mixer outfitted with the whisk attachment and beat on medium speed until smooth, about 4 minutes. Add the vanilla and milk and beat until smooth. Add the dry ingredients and beat on low speed until combined. Stir in any remaining chopped apricots. Pour the batter into the prepared pan, place on a sheet tray, and bake until golden brown and a toothpick inserted into the center of the cake comes out clean, about 35 minutes. Let cool for 20 minutes before turning out onto a cake plate. Slice and serve.