Fill a heavy-bottomed pot with 2 inches of vegetable oil and heat to 350 degrees F.
Set up three bowls. Put the flour and cayenne in one, the beaten egg in another, and the panko in the last. Roll each olive in the flour to coat completely, shaking off any excess; then dip in the egg to coat; and, finally, roll in the panko. Carefully add the olives, about 10 or so at a time, to the oil. Fry the olives until golden brown, about 3 minutes. Transfer to a paper-towel-lined plate to absorb any excess oil, and season with salt as they come out of the fryer. Repeat until all of the olives are fried. Serve warm.