Place the potatoes in a saucepan, add water to cover and bring to a boil. Cook, partially covered, until the potatoes are fork-tender, about 10 minutes.
Drain the potatoes and transfer them to a salad bowl. While the potatoes are still warm, season them with the salt, cayenne, black pepper and lemon juice. Toss well to combine. Add the celery, green onions, parsley and eggs.
In a small bowl, stir the mayonnaise and mustard together. Add this to the salad. Sprinkle the bacon over the salad, and toss gently but thoroughly to mix well.
Serve immediately, or refrigerate, covered, and serve slightly chilled.