Line a small loaf pan or other mold with plastic wrap, leaving some plastic wrap hanging over the edges.
Combine jalapeno, whole milk and 1/2 cup of the sugar in a saucepan and heat until steaming, stirring often. Remove milk from heat and let cool to room temperature.
Peel and grate ginger and squeeze the gratings to release ginger juice. Stir into milk mixture.
Combine heavy cream, remaining sugar and vanilla extract and whip to soft peaks. Fold into milk mixture.
Transfer to prepared pan and freeze until set.
Remove semifreddo by pulling evenly on opposite ends of the plastic wrap lining.