The base of the pumpkin latte — espresso and milk, better known as a latte — is arguably not bad for you. Coffee in moderation has some health benefits. And milk has benefits, too, especially if you’re going for low-fat or nonfat (it delivers protein, calcium and some other vitamins).

Unfortunately, the thing that makes a pumpkin latte pumpkin-y is syrup. One variety of syrup that I looked at is made from condensed milk (pre-sweetened milk), two kinds of sugar (sugar and high-fructose corn syrup) and caramel coloring (which is linked to an increased cancer risk). A couple pumps of the stuff will add 106 calories and 24 grams of sugar (which is equivalent to 2 tablespoons of sugar!) to your latte.

It’s clear that the pumpkin latte is no health drink. But what really determines whether it takes a toll on your health and waistline is how big and how often your order it.

Size really does matter when it comes to sweetened beverages. Whether you choose skim or whole milk also makes a difference. A totally reasonable 8-ounce pumpkin latte with skim milk has 130 calories and 24 grams of sugar — some of which is from the natural sugars in milk. While a 12-ounce beverage usually clocks in at less than 200 calories (100 for milk, 100 for sugar), go up in size and you’ll start to see those numbers rise. A 16-ounce drink with whole milk has about 440 calories and 66 grams of sugar (that’s 5 tablespoons of sugar!). One more tip adds 60 calories, which might not seem like much, but it all adds up.

The second factor is frequency. Treat a pumpkin latte like dessert — a treat to have and enjoy once in a while — rather than make it your everyday coffee order (even if it’s just every day in the fall). Or you can take my route: Order your plain latte and celebrate pumpkin season with one of these healthy pumpkin recipes.

Kerri-Ann is a registered dietitian and nutrition coach who writes on food and health trends. Find more of her work at kerriannjennings.com or follow her on Twitter @kerriannrd or Facebook.