So, sugar itself doesn't cause cavities, ACID does. The bacteria in your mouth basically eats the sugar... excretes acid... and THAT'S what breaks down your enamel.
That's not to say candy is GOOD for your teeth. It's not. But according to a dentist, there are five foods that are as bad for your teeth as candy, or worse...
1. Crackers and bread. Things like Cheez-Its, Goldfish, and Saltines are made of simple starch, which turns to sugar really fast. And they get stuck in your teeth, so the bacteria has even more time to turn that sugar into acid.
2. Dried fruit. It's as bad as candy because it basically IS candy. And it's stickier than most candy is, so it stays on your teeth longer.
3. Grapefruit. There's sugar in it, and it's naturally very acidic. Lemons, limes, cranberries, and grapes are really acidic too. Oranges actually aren't as bad.
4. Coffee. Again, there's a lot of acid in there. Plus it can stain your teeth.
5. Diet soda. It's worse than regular soda, because they add phosphoric acid to make it tangy. So even without sugar, there's more acid to rip up your enamel.
From Business Insider - A dentist reveals 8 foods and drinks that are worse for your teeth than candy