It's not that it's going to break anything as there is nothing locked together. It's more of a drag concern on the vehicle, with very little concern for wear of items since they are engaged (ie. bearings, seals, cv's, boots, etc.). Most of these items are not in motion constantly so wear will occur. Now to measure the wear is something else and have an accurate measurement forget about that!! LOL.
But a need to replace a cv boot, rather than the cv itself, could be argued due to the rotation of the shaft and constant stretch and compression during turning.
Think of it like why does the front tires wear when they are not driven (if driven constantly straight)? Why does the rear tires wear on a front wheel drive vehicle? Drag is a wear item.