I would think that the spacer would provide an extra weak link, but I have nothing to back that up. I used spacers once on an old Cutlass I had but that is totally different from an offroad setting.
I think alot depends on the specifics....how big a spacer...a 1/4 inch is probably no big deal, but spacers, just like offset wheels put more stress on the bearings the farther you go.
I would say they should be ok unless you are getting into some hardcore wheel'n or rock crawling, but once again I am not sure....
Sorry for the lack of answer there....Joe