I run 125ft of 5/16" steel. Rope is cool but not tough enough for my liking.
Plus I occasionally run a Zipline on my cable.
I run the Warn 9.5xp 6hp motor. It's wicked strong and fast. I've also ran the stock 2.5hp, and 4.0hp. I think the standard series winch motor is 4.7hp these days. I'd stick with the Warn brand. Pay attention if your motor uses a keyway or splines on the gear. If you order a new motor, it'll likely be splines, and if your winch is old, it may be a keyway. Worst case, you may have to order a new gear for the motor...