Mine is doing the same. If I hold the key on, then it will finally start. Normally, when the engine is cold, it works everytime, but gets worse if it's hot.
My starter is under warranty, so i was just going to change it out again. My old starter never did this, but this newer one started doing it after I changed the tranny out the last time. Maybe the stud terminals are loose or the wiring that attaches to them.