Do you have a body lift?
Mine was popping out of 4L due to the rubber boot acting as a "spring" of sorts. When i'd push the shifter up into 4L, the boot would be highly compressed on that end, and want to go back into the neutral position. You have to either change the shifter (bend or weld an extension), or flip the boot opposite to how it sits normally, then use a stock shifter.
You might have wear of the case itself, but it's not so common in the kicks as it is in the samurai's.