Here's the deal:
You cannot tell if the crank accessory pulley ihas come loose from the tcrank timing belt pulley
OR if the keyway is becoming wallowed until you take it apart!
From what you have written, it is too early to tell if the keyway is wallowed as you have not mentioned the sudden loss of power from a keyway wallow (the timing goes out because the camshaft is no longer precisely in time with the crankshaft) THIS IS A GOOD SIGN!

Here is what you need to do:
Download the Samurai FSM at Ack's FAQ (Reprehensible self-promotion!). Use the link in my signature to go there and do a search for
Samurai fsm. The download page should help you download the part that has the engine in it - so you don't have to download the whole thing if don't want to.
Study the timing belt parts of the engine section to get an idea of what things look like and how they go together.
To get at the crank accessory pulley for disassembly, remove the four 8mm nuts that hold the fan in place and carefully wiggle it out of it's home. Be sure to hod your mouth right!

Remove the radiator shroud's four 10mm bolts and lift it out.
At this point you should have enough room to get to the four bolts that attach the crankshaft accessory pulley to the crankshaft timing belt pulley beneath it. Tighten those bolts to remove the wiggle that the pulley
may have. If this fixes the problem, you are
home free!
If it does not, you may have a bit of wallowing but it is good that you have caught it early!
To fix that, you'll need to remove the timing cover (removing the accessory pulley is the first
and correct step!) followed by turning the crankshaft with a 17mm socket until you can actually see the key in the crankshaft that holds the timing pulley in place. Keep turning the crankshaft until it is pointing straight up. If there is extensive damage, you'll see it.
Here is where it gets touchy...
In order to remove the timing belt pulley, you must
1. keep the engine from turning. I remove the dust cover over the bottom of the transmission bellhousing (this exposes the flywheel) and clamp vice-grips to the flywheel on the
driver's edge of the opening. This will keep the engine from turning as you remove the 17mm crank bolt.
2. to remove the 17mm crank bolt, apply heat to the end of the bolt with a propane tourch like you would use to solder copper pipe together - you want to get the bolt warm, not melt it. This will cause the locking goo installed at the factory to break down making the removal of the bolt a lot easier. Use a 1/2' rathchet with a pipe extension on the end to break the bolt loose. I do not reccomend using an air tool.
3. Carefully remove the bolt then remove the camshaft pulley and finally the key. If the key or pulley are damaged, you may need replacementsif the keyway is damaged, you can try cleaning it up a bit with a file, remove all grease/contaminents applying JB weld then reassembling everything using the manufacturer's torque specificatons. A fresh 17mm bolt and some locktitie is recommended.
AT NO TIME SHOULD YOU ROTATE EITHER THE CRANKSHAFT OR THE CAMSHAFT DURING THIS PROCESS!
[
EDIT]
If you do, you'll have to go through the procedure for re-syncing all the timing.
Also, keep an eye on mechanical fuel pump. It's located on the intake side of the engine nat the firewall end. The tube that has no hose on it should not have any "wetness" on the end of it. If it does, you have a leaky pump!