I have worked with fibre-glass in boat building. It is a grim material, with unpleasant properties, and on the whole, I would opt to weld rather than use it (and my welding is really poor). I think that your efforts are to be commended, firstly for having attempted it, and secondly for having a reasonable result.
In order to get the elusive fat finish that you are probably looking for, I would work from below, cutting a single piece of woven-roven to size and stippling it in-place with resin (dont overdo the catalyst though. I would then support this with a piece of ply, painted in release agent.
Once this has dried you can work from inside the car, cutting and applying 2 layers, the first should be a thicker layer of matting, cut to fill the hole, and the second a thin (gossamer) layer with a half inch overlap around the hole. This should be stippled with resin and will give a smoother surface than the woven roven (and it's dirt cheap). If you want a better finish, paint with a filler paint and then falt b ack and spray.
This is a much more arduous repair than just chopping out the floor and having a friend (but not me) weld a relacement panel back in.
Cheers
