I am watching such xvids regularly, with buffering no more than 5 secs. So everything looks OK from this side too :-)
Another idea, where are the videos stored? I am using an external WD MyBook, and if it has not been accessed for a longer time I believe it goes into sleep mode, and after you initiate the movie it takes a few more seconds for it to go into regular working mode.
If you feel like experimenting, you can also try to connect the PC and the TV directly with a crossover cable, no router, no switch. Set up fixed IP addresses and try streaming through DLNA.