There are two problems with the VGA output:
- The pixelclock is currently fixed, that means unless you are using the default 800x600x72 mode, the pixelclock will not match the exact requirements of the VESA modes, and thus the monitor might get confused about it and not sample it correctly.
- The sync polarity is wrong in some modes.
- Unless your monitor has a native 800x600 resolution, it will do scaling and everything bad connected to it. Depending on how picky it is on clock detection and sync phase/polarity, it will even do more bad things.
That said, Peter is currently working on the VGA output part of the core, essentially rewriting everything, so we have a async clock now, perfect matching pixelclocks and syncs, and essentially much better compatibility (hopefully). So chances are the next update will fix most (hopefully all) of those related problems.