Switching the core to use a better audio DAC might help, too - the one it uses at the moment will be quite noisy in its current configuration.
I suggest to use at least 50MHz sample rate for the Delta-Sigma DAC, otherwise it'll sound weird. You probably have a 100MHz-ish clock domain for SD-Ram, so that may be a good one. The FPGA is definitely fast enough for Delta-Sigma at SD-Ram speed.