and indeed, the stepping time should be enough to read it from sd card.
OK, if an SD-card in SPI mode can serve random data in under 5ms (shortest head settle time I've seen on any drive), this may work. We should maybe cross-check with the machine-specific times, as I know that Amiga documentation also gives recommendations for step and head settle times that are *very* conservative. My expectation would be that most programmers go with conservative timings, because they want their software to work on most (if not all) machines out there. So the 5ms timing may even be stretched.
A single cylinder (both sides) would be 28k if you agree with my 445-snippets-per-track suggestion. This would exceed the available Blockram of the Chameleon FPGA, so we're back in SD-Ram.