When I modded my N3dsxl, I figured I better put in a 128g card. It took forever to boot up and load stuff. I swapped it for a 64g card and it's much faster now. So I guess bigger isn't always better.
It's due in part to the size and speed of your SD card, and how much storage is being used. No matter what, a bigger SD card with more storage will cause the system to slow down while powering on and off.