EDIT
It was the OTG adapter itself, see here: https://www.reddit.com/r/miniSNESmods/comments/7twsww/larger_neo_geo_games_taking_ages_to_load_up_via/dtprs88/

So to sums up:

• USB-HOST installed with hakchi-gui 1-5-18.
• Tried with Garou MotW (garou.zip, ~80MB), works perfectly fine on PC with FBA, works fine on SNESC but launches in 1 minutes and 20 seconds! No problem with smaller roms like Windjammers.
• Tried different FBA versions (2016 from KMFD and 2016 from Cluster).
• Tried different RA versions (1.6.7, 1.7.0 from KMFD and I think 1.4.1 from Cluster).
• Tried with 2 different USB drives (SanDisk Extreme 32GB and Transcend JetFlash 16GB).
• Tried with different power adapter.
• Tried with FAT32 and EXT4.
• Tried with the bios in /etc/libretro/system and inside the game zip.
• Tried .zip without any compression, and .7z.
• Tried the "dummy" method.

To avoid any conflict, I used the uninstall file every time.

And I can load a 2GB+ file (FFVIII eboot) in just a few seconds.

Here's a VIDEO (Windjammers, then Garou then a PS1 game for comparison), I know I have ~40 items in that folder, but when I tried with the Transcend drive it was THE ONLY game installed and the results were the same.

I think that's something else, I tried with these 2 drives:

• SanDisk Extreme 32GB
• Transcend JetFlash 16GB

They work great on my PC and they are pretty fast.