I personally use an N64/GameCube to HDMI adapter and an Elgato HD60 S HDMI capture card. You could also use the AV cables that came with the GameCube with an AV to HDMI adapter but you would still need an HDMI capture card. If you don't own a HDMI capture card and you don't want to spend a lot of money on an Elgato there are cheaper "HDMI to USB video capture" options online for less than $20 that aren't as good but will still work okay.

N64 to HDMI Converter Converts N64 Game Console Video Signal to HDMI Signal Easily Connect the Game Console to HDTV N64 Adapter https://www.amazon.com/dp/B085TKRPCC/ref=cm_sw_r_apan_glt_fabc_GQCWVNTC6JYD3HJWCZX0?_encoding=UTF8&psc=1