With a $500 to $700 budget I might suggest the following. Steer clear of the "packaged podcast stuff". And others will have their own opinions and thoughts but here are mine.

Recorder: Zoom H4nPro $230. Portable, flexible, will accept up to 4 inputs. Records on a SD Card.

Software: Audacity. It is free and allows you to manipulate and put together a cohesive product.

Mics: 2 of the ATR 2100 USB $150. Again, flexible and a decent beginner mic.

Accessories needed. 2 wind screen/pop filters for mics ($20), 2 mic stands ($20), 2 XLR cables for mics ($25)

Headphones: Sony MDR7560 $80 2 might be nice, but 1 required.

That is about $550 to $600 at this point.

If you wanted a studio mic, a decent starter mic is the MXL 990 at $100/ea. But beware that this is a condenser mic and it will pick up a ton of ambient sound especially in a non-treated room.

You may want to add a mixer into the mix at some point. I have a Behringer Xenyx 1204USB $139, but wish I had known more when I bought it and I would have bought the Behringer UFX1204 with the difference being that the one I have sends out a single stereo track to the recorder, but the latter has the ability to send separate tracks. Woudl be very handy if your guest is VERY soft spoken, there is a lot of talk over one another, etc.