You want a linear voltage regular that drops the voltage to 3.3v. Since the battery voltage is already relatively close to 3.3v, you need an LDO (low dropout) regulator. Here's an example: HT7333-A. Read the datasheet and verify that it meets your amperage needs. I don't use the esp32's wifi, so I can't confirm it'll do the job in every scenario. You can connect the LDO to the 3.3v pin. If you have reliability trouble, a 10μF cap on each input/output might help deal with voltage fluctuation.

The battery you link to appears to have a protection circuit built in. This is good as you can connect up the battery and the LDO directly to the device and you won't need to worry quite as much about destroying the life of the battery via over discharge or starting a fire with a short circuit.

For charging, consider a charging board containing a TP4056. An example is found here. The board I've linked to does charging and also has battery protection. You can charge via the USB or even hook up a small solar panel. There are other similar boards rocking a TP4056 which don't have battery protection, those might work for you, but perhaps just go with the type I linked so we don't need to get into the details.

Esp32 with an LDO connected to 3v3, the LDO connected to the OUT of the TP4056-based charging board, the lipo connected to the BAT of the charging board, and either USB or a 6v solar connected to the IN of the charging board.