It depends what type of lights they are. A multimeter will not help find which bulbs are bad though unless you plan to test one bulb at a time. Mini lights are cheap and finding the failed bulbs can be a PIA. There are specialized light tools for different types of lights. Here is one for LED lights:

https://www.amazon.com/LED-Keeper-Light-Repair-Tool/dp/B007H4I8HY/ref=asc_df_B007H4I8HY/?tag=hyprod-20&linkCode=df0&hvadid=167158513139&hvpos=&hvnetw=g&hvrand=15375820031832196349&hvpone=&hvptwo=&hvqmt=&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=9031927&hvtargid=pla-308259418710&psc=1