I've found that when the band is too tight, because my waist is smaller, it pulls down.
try a bra extender, but the kind without elastic so it doesn't extend TOO much! I recently got a pack with both kinds.

I use the stretchy kind to make the band bigger for sleeping, because I like to sleep with a bra on, and my ribs expand when I lay down. there are 3 hook packs too, and 4 hook ones.