Okay, I think I've got it right, at least for today.
Changed the pattern again, got rid of the back strap and went with the ear loops only.
No falling while talking, no side gaps, still adjustable. The only thing I don't like about it is I want to keep it around my neck when not in use. I'm working on it, but for now I'm pretty happy with the design!