I have a different take on this Bible verse than you would usually hear. Not only do we have to stay clear of people who may negatively affect our relationship Christ, but also those who affect our self-esteem.
Christ wants us to all have a positive self-esteem, since we are children of God. If we choose to be around people who are always putting us down, making us feel bad about ourselves or our weight, pointing out all our flaws while minimizing the good things we do, maybe we are yoking ourselves with unrighteousness without thinking about it.
This is something to be prayerful about. Sometimes it's difficult, because sometimes these people may not be friends that you can stop choosing to visit. Sometimes these are family members, so how you deal with this needs to be be prayerfully contemplated. I do think that it's vitally important that you start praying for these negative people and turn over the impact their statements have on you. You are a child of God and He's telling us in this Bible verse why should light hang around with darkness. (paraphrased) Maybe you can structure your encounters so that there's less opportunity for those people to be a negative influence in your life. Maybe, it's time to think whether this relationship is worth perpetuating. Maybe, it's time to find people who really appreciate the wonderful person that you are!