Medyo mahaba, bear with me.

I'm currently in a relationship with a married (separated for 9-10 years, not annulled) man with 2 children with his wife and 1 lovechild he had with a live-in partner after we broke up 6-7 years ago. We just got back together May of last year and recently celebrated our anniversary.
He is a very funny, smart and charming guy who takes care of me very well. If soulmates do exist, he's probably my soulmate.
Last holy week, he brought me to their province to (re)introduce me to his family.... that's when I saw how integral his role was to the
entire clan. His mother also called me in for a "talk". She was discouraging me from dating his son, which wasn't an issue at all since she was only thinking about my welfare.
A rush of emotions enveloped me since... anger, guilt, annoyance (not at him nor his mom). Despite the fact that were a wonderful couple and strategically laid-out life plans... there are too many trade-offs for us to continue our relationship, mostly at the expense of our families.
Last week, someone tipped the scales further: My "abusive" Ex. He did not hurt me physically, but the psychological trauma I sustained wrecked me. Neither one of us wants a relationship with each other, we merely wanted to catch up and keep communication lines open.
We were together for 2 years and had a nasty breakup with no closure. I had tried to look for closure during the year that I started dating again but never got it. Now, 2 years later, just when I thought I could handle it and I was over him ... I felt instantaneously depressed the day after we met.
I realized that I still love him. It makes me wonder how I could love my ex (so much) when I am also in love with a man who loves me more than life itself.
I am not hoping to get back together with the EX but I feel extremely sad about my complicated relationship. I feel like my partner and I will never make it through. I feel as if at some point I will eventually snap because of all the responsibilities he has to fulfill before he could attend to my needs.
Should I just ditch them both out of my life so I could clear my head? :/