ENB,
I'm so glad you found us but sorry you had reason to. There is a wealth of experience and perspectives here, so take to heart what serves you best.
One of the reasons you may be caught in yoyo-ing cycles of trying to be together and heal and then be apart and wonder if that might be better is perhaps that your wayward husband (referred to around here as WH) hasn't dug into and owned his reasons for the affair and worked to fix his own core issues to become a safer partner for you.
You seem to describe what we would call marriage counseling (MC) but not individual counseling. Please consider looking for individual counseling (IC) for yourself with someone who specializes in betrayal trauma. If you do seek counseling and your counselor begins to nudge you toward a "shared responsibility" model for the affair (as in "what did you do that contributed to your husband's affair?"), find a new counselor.
Was your relationship perfect before he decided to cheat? Of course not. But it wasn't perfect for you either (and no relationship is ever perfect), and yet you didn't choose to seek an outside affair. So, affairs aren't "caused" by marital dissatisfaction. Yet, cheaters often justify affairs based on this.
Please be assured that the affair was in no way your fault. It was a result of repeated self-focused choices on the part of your husband. It was caused by his own "whys"--likely a combination of self-focus, lack of self-reflection, avoidance, and lack of empathy. Cheating spouses do a lot of lying to themselves (along with lying to others).
So, has he owned and consistently worked on the core issues he has that allowed him to want to break his vows and to justify it to himself?
Has he sought individual counseling with someone who will hold him accountable?
It's concerning that he is so absorbed by the idea of you cheating when you focus on yourself and your own healing. Seems like he's projecting his own thinking and behavior on you. His focus on you cheating with him if you were to divorce and remarry also seems like a huge red flag that his basic thinking about infidelity hasn't shifted.
Along with seeking a supportive counselor for yourself, please consider giving yourself the space and time you really need to focus on yourself and your own healing. This can be done by actually living separately or even by living separately in the same house but using a "180" approach where you draw strong boundaries that focus on you and your kids (emotionally and physically) and avoid giving him traction in your life. Don't cook for him or do his laundry. Don't engage with him beyond what's required for coordinating your kids schedules, etc. If he wants to accuse you of cheating, ignore him. Don't engage.
Look for the Healing Library in the menu to this website. There is a lot of good information there, including the 180 approach.
You need time to really get your feet under you and heal. Taking care of yourself is critical.
Please don't confuse rugsweeping with healing. If you consider reconciliation, he has a ton of work to do. Even if he does the work, you have to decide if you want to be in a marriage with someone capable of making such destructive and selfish choices.
Whatever you decide is okay. Whatever timeline you have is okay.
Take care of you and your kids and do what you need to do to be safe from emotional abuse and trauma.
(((Hugs to you)))
[This message edited by BreakingBad at 11:59 AM, Monday, April 8th]