I'm not single but if I was, I'd walk away at the first sign. I think it's a cop out to say that he'll just go and do it with someone else if it's not you. You can also find another person to sleep with, if that's what you're looking for. It doesn't have to be him.
Why put one night of pleasure over the time (likely years) that his wife has spent with him. You don't have to know or care about her to respect her and to do right by her. Why take part in something that'd likely devastate her? Just because it will happen anyway, it doesn't mean that you should be involved just for the sake of some (potentially) good sex and just because you're involved indirectly and won't have to deal with the consequences.
What would you do if you knew he had children? How many people have to be hurt by his infidelity before it would seem like too immoral an act to take part in?
Would you have any respect for a closeted man cheating on his family? If not, then why have sex with someone that you have no respect for?
A friend of mine's attitude was that it was "not his problem." After a couple of conversations, I shamed him into changing his tune
and believe me, he was actually shamed...I wish I could remember all the points I made to him!