Threatened to get hit by my husband's friend... what to do?

It's a HUGE mess... I'll try summarizing. I had an affair (1 month) with my husband's friend at work, but I felt terrible and regretted and confessed to my husband. He forgave me and we decided to stay together and make it work (there's a lot behind this affair... my husband was neglectful and pushed me to his friend... gave him my cell number, sent him my pics, I was vulnerable too, etc). Ok my husband then confronts his friend and the guy denies everything and gets mad at me and tells me I broke up their friendship! Then a common friend of ours called me today saying this guy wants to hit me when he sees me (although he never hit a woman, he said), and he's also going to fight with my husband. He also said he's gonna sue us for slandering (!!!) and defamation of character -- which he said we spread lies about him at their job. I don't even work at their workplace and hardly go there except to pick up or drop my husband off. And my husband is never there and doesn't talk about this guy (they're truck drivers). So my friend (and also still talks to this guy) tells me be careful, my husband and I. We never called or texted him after my husband confronted him about 3 weeks ago, and he was still calling my husband until 2 weeks ago, but stopped altogether. So we are clearly not harming him. We are working on our marriage and we want to be left alone. So what can be done legally at this point to protect ourselves? Thank you.