How to deal with this person who is affecting my life?

I use to work with this guy whom I was becoming friends with. He use flirt with me and even sexual harass me but I never let him do anything to me. After things got out of hand professionally at work with my job concerns and with him also, I thought we could remain friends afterward. He use to call me everyday just to see how I was doing when we were working together. Now after I quit he doesn't act like he use to. He is acting different toward me on the phone, its all about him and nothing about me. I have shown a lot of support for him and he doesn't ask interested in me and my life. I feel that I am going in a different direction in life right now, focusing on a career and not him or anyone else. So he called me today for the first time in eleven days and said he's sorry for not calling and wants me to call him back. I really don't want to talk to him anymore because I feel that he may try to bring my good energy down by talking about himself and not acting interested in me. I need your advice. What should I do concerning his message, should I return his call? Should I release him out of my life and continue to focus on me and my career and never call him back?