The 70-year-old Uruguayan Armenian succeeded Italian Claudio Ranieri in February but he handed in his resignation just five months into the job.
Greece suffered a second humiliating defeat to the Faroe Islands in June that left them bottom of Group F and with no hope of qualifying for Euro 2016.
“I came to Greece for the national team at a critical moment and took on a difficult job,” Markarian said in a statement.
“I gave my all from the day I started, working long hours with great love for the work I do.
“I apologise to the Greek fans because the results in our matches were not as good as we had hoped. Now I am making the difficult decision to end my collaboration with the team.”
EPO president George Gkirtzikis said he wanted Markarian to stay.
“I have not accepted it (Markarian’s resignation), I continue to believe in him and trust him,” Gkirtzikis told SportFM radio.
Markarian, nicknamed El Mago (The Magician) for his tactical nous, won domestic titles with clubs in Paraguay, Chile and Peru and his last job was with the Peruvian national team.
During his previous spell in Greece, where he coached Ionikos, Panathinaikos and Iraklis, he was known for his so-called “tsuku tsuku” football which involved grinding out low-scoring wins.