Critics of the accord say the deal did not go far enough in holding Japan responsible for wartime abuses
Tensions spiked on Friday when Tokyo said it would recall its envoy over the statue which was placed outside its consulate in Busan last month, symbolising women forced to work in Japanese military brothels mostly during World War II.
Japan argues it is against a 2015 agreement between the neighbours meant to put an end to the hugely emotional and decades-long "comfort women" issue with a Japanese apology and payment of money.
"It is extremely regrettable that the girl statue was set," ambassador Yasumasa Nagamine told reporters ahead of his departure from Seoul Monday morning, adding that he would discuss the issue with government officials, possibly including Foreign Minister Fumio Kishida.
The plight of the women has marred relations for decades but the two governments reached an agreement in late 2015 to finally resolve it.
Under that accord, which both countries described as "final and irreversible," Japan offered an apology and a one-billion yen ($8.6 billion) payment to surviving Korean comfort women.
Critics of the accord say the deal did not go far enough in holding Japan responsible for wartime abuses during its 1910-45 colonial rule over the Korean peninsula.
Mainstream historians say up to 200,000 women, mostly from Korea but also other parts of Asia including China, were forced to work in Japanese military brothels during World War II.
The statue in Busan was initially removed by local authorities but after the Japanese defence minister paid homage at Yasukuni Shrine last month -- a spot where senior convicted war criminals are honoured -- Seoul allowed activists to put the statue back up.