The tree looks pretty bad, it lost a lot of leaves, but like I said, had completely stopped growing, and then started again after I moved it to a larger pot.
I don't think it looks worse than mine, which is now also yellowing and dropping leaves.
The pot seems large enough, and it started to grow in it, so I think there's no real need to put it in the ground.
Let's see what others say.
I would get rid of that grass and hair moss.