Don't worry too much; In spring I brought our Olive tree out too early. It received a nice doce of early-spring night frost, and lost all the leaves. It took 5 months to recover, and only in late summer it started growing again. These plants are quite hardy, but are slow growers, and need time.
The key in our case was water: When I started watering the plant every day, all of the sudden new grow popped out.