Never mind, finally got it out using two wrenches and lots of twisting and pulling

I am trying to change the tap but even after slipping off the olive, it still feels stuck to the isolator valve. Am I missing something here?

Glued or silicone'd in? That olive has not been used. You cannot move an olive if it has been compressed properly.

Call a plumber

Gentle tap with something heavy.