The X5 Pro was precisely designed so that if you are not pulling straight up, the force to the side can cause the pin to jam. Therefore, if you either grip the sides of the handle or directly in the center the handle comes off. It does not take much force so if you are forcing it, reposition your hand and try to pull straight up from the center.
When you remove your handle you can put a little chapstick around the pin area and on the backside. This will help until the unit is broken in.
If after trying to no avail, please contact us and we will be happy to help correct the issue.