I agree in the situation you describe, but you are assuming other tables readily available that allow for similar EV with practically no loss of time.
In this particular case there were no other playable tables at that casino, I would have lost a lot of time going elsewhere, and might not have found playable games even then. Also in my particular case, my max bet is at least a purple, and sometimes twice that depending on the casino tolerance, so I could easily have lost more in EV due to down time or playing an inferior game than the toke. I sized the guy up as not staying long, which fortunately turned out to be the case, so I decided to compromise on my alternatives. I chose to move over a spot at the cost of another hand eating cards in exchange to keep peace. But I figured the high EV of that table and my lost time to relocate was worth the risk him getting upset at my play especially in light of my guess he would be soon moving on anyway.
Hey, if the tightwad had flipped me a gray instead, I'd had no problem hitting the door... ;)