BJ Etiquette -- joining/leaving during a shoe

So I was off at my local casino this past weekend, playing a modified Basic Strategy and providing some much-needed financial support to the Mashantucket Pequot Indian Nation.

Red-chipping it at 2:00 am, we had a lively table going. A couple of people came to the empty holes, played a couple of hands (first time I've seen anyone split a pair of fives with a pile of green-chips) and left. We got to discussing the effect that such a player has on the game, with various opinions as to how a player joining or leaving in the middle of the shoe "destroyed the rhythm of the game", "ruined the flow" and "broke up the game."

Now I'm just a poor beginner (I'd like to think I'm better than the average ploppie) but I don't see mathematically why varying the number of players would have an effect.

So is it a matter of some number theory which I have not figured out yet?
Is it a superstition (and at 2 in the morning, I heard some wonderful ones)?
Or is it just "good manners" and not something a real BJ player would do?

(and if anyone cares, I actually managed to break even for the weekend after being deep in a hole at one point (mostly green-chipping with some $15 late-night BJ) _and_ got enough of their "frequent loser" points to get comped a most excellent meal.

An average Joe jumping in and out of the game is not going to affect the game overall, it's just superstition. People get jealous when other players successfully dive in and out of games and they appear to make money. They assume the money is made at their expense.

I can speculate one mathematical argument though. If a counter wongs into a game at a high count, it can reduce the number of hands the ploppies play at high counts since they are sharing the deck with him. The AP who does this most likely leaves as soon as the RC tanks. The players remaining are then left with a deck with fewer potential BJs, more dealer wins with stiff starts, and less successful doubles/splits therefore making the cards "bad". This statement would apply for a skillful AP, but an average Joe probably wouldn't cause this scenario anymore than he causes the opposite scenario.