We all have an unconscious selective memory
However, I firmly believe that just the oposite happens to us and we get our azzes saved just as often.
Look at it this way. Let's assume the composition of the remainder of a deck is as follows:
4 of hearts, 5 of c, 5 of s, K of d, J of d, K of s, Qs, 10c, 10s, Jh & a 10h.
With this scenario, we have a high count situation. If we mix these cards a gazillion times, there is still just as much chance of us or the dealer getting a 5 of clubs as there is of getting a Queen of spades. So not necessarily are the big cards going to fall first and then leave the small ones for the dealer.