If your virtual hands don't show up in the same place as your real hands and move through space like real hands, it'll feel weird and be hard to use. Since hand joints (usually) curl only in one direction, and if most of the cross products point towards the thumb, we guess it's a right hand; otherwise, we guess it's a left hand. The official Mediapipe implementation only runs the hand detection model once every now and then, then for all the subsequent frames it predicts where the new hand should be in pixel coordinates, just predicting based on the keypoints predicted in the past two frames.