1) Chronic stressStress can wreak havoc on your immune system, the pharmacist said. 2) Poor sleepSleep is key to recovery, Bhogal added. “When you’re sleep deprived, this process doesn’t happen properly. Immune system recovery happens impressively quickly after quitting smoking, he added. 6) Underlying health conditionsSometimes, recurring colds can signal something else is going on with your immune system, said Bhogal.