This article explores the application of information theory—originally established by Claude Shannon in 1948—to machine learning, specifically focusing on its role in feature selection and model training. The author defines information as the reduction of uncertainty and details four core mathematical concepts: entropy, information gain, cross-entropy, and KL divergence. Entropy $H(X) = -\sum P(x) \log_2 P(x)$ quantifies the average surprise or uncertainty in a variable, where a fair coin represents 1 bit of entropy. In decision tree models like XGBoost, features are selected based on information gain, which measures the reduction in entropy achieved by a specific split. Cross-entropy serves as the standard classification loss function because it penalizes confident, incorrect predictions more heavily than soft errors, effectively acting as negative log-likelihood. Furthermore, the article explains that KL divergence, $KL(P \parallel Q) = \sum P(x) \log(P(x)/Q(x))$, measures the information loss when one distribution approximates another, though it remains asymmetric. For practitioners, the author emphasizes using mutual information to detect non-linear dependencies that standard correlation might miss, and warns that the maximum edge in binary prediction is capped by the target's entropy, which for markets like the Nifty often approaches the 1-bit ceiling.