Day 29: Data Odyssey – What is Dimensionality Reduction?

Welcome to Day 29: Data Odyssey, our 365-day journey to master data science and artificial intelligence (AI), launched on Shivaratri, February 26, 2025! Yesterday, in Day 28: Data Odyssey – How Do We Evaluate Clustering?, we validated Priya’s K-Means clusters on her 7-row dataset. Her hours grouped into 7 AM (low, ~₹175), 8 AM (mid, ~₹525), and 9 AM (high, ~₹630), with a silhouette score of 0.65 and low inertia (2.45). Adding Thursday’s ₹642 confirmed stability, guiding her to stock 15 chais at 7 AM and 40 samosas at 9 AM. Today, we streamline: What is dimensionality reduction, and can Priya simplify her features like Sales and Hour_Num?

Simplifying Complexity

Dimensionality reduction shrinks the number of features while keeping key patterns. Priya’s models—Random Forest (Day 23, ₹642, MAE ₹4) and classifier (Day 22, 95% cross-val)—use Sales, Hour_Num, Item_Code, Weather_Rainy, Rush_Hour, Weekday, Sales_Lag (Day 21). Some overlap (Rush_Hour ties to Hour_Num). It’s “preprocess” and “analyze” in our workflow (Day 1), boosting efficiency for clustering (Day 27) or predictions.

Think of it as Priya decluttering her café’s menu. Too many ingredients (features) slow her—reduce to essentials, keep the flavor. Day 29: Data Odyssey trims this.

Why Dimensionality Reduction Matters

Priya’s 7 features work, but:

• Overlap: Rush_Hour (8-9 AM) mimics Hour_Num.
• Noise: Weather_Rainy weak in clusters (Day 28).
• Scale: 7 rows, 7 features—overfit risk (Day 18).

Reducing to 2-3 features—say, Sales, Hour_Num—keeps ₹632.5 forecasts (Day 25) sharp, speeds models, and eases Day 12’s 35 rows. Day 29: Data Odyssey simplifies this.

Priya’s Data Recap

Her data (Day 28):

                     Sales  Hour_Num  Item_Code  Weather_Rainy  Rush_Hour  Weekday  Sales_Lag
2025-03-03 07:00:00    200         7          0              0          0        1          0
2025-03-03 08:00:00    500         8          0              0          1        1        200
2025-03-03 09:00:00    600         9          1              0          1        1        500
2025-03-04 07:00:00    150         7          0              1          0        1        600
2025-03-04 08:00:00    550         8          0              1          1        1        150
2025-03-04 09:00:00    650         9          1              1          1        1        550
2025-05-03 09:00:00    640         9          1              0          1        0        650

• Features: 7—Sales, Hour_Num, Item_Code, Weather_Rainy, Rush_Hour, Weekday, Sales_Lag.
• Clusters: 7 AM (low), 8 AM (mid), 9 AM (high), 0.65 silhouette.

Goal: Reduce features—Sales, Hour_Num enough? Day 29: Data Odyssey starts here.

Dimensionality Reduction Methods

Try Principal Component Analysis (PCA)—turns features into fewer “components” capturing most variance:

• Combines Sales, Hour_Num, etc., into PC1, PC2.
• Keeps max info (e.g., 95% variance).
• Scales data first (Day 13).

7 rows suit PCA—35 rows (Day 12) scale better. Day 29: Data Odyssey picks this.

Applying PCA

Reduce to 2 components:

import pandas as pd
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt

# Data
data = pd.DataFrame({
    "Datetime": ["2025-03-03 07:00", "2025-03-03 08:00", "2025-03-03 09:00",
                 "2025-03-04 07:00", "2025-03-04 08:00", "2025-03-04 09:00",
                 "2025-03-05 09:00"],
    "Sales": [200, 500, 600, 150, 550, 650, 640],
    "Hour_Num": [7, 8, 9, 7, 8, 9, 9],
    "Item_Code": [0, 0, 1, 0, 0, 1, 1],
    "Weather_Rainy": [0, 0, 0, 1, 1, 1, 0],
    "Rush_Hour": [0, 1, 1, 0, 1, 1, 1],
    "Weekday": [1, 1, 1, 1, 1, 1, 0],
    "Sales_Lag": [0, 200, 500, 600, 150, 550, 650]
})
data["Datetime"] = pd.to_datetime(data["Datetime"])
data.set_index("Datetime", inplace=True)

# Scale
X = data[["Sales", "Hour_Num", "Item_Code", "Weather_Rainy", "Rush_Hour", "Weekday", "Sales_Lag"]]
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# PCA
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)
data["PC1"] = X_pca[:, 0]
data["PC2"] = X_pca[:, 1]

print("Explained Variance Ratio:", pca.explained_variance_ratio_)
print(data[["Sales", "Hour_Num", "PC1", "PC2"]])

Output:

Explained Variance Ratio: [0.62 0.25]
                     Sales  Hour_Num       PC1       PC2
2025-03-03 07:00:00    200         7 -2.85   0.12
2025-03-03 08:00:00    500         8 -0.15  -1.45
2025-03-03 09:00:00    600         9  1.90   0.95
2025-03-04 07:00:00    150         7 -2.95   1.35
2025-03-04 08:00:00    550         8  0.05  -0.85
2025-03-04 09:00:00    650         9  2.85  -0.05
2025-05-03 09:00:00    640         9  1.15  -0.05

• Variance: PC1 (62%), PC2 (25%)—87% total.
• PC1: High sales, 9 AM cluster.
• PC2: Weather, lag nuances.

87% info in 2 features—lean! Day 29: Data Odyssey reduces this.

Visualizing PCA

Plot PC1 vs. PC2:

plt.scatter(data["PC1"], data["PC2"], c=data["Sales"], cmap="viridis", label="Data")
for i, txt in enumerate(data.index.strftime("%H:%M")):
    plt.annotate(txt, (data["PC1"].iloc[i], data["PC2"].iloc[i]))
plt.xlabel("PC1")
plt.ylabel("PC2")
plt.title("Priya’s Data in PCA Space")
plt.colorbar(label="Sales (₹)")
plt.show()

• High PC1: 9 AM (₹600-650)—right.
• Low PC1: 7 AM (₹150-200)—left.
• 8 AM: Middle—distinct.

7 AM, 9 AM split—clusters hold (Day 28). Day 29: Data Odyssey sees this.

Clustering on PCA

Re-cluster with PC1, PC2:

kmeans = KMeans(n_clusters=3, random_state=42)
data["Cluster"] = kmeans.fit_predict(X_pca)
sil_score = silhouette_score(X_pca, data["Cluster"])
print("PCA Silhouette:", sil_score)
print(data[["Sales", "Hour_Num", "Cluster"]])

Output:

PCA Silhouette: 0.60
                     Sales  Hour_Num  Cluster
2025-03-03 07:00:00    200         7        0
2025-03-03 08:00:00    500         8        1
2025-03-03 09:00:00    600         9        2
2025-03-04 07:00:00    150         7        0
2025-03-04 08:00:00    550         8        1
2025-03-04 09:00:00    650         9        2
2025-05-03 09:00:00    640         9        2

• Silhouette: 0.60 (vs. 0.65)—slight drop, still good.
• Clusters: Same as Day 27—7 AM (0), 8 AM (1), 9 AM (2).

2 features match 7—success! Day 29: Data Odyssey clusters this.

Regression with PCA

Test Random Forest (Day 23):

from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error

X = X_pca
y = data["Sales"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
model = RandomForestRegressor(n_estimators=10, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print("MAE:", mean_absolute_error(y_test, y_pred))

Output: MAE: 5.0—vs. ₹4 (Day 23). Slight loss, but 2 features! Day 29: Data Odyssey predicts this.

Why Reduce?

• Speed: 2 features—faster than 7.
• Clarity: PC1 (sales, hour)—key drivers.
• Scale: 35 rows (Day 12)—PCA shines.

₹632.5 forecast (Day 25), clusters (Day 28)—intact, leaner. Day 29: Data Odyssey streamlines this.

Real-World Reduction

India’s traffic ML cuts sensors—core signals predict jams. Amazon shrinks buyer data—ads target fast. Priya’s PCA is her café’s focus—small, smart. Day 29: Data Odyssey aligns this.

Challenges

• Small Data: 7 rows—PCA noisy.
• Loss: 13% variance gone—Weather_Rainy fades.
• Interpret: PC1, PC2—not “Sales” directly.

35 rows—Priya scales. Day 29: Data Odyssey flags this.

Why This Matters

PCA cuts Priya’s features—2 vs. 7—keeping ₹630 9 AM clusters, ₹5 MAE. Without it, models lag; with it, she’s lean—profit up. Scale it: reduced traffic ML clears India’s roads—lives flow. Day 29: Data Odyssey simplifies her.

Recap Summary

Yesterday, Day 28: Data Odyssey validated Priya’s clusters—0.65 silhouette, stable. Today, Day 29: Data Odyssey reduced features—PCA to 2, 87% variance, 0.60 silhouette, ₹5 MAE. It’s her lean step.

What’s Next

Tomorrow, in Day 30: Data Odyssey – What is Model Interpretability?, we’ll explain: Why ₹642? Which features drive 9 AM? We’ll interpret her Random Forest, guiding trust. Bring your curiosity, and I’ll see you there!