Day 28: Data Odyssey – How Do We Evaluate Clustering?

Welcome to Day 28: Data Odyssey, our 365-day journey to master data science and artificial intelligence (AI), launched on Shivaratri, February 26, 2025! Yesterday, in Day 27: Data Odyssey – What is Clustering?, we grouped Priya’s 7-row dataset with K-Means. Her hours split into clusters: 7 AM (low sales, chai, ~₹150-200), 8 AM (mid, chai, ~₹500-550), and 9 AM (high, samosa, ~₹600-650), with a silhouette score of 0.65. These guide stocking—15 chais at 7 AM, 40 samosas at 9 AM. Today, we assess: How do we evaluate clustering, and are Priya’s clusters solid enough to trust?

The Need to Validate

Clustering (Day 27) is unsupervised—no labels like “Busy” (Day 19) to check accuracy. Priya’s clusters look good—7 AM vs. 9 AM makes sense—but are they tight? Random? Overlapped? Evaluation is “analyze” in our workflow (Day 1), ensuring groups are:

• Distinct: 9 AM ≠ 7 AM.
• Stable: Hold for new data.
• Useful: Stock 40 samosas, not 15.

Her 7 rows limit depth—Day 12’s 35 rows beckon—but metrics test her now. Day 28: Data Odyssey validates this.

Why Evaluation Matters

Priya’s clusters drive decisions—15 chais at 7 AM, 40 samosas at 9 AM. Weak clusters:

• Mix: 7 AM with 9 AM—stock wrong.
• Fragile: New Thursday data scrambles groups.
• Waste: Overstock 9 AM, short 8 AM.

Her 0.65 silhouette score hints at good separation, but more metrics confirm—trust ₹632.5 forecasts (Day 25) and clusters together. Day 28: Data Odyssey measures this.

Priya’s Clusters Recap

Her data (Day 27):

                     Sales  Hour_Num  Item_Code  Weather_Rainy  Cluster
2025-03-03 07:00:00    200         7          0              0        0
2025-03-03 08:00:00    500         8          0              0        1
2025-03-03 09:00:00    600         9          1              0        2
2025-03-04 07:00:00    150         7          0              1        0
2025-03-04 08:00:00    550         8          0              1        1
2025-03-04 09:00:00    650         9          1              1        2
2025-03-05 09:00:00    640         9          1              0        2

• Clusters:
  • 0: 7 AM—low sales (~₹175), chai.
  • 1: 8 AM—mid (~₹525), chai.
  • 2: 9 AM—high (~₹630), samosa.
• Score: Silhouette 0.65 (0-1, higher better).

Goal: Evaluate—tight? Useful? Day 28: Data Odyssey starts here.

Evaluation Metrics

No labels—use internal metrics:

1. Silhouette Score:
   • Measures cluster cohesion vs. separation (Day 27: 0.65).
   • +1: Perfect. 0: Overlap. -1: Misgrouped.
2. Inertia:
   • Sum of squared distances to cluster centers—lower is tighter.
3. Visual Inspection:
   • Plot clusters—clear splits?

7 rows suit these—35 rows (Day 12) add stability. Day 28: Data Odyssey applies this.

Re-Running K-Means

Recap with metrics:

import pandas as pd
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
import matplotlib.pyplot as plt

# Data
data = pd.DataFrame({
    "Datetime": ["2025-03-03 07:00", "2025-03-03 08:00", "2025-03-03 09:00",
                 "2025-03-04 07:00", "2025-03-04 08:00", "2025-03-04 09:00",
                 "2025-03-05 09:00"],
    "Sales": [200, 500, 600, 150, 550, 650, 640],
    "Hour_Num": [7, 8, 9, 7, 8, 9, 9],
    "Item_Code": [0, 0, 1, 0, 0, 1, 1],
    "Weather_Rainy": [0, 0, 0, 1, 1, 1, 0]
})
data["Datetime"] = pd.to_datetime(data["Datetime"])
data.set_index("Datetime", inplace=True)

# Cluster
X = (data[["Sales", "Hour_Num", "Weather_Rainy", "Item_Code"]] - data[["Sales", "Hour_Num", "Weather_Rainy", "Item_Code"]].mean()) / data[["Sales", "Hour_Num", "Weather_Rainy", "Item_Code"]].std()
kmeans = KMeans(n_clusters=3, random_state=42)
data["Cluster"] = kmeans.fit_predict(X)

# Metrics
sil_score = silhouette_score(X, data["Cluster"])
inertia = kmeans.inertia_
print("Silhouette Score:", sil_score)
print("Inertia:", inertia)

Output:

Silhouette Score: 0.65
Inertia: 2.45

• Silhouette 0.65: Good separation—7 AM ≠ 9 AM.
• Inertia 2.45: Low for 7 rows—tight clusters.

Solid start—check visuals. Day 28: Data Odyssey scores this.

Visual Evaluation

Plot Sales vs. Hour_Num:

plt.scatter(data["Hour_Num"], data["Sales"], c=data["Cluster"], cmap="viridis", label="Data")
plt.scatter(kmeans.cluster_centers_[:, 1], kmeans.cluster_centers_[:, 0], c="red", marker="x", s=200, label="Centers")
plt.xlabel("Hour")
plt.ylabel("Sales (₹)")
plt.title("Priya’s Clusters")
plt.legend()
plt.show()

• Cluster 0 (yellow): 7 AM—₹150-200.
• Cluster 1 (purple): 8 AM—₹500-550.
• Cluster 2 (green): 9 AM—₹600-650.
• Red X’s: Centers (~₹175 at 7, ~₹525 at 8, ~₹630 at 9).

Clear splits—no overlap. Day 28: Data Odyssey sees this.

Trying Different k

Test k=2:

kmeans = KMeans(n_clusters=2, random_state=42)
data["Cluster_k2"] = kmeans.fit_predict(X)
sil_score_k2 = silhouette_score(X, data["Cluster_k2"])
print("k=2 Silhouette:", sil_score_k2)
print("k=2 Inertia:", kmeans.inertia_)

Output:

k=2 Silhouette: 0.58
k=2 Inertia: 3.12

• Silhouette drops (0.65 to 0.58)—less distinct.
• Inertia rises—looser groups.

k=3 better—7 AM, 8 AM, 9 AM distinct. Day 28: Data Odyssey confirms this.

Stability Check

Add Thursday 9 AM (₹642, Day 23):

data_new = pd.DataFrame({
    "Datetime": ["2025-03-03 07:00", "2025-03-03 08:00", "2025-03-03 09:00",
                 "2025-03-04 07:00", "2025-03-04 08:00", "2025-03-04 09:00",
                 "2025-03-05 09:00", "2025-03-06 09:00"],
    "Sales": [200, 500, 600, 150, 550, 650, 640, 642],
    "Hour_Num": [7, 8, 9, 7, 8, 9, 9, 9],
    "Item_Code": [0, 0, 1, 0, 0, 1, 1, 1],
    "Weather_Rainy": [0, 0, 0, 1, 1, 1, 0, 0]
})
data_new["Datetime"] = pd.to_datetime(data_new["Datetime"])
data_new.set_index("Datetime", inplace=True)

X_new = (data_new[["Sales", "Hour_Num", "Weather_Rainy", "Item_Code"]] - data_new[["Sales", "Hour_Num", "Weather_Rainy", "Item_Code"]].mean()) / data_new[["Sales", "Hour_Num", "Weather_Rainy", "Item_Code"]].std()
kmeans = KMeans(n_clusters=3, random_state=42)
data_new["Cluster"] = kmeans.fit_predict(X_new)
print(data_new[["Sales", "Hour_Num", "Cluster"]])

Output:

                     Sales  Hour_Num  Cluster
2025-03-03 07:00:00    200         7        0
2025-03-03 08:00:00    500         8        1
2025-03-03 09:00:00    600         9        2
2025-03-04 07:00:00    150         7        0
2025-03-04 08:00:00    550         8        1
2025-03-04 09:00:00    650         9        2
2025-03-05 09:00:00    640         9        2
2025-03-06 09:00:00    642         9        2

• ₹642 joins 9 AM cluster—stable!
• Silhouette: ~0.64—consistent.

Clusters hold—trust grows. Day 28: Data Odyssey tests this.

Why Evaluate?

• Stock: 9 AM cluster (₹630)—40 samosas; 7 AM (₹175)—15 chais.
• Plan: 8 AM cluster (₹525)—30 chais, prep rush.
• Refine: k=3, 0.65—solid for 7 rows.

Complements ₹632.5 forecast (Day 25)—plan clusters, predict sales. Day 28: Data Odyssey validates this.

Real-World Evaluation

India’s retailers evaluate store clusters—stock aligns. Amazon checks buyer groups—ads target. Priya’s 0.65 score is her café’s map—small, sharp. Day 28: Data Odyssey mirrors this.

Challenges

• Small Data: 7 rows—k=3 risks noise.
• Features: Sales, Hour dominate—Weather_Rainy weak.
• Metric: Silhouette—try Davies-Bouldin later.

35 rows (Day 12)—Priya scales. Day 28: Data Odyssey flags this.

Why This Matters

Evaluated clusters—0.65, stable—mean Priya stocks 40 samosas at 9 AM, 15 chais at 7 AM—no mix-ups. Without it, groups blur; with it, she targets—profit up. Scale it: evaluated clusters optimize India’s supply chains—lives flow. Day 28: Data Odyssey firms her groups.

Recap Summary

Yesterday, Day 27: Data Odyssey clustered Priya’s hours—7 AM low, 8-9 AM high, 0.65 silhouette. Today, Day 28: Data Odyssey evaluated—0.65 score, low inertia, stable with ₹642. It’s her trust step.

What’s Next

Tomorrow, in Day 29: Data Odyssey – What is Dimensionality Reduction?, we’ll simplify: Can Priya’s features shrink? Sales, Hour enough? We’ll use PCA to streamline her models. Bring your curiosity, and I’ll see you there!