Fundamentals of Data Science: the questions likely to come

Data Munging

Data munging (also called data wrangling) is the process of transforming and mapping raw, messy data into a clean, structured, usable format suitable for analysis and modeling. It typically consumes 60–80% of a data scientist's time and includes detecting, correcting, reshaping, and enriching data.

Common issues in real-world data and how to handle them

1. Missing values

Real data often has gaps (empty fields, NaN). Handling: Delete rows/columns if missingness is small; otherwise impute — mean/median for numeric, mode for categorical, or model-based (KNN, regression) imputation. Add a missing-indicator flag if missingness is informative.

2. Outliers and noise

Extreme or erroneous values distort statistics and models. Handling: Detect via box plots, IQR rule (outside $Q1-1.5\,\text{IQR}$ to $Q3+1.5\,\text{IQR}$), or z-scores ($|z|>3$). Treat by removal, capping/winsorizing, or transformation (log).

3. Inconsistent / non-standard formats

Dates, units, capitalization, and encodings differ across sources (e.g., "NP" vs "Nepal", "2082-01-01" vs "01/01/2082"). Handling: Standardize formats, normalize units, trim whitespace, fix casing, and apply consistent categorical labels.

4. Duplicate records

Same entity appears multiple times. Handling: Identify with key/fuzzy matching and deduplicate, keeping the most complete or most recent record.

5. Structural problems

Data may be in the wrong shape (wide vs long), nested, or split across files. Handling: Reshape (pivot/melt), join/merge tables on keys, parse nested fields.

6. Incorrect data types

Numbers stored as strings, dates as text. Handling: Cast columns to correct types (numeric, datetime, category).

7. Scaling differences

Features on very different scales bias distance-based models. Handling: Apply normalization (min–max to [0,1]) or standardization (z-score).

General workflow

1. Discover/profile the data and its problems.
2. Structure — reshape into tidy format.
3. Clean — fix missing values, outliers, duplicates, types.
4. Enrich — derive new features, merge external data.
5. Validate — verify consistency and ranges.
6. Publish the clean dataset for analysis.

Conclusion: Good munging directly determines model quality — garbage in, garbage out — so each issue must be diagnosed during EDA and resolved with a documented, repeatable step.