Core Strategy
Start by defining which variation is noise and which carries predictive information:
-
Normalize safely: Decode HTML entities, standardize Unicode, normalize whitespace, and lowercase text when case does not matter.
-
Replace metadata: Convert URLs to URL and mentions to USER while preserving their positions in the sentence.
-
Preserve meaningful content: Keep hashtags, emojis, slang, and intensifiers when they indicate topics or sentiment.
-
Handle negation: Preserve words such as not and never; removing them can reverse a label.
-
Reduce accidental repetition: Carefully collapse repeated characters or punctuation when they clearly result from informal typing.
-
Detect duplicates: Remove exact duplicates and consider grouping near-duplicates before splitting data.
-
Address label noise: Correct or relabel examples using annotator agreement, adjudication, or confidence-based filtering.
Normalization is not sufficient by itself. Misspellings, domain-specific slang, multilingual content, and unfamiliar hashtags may require dictionaries, domain adaptation, or subword models.
Classic NLP Pipeline
The traditional text pipeline remains useful for small datasets and interpretable baselines:
-
Tokenization: Split text into words, subwords, or characters.
-
Normalization: Standardize casing, Unicode, URLs, mentions, and whitespace.
-
Stop-word handling: Remove common words only when they add little to the task; avoid removing negation or task-critical words.
-
Stemming or lemmatization: Stemming is faster but can produce nonstandard roots; lemmatization uses linguistic rules and generally produces dictionary words.
-
Feature extraction: Bag-of-words records word presence and counts, while TF-IDF reduces the influence of words common across the corpus.
-
N-grams: Unigrams capture individual words, while bigrams capture phrases such as not good and highly recommended.
-
Modeling: Apply algorithms such as logistic regression, SVMs, or linear classifiers to sparse vectors.
Splitting data before fitting the vectorizer is essential. Otherwise, vocabulary and inverse document frequencies can leak test information into training.
Modern NLP Approach
Word embeddings represent words as dense vectors:
-
word2vec: Learns local co-occurrence patterns using skip-gram or continuous bag-of-words training.
-
GloVe: Uses global co-occurrence statistics.
-
fastText: Extends word embeddings to character-level subwords, helping with rare and misspelled words.
Contextual models produce different representations for the same word according to its context. BERT is primarily used for encoding and classification, while GPT-style models are designed for generation. Their tokenizers commonly use BPE or WordPiece, which split words into reusable subword units and reduce out-of-vocabulary problems.
For most real-world NLP projects, using a pretrained model with fine-tuning is a strong default. Fine-tuning should use task-specific training data, while validation data remains reserved for model selection and early stopping.
Evaluation
Choose metrics according to the output and class distribution. For classification, report precision, recall, and F1; for imbalanced data, also examine ROC-AUC and PR-AUC. Accuracy can be misleading when a rare positive class contributes little to the overall score.
| Task |
Typical output |
Suitable evaluation |
| Text classification or sentiment |
Label and class probabilities |
Accuracy, precision, recall, macro/weighted F1 |
| Named entity recognition |
Entity spans and types |
Entity-level precision, recall, F1 |
| Part-of-speech tagging |
One tag per token |
Accuracy or macro F1 |
| Question answering |
Answer span or generated answer |
Exact match, token F1, human review |
| Summarization or translation |
Generated text |
ROUGE, BLEU, BERTScore plus human evaluation |
| Information retrieval |
Ranked relevant documents |
MRR or nDCG |
Automatic generation metrics are imperfect because they may not recognize factual errors, omissions, or meaning changes. Human evaluation remains important.
Code Example
This example trains a TF-IDF and logistic-regression classifier on noisy, inline social media text. The vectorizer is inside a pipeline so that it is fitted only on training data. The cleaner now also collapses repeated characters and preserves emojis in the text (though the TF-IDF token pattern used here does not capture emojis; for emoji-aware models, use a tokenizer that includes them).
import html
import re
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import (
StratifiedKFold,
cross_validate,
train_test_split,
)
from sklearn.pipeline import Pipeline
texts = np.array([
"I hate this game!!! 😡",
"Worst match ever https://example.com",
"This app is terrible #fail",
"Awful service and broken UI",
"Really disappointed @support",
"Not good at all",
"Too many bugs!!!",
"The update is awful",
"Great game!!! #NBA 😊",
"Not bad at all",
"This app works great",
"Love the new update",
"Really happy with the service",
"Excellent and highly recommended",
"The new feature is great",
"Not terrible, actually quite good",
])
labels = np.array([0] * 8 + [1] * 8)
def clean_text(text: str) -> str:
text = html.unescape(text)
text = re.sub(r"https?://\S+|www\.\S+", " URL ", text, flags=re.IGNORECASE)
text = re.sub(r"@\w+", " USER ", text)
text = re.sub(r"#(\w+)", r" \1 ", text)
# Collapse 3+ repeated characters to 2 (e.g., "!!!" -> "!!")
text = re.sub(r"(.)\1{2,}", r"\1\1", text)
text = re.sub(r"\s+", " ", text.casefold()).strip()
return text
X_train, X_test, y_train, y_test = train_test_split(
texts,
labels,
test_size=0.25,
stratify=labels,
random_state=42,
)
model = Pipeline([
(
"tfidf",
TfidfVectorizer(
preprocessor=clean_text,
token_pattern=r"(?u)\b\w+\b",
ngram_range=(1, 2),
sublinear_tf=True,
),
),
(
"classifier",
LogisticRegression(
max_iter=1000,
class_weight="balanced",
solver="liblinear",
random_state=42,
),
),
])
cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
cv_results = cross_validate(
model,
X_train,
y_train,
cv=cv,
scoring="f1_macro",
)
model.fit(X_train, y_train)
predictions = model.predict(X_test)
print("Cross-validation macro F1:", round(cv_results["test_score"].mean(), 2))
print(classification_report(
y_test,
predictions,
labels=[0, 1],
target_names=["negative", "positive"],
zero_division=0,
))
print("Confusion matrix:")
print(confusion_matrix(y_test, predictions, labels=[0, 1]))
Common Pitfalls
-
Over-cleaning: Removing every emoji, hashtag, or unusual token can discard useful sentiment or topic signals.
-
Negation loss: Stop-word lists and naive stemming can change the meaning of a sentence.
-
Data leakage: Fit TF-IDF, normalization dictionaries, and imputers only on training folds.
-
Duplicate leakage: Group duplicated or campaign-related posts before splitting; otherwise identical content can appear in both training and test sets.
-
Wrong metric: Accuracy may conceal severe minority-class failure; inspect per-class precision and recall.
-
Domain shift: A model trained on formal text may perform poorly on slang, abbreviations, or newly emerging topics.
-
Multilingual data: Identify language before applying language-specific stop words, tokenizers, or embeddings.
-
Bias and privacy: Pretrained models may reproduce social stereotypes, and raw social media can contain sensitive or personally identifiable information. Minimize retained data, apply appropriate access controls, and document filtering decisions.
Follow-up questions interviewers often ask: - How would you handle multilingual social media text in a single pipeline? - What are the trade-offs of using a rule-based cleaner versus a learned normalization model? - How do you evaluate the impact of cleaning choices on downstream task performance? - When would you choose character-level or subword tokenization over word-level for noisy text? - How do you prevent leakage when the cleaning step uses global statistics (e.g., a slang dictionary built from the whole corpus)?