Skip to content
Machine Learning & Data Science

NLP interview questions and answers

50 questions, 38 with code. About 4 h 23 min to read them all.

By TestLab · Updated

Plan for an interview
50 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What is Natural Language Processing (NLP) and why does it matter?

Natural Language Processing (NLP) is the field of AI and computational linguistics that builds systems to understand, analyse and generate human language in text or speech, covering tasks such as text classification, sentiment analysis, named entity recognition, translation, question answering, summarization, semantic search and conversational assistants. It matters because most of the world’s information and most human interaction is in unstructured language: emails, documents, support tickets, reviews, medical notes, contracts, code comments and conversations. NLP turns that text into structured signals that software can search, route, summarize and act on, and it powers products people use every day, from search engines and spam filters to machine translation, voice assistants and large language models. The field has moved through three eras: rule-based systems (hand-written grammars and patterns), statistical machine learning (n-grams, Naive Bayes, CRFs over features like bag-of-words and TF-IDF) and neural NLP (word embeddings, RNNs and now transformers such as BERT for understanding and GPT-style models for generation). Today the default approach is to start from a pretrained transformer and fine-tune or prompt it, while classic methods remain strong, cheap baselines. Language is hard for computers because it is ambiguous (“bank” of a river or for money), context-dependent, full of idioms, sarcasm and slang, and varies across domains and thousands of languages.

Read full answer

Applies toPython 3.10+, scikit-learn 1.x

Core Tasks

Task Input → output Example
Text classification Text → label Spam vs not spam
Sentiment analysis Text → polarity “Loved it” → positive
Named entity recognition Text → spans + types “Paris” → LOCATION
Part-of-speech tagging Tokens → tags “runs” → VERB
Machine translation Text → text in another language English → Hindi
Question answering Question + context → answer Extract the answer span
Summarization Long text → short text News article → 3 lines
Semantic search Query → ranked documents Embedding similarity

Code Example: A Classic NLP Baseline

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

texts = [
    "win a free prize now", "limited offer click to claim cash",
    "free money waiting claim today", "meeting moved to monday",
    "please review the attached report", "lunch with the team tomorrow",
]
labels = ["spam", "spam", "spam", "ham", "ham", "ham"]

model = make_pipeline(TfidfVectorizer(), LogisticRegression())
model.fit(texts, labels)
print(model.predict(["claim your free cash prize", "report for the monday meeting"]).tolist())
# ['spam', 'ham']

A modern alternative is a pretrained transformer, for example transformers.pipeline("sentiment-analysis"), which downloads a model on first use.

Why It’s Hard

Challenge Example
Ambiguity “I saw the man with the telescope”
Context “It’s cold” (weather or a person’s tone?)
Variation Slang, typos, code-switching
Domain shift Legal vs medical vs social media text
Low-resource languages Little labelled data

Common Pitfalls

  • Skipping a simple baseline (TF-IDF + logistic regression) before using large models.

  • Ignoring domain shift between training and real text.

  • Treating model outputs as unbiased; language models reflect biases in their training data.

Interview Tips

  • Define NLP, list core tasks and explain why language is hard.

  • Describe the shift from rules to statistics to transformers, and the pretrain-then-fine-tune default.

2

What are a corpus, tokenization and stopwords in NLP?

A corpus is a collection of texts used to train, evaluate or study NLP systems (for example Wikipedia, a set of product reviews or a company’s support tickets); tokenization is the step that splits raw text into units called tokens, such as words, punctuation marks or subword pieces; and stopwords are very frequent function words like “the”, “is”, “and” and “of” that carry little topical meaning and are often removed in classic pipelines. A corpus can be labelled (each review tagged positive or negative) or unlabelled, and its domain, size, language and quality largely determine what a model learns. Tokenization looks simple but has many decisions: handling punctuation and contractions (“don’t” → “do” + “n’t”), hyphens, URLs, emojis and numbers, and languages without spaces between words (Chinese, Japanese, Thai). Modern transformers use subword tokenization (BPE, WordPiece, SentencePiece/Unigram), which splits rare words into frequent pieces (“tokenization” → “token” + “##ization”), so any word can be represented with a fixed vocabulary and no word is out of vocabulary. Stopword removal shrinks bag-of-words features and helps topic-oriented tasks such as search or topic modelling, but it can hurt tasks where those words matter: removing “not” flips sentiment (“not good”), and question words matter for question answering. Transformers usually keep every token, because they learn from context which words matter.

Read full answer

Applies toPython 3.10+, scikit-learn 1.x

Comparison

Term What it is Example
Corpus Collection of documents 10,000 movie reviews
Token A unit of text “cats”, “!”, “##ing”
Tokenization Splitting text into tokens “I’m here” → [“I”, “‘m”, “here”]
Stopwords Frequent function words the, a, is, of, and
Vocabulary Set of distinct tokens {cat, sat, mat, …}

Code Example

import re
from collections import Counter

from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS

corpus = [
    "The cats aren't sitting on the mat.",
    "A cat sat on the mat, and the dog barked!",
]

def tokenize(text):
    return re.findall(r"\w+|[^\w\s]", text.lower())       # words or single punctuation marks

tokens = [tokenize(doc) for doc in corpus]
print(tokens[0])
# ['the', 'cats', 'aren', "'", 't', 'sitting', 'on', 'the', 'mat', '.']

stop = {"the", "a", "on", "and"}
content = [t for t in tokens[1] if t.isalnum() and t not in stop]
print(content)
# ['cat', 'sat', 'mat', 'dog', 'barked']

freq = Counter(t for doc in tokens for t in doc if t.isalnum())
print(freq.most_common(2))
# [('the', 4), ('on', 2)]

print("not" in ENGLISH_STOP_WORDS, len(ENGLISH_STOP_WORDS))  # careful: 'not' is a stopword here
# True 318

A subword tokenizer from Hugging Face (AutoTokenizer.from_pretrained("bert-base-uncased")) would split rare words into known pieces, for example “tokenization” into “token” and “##ization” (the model files download on first use).

“the” appears 4 times; “on” and “mat” tie at 2, and most_common returns ties in first-seen order, so “on” comes first.

Common Pitfalls

  • Naive whitespace splitting that keeps punctuation glued to words (“mat.”).

  • Removing stopwords such as “not” or “no” in sentiment tasks.

  • Using a tokenizer at inference time that differs from the one used in training.

  • Applying English tokenization rules to other languages.

Interview Tips

  • Define all three terms with examples.

  • Mention subword tokenization and when stopword removal hurts.

3

What is the difference between morphology and syntax in NLP?

Morphology studies the internal structure of words, how they are built from morphemes (the smallest meaningful units such as roots, prefixes and suffixes, as in “un-happi-ness” or “walk-ed”), while syntax studies how words combine into phrases and sentences, the grammatical structure that determines which word is the subject, which is the object and what modifies what. In NLP, morphological processing includes stemming and lemmatization (reducing “studies” to “studi” or “study”), morphological analysis (identifying features such as number, tense, person and case, for example “cats” = cat + plural), and handling compounding and agglutination, which matters greatly in morphologically rich languages like Turkish, Finnish, Hindi or German, where one word can carry what English expresses with a whole phrase. Syntactic processing includes part-of-speech tagging, constituency parsing (nested phrases, NP and VP, from context-free grammars) and dependency parsing (head–dependent relations like nsubj and obj). The two interact: morphology supplies the features (agreement, case) that syntax uses, and errors in one propagate to the other. Modern subword tokenizers capture part of morphology implicitly, and transformers learn much syntax without explicit parsers, but explicit morphological and syntactic analysis is still useful for information extraction, grammar checking, low-resource languages and interpretable pipelines.

Read full answer

Applies toPython 3.10+, spaCy 3.x

Comparison

Morphology Syntax
Unit Word (morphemes) Sentence (words, phrases)
Question How is a word formed? How are words arranged and related?
Example “unhappiness” = un + happy + ness “The dog chased the cat”: dog = subject
NLP tasks Stemming, lemmatization, morphological tagging POS tagging, constituency and dependency parsing
Features Tense, number, case, gender Subject, object, modifier, phrase structure

Code Example

import re

import spacy  # requires: python -m spacy download en_core_web_sm

# Toy morphological analysis with suffix rules (real analyzers use lexicons and finite-state transducers)
RULES = [(r"(\w+)ies$", r"\1y", "Number=Plur"), (r"(\w+)ed$", r"\1", "Tense=Past"),
         (r"(\w+)s$", r"\1", "Number=Plur")]

def analyze(word):
    for pattern, repl, feat in RULES:
        if re.match(pattern, word):
            return re.sub(pattern, repl, word), feat
    return word, "-"

print([analyze(w) for w in ["studies", "walked", "cats", "run"]])
# [('study', 'Number=Plur'), ('walk', 'Tense=Past'), ('cat', 'Number=Plur'), ('run', '-')]

# Syntax with spaCy
nlp = spacy.load("en_core_web_sm")
doc = nlp("The dogs chased the cat.")
for tok in doc:
    print(tok.text, tok.lemma_, str(tok.morph), tok.dep_, tok.head.text)
# Typical output (may vary slightly by model version):
# The the Definite=Def|PronType=Art det dogs
# dogs dog Number=Plur nsubj chased
# chased chase Tense=Past|VerbForm=Fin ROOT chased
# the the Definite=Def|PronType=Art det cat
# cat cat Number=Sing dobj chased
# . . PunctType=Peri punct chased

The toy rules show why real morphology needs more than suffixes: they would wrongly turn “bus” into “bu”.

Common Pitfalls

  • Treating morphology as just stemming; features like tense and case matter.

  • Ignoring rich morphology when working with non-English languages.

  • Assuming one correct parse; syntax is often ambiguous (prepositional phrase attachment).

Interview Tips

  • Define both with examples (morphemes vs sentence structure).

  • Name the NLP tasks for each and mention morphologically rich languages.

4

Why is part-of-speech (POS) tagging important in NLP?

Part-of-speech tagging assigns each token its grammatical category, such as noun, verb, adjective, adverb, pronoun, preposition or determiner, using the surrounding context to resolve ambiguity, so “book” is tagged as a noun in “read the book” and as a verb in “book a flight”. It is significant because many downstream steps depend on it: lemmatization needs the POS to pick the right base form (“saw” → “see” as a verb, “saw” as a noun), dependency and constituency parsers build on tags, named entity recognition and information extraction use them as features (for example proper nouns), keyword and noun-phrase extraction rely on patterns like adjective + noun, text-to-speech uses them for pronunciation (“record” as a noun vs verb), and grammar checkers and linguistic research analyse them directly. Tag sets include the coarse Universal POS tags (NOUN, VERB, ADJ, ADP, DET, PROPN …, used across 100+ languages) and the finer Penn Treebank tags for English (NN, NNS, VBD, VBZ, JJ …). Methods evolved from rules and Hidden Markov Models (with Viterbi decoding) to CRFs and now neural taggers; accuracy on standard English news text is around 97 to 98 percent, lower on social media or other domains. In end-to-end transformer models, explicit POS tags are rarely needed as input because the model learns syntactic information internally, but tagging remains a cheap, interpretable tool in pipelines.

Read full answer

Applies toPython 3.10+, spaCy 3.x

Example Tags

Word Universal POS Penn Treebank
The DET DT
quick ADJ JJ
fox NOUN NN
jumps VERB VBZ
over ADP IN
London PROPN NNP

Code Example

# Requires: pip install spacy && python -m spacy download en_core_web_sm
import spacy

nlp = spacy.load("en_core_web_sm")

for sent in ["I want to book a flight.", "I read a good book."]:
    doc = nlp(sent)
    print([(t.text, t.pos_, t.tag_) for t in doc if t.text == "book"])
# Typical output:
# [('book', 'VERB', 'VB')]
# [('book', 'NOUN', 'NN')]

doc = nlp("The striped bats were hanging on their feet.")
print([(t.text, t.pos_, t.lemma_) for t in doc if t.pos_ in {"NOUN", "VERB"}])
# Typical output:
# [('bats', 'NOUN', 'bat'), ('hanging', 'VERB', 'hang'), ('feet', 'NOUN', 'foot')]

noun_phrases = [chunk.text for chunk in nlp("Large language models need clean training data.").noun_chunks]
print(noun_phrases)
# Typical output: ['Large language models', 'clean training data']

Exact tags can differ between spaCy model versions, especially for ambiguous words.

Common Pitfalls

  • Expecting news-level accuracy on tweets or domain-specific text.

  • Mixing Universal and Penn Treebank tag sets.

  • Feeding untokenized or poorly tokenized text to a tagger.

Interview Tips

  • Define POS tagging with an ambiguity example and list downstream uses.

  • Mention tag sets, HMM/CRF/neural methods and typical accuracy.

5

What are lemmatization and stemming, and when do you use each?

Both reduce inflected words to a common base so that “running”, “runs” and “ran” can be treated as related, but stemming does it by chopping suffixes with heuristic rules (the Porter or Snowball stemmers turn “studies” into “studi” and “running” into “run”), while lemmatization uses a vocabulary and morphological analysis, usually with the word’s part of speech, to return a real dictionary form called the lemma (“studies” → “study”, “better” as an adjective → “good”, “ran” as a verb → “run”). Stemming is fast, simple and language-specific but crude: it produces non-words, over-stems unrelated words to the same stem (“universe” and “university” both become “univers”) and under-stems irregular forms (“ran” stays “ran”). Lemmatization is more accurate and gives readable output but is slower, needs language resources (WordNet, spaCy models) and depends on correct POS tags. Use stemming when you need speed and higher recall in search or bag-of-words features and readability doesn’t matter (classic information retrieval, quick baselines on large corpora); use lemmatization when meaning and readable output matter (chatbots, analytics, topic models people will read, information extraction) or for morphologically rich languages. With transformer models, neither is usually applied, since subword tokenizers and context handle word forms.

Read full answer

Applies toPython 3.10+, NLTK 3.8+

Comparison

Stemming Lemmatization
Method Suffix-stripping rules Dictionary + morphology + POS
Output May be a non-word (“studi”) Real word (“study”)
Irregular forms Not handled (“ran”) Handled (“ran” → “run”)
Speed Very fast Slower
Resources Rules only Lexicon / model
Use Search, quick BoW features Analytics, readable output, extraction

Code Example

# Requires: pip install nltk; WordNet data is downloaded once with nltk.download("wordnet")
import nltk
from nltk.stem import PorterStemmer, WordNetLemmatizer

nltk.download("wordnet", quiet=True)
stemmer, lemmatizer = PorterStemmer(), WordNetLemmatizer()

words = ["running", "studies", "flies", "caring", "ran"]
print([stemmer.stem(w) for w in words])
# ['run', 'studi', 'fli', 'care', 'ran']

print([lemmatizer.lemmatize(w, pos="v") for w in ["running", "ran", "studies"]])
# ['run', 'run', 'study']
print(lemmatizer.lemmatize("flies"), lemmatizer.lemmatize("better", pos="a"), lemmatizer.lemmatize("better"))
# fly good better

print(stemmer.stem("universe"), stemmer.stem("university"))   # over-stemming
# univers univers

Without a POS argument the WordNet lemmatizer assumes a noun, which is why “better” stays “better” in the last call.

Common Pitfalls

  • Lemmatizing without POS tags, which silently treats every word as a noun.

  • Stemming text that users will read.

  • Applying English stemmers to other languages.

  • Normalizing text for transformer models, which can remove useful information.

Interview Tips

  • Contrast rules vs dictionary with “studies” → “studi”/“study”.

  • Explain the speed vs accuracy trade-off and give a use case for each.

6

What is a named entity, and how is NER useful?

A named entity is a real-world object referred to by a proper name or a specific expression, such as a person (“Ada Lovelace”), organization (“Infosys”), location (“Mumbai”), date (“12 March 2026”), money amount (“$1 billion”), product or event; Named Entity Recognition (NER) is the task of finding these spans in text and classifying their type. NER turns unstructured text into structured data, which makes it useful across many applications: information extraction (who acquired whom, for how much, when), populating knowledge graphs and CRMs, search and question answering (boosting documents that mention the queried company), content recommendation and tagging of news, resume parsing, customer-support routing, financial and legal document analysis, and de-identifying personal data in medical or legal records for privacy compliance. It is usually framed as sequence labelling with BIO tags (B-PER for the beginning of a person name, I-PER inside it, O outside), solved historically with CRFs on hand-crafted features, then BiLSTM-CRFs, and now fine-tuned transformers such as BERT or spaCy’s pipelines, with LLM prompting for zero-shot extraction. Evaluate it with entity-level precision, recall and F1, where a prediction counts only if both the span boundaries and the type match exactly. Hard cases include ambiguity (“Jordan” as a person or country, “Apple” the company or fruit), nested entities, new names unseen in training, and domain-specific types (genes, drugs, legal clauses) that require custom training data. Entity linking is the follow-up step that maps a mention to a specific entry in a knowledge base.

Read full answer

Applies toPython 3.10+, spaCy 3.x

Common Entity Types

Type Examples
PERSON Ada Lovelace, Virat Kohli
ORG Infosys, United Nations
GPE / LOC India, Mumbai, the Himalayas
DATE / TIME 12 March 2026, tomorrow morning
MONEY / PERCENT $1 billion, 15%
PRODUCT / EVENT iPhone, Olympics

Code Example

# Requires: python -m spacy download en_core_web_sm
import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")
print([(ent.text, ent.label_) for ent in doc.ents])
# Typical output: [('Apple', 'ORG'), ('U.K.', 'GPE'), ('$1 billion', 'MONEY')]

print([(t.text, t.ent_iob_, t.ent_type_) for t in doc][:3])     # token-level BIO tags
# Typical output: [('Apple', 'B', 'ORG'), ('is', 'O', ''), ('looking', 'O', '')]

def entity_f1(gold, pred):
    gold, pred = set(gold), set(pred)
    tp = len(gold & pred)
    p = tp / len(pred) if pred else 0.0
    r = tp / len(gold) if gold else 0.0
    return round(2 * p * r / (p + r), 3) if p + r else 0.0

gold = [(0, 5, "ORG"), (27, 31, "GPE"), (44, 54, "MONEY")]          # (start, end, type) character spans
pred = [(0, 5, "ORG"), (27, 31, "LOC"), (44, 54, "MONEY")]          # one wrong type
print(entity_f1(gold, pred))
# 0.667

With one of three entities mislabelled, precision and recall are both 2/3, so entity-level F1 is 0.667.

Common Pitfalls

  • Using a general model for domain entities (drugs, legal terms) without fine-tuning.

  • Evaluating per token instead of per entity, which overstates quality.

  • Ignoring ambiguity and entity linking.

Interview Tips

  • Define named entities, give types and real applications.

  • Explain BIO tagging, model types and entity-level F1.

7

What is sentiment analysis, and where is it used?

Sentiment analysis (opinion mining) is the NLP task of determining the attitude expressed in text, most often its polarity (positive, negative or neutral), sometimes a score or star rating, and in richer forms the emotion (joy, anger, fear) or the sentiment toward specific aspects of an entity (“the battery is great but the camera is poor”, which aspect-based sentiment analysis splits into battery: positive and camera: negative). Applications include monitoring brand perception on social media, analysing product and app reviews to find what customers like or complain about, prioritizing and routing support tickets and detecting angry customers, measuring employee feedback, market research and financial signals from news and earnings calls, and moderating content. Approaches range from lexicon-based methods (word lists with polarity scores, such as VADER for social media) through classic machine learning (bag-of-words or TF-IDF with logistic regression or Naive Bayes) to fine-tuned transformers, which handle context and negation best, and LLM prompting for zero-shot labels. Evaluate with accuracy and macro-F1 on a labelled test set, checking each class, since neutral and mixed texts are often confused. The hard cases are negation (“not bad”), sarcasm and irony (“great, another delay”), comparative statements, domain-specific words (“unpredictable” is good for a film plot and bad for car steering), emojis and slang, and mixed opinions in one text.

Read full answer

Applies toPython 3.10+, scikit-learn 1.x

Approaches

Approach Pros Cons
Lexicon (e.g. VADER) No training, fast, interpretable Misses context, domain words
TF-IDF + linear model Cheap, strong baseline Weak on negation, sarcasm
Fine-tuned transformer Best accuracy, handles context Needs GPU, labelled data
LLM prompting Zero-shot, aspect-level Cost, latency, consistency

Code Example

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

train = [
    "i love this phone, the battery is great", "excellent service and friendly staff",
    "absolutely wonderful experience", "great value, highly recommend",
    "terrible battery and awful screen", "the worst service i have had",
    "very disappointing and slow", "awful experience, never again",
]
y = ["pos"] * 4 + ["neg"] * 4

clf = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)), LogisticRegression())
clf.fit(train, y)
print(clf.predict(["great staff and wonderful value", "slow and awful"]).tolist())
# ['pos', 'neg']

# Pretrained alternative (downloads a model on first use):
# from transformers import pipeline
# pipeline("sentiment-analysis")("The battery is not bad at all")
# -> [{'label': 'POSITIVE', 'score': ...}]

Bigrams (ngram_range=(1, 2)) let a linear model learn phrases such as “not good”, but only if they appear in training data; transformers handle negation far better.

Common Pitfalls

  • Removing stopwords like “not”, which reverses meaning.

  • Training on one domain (movie reviews) and applying to another (finance).

  • Reporting accuracy on imbalanced data instead of macro-F1.

  • Ignoring sarcasm and mixed sentiment.

Interview Tips

  • Define polarity, emotion and aspect-based sentiment, with applications.

  • Compare lexicon, classical ML and transformer approaches and the hard cases.

8

How does a dependency parser work, and what does it show?

A dependency parser analyses the grammatical structure of a sentence as a tree of directed, labelled links between words: every word except the root has exactly one head word that it depends on, and each link carries a relation such as nsubj (nominal subject), obj or dobj (direct object), amod (adjective modifier), det (determiner) or prep/pobj (prepositional attachment), so in “She ate the pizza” the verb “ate” is the root, “She” is its nsubj and “pizza” its object. Most practical parsers, including spaCy’s, are transition-based: they read the sentence left to right with a stack and a buffer and apply actions (SHIFT moves the next word to the stack, LEFT-ARC and RIGHT-ARC add an arc between the top items and remove the dependent), with a neural network choosing the next action from features of the current state, which gives linear-time parsing. Graph-based parsers instead score every possible head–dependent arc (often with a biaffine scorer on BiLSTM or transformer embeddings) and pick the highest-scoring tree with a maximum spanning tree algorithm, which is often a little more accurate on long-distance links. The output tells you who did what to whom, which powers relation and information extraction (subject–verb–object triples), question answering, aspect-based sentiment (which adjective modifies which noun), grammar checking and features for other models. Quality is measured by the unlabelled and labelled attachment scores (UAS, LAS), the share of words with the correct head (and label); Universal Dependencies provides consistent annotation for more than 100 languages.

Read full answer

Applies toPython 3.10+, spaCy 3.x

Transition-Based Parsing (Arc-Standard) for “She ate pizza”

Step Action Stack Buffer New arc
0 — [ROOT] She ate pizza
1 SHIFT [ROOT, She] ate pizza
2 SHIFT [ROOT, She, ate] pizza
3 LEFT-ARC(nsubj) [ROOT, ate] pizza ate → She
4 SHIFT [ROOT, ate, pizza]
5 RIGHT-ARC(obj) [ROOT, ate] ate → pizza
6 RIGHT-ARC(root) [ROOT] ROOT → ate

Code Example

# spaCy part requires: python -m spacy download en_core_web_sm
import spacy


def arc_standard(words, actions):
    stack, buffer, arcs = ["ROOT"], list(words), []
    for act in actions:
        if act == "SHIFT":
            stack.append(buffer.pop(0))
        else:
            kind, label = act.split(":")
            if kind == "LEFT":                       # top of stack is the head of the item below it
                dep = stack.pop(-2)
                arcs.append((stack[-1], label, dep))
            else:                                    # RIGHT: item below is the head of the top
                dep = stack.pop()
                arcs.append((stack[-1], label, dep))
    return arcs

print(arc_standard(["She", "ate", "pizza"],
                   ["SHIFT", "SHIFT", "LEFT:nsubj", "SHIFT", "RIGHT:obj", "RIGHT:root"]))
# [('ate', 'nsubj', 'She'), ('ate', 'obj', 'pizza'), ('ROOT', 'root', 'ate')]

# With a trained parser
nlp = spacy.load("en_core_web_sm")
doc = nlp("The hungry student ate a large pizza.")
print([(t.text, t.dep_, t.head.text) for t in doc])
# Typical output:
# [('The', 'det', 'student'), ('hungry', 'amod', 'student'), ('student', 'nsubj', 'ate'),
#  ('ate', 'ROOT', 'ate'), ('a', 'det', 'pizza'), ('large', 'amod', 'pizza'),
#  ('pizza', 'dobj', 'ate'), ('.', 'punct', 'ate')]

triples = [(t.head.text, t.text) for t in doc if t.dep_ in {"nsubj", "dobj"}]
print(triples)
# Typical output: [('ate', 'student'), ('ate', 'pizza')]

In the parser, a neural classifier predicts each action; here the action sequence is given by hand to show the mechanics.

Common Pitfalls

  • Trusting attachments of prepositional phrases, the classic ambiguity (“ate pizza with a fork” vs “with cheese”).

  • Expecting news-quality parses on noisy text.

  • Mixing label schemes (spaCy’s English dobj vs Universal Dependencies obj).

Interview Tips

  • Explain heads, dependents and labels with an example tree.

  • Describe transition-based vs graph-based parsing and UAS/LAS.

9

What are n-grams, and how do they help language modeling?

An n-gram is a contiguous sequence of n items, usually words or characters, taken from text: unigrams are single words, bigrams are pairs (“machine learning”), trigrams are triples (“natural language processing”). In language modelling, which assigns probabilities to sequences of words, an n-gram model makes the Markov assumption that each word depends only on the previous n − 1 words, so the probability of a sentence is the product of conditional probabilities like P(w_i | w{i−1}) for a bigram model, each estimated from corpus counts as count(w{i−1}, w_i) / count(w_{i−1}). This makes language modelling tractable and gives useful tools for speech recognition, spelling correction, autocomplete, machine translation (in the statistical era) and text generation. Its weaknesses are sparsity and short context: most possible n-grams never appear in training, so unseen sequences get probability zero unless you apply smoothing (add-one/Laplace, Good–Turing, Kneser–Ney) or backoff and interpolation to lower orders; the number of possible n-grams grows exponentially with n, so practical models rarely go beyond 5-grams; and the fixed window can’t capture long-range dependencies. Language models are compared with perplexity, the exponentiated average negative log-likelihood per word, where lower is better. Neural language models and transformers replaced n-gram models for generation, but n-grams remain everywhere as features (bigrams in TF-IDF classifiers capture phrases like “not good”), in character n-gram models for language identification and fastText embeddings, in BLEU and ROUGE evaluation metrics, and in deduplication.

Read full answer

Applies toPython 3.10+

Example for “I like NLP”

n n-grams
1 I, like, NLP
2 (I, like), (like, NLP)
3 (I, like, NLP)

Code Example

import math
from collections import Counter

corpus = ["i like nlp", "i like python", "i love nlp"]
sents = [["<s>"] + s.split() + ["</s>"] for s in corpus]

unigrams = Counter(w for s in sents for w in s)
bigrams = Counter((a, b) for s in sents for a, b in zip(s, s[1:]))

def p_mle(prev, word):
    return bigrams[(prev, word)] / unigrams[prev]

print(round(p_mle("i", "like"), 3), p_mle("like", "nlp"))
# 0.667 0.5

def sentence_prob(sentence, prob):
    toks = ["<s>"] + sentence.split() + ["</s>"]
    return math.prod(prob(a, b) for a, b in zip(toks, toks[1:]))

print(round(sentence_prob("i like nlp", p_mle), 3), sentence_prob("i love python", p_mle))
# 0.333 0.0   (unseen bigram "love python" gives zero without smoothing)

V = len(unigrams) - 1                              # vocabulary size without <s>, which is never predicted

def p_laplace(prev, word):
    return (bigrams[(prev, word)] + 1) / (unigrams[prev] + V)

print(sentence_prob("i love python", p_laplace) > 0)
# True

def perplexity(sentence, prob):
    toks = ["<s>"] + sentence.split() + ["</s>"]
    logs = [math.log(prob(a, b)) for a, b in zip(toks, toks[1:])]
    return math.exp(-sum(logs) / len(logs))

print(round(perplexity("i like nlp", p_mle), 3))
# 1.316

P(i like nlp) = P(i|<s>) × P(like|i) × P(nlp|like) × P(</s>|nlp) = 1 × 2/3 × 1/2 × 1 = 1/3, and the perplexity over its four predictions is (1/3)^(−1/4) ≈ 1.316.

Common Pitfalls

  • Zero probabilities for unseen n-grams without smoothing.

  • Using large n with little data (extreme sparsity).

  • Forgetting sentence boundary tokens.

  • Comparing perplexities computed with different vocabularies.

Interview Tips

  • Define n-grams and the Markov assumption with the count-ratio estimate.

  • Discuss smoothing, perplexity, limits and current uses as features and metrics.

10

What is the bag-of-words model, and what are its limitations?

The bag-of-words (BoW) model represents a document as a vector of word counts (or binary presence, or TF-IDF weights) over a fixed vocabulary, ignoring grammar and word order: you tokenize the corpus, build the vocabulary of distinct words, and for each document record how often each vocabulary word occurs, giving a sparse document–term matrix with one row per document and one column per word. It is simple, fast, interpretable and surprisingly effective for document-level tasks such as spam filtering, topic classification, sentiment baselines, search and clustering, especially combined with a linear model or Naive Bayes, and scikit-learn implements it with CountVectorizer (and TfidfVectorizer for weighted counts). Its limitations follow from what it throws away and how it represents words. Word order is lost, so “dog bites man” and “man bites dog” get identical vectors, and negation and context (“not good”) are poorly captured unless n-grams are added. There is no notion of meaning: synonyms (“car” and “automobile”) are unrelated dimensions, and polysemous words (“bank”) get one column for every sense. Vectors are high-dimensional and sparse (tens of thousands of columns), raw counts favour frequent words and long documents (hence TF-IDF and normalization), and words not seen during fitting are simply ignored at prediction time. Dense word embeddings and contextual transformer representations address these issues, but BoW remains a strong, cheap baseline.

Read full answer

Applies toPython 3.10+, scikit-learn 1.x

Strengths and Limitations

Strengths Limitations
Simple, fast, interpretable Ignores word order and syntax
Strong baseline for classification No semantics: synonyms unrelated
Works with linear models, Naive Bayes Sparse, very high-dimensional
Easy to extend with n-grams, TF-IDF Out-of-vocabulary words ignored
Raw counts biased toward frequent words, long documents

Code Example

from sklearn.feature_extraction.text import CountVectorizer

docs = ["the cat sat", "the cat sat on the mat", "the dog sat"]
vec = CountVectorizer()
X = vec.fit_transform(docs)                        # sparse document-term matrix
print(vec.get_feature_names_out().tolist())
# ['cat', 'dog', 'mat', 'on', 'sat', 'the']
print(X.toarray().tolist())
# [[1, 0, 0, 0, 1, 1], [1, 0, 1, 1, 1, 2], [0, 1, 0, 0, 1, 1]]

a, b = vec.transform(["dog bites man", "man bites dog"]).toarray()
print((a == b).all(), a.tolist())                  # word order lost; unseen words ignored
# True [0, 1, 0, 0, 0, 0]

bigram_vec = CountVectorizer(ngram_range=(1, 2))
bigram_vec.fit(["not good", "good"])
print(bigram_vec.get_feature_names_out().tolist())  # bigrams recover some local order
# ['good', 'not', 'not good']

syn = CountVectorizer().fit(["car", "automobile"])
print(syn.transform(["car"]).toarray().tolist(), syn.transform(["automobile"]).toarray().tolist())
# [[0, 1]] [[1, 0]]   (synonyms share no dimension)

Common Pitfalls

  • Fitting the vocabulary on test data (leakage).

  • Huge vocabularies without min_df, max_df or max_features.

  • Expecting BoW to handle negation, word order or synonyms.

Interview Tips

  • Explain vocabulary, counts and the document–term matrix with an example.

  • List limitations and the fixes: n-grams, TF-IDF, embeddings, transformers.

11

How is the Naive Bayes classifier used in NLP?

Naive Bayes classifies a document by applying Bayes’ theorem, P(class | document) ∝ P(class) × Π P(word | class), with the “naive” assumption that words are conditionally independent given the class; it estimates the prior from class frequencies and each word’s likelihood from how often it appears in documents of that class, then predicts the class with the highest posterior. In NLP it is a classic, very fast baseline for text classification: spam filtering (its most famous use), sentiment analysis, topic and news categorization, language identification and intent detection. The usual variant for word counts is Multinomial Naive Bayes; Bernoulli Naive Bayes models word presence or absence and suits short texts, and Complement Naive Bayes is more robust on imbalanced classes. Two practical details are essential: Laplace (add-α) smoothing, so a word never seen with a class doesn’t force that class’s probability to zero, and computing in log space (summing log probabilities) to avoid numerical underflow with long documents. Strengths are training in a single pass over the data, good accuracy with little training data, high-dimensional sparse features handled well, and interpretability (the most indicative words per class are easy to list). Weaknesses are the independence assumption (which ignores word order and correlated words, and makes predicted probabilities poorly calibrated, often extreme), and lower accuracy than logistic regression, SVMs or transformers when data is plentiful.

Read full answer

Applies toPython 3.10+, scikit-learn 1.x

Worked Example (Laplace Smoothing, α = 1)

Training: spam = “free prize”, “free money”; ham = “project meeting”, “money meeting”. Vocabulary of 5 words; each class has 4 word tokens, so P(w | c) = (count + 1) / (4 + 5).

Word P(word given spam) P(word given ham)
free 3/9 1/9
money 2/9 2/9
meeting 1/9 3/9

For “free money”: spam ∝ 0.5 × 3/9 × 2/9, ham ∝ 0.5 × 1/9 × 2/9, so P(spam) = 6 / (6 + 2) = 0.75.

Code Example

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

docs = ["free prize", "free money", "project meeting", "money meeting"]
y = ["spam", "spam", "ham", "ham"]

vec = CountVectorizer()
X = vec.fit_transform(docs)
nb = MultinomialNB(alpha=1.0).fit(X, y)

test = vec.transform(["free money"])
print(nb.classes_.tolist(), nb.predict_proba(test).round(3).tolist(), nb.predict(test).tolist())
# ['ham', 'spam'] [[0.25, 0.75]] ['spam']

vocab = vec.get_feature_names_out()
log_ratio = nb.feature_log_prob_[1] - nb.feature_log_prob_[0]      # spam vs ham
print([vocab[i] for i in log_ratio.argsort()[::-1][:2]])            # most spam-indicative words
# ['free', 'prize']

“prize” comes second: its likelihood ratio is (2/9) / (1/9) = 2, against 3 for “free”.

Common Pitfalls

  • No smoothing, so unseen words zero out a class.

  • Multiplying many small probabilities instead of summing logs.

  • Trusting predicted probabilities, which are badly calibrated.

  • Using TF-IDF features without checking (MultinomialNB expects count-like, non-negative features; it works with TF-IDF in practice but the probabilistic interpretation weakens).

Interview Tips

  • Write the Bayes formula with the independence assumption and Laplace smoothing.

  • Name variants and when NB is a good baseline.

12

How are Hidden Markov Models used in NLP?

A Hidden Markov Model (HMM) models a sequence of observed tokens as being generated by a sequence of hidden states, where each state depends only on the previous one (transition probabilities) and each word depends only on the current state (emission probabilities); in NLP, the hidden states are typically labels such as part-of-speech tags or named-entity tags, and the observations are the words. An HMM is defined by initial state probabilities, a transition matrix P(tag_t | tag_{t−1}) and an emission matrix P(word | tag), all of which can be estimated by counting in a labelled corpus (with smoothing), or learned from unlabelled text with the Baum–Welch (expectation–maximization) algorithm. Three algorithms answer the standard questions: the forward algorithm computes the probability of a sentence, the Viterbi algorithm finds the single most likely tag sequence using dynamic programming in O(T × N²) time, and forward–backward gives per-position tag probabilities. Classic NLP applications are POS tagging (HMM taggers reached about 96 percent accuracy on English), named entity recognition and chunking, speech recognition (where HMMs with Gaussian emissions dominated for decades), word segmentation and simple information extraction. Their limitations are the strong independence assumptions: a word depends only on its tag, so rich overlapping features (capitalization, suffixes, neighbouring words) can’t be used easily. Discriminative models such as CRFs, and later BiLSTM-CRFs and transformers, replaced HMMs for most tasks, but HMMs remain a clear way to understand sequence labelling.

Read full answer

Applies toPython 3.10+, NumPy 2.x

HMM Components for POS Tagging

Component Meaning Example
States Tags NOUN, VERB
Observations Words “fish”, “sleep”
Initial π P(first tag) P(NOUN) = 0.6
Transition A P(tag_t given tag_{t−1}) P(VERB given NOUN) = 0.7
Emission B P(word given tag) P(“fish” given NOUN) = 0.7
Viterbi Best tag sequence NOUN VERB

Code Example: Viterbi Decoding

import numpy as np

states = ["N", "V"]
vocab = {"fish": 0, "sleep": 1}
pi = np.array([0.6, 0.4])
A = np.array([[0.3, 0.7],          # from N to N, V
              [0.6, 0.4]])         # from V to N, V
B = np.array([[0.7, 0.3],          # N emits fish, sleep
              [0.4, 0.6]])         # V emits fish, sleep

def viterbi(words):
    obs = [vocab[w] for w in words]
    delta = pi * B[:, obs[0]]
    back = []
    for o in obs[1:]:
        scores = delta[:, None] * A              # previous state x next state
        back.append(scores.argmax(axis=0))
        delta = scores.max(axis=0) * B[:, o]
    path = [int(delta.argmax())]
    for bp in reversed(back):
        path.append(int(bp[path[-1]]))
    return [states[s] for s in reversed(path)], float(delta.max())

tags, p = viterbi(["fish", "sleep"])
print(tags, round(p, 4))
# ['N', 'V'] 0.1764

Step 1: δ(N) = 0.6 × 0.7 = 0.42 and δ(V) = 0.4 × 0.4 = 0.16. Step 2: the best way to reach V is from N, 0.42 × 0.7 = 0.294, times P(“sleep” | V) = 0.6, giving 0.1764, larger than the best score for N (0.42 × 0.3 × 0.3 = 0.0378).

Common Pitfalls

  • Unseen words with zero emission probability; smooth or use suffix-based unknown-word models.

  • Underflow from multiplying probabilities over long sentences; use log probabilities.

  • Expecting HMMs to use rich features; that is what CRFs are for.

Interview Tips

  • Define states, observations, transition and emission probabilities.

  • Explain Viterbi, applications such as POS tagging, and why CRFs and neural models replaced HMMs.

13

What role do SVMs play in text classification?

Support Vector Machines find the hyperplane that separates classes with the largest margin, and they became one of the strongest classic text classifiers because text represented as TF-IDF vectors is very high-dimensional and sparse, a setting where classes are often close to linearly separable and a maximum-margin linear model generalizes well even with tens of thousands of features and modest data. In practice you combine TfidfVectorizer (often with word bigrams or character n-grams) and a linear SVM (LinearSVC or SGDClassifier(loss="hinge")), tune the regularization parameter C (smaller C means a wider margin and more regularization), and use one-vs-rest for multiple classes. Applications include topic and news categorization, spam detection, sentiment analysis, intent classification, language identification and authorship attribution. Linear kernels are almost always preferred for text; non-linear kernels like RBF are expensive on large sparse data and rarely help. Strengths are high accuracy for a classic method, robustness to irrelevant features, fast training and prediction with linear solvers, and interpretable weights per word. Limitations: SVMs produce decision scores, not probabilities (wrap them with CalibratedClassifierCV if you need probabilities), they depend on bag-of-words features with no semantics or long-range context, and they need retraining for new domains. Compared with Naive Bayes they usually win when there is more data; compared with fine-tuned transformers they are less accurate but far cheaper and remain an excellent baseline.

Read full answer

Applies toPython 3.10+, scikit-learn 1.x

Why SVMs Suit Text

Property of text SVM advantage
Very high-dimensional Margin maximization regularizes well
Sparse vectors Linear solvers are fast
Often nearly linearly separable Linear kernel suffices
Many irrelevant words Robust; few support vectors matter

Code Example

from sklearn.calibration import CalibratedClassifierCV
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import make_pipeline
from sklearn.svm import LinearSVC

texts = [
    "the team won the football match", "a great goal in the final minute",
    "the striker scored twice in the league", "fans cheered the cricket win",
    "the new laptop has a faster processor", "update your phone software today",
    "the app crashed after the update", "a cheaper graphics card for gaming pcs",
]
labels = ["sports"] * 4 + ["tech"] * 4

svm = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)), LinearSVC(C=1.0))
svm.fit(texts, labels)
print(svm.predict(["the striker scored a late goal", "my laptop software crashed"]).tolist())
# ['sports', 'tech']

scores = svm.decision_function(["a football match on my phone"])
print(scores.shape)                                  # a signed distance, not a probability
# (1,)

calibrated = make_pipeline(TfidfVectorizer(), CalibratedClassifierCV(LinearSVC(), cv=2))
calibrated.fit(texts, labels)
print(calibrated.predict_proba(["the team won"]).shape)
# (1, 2)

Common Pitfalls

  • Using RBF kernels on large sparse text.

  • Treating decision scores as probabilities.

  • Not tuning C, or fitting the vectorizer on all data before cross-validation.

Interview Tips

  • Explain maximum margin and why linear SVMs fit sparse, high-dimensional TF-IDF.

  • Mention C, one-vs-rest, calibration and comparison with NB and transformers.

14

What are the advantages of random forests in NLP?

A random forest is an ensemble of decision trees, each trained on a bootstrap sample of the data and choosing splits from a random subset of features, with predictions combined by voting; in NLP it is useful mainly when text is turned into a moderate number of informative features, and its advantages are robustness to overfitting compared with a single tree, handling of non-linear interactions between features, little need for feature scaling, built-in feature importance and out-of-bag error estimates, and the ability to mix text features with numeric and categorical metadata. That last point is the most common practical reason to use one: a model that predicts ticket priority or fake reviews from TF-IDF or embedding features plus the review length, star rating, account age and number of links. Forests are also strong on dense, low-dimensional text features such as sentence-embedding vectors, topic proportions from LDA, readability scores or handcrafted counts, and they are parallelizable and easy to use. Their weaknesses on raw text are real: with tens of thousands of sparse TF-IDF columns, random feature subsets rarely contain the few informative words, trees grow large and slow, and linear models (logistic regression, linear SVMs) or Naive Bayes usually match or beat them at a fraction of the cost; gradient-boosted trees (XGBoost, LightGBM) also tend to beat random forests on the same features. Feature importances from impurity are biased toward high-cardinality features, so check permutation importance before drawing conclusions.

Read full answer

Applies toPython 3.10+, scikit-learn 1.x

Advantages and Limitations

Advantages Limitations
Captures non-linear feature interactions Weak on very sparse, high-dimensional BoW
Robust, little tuning, no scaling needed Slower and larger than linear models
Mixes text and metadata features No word order or semantics by itself
Feature importance, out-of-bag score Impurity importance can mislead
Parallel training Usually beaten by boosting or transformers

Code Example: Text Plus Metadata

import numpy as np
from scipy.sparse import hstack
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_extraction.text import TfidfVectorizer

reviews = [
    "best product ever buy now", "amazing deal click link buy", "buy now limited offer best",
    "solid build, battery lasts two days", "works fine after a month of use",
    "the strap broke but support replaced it",
]
links = np.array([[3], [4], [2], [0], [0], [0]])         # number of links in the review
account_age_days = np.array([[1], [2], [1], [400], [250], [900]])
y = np.array([1, 1, 1, 0, 0, 0])                         # 1 = fake review

tfidf = TfidfVectorizer(max_features=50)
X = hstack([tfidf.fit_transform(reviews), links, account_age_days]).tocsr()

rf = RandomForestClassifier(n_estimators=200, random_state=42, oob_score=True)
rf.fit(X, y)
print(X.shape[1] == len(tfidf.vocabulary_) + 2, round(float(rf.feature_importances_.sum()), 6))
# True 1.0

new = hstack([tfidf.transform(["buy now best deal"]), np.array([[5]]), np.array([[3]])]).tocsr()
print(rf.predict(new).tolist())
# [1]

With six examples this is only a demonstration; real use needs hundreds of labelled examples and cross-validation.

Common Pitfalls

  • Feeding 50,000 sparse TF-IDF columns and expecting better results than a linear model.

  • Reading impurity-based importances as causal.

  • Leakage when metadata encodes the label (for example a “flagged” field).

Interview Tips

  • Explain bagging plus random feature subsets and the resulting robustness.

  • Emphasize mixed text + metadata features and the limits on sparse text.

15

How are decision trees used for NLP problems?

A decision tree classifies text by asking a sequence of yes/no questions about features, such as “does the document contain the word ‘refund’?” or “is the TF-IDF weight of ‘free’ above 0.3?”, choosing at each node the feature and threshold that best separate the classes (by Gini impurity or information gain), and assigning the majority label at each leaf. In NLP they are used for small, interpretable classifiers (routing support tickets, flagging spam or abusive messages, intent detection with keyword features), as rule-discovery tools whose paths read like if–then rules that domain experts can check, for tasks with a few handcrafted linguistic features (capitalization, suffixes, punctuation, sentence length, presence of negation) such as sentence boundary detection or simple word-sense disambiguation, and above all as the building blocks of random forests and gradient-boosted trees. Their advantages are transparency, no need for feature scaling and the ability to model interactions between features. Their weaknesses on text are significant: a single deep tree overfits easily on thousands of sparse word features (memorizing rare words), it is unstable (small data changes give a different tree), it can only use a few words per decision path, and it has no notion of meaning or word order. Limit depth with max_depth or min_samples_leaf, use feature selection or binary keyword features, and prefer ensembles or linear models for accuracy.

Read full answer

Applies toPython 3.10+, scikit-learn 1.x

Pros and Cons for Text

Pros Cons
Readable rules Overfits sparse, high-dimensional features
Handles feature interactions Unstable to small data changes
No scaling needed Uses only a few words per path
Fast prediction Usually less accurate than linear models
Basis for forests and boosting No semantics

Code Example

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.tree import DecisionTreeClassifier, export_text

docs = ["free prize", "free cash", "free trip", "prize ceremony", "cash report", "trip plan"]
y = ["spam", "spam", "spam", "ham", "ham", "ham"]

vec = CountVectorizer(binary=True)
X = vec.fit_transform(docs)
tree = DecisionTreeClassifier(max_depth=3, random_state=42).fit(X, y)
print(export_text(tree, feature_names=list(vec.get_feature_names_out())))
# |--- free <= 0.50
# |   |--- class: ham
# |--- free >  0.50
# |   |--- class: spam

print(tree.predict(vec.transform(["free trip to win", "trip report"])).tolist())
# ['spam', 'ham']

“free” is the only word that appears in every spam message and no ham message, so a single split separates the classes perfectly and the tree stops at depth 1.

Common Pitfalls

  • Unlimited depth on bag-of-words features (memorization).

  • Trusting one tree’s rules as stable truths.

  • Ignoring that unseen words at prediction time are invisible to the tree.

Interview Tips

  • Explain splits on word features and the interpretability benefit.

  • Discuss overfitting on sparse text and the move to forests, boosting or linear models.

16

What are word embeddings, and why do they matter in NLP?

Word embeddings are dense, low-dimensional vectors (typically 100 to 1,000 numbers) that represent words so that words used in similar contexts get similar vectors, following the distributional hypothesis that “a word is known by the company it keeps”; they replace one-hot or bag-of-words representations, where every word is an independent dimension and “car” and “automobile” look completely unrelated. Static embeddings such as word2vec (predicting words from their neighbours, with skip-gram or CBOW), GloVe (factorizing global co-occurrence statistics) and fastText (adding character n-grams so rare and misspelled words get vectors) give one vector per word, and their geometry captures meaning: cosine similarity measures relatedness, and vector arithmetic captures some relations, the famous example being king − man + woman ≈ queen. They matter because they let models generalize across synonyms and related words, shrink input dimensionality, transfer knowledge learned from huge unlabelled corpora to small labelled tasks, and became the input layer of neural NLP models. Their main limitation is one vector per word regardless of context (“bank” of a river vs a bank account), which contextual embeddings from transformers such as BERT solve by computing a different vector for each occurrence; sentence-embedding models (for example Sentence-BERT) extend the idea to whole sentences for semantic search and clustering. Embeddings also absorb social biases from their training text (gender stereotypes in occupation words), which must be measured and mitigated.

Read full answer

Applies toPython 3.10+, NumPy 2.x

Types

Type Examples Vector per
One-hot / BoW CountVectorizer Word (sparse, no meaning)
Static embeddings word2vec, GloVe, fastText Word type
Contextual embeddings BERT, RoBERTa, GPT hidden states Word occurrence
Sentence embeddings Sentence-BERT, E5, OpenAI/Cohere embeddings Sentence or passage

Code Example (Toy Vectors)

import numpy as np

# Toy 3-D embeddings with dimensions roughly meaning (royal, male, person)
emb = {
    "king":  np.array([1.0, 1.0, 1.0]),
    "queen": np.array([1.0, 0.0, 1.0]),
    "man":   np.array([0.0, 1.0, 1.0]),
    "woman": np.array([0.0, 0.0, 1.0]),
    "apple": np.array([0.1, 0.0, 0.0]),
}

def cos(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

print(round(cos(emb["king"], emb["queen"]), 3), round(cos(emb["king"], emb["apple"]), 3))
# 0.816 0.577

target = emb["king"] - emb["man"] + emb["woman"]
best = max((w for w in emb if w not in {"king", "man", "woman"}), key=lambda w: cos(target, emb[w]))
print(best)
# queen

one_hot_car, one_hot_automobile = np.eye(2)
print(cos(one_hot_car, one_hot_automobile))         # one-hot vectors: every pair of words is unrelated
# 0.0

Real embeddings are learned, for example with gensim.models.Word2Vec(sentences, vector_size=100) or downloaded pretrained GloVe or fastText vectors; analogy arithmetic works only approximately on them.

Common Pitfalls

  • Using static embeddings where word sense depends on context.

  • Out-of-vocabulary words with word2vec/GloVe (fastText handles them better).

  • Averaging word vectors for long documents and losing information.

  • Ignoring biases embedded in the vectors.

Interview Tips

  • Define embeddings via the distributional hypothesis and contrast with one-hot.

  • Name word2vec, GloVe, fastText and contextual embeddings, and mention bias.

17

How do RNNs work, and how are they used in NLP?

A recurrent neural network processes a sequence one token at a time while carrying a hidden state that summarizes everything seen so far: at step t it combines the current input embedding x_t with the previous hidden state h{t−1}, h_t = tanh(W_x x_t + W_h h{t−1} + b), using the same weights at every position, so it handles sequences of any length. For NLP, tokens are mapped to embeddings, the RNN produces a hidden state per token, and different heads use those states: a classifier on the final (or pooled) state for sentence-level tasks like sentiment or intent classification (many-to-one), a per-token classifier for sequence labelling like POS tagging and NER (many-to-many, often with a bidirectional RNN that reads in both directions and a CRF on top), a next-token softmax for language modelling and text generation, and encoder–decoder (seq2seq) pairs for machine translation and summarization, which with attention were the state of the art around 2015–2017. Training uses backpropagation through time. Vanilla RNNs suffer from vanishing and exploding gradients, which makes long-range dependencies hard to learn, so in practice the gated variants LSTM and GRU are used, with gradient clipping. RNNs process tokens sequentially, which prevents parallelization over time and limits speed on GPUs, and they struggle with very long contexts; transformers replaced them for most NLP because self-attention parallelizes across positions and connects distant tokens directly. RNNs remain useful for streaming input, on-device models with small memory, and some time-series problems.

Read full answer

Applies toPython 3.10+, PyTorch 2.x

RNN vs 1-D CNN vs Transformer

RNN / LSTM 1-D CNN Transformer
Processing Sequential Parallel windows Parallel, all pairs
Long-range context Weak (better with LSTM) Limited by receptive field Direct via attention
Training speed Slow Fast Fast on GPUs
Memory for long inputs Constant per step Linear Quadratic in length
Best fit today Streaming, small models Fast local features Most NLP tasks

Code Example

import torch
from torch import nn

class RNNClassifier(nn.Module):
    def __init__(self, vocab_size=1000, emb=32, hidden=64, classes=2):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, emb, padding_idx=0)
        self.rnn = nn.RNN(emb, hidden, batch_first=True)
        self.head = nn.Linear(hidden, classes)

    def forward(self, ids):
        out, h_n = self.rnn(self.embed(ids))     # out: every step, h_n: last hidden state
        return self.head(h_n[-1])

torch.manual_seed(42)
ids = torch.randint(1, 1000, (4, 12))            # batch of 4 sentences, 12 token ids each
model = RNNClassifier()
logits = model(ids)
out, h_n = model.rnn(model.embed(ids))
print(out.shape, h_n.shape, logits.shape)
# torch.Size([4, 12, 64]) torch.Size([1, 4, 64]) torch.Size([4, 2])
print(torch.allclose(out[:, -1], h_n[0]))        # last output equals the final hidden state
# True

tagger = nn.Linear(64, 17)                        # per-token tags (for example 17 UPOS tags)
print(tagger(out).shape)
# torch.Size([4, 12, 17])

For padded batches, use nn.utils.rnn.pack_padded_sequence so padding doesn’t affect the final state.

Common Pitfalls

  • Vanilla RNNs on long sequences (use LSTM/GRU and gradient clipping).

  • Taking the last time step of padded sequences as the sentence representation.

  • Expecting RNNs to beat pretrained transformers on standard NLP tasks.

Interview Tips

  • Write the recurrence and show many-to-one, many-to-many and seq2seq uses.

  • Explain vanishing gradients and why transformers replaced RNNs, with the table.

18

How do LSTM networks work, and when would you use them?

A Long Short-Term Memory network is a recurrent network designed to remember information over long sequences: besides the hidden state h_t, each step keeps a cell state c_t, a memory line that is updated additively and controlled by three learned gates with sigmoid outputs between 0 and 1: the forget gate decides how much of the old cell state to keep, the input gate decides how much of a new candidate (computed with tanh from the current input and previous hidden state) to write, and the output gate decides how much of the cell state to expose as the new hidden state. In equations, f = σ(W_f[h{t−1}, x_t] + b_f), i = σ(W_i[…] + b_i), c̃ = tanh(W_c[…] + b_c), c_t = f ⊙ c{t−1} + i ⊙ c̃, o = σ(W_o[…] + b_o), and h_t = o ⊙ tanh(c_t). Because the cell state changes through multiplication by gates close to 1 and addition rather than repeated squashing, gradients can flow across many steps, which largely solves the vanishing-gradient problem of vanilla RNNs. The GRU is a simpler variant with two gates and no separate cell state that often performs similarly. LSTMs were the workhorse of NLP before transformers: bidirectional LSTMs (often with a CRF layer) for NER and POS tagging, LSTM language models, seq2seq translation with attention, and sentiment classifiers, and they powered speech recognition and handwriting recognition. Use them today when inputs arrive as a stream and you need constant memory per step, for small on-device models, for time series and sensor data, or when sequences are long but data is too small for a transformer; for most text tasks, a fine-tuned pretrained transformer is more accurate.

Read full answer

Applies toPython 3.10+, PyTorch 2.x

Gates

Gate Formula Role
Forget f σ(W_f [h, x] + b) Keep or erase old memory
Input i σ(W_i [h, x] + b) How much new info to write
Candidate c̃ tanh(W_c [h, x] + b) New content
Cell c_t f ⊙ c_{t−1} + i ⊙ c̃ Long-term memory
Output o σ(W_o [h, x] + b) What to expose
Hidden h_t o ⊙ tanh(c_t) Short-term output

Code Example

import torch
from torch import nn

torch.manual_seed(42)
embed = nn.Embedding(5000, 64, padding_idx=0)
lstm = nn.LSTM(64, 128, num_layers=2, batch_first=True, bidirectional=True, dropout=0.2)

ids = torch.randint(1, 5000, (8, 20))                  # 8 sentences, 20 tokens
out, (h_n, c_n) = lstm(embed(ids))
print(out.shape, h_n.shape, c_n.shape)
# torch.Size([8, 20, 256]) torch.Size([4, 8, 128]) torch.Size([4, 8, 128])

sentence_vec = torch.cat([h_n[-2], h_n[-1]], dim=1)    # last layer, forward + backward
print(sentence_vec.shape)
# torch.Size([8, 256])

ner_head = nn.Linear(256, 9)                           # BIO tags for 4 entity types + O
print(ner_head(out).shape)
# torch.Size([8, 20, 9])

cell = nn.LSTMCell(64, 128)                            # step by step, e.g. for streaming input
h, c = torch.zeros(1, 128), torch.zeros(1, 128)
for t in range(3):
    h, c = cell(embed(ids[:1, t]), (h, c))
print(h.shape, bool(((h > -1) & (h < 1)).all()))       # h = o * tanh(c) stays in (-1, 1)
# torch.Size([1, 128]) True

h_n has shape (num_layers × directions, batch, hidden) = (4, 8, 128); the last two entries are the top layer’s forward and backward states.

Common Pitfalls

  • Confusing out (all steps, top layer) with h_n (last step, all layers).

  • Not packing padded sequences, so padding affects the final states.

  • Forgetting gradient clipping; exploding gradients can still occur.

  • Choosing LSTMs over pretrained transformers for standard text tasks with enough data.

Interview Tips

  • Explain the cell state and the three gates with equations.

  • Say why gradients flow better, mention GRU and give current use cases.

19

What are the benefits of attention mechanisms in NLP?

Attention lets a model compute, for each position it is processing, a weighted combination of all other positions, where the weights are learned relevance scores; instead of squeezing a whole sentence into one fixed-size vector, the model looks back directly at the parts of the input that matter for the current step. In scaled dot-product attention, each token produces a query, key and value vector; the scores are the dot products of a query with every key divided by √d_k, a softmax turns them into weights that sum to 1, and the output is the weighted sum of values: Attention(Q, K, V) = softmax(QKᵀ / √d_k) V. The benefits are: long-range dependencies are handled directly, since any two tokens are one step apart regardless of distance; there is no fixed-length bottleneck, which made attention-based seq2seq translation much better on long sentences (Bahdanau et al., 2014); self-attention computes all positions in parallel, which trains far faster on GPUs than recurrent networks and enabled the transformer and large pretrained models; multi-head attention lets different heads capture different relations (syntax, coreference, position); representations become contextual, so the same word gets different vectors in different sentences; and the weights offer some insight into what the model used, though attention weights are not a reliable explanation on their own. The main cost is that self-attention is quadratic in sequence length in time and memory, which motivates efficient variants such as FlashAttention (exact but memory-efficient), sparse and sliding-window attention, and linear-attention approximations.

Read full answer

Applies toPython 3.10+, NumPy 2.x

Benefits

Benefit Why
Long-range dependencies Direct connections between all positions
No bottleneck Decoder attends to every encoder state
Parallel training No step-by-step recurrence
Contextual representations Each token mixes information from the whole sequence
Multi-head flexibility Heads specialize in different relations
Some interpretability Inspect weights (with caution)

Code Example

import numpy as np

def softmax(x, axis=-1):
    e = np.exp(x - x.max(axis=axis, keepdims=True))
    return e / e.sum(axis=axis, keepdims=True)

def attention(Q, K, V, mask=None):
    scores = Q @ K.T / np.sqrt(K.shape[-1])
    if mask is not None:
        scores = np.where(mask, scores, -1e9)
    w = softmax(scores)
    return w @ V, w

Q = np.array([[1.0, 0.0]])
K = np.array([[1.0, 0.0], [0.0, 1.0]])
V = np.array([[10.0, 0.0], [0.0, 10.0]])
out, w = attention(Q, K, V)
print(np.round(w, 4).tolist(), np.round(out, 3).tolist())
# [[0.6698, 0.3302]] [[6.698, 3.302]]

rng = np.random.default_rng(42)
X = rng.normal(size=(5, 8))                         # 5 tokens, 8-dim embeddings
Wq, Wk, Wv = (rng.normal(size=(8, 8)) for _ in range(3))
out, w = attention(X @ Wq, X @ Wk, X @ Wv)
print(out.shape, np.allclose(w.sum(axis=1), 1.0))   # self-attention: each row of weights sums to 1
# (5, 8) True

causal = np.tril(np.ones((5, 5), dtype=bool))       # decoder: no peeking at future tokens
_, wc = attention(X @ Wq, X @ Wk, X @ Wv, mask=causal)
print(np.allclose(np.triu(wc, k=1), 0))
# True

The query matches the first key, so its score is 1/√2 ≈ 0.707 against 0 for the second; the softmax gives weights 0.67 and 0.33, and the output mixes the two values in that proportion.

Common Pitfalls

  • Forgetting the √d_k scaling, which makes softmax saturate.

  • Missing padding or causal masks.

  • Treating attention weights as faithful explanations.

  • Ignoring the quadratic cost for long inputs.

Interview Tips

  • Write softmax(QKᵀ/√d_k)V and explain queries, keys and values.

  • List benefits (long range, parallelism, no bottleneck) and the quadratic cost.

20

What are Transformer models like BERT and GPT capable of?

The Transformer (Vaswani et al., 2017) is a neural architecture built from stacked layers of multi-head self-attention and position-wise feed-forward networks, with residual connections, layer normalization and positional information, that processes all tokens of a sequence in parallel; BERT and GPT are two families built from it, BERT from the encoder stack and GPT from the decoder stack, and both are pretrained on huge text corpora and then adapted to tasks. BERT (Bidirectional Encoder Representations from Transformers) is pretrained with masked language modelling, predicting randomly masked tokens from context on both sides (plus next-sentence prediction in the original paper), which makes its contextual token embeddings excellent for understanding tasks: text classification, NER, extractive question answering, semantic similarity and retrieval after fine-tuning; descendants include RoBERTa, DistilBERT, DeBERTa and many sentence-embedding models. GPT (Generative Pretrained Transformer) is trained autoregressively to predict the next token with a causal mask, so each token sees only earlier tokens, which makes it a generator: at scale (GPT-3, GPT-4 and similar large language models from many labs), plus instruction tuning and reinforcement learning from human feedback, such models write and summarize text, translate, answer questions, write code, follow instructions and learn new tasks from a few examples in the prompt. Encoder–decoder transformers (T5, BART) combine both for sequence-to-sequence tasks like translation and summarization. Key practical points: inputs are split into subword tokens (WordPiece for BERT, byte-level BPE for GPT), context length is limited and attention cost grows quadratically with it, and the standard workflow is to start from a pretrained checkpoint (for example via Hugging Face Transformers) and fine-tune, use adapters such as LoRA, or prompt, rather than train from scratch. Their limitations include hallucination in generative models, biases from training data, high compute cost and the need for careful evaluation.

Read full answer

Applies toPython 3.10+, Hugging Face Transformers 4.x

BERT vs GPT

BERT GPT
Stack Encoder Decoder
Attention Bidirectional Causal (left-to-right)
Pretraining Masked language modelling Next-token prediction
Strength Understanding, embeddings Generation, instruction following
Typical use Classification, NER, extractive QA, retrieval Chat, summarization, code, few-shot tasks
Adaptation Fine-tune a task head Prompting, instruction tuning, fine-tuning
Tokenizer WordPiece Byte-level BPE

Code Example

# Requires: pip install transformers torch; models download on first use
import numpy as np
from transformers import pipeline

fill = pipeline("fill-mask", model="bert-base-uncased")
print(fill("The capital of France is [MASK].")[0]["token_str"])
# Typical output: paris

gen = pipeline("text-generation", model="gpt2")
print(gen("Machine learning is", max_new_tokens=10, do_sample=False)[0]["generated_text"][:19])
# Machine learning is

# The difference in one picture: which positions each token may attend to (4 tokens)
bidirectional = np.ones((4, 4), dtype=int)           # BERT: everyone sees everyone
causal = np.tril(np.ones((4, 4), dtype=int))         # GPT: only the past and itself
print(causal.tolist())
# [[1, 0, 0, 0], [1, 1, 0, 0], [1, 1, 1, 0], [1, 1, 1, 1]]
print(int(bidirectional.sum()), int(causal.sum()))
# 16 10

GPT-2 continues the prompt with its own text; with do_sample=False the output is deterministic for a given model version, and the generated text always starts with the prompt.

Common Pitfalls

  • Using GPT-style generation where a small fine-tuned BERT classifier is cheaper and better.

  • Exceeding the context length and silently truncating input.

  • Trusting generated facts without verification.

  • Training from scratch instead of starting from a pretrained checkpoint.

Interview Tips

  • Describe the transformer block, then contrast BERT (encoder, MLM) with GPT (decoder, next-token).

  • Give tasks for each and mention fine-tuning, LoRA, prompting and limitations.

21

What is TF-IDF in document retrieval?

TF-IDF is a weighting statistic that measures how important a term is to a document relative to an entire collection. It increases a term’s weight when it occurs frequently in one document but rarely elsewhere, helping retrieval systems rank documents by discriminative keyword overlap. Its main trade-off is that it captures lexical similarity but does not understand synonyms, context, or semantic relationships.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

A common TF-IDF formulation is:

weight(t, d) = tf(t, d) * idf(t)

  • t: the term being evaluated.

  • d: the document containing the term.

  • tf(t, d): the term’s frequency in the document, optionally normalized by document length.

  • idf(t): the term’s inverse document frequency.

  • N: the total number of documents.

  • df(t): the number of documents containing the term.

A common inverse document frequency is log(N / df(t)). Scikit-learn uses the smoothed version:

idf(t) = log((1 + N) / (1 + df(t))) + 1

The smoothing prevents zero weights and avoids division-by-zero errors. Terms appearing in nearly every document receive a low IDF weight, while rare terms receive a high weight.

The classic NLP pipeline usually performs tokenization, normalization, stop-word handling, stemming or lemmatization, and n-gram construction before creating a bag-of-words representation. TF-IDF assigns weights to those resulting features:

  • Stemming truncates words to a base form and is fast but approximate.

  • Lemmatization maps words to dictionary-based forms and preserves more meaning.

  • N-grams represent phrases such as “customer service” when unigrams are insufficient.

Significance in Document Retrieval

Documents and queries are represented as sparse TF-IDF vectors. A retrieval system typically computes cosine similarity:

cos(q, d) = (q · d) / (||q|| * ||d||)

Here, q and d are the query and document vectors, · denotes their dot product, and || || denotes vector magnitude. Ranking by this score favors documents containing rare query terms that also occur frequently in the document.

This works particularly well when:

  • Queries contain distinctive names, identifiers, or phrases.

  • Exact term matching is important.

  • The collection is large enough for document frequencies to be meaningful.

  • Interpretability matters because matched terms explain the ranking.

TF-IDF and Modern Retrieval

Representation How It Works Strengths Weaknesses Typical Use or Scale
TF-IDF Counts terms and weights them by corpus rarity Fast, interpretable, inexpensive No semantic matching; sparse Large keyword indexes; sparse storage proportional to non-zero entries
word2vec, GloVe, fastText Maps tokens to static dense vectors Captures some semantic similarity One fixed vector per word; weak context handling Classification, similarity, and compact feature representations
BERT-style encoders Produces contextual token representations Handles synonyms, context, and long-range dependencies Higher compute and memory cost Modern semantic retrieval; transformer self-attention is roughly O(L²d) per layer for sequence length L and hidden size d

word2vec, GloVe, and fastText generate static embeddings, whereas transformer models generate contextual embeddings. GPT-style models are primarily generative rather than conventional document encoders, but can support query expansion or answer generation. For semantic tasks, using a pretrained transformer with task-specific fine-tuning is generally preferable to relying only on TF-IDF.

Evaluation and Practical Guidance

  • Retrieval: Use Precision@k, Recall@k, mean reciprocal rank, and nDCG when documents have relevance judgments.

  • Classification: Report accuracy and macro or weighted F1; accuracy alone can be misleading with imbalanced classes.

  • Other NLP tasks: Use entity-level F1 for NER, and BLEU, ROUGE, BERTScore, plus human evaluation for generation.

  • Data leakage: Fit tokenization, vocabulary, IDF values, and other learned preprocessing on training data only. In production search indexing, fitting TF-IDF over the complete index is normally intentional because IDF is an unlabeled corpus statistic.

  • Scalability: TF-IDF remains attractive for large indexes because inverted indexes and sparse matrices are efficient, but vocabulary growth and long documents can increase memory use.

Common Pitfalls

  • Treating all terms as equally informative by ignoring TF-IDF weighting.

  • Removing stop words mechanically when phrases such as “not good” depend on them.

  • Computing IDF on validation or test data during offline evaluation.

  • Assuming high lexical similarity implies semantic equivalence.

  • Ignoring rare queries, vocabulary mismatch, multilingual content, or domain shift.

  • Treating retrieval scores as probabilities; cosine scores are ranking signals, not calibrated relevance probabilities.

  • Overlooking privacy, sensitive content, and demographic bias in documents or pretrained models.

  • Expecting transformer models to replace lexical retrieval automatically; hybrid sparse-and-dense retrieval often performs best.

22

How does Latent Semantic Analysis capture meaning in NLP?

Latent Semantic Analysis (LSA) reduces a large term-document matrix to a lower-dimensional representation in which documents with similar vocabulary can be compared even when they do not share exactly the same terms. It solves the sparsity and vocabulary-mismatch problems of basic bag-of-words models, at the cost of discarding some information and producing semantic dimensions that are difficult to interpret.

Read full answer

Applies tomachine learning concepts (LSA; examples in Python 3.12)

Core Idea

LSA starts with a matrix A containing TF-IDF values, where each row represents a term, each column represents a document, and each cell measures the term’s importance in that document. It applies Singular Value Decomposition (SVD):

A ≈ U × Σ × Vᵀ

  • U: latent dimensions associated with terms.

  • Σ: singular values, which indicate the importance or strength of each latent dimension.

  • Vᵀ: latent dimensions associated with documents.

  • k: number of retained dimensions, usually much smaller than the original vocabulary size.

Keeping only the k most important dimensions creates an approximation of the original matrix. Documents can then be represented by their coordinates in this latent space and compared with cosine similarity.

For example, “bank” might be close to both “loan” and “river” in the original vocabulary. LSA may create a latent dimension representing the financial sense of “bank,” allowing semantically related documents to have a higher similarity than documents sharing no terms.

Processing Pipeline

  1. Tokenization: Split text into terms, words, or subword tokens.

  2. Normalization: Standardize case and, when appropriate, remove punctuation or apply stemming.

  3. Stop-word handling: Remove common words such as “the” or “is.” This reduces noise but may hurt tasks where function words carry meaning.

  4. Stemming or lemmatization: Stemming is faster and more aggressive; lemmatization maps words to dictionary forms and generally preserves more meaning.

  5. Document representation: Build a bag-of-words or TF-IDF matrix. N-grams can capture phrases such as “not good,” but increase matrix dimensionality.

  6. Truncated SVD: Retain a limited number of latent dimensions.

  7. Downstream task: Use the reduced document or term vectors for classification, clustering, retrieval, or similarity.

For a sparse term-document matrix, iterative truncated-SVD methods typically require approximately O(nnz × k) work per iteration, where nnz is the number of nonzero matrix entries and k is the number of retained dimensions.

LSA and Modern NLP Methods

Approach How it works Strengths Weaknesses Typical use
Bag-of-words Counts exact terms Simple and interpretable Sparse and sensitive to wording Baseline classification
TF-IDF Weights terms by importance across documents Reduces common-term influence Still cannot match different words reliably Classical text classification
LSA Applies truncated SVD to TF-IDF Captures latent term-document relationships Static, lossy, and hard to interpret Similarity, clustering, retrieval
Word2Vec, GloVe, fastText Learn static vector embeddings Efficient comparison and transfer One vector per word; limited context Legacy embeddings and feature initialization
BERT-style models Produce contextual token embeddings Handles context, syntax, and rare words More computation and model complexity Classification, retrieval, extraction, fine-tuning
GPT-style models Generate or encode text autoregressively Strong language generation and reasoning Higher resource cost; generation risks Summarization, translation, question answering

Modern contextual embeddings such as those produced by BERT can represent the same word differently in different sentences. LSA instead derives document-level latent structure from co-occurrence patterns and does not explicitly model token context.

Practical Use and Evaluation

LSA is most useful when a lightweight, fully local method is needed for document similarity, clustering, or traditional classification. Common practice is to fit the vectorizer and dimensionality reduction only on training data, then transform validation and test data to prevent data leakage.

Evaluate LSA through the downstream task rather than reconstruction error alone:

  • Text classification: Accuracy for balanced classes; precision, recall, and F1 for imbalanced classes.

  • Retrieval or similarity: Precision at k, recall at k, and mean average precision.

  • Clustering: Silhouette score plus cluster inspection for usefulness.

For current NLP systems, using pretrained transformer models with fine-tuning is generally preferred. LSA remains valuable as an interpretable baseline and can work better than large models on small, domain-specific datasets.

Common Pitfalls

  • Choosing too many dimensions: The reduced representation may approach the original sparse matrix and lose the efficiency benefit.

  • Choosing too few dimensions: Important topics or distinctions may disappear.

  • Ignoring leakage: Fitting TF-IDF or SVD on the full dataset exposes test information to preprocessing.

  • Treating dimensions as literal concepts: LSA components are statistical directions and are rarely cleanly human-readable.

  • Using accuracy for severe class imbalance: A model can achieve high accuracy by predicting only the majority class; inspect precision, recall, F1, and PR-AUC.

  • Ignoring domain and language differences: LSA quality depends on representative documents. Domain shift, multilingual text, biased training data, and privacy-sensitive corpora require careful evaluation.

23

What are the differences between word2vec and GloVe?

Word2vec and GloVe are two widely adopted techniques for learning dense vector representations of words from a text corpus. They both produce embeddings that capture semantic and syntactic relationships, enabling downstream NLP tasks, but they differ fundamentally in their training objectives and the way they exploit corpus statistics. The main trade-off is between predictive-based learning (word2vec) that uses local context windows and count-based factorization (GloVe) that leverages global co-occurrence patterns.

Read full answer

Applies toword2vec and GloVe embedding methods (general concept)

How They Work

  • Word2vec: A neural network that learns embeddings by predicting a target word from its context (CBOW) or predicting context words from a target word (Skip-gram). It uses local context windows and stochastic gradient descent, producing embeddings that capture linear regularities (e.g., king - man + woman ≈ queen).

  • GloVe: Stands for Global Vectors; it factorizes a word-word co-occurrence matrix constructed from the entire corpus. The objective is to minimize a weighted least-squares error on the log co-occurrence counts, combining global statistics with a weighting function to handle rare and frequent words.

Key Differences

Aspect word2vec GloVe
Training objective Predict context words (Skip-gram) or predict target word from context (CBOW) Factorize word co-occurrence matrix to reconstruct log co-occurrence counts
Data usage Local context windows; each training example is a (target, context) pair Global co-occurrence statistics; uses entire matrix of word pairs
Training algorithm Stochastic gradient descent on a shallow neural network Matrix factorization (can be done with SGD or alternating least squares)
Out-of-vocabulary handling Cannot handle unseen words unless subword information is used (e.g., fastText) Cannot handle unseen words; requires a fallback (e.g., unknown token)
Analogy performance Excels at linear analogies (king - man + woman ≈ queen) due to the additive property Also captures analogies, but often slightly less sharp than Skip-gram
Complexity O(n * d * C) where n is vocabulary size, d embedding dim, C context size; memory efficient with negative sampling O(n^2) to store co-occurrence matrix; training cost O(n * d * k) with k non-zeros per row
Typical use cases When you have a large corpus and want fast training; good for downstream tasks requiring semantic similarity When you want to leverage global co-occurrence and have moderate vocabulary size; often used as a baseline

Strengths and Weaknesses

  • Word2vec strengths: Fast training, captures fine-grained semantic relationships, widely available in libraries (Gensim). Weaknesses: Sensitive to local context only, may miss global word associations, poor with rare words unless subword models are used.

  • GloVe strengths: Incorporates global co-occurrence, often performs well on word similarity tasks, less prone to overfitting on local patterns. Weaknesses: Requires building a large co-occurrence matrix, which can be memory heavy for large vocabularies; training can be slower; does not handle OOV words.

When to Use Which

  • Use word2vec (especially Skip-gram with negative sampling) when you need to train quickly on a large corpus, want to capture analogies, or have a limited memory budget. It is also the go-to choice when you plan to use subword extensions (fastText).

  • Use GloVe when you have a moderate-sized vocabulary and want to explicitly model global word co-occurrences, or when you want a simple baseline that often performs comparably to word2vec on similarity benchmarks.

Common Pitfalls

  • Ignoring preprocessing: Both methods are sensitive to lowercasing, punctuation, and stop words; inconsistent preprocessing can hurt embedding quality.

  • Using the same embedding dimension for all tasks: Larger dimensions may overfit on small corpora; choose dimensionality based on corpus size and downstream task.

  • Evaluating on in-vocabulary words only: GloVe and word2vec both fail on unseen words; always include an out-of-vocabulary strategy (e.g., character n-grams or a fallback token).

  • Confusing training objective with evaluation: A model that optimizes a different objective may still perform well on downstream tasks; always validate on a task-specific metric (e.g., F1 for classification) rather than relying solely on word similarity scores.

  • Not using pre-trained embeddings: Training from scratch requires a large corpus; when possible, start from pre-trained embeddings (e.g., GoogleNews word2vec, GloVe from Stanford) and fine-tune.

24

What challenges arise with vector space models for semantic analysis?

vector space models represent text as dense vectors in a high-dimensional space, enabling similarity computations, but they present several challenges when used for semantic analysis. These challenges can degrade the accuracy and interpretability of results if not addressed.

Read full answer

Applies tovector space models (conceptual, independent of specific library versions)

Dimensionality and the Curse of Dimensionality

  • As the number of dimensions increases, data points become increasingly sparse, making distance metrics less meaningful.

  • The volume of the space grows exponentially, requiring exponentially more data to achieve reliable density estimates.

Semantic Ambiguity and Polysemy

  • Words with multiple senses are represented by a single vector, conflating distinct meanings.

  • Synonyms may receive separate vectors, reducing the similarity score between them.

Context and Scope Limitations

  • Bag-of-words and simple TF-IDF models discard word order and syntactic relationships.

  • Static embeddings cannot capture context-dependent meanings, limiting their applicability to tasks requiring nuance.

Computational and Storage Costs

  • Training or loading large embedding matrices consumes substantial memory and processing time.

  • Storing vectors for a large vocabulary scales linearly with vocabulary size, posing deployment challenges.

Evaluation and Interpretation Challenges

  • There is no direct intrinsic metric for vector quality; evaluation typically relies on downstream task performance.

  • Interpreting vector distances requires domain expertise and can be misleading without context.

Common Pitfalls

  • Applying distance measures to high-dimensional sparse vectors without dimensionality reduction.

  • Using raw count vectors instead of normalized representations like TF-IDF or pre-trained embeddings.

  • Overlooking out-of-vocabulary words, which can cause silent failures or biased results.

  • Relying on off-the-shelf embeddings without fine-tuning for the specific domain.

  • Mistaking proximity in vector space for semantic equivalence.

25

What is semantic similarity, and how is it computed in NLP?

Semantic similarity measures how close two pieces of text (words, sentences or documents) are in meaning rather than in surface form, so “How do I reset my password?” and “I forgot my login credentials” should score high despite sharing almost no words. Computational approaches form a spectrum. Lexical overlap measures (Jaccard similarity of word sets, edit distance, BM25 or TF-IDF cosine) are fast and transparent but miss paraphrases. Knowledge-based measures use lexical resources such as WordNet, scoring word pairs by path length or information content in the taxonomy (Wu–Palmer, Lin), which works for single words but scales poorly to sentences. Distributional methods use static embeddings (word2vec, GloVe, fastText), averaged or weighted into sentence vectors, or LSA, and compare them with cosine similarity, which captures synonyms but blurs word order. Modern approaches use transformer models: bi-encoders such as Sentence-BERT and other sentence-embedding models encode each text independently into a vector, enabling fast large-scale semantic search with approximate nearest-neighbour indexes (FAISS, vector databases), while cross-encoders feed both texts into one transformer and output a score directly, which is more accurate but too slow to compare against millions of candidates, so production systems retrieve with a bi-encoder or BM25 and then re-rank the top results with a cross-encoder. Similarity models are evaluated on benchmarks such as STS-B (Spearman correlation with human ratings) and on retrieval metrics like recall@k and nDCG. Applications include semantic search, duplicate question detection, FAQ matching, clustering, plagiarism detection, recommendation and retrieval-augmented generation.

Read full answer

Applies toPython 3.10+, scikit-learn 1.x, sentence-transformers 3.x

Approaches

Approach Example Captures paraphrase? Speed
Lexical overlap Jaccard, TF-IDF cosine, BM25 No Very fast
Knowledge-based WordNet path, Wu–Palmer Words only Fast
Static embeddings Average GloVe vectors Partly Fast
Bi-encoder Sentence-BERT embeddings + cosine Yes Fast at scale
Cross-encoder Pair-scoring transformer Best Slow (re-ranking only)

Code Example

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

def jaccard(a, b):
    A, B = set(a.lower().split()), set(b.lower().split())
    return len(A & B) / len(A | B)

print(round(jaccard("the cat sat on the mat", "a cat sat on a mat"), 3))
# 0.667

pair = ["How do I reset my password?", "I forgot my login credentials"]
tfidf = TfidfVectorizer().fit(pair)
print(round(float(cosine_similarity(tfidf.transform([pair[0]]), tfidf.transform([pair[1]]))[0, 0]), 3))
# a small value: the only shared word is "my" (the one-letter "i" is dropped by the default tokenizer)

Lexical scores barely register this paraphrase. A sentence-embedding model captures it:

# Requires: pip install sentence-transformers (downloads the model on first use)
from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("all-MiniLM-L6-v2")
emb = model.encode(["How do I reset my password?", "I forgot my login credentials",
                    "What is the weather today?"], normalize_embeddings=True)
print(util.cos_sim(emb[0], emb[1]).item() > util.cos_sim(emb[0], emb[2]).item())
# True: the paraphrase scores much higher than the unrelated question

Common Pitfalls

  • Using lexical overlap where paraphrases matter.

  • Comparing cosine scores across different embedding models (scales differ).

  • Running a cross-encoder over millions of pairs instead of retrieve-then-rerank.

  • Not evaluating on in-domain pairs.

Interview Tips

  • Define semantic vs lexical similarity and walk through the approaches.

  • Explain bi-encoders vs cross-encoders and retrieve-then-rerank.

26

How do you use spaCy for NLP tasks?

spaCy is a Python library for building efficient NLP pipelines that tokenize text and extract linguistic information such as lemmas, dependencies, entities, and labels. It solves structured language-processing tasks with configurable components, trading some of the contextual flexibility of large language models for speed, control, and relatively low resource usage.

Read full answer

Applies tospaCy 3.8.x (Python 3.12)

Core Processing Pipeline

A typical spaCy pipeline performs these steps:

  1. Tokenization: nlp(text) splits text into Token objects. Tokens expose properties such as text, lemma_, pos_, dep_, is_stop, and ent_type_.

  2. Normalization: spaCy generally preserves original case, punctuation, and stop words. Lowercasing, Unicode normalization, stop-word removal, or custom token-boundary rules require preprocessing, a custom tokenizer, or a custom component.

  3. Linguistic analysis: Trained pipelines can assign part-of-speech tags, syntactic dependencies, and named entities.

  4. Custom processing: Components can add sentiment, text categories, custom entities, or application-specific fields to each Doc.

  5. Batch processing: nlp.pipe() applies the pipeline efficiently to multiple documents.

A Doc represents one document, a Token represents one word or symbol, and a Span represents a contiguous token sequence. Named entities are available through doc.ents.

Common NLP Tasks

  • Text classification and sentiment analysis: textcat and textcat_multilabel components produce category scores through doc.cats.

  • Named entity recognition: The ner component identifies entity spans such as people, organizations, locations, and dates.

  • Part-of-speech tagging: Tokens receive universal or detailed tags such as NOUN, VERB, and PROPN.

  • Question answering: spaCy has no built-in QA component. Implement a custom component or call a separate QA model and expose its selected answer span and score through your own API.

  • Summarization and translation: These generally require an additional generative or translation model rather than the standard spaCy pipeline.

  • Semantic search: Represent passages with spaCy vectors or an integrated transformer and rank them using a suitable similarity measure.

For classification, report accuracy and macro, weighted, or micro F1 as appropriate; accuracy can mislead with imbalanced classes. Evaluate NER with entity-level precision, recall, and F1. For generation, BLEU is common for translation, ROUGE for summarization, and BERTScore for semantic similarity, but each captures only part of quality, so human evaluation remains important.

Classic and Modern NLP

spaCy can support both traditional and modern NLP systems:

Approach How it works Strengths Weaknesses When to use Complexity or scale
TF-IDF and n-grams Represents word or n-gram frequencies Fast and interpretable Sparse and limited semantic context Moderate-scale classification Roughly linear in corpus text during fitting; sparse matrices
spaCy linguistic pipeline Tokenizes and extracts structured attributes Efficient and customizable Limited contextual reasoning Extraction and rule-based systems Roughly linear per document plus pipeline memory
Transformer models Produces contextual representations Strong context-sensitive understanding More compute and operational complexity Difficult classification, QA, and generation Model-dependent; cost grows with model size and sequence length

Traditional pipelines use bag-of-words, TF-IDF, and character or word n-grams. Stemming reduces words to root forms, while lemmatization maps them to dictionary-based base forms; spaCy provides lemmatization, but an external stemmer such as Snowball is needed for stemming.

word2vec, GloVe, and fastText are static embeddings: each word has one fixed vector, and fastText also represents words through character subwords. BERT produces contextual embeddings for encoding, while GPT-style models are designed for generation. spaCy does not train those architectures itself, but spacy-transformers can integrate transformer components. Many transformer tokenizers use BPE or WordPiece subword units, allowing rare words to be represented through smaller pieces.

Code Example

Install spaCy and download the small English model once:

python -m pip install "spacy>=3.8,<3.9"
python -m spacy download en_core_web_sm

Process a small corpus and print extracted linguistic information:

import spacy

# Load a pretrained pipeline; the model is downloaded separately.
nlp = spacy.load("en_core_web_sm")

texts = [
    "Apple released the new phone in San Francisco.",
    "Ada Lovelace worked in London.",
]

# Process multiple documents in batches.
for doc in nlp.pipe(texts, batch_size=8):
    print(f"Text: {doc.text}")

    for token in doc:
        print(
            f"  token={token.text!r}, lemma={token.lemma_!r}, "
            f"pos={token.pos_}, dependency={token.dep_}, stop={token.is_stop}"
        )

    for entity in doc.ents:
        print(
            f"  entity={entity.text!r}, label={entity.label_}, "
            f"start={entity.start}, end={entity.end}"
        )

    print()

Register custom components with @Language.component or add configured factories with nlp.add_pipe(). Use spacy train for supported trainable pipeline components; train custom components in a separate Python training loop or script.

Common Pitfalls

  • Data leakage: Fit TF-IDF vocabulary selection on training data only. Use a scikit-learn Pipeline when combining TF-IDF with a model.

  • Overprocessing: Removing stop words or punctuation can remove task-relevant signals.

  • Incorrect model scope: en_core_web_sm is useful for learning but may perform poorly on specialized domains or languages. Prefer pretrained models and fine-tune them on representative labeled data when needed.

  • Multilingual text and domain shift: Use models trained for the target language and domain, and evaluate separately on representative data because changes in language or writing style can reduce performance.

  • Document boundaries: nlp.pipe() processes each input string independently; combine sentences or paragraphs explicitly when document context matters.

  • Operational safeguards: Apply access controls and retention policies to private text, and evaluate demographic or domain bias before deployment.

  • Weak generation evaluation: BLEU, ROUGE, and BERTScore capture only parts of quality; human evaluation remains important.

27

What is a typical NLTK text-processing workflow in Python?

A typical NLTK text-processing workflow tokenizes and normalizes raw text, converts it into numerical features, trains a model, and evaluates its predictions. NLTK is strongest for transparent classical NLP pipelines, while pretrained transformers are the usual default for context-sensitive tasks. The main trade-off is interpretability and simplicity versus semantic accuracy, compute, and engineering complexity.

Read full answer

Applies toNLTK 3.9.1 and scikit-learn 1.5+ (Python 3.12)

Typical Pipeline

  1. Inspect the corpus

  2. Check for missing or empty documents, duplicates, class balance, and unusual document lengths.

  3. Split training, validation, and test data before fitting any preprocessing or model.

  4. Tokenize

  5. Split text into sentences, words, or subword units.

  6. NLTK provides wordpunct_tokenize, TreebankWordTokenizer, and word_tokenize.

  7. word_tokenize requires the punkt_tab resource in NLTK 3.9.1; wordpunct_tokenize requires no downloaded corpus.

  8. Normalize

  9. Usually lowercase the text, normalize Unicode, and remove unwanted whitespace, punctuation, URLs, or email addresses.

  10. Handle contractions, abbreviations, dates, and domain-specific variants consistently between training and production.

  11. Remove stop words and reduce word forms

  12. Stop-word removal discards frequent low-information words. It can help sparse models but can remove useful negation such as “not good”.

  13. Stemming is fast but approximate: “running” may become “run” and “cars” becomes “car”.

  14. Lemmatization uses vocabulary and part-of-speech rules: “better” becomes “good” when treated as an adjective. NLTK stop-word lists require the stopwords corpus.

  15. Create numerical features

  16. Bag-of-words represents documents with token counts.

  17. TF-IDF reduces the weight of terms common across the corpus.

  18. N-grams retain short sequences, such as bigrams for phrases like “not good”.

  19. Train and evaluate

  20. Put vectorization and the estimator in a scikit-learn Pipeline so fitted preprocessing sees only training folds.

  21. Use StratifiedKFold cross-validation for classification and retain an untouched test set for final evaluation.

Classic and Modern Approaches

Method How it works Strengths Weaknesses Use when Typical scale
TF-IDF or bag-of-words Sparse token or n-gram counts and weights Fast, interpretable, strong baseline Ignores word order and deep context Limited data or classical baselines Linear in corpus text size; sparse, high-dimensional features
Word2Vec, GloVe, fastText Assigns each word one fixed vector Fast lookup and useful similarity Context-insensitive; limited handling of unseen words Similarity, clustering, or initialization Roughly O(n*d) to build corpus representations
BERT or GPT-style models Produces contextual representations from subword tokens Strong semantics and broad task support Compute-intensive; possible bias and hallucinations Classification, QA, extraction, search, or generation Architecture-dependent; often benefits from a GPU

Modern tokenizers commonly use BPE or WordPiece. They represent rare or unseen words as smaller familiar units, reducing out-of-vocabulary problems. For context-sensitive production tasks, using a pretrained model with fine-tuning is generally preferable to training representations from scratch.

Tasks and Evaluation

Task Typical output Suitable evaluation
Text classification One or more labels Accuracy and macro/weighted F1
Sentiment analysis Sentiment label Precision, recall, and F1
Named entity recognition Entity text, type, and offsets Entity-level F1
Part-of-speech tagging A tag for each token Accuracy or macro F1
Question answering Answer span or generated answer Exact match, token F1, and human review
Summarization Generated summary ROUGE, BERTScore, and human evaluation
Machine translation Translated text BLEU or chrF, and human evaluation
Search or retrieval Ranked relevant documents Precision@k, recall@k, MRR, or nDCG

Accuracy can mislead when classes are imbalanced; binary classification may instead require precision, recall, F1, ROC-AUC, and PR-AUC. BLEU, ROUGE, and BERTScore are proxies that do not reliably measure factual correctness, helpfulness, or harmful bias, so generation systems also require human evaluation.

Code Example

This offline example uses NLTK tokenization and stemming with a leakage-safe TF-IDF and logistic-regression pipeline.

from nltk.stem import PorterStemmer
from nltk.tokenize import wordpunct_tokenize
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import (
    StratifiedKFold,
    cross_validate,
    train_test_split,
)
from sklearn.pipeline import Pipeline

corpus = [
    "The product works very well and is easy to use",
    "Excellent quality and quick delivery",
    "I love this product and would buy it again",
    "Fast shipping and great value",
    "Reliable and simple to use every day",
    "The build quality is excellent",
    "Great price and dependable performance",
    "Delivery was quick and the item works well",
    "I am happy with this purchase",
    "Useful product with friendly support",
    "The product broke after one day",
    "Terrible quality and poor customer service",
    "I dislike this item and would not recommend it",
    "Slow delivery and a disappointing purchase",
    "The item stopped working very soon",
    "Bad quality and unreliable performance",
    "Support was slow and not helpful",
    "Delivery was late and the package was damaged",
    "I am unhappy with this purchase",
    "Useless item with a poor experience",
]
labels = [1] * 10 + [0] * 10

X_train, X_test, y_train, y_test = train_test_split(
    corpus,
    labels,
    test_size=0.25,
    stratify=labels,
    random_state=42,
)

stemmer = PorterStemmer()

def analyze(text: str) -> list[str]:
    tokens = wordpunct_tokenize(text.lower())
    words = [token for token in tokens if token.isalpha()]
    return [stemmer.stem(word) for word in words]

model = Pipeline([
    ("tfidf", TfidfVectorizer(analyzer=analyze)),
    ("classifier", LogisticRegression(max_iter=1000, random_state=42)),
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_validate(
    model,
    X_train,
    y_train,
    cv=cv,
    scoring=("accuracy", "f1"),
)

model.fit(X_train, y_train)
predictions = model.predict(X_test)

print(f"CV accuracy: {scores['test_accuracy'].mean():.3f}")
print(f"CV F1: {scores['test_f1'].mean():.3f}")
print(classification_report(y_test, predictions, zero_division=0))

Common Pitfalls

  • Data leakage: Fit TF-IDF, stop-word selection, and feature selection only on training folds; run cross-validation only on the development split.

  • Overaggressive cleaning: Removing stop words can erase negation, names, or domain-specific meaning.

  • Wrong metrics: Accuracy alone can conceal poor minority-class performance.

  • Overfitting: Repeatedly tuning against the validation set makes its results optimistic; preserve the final test set.

  • Resource handling: NLTK tokenizers such as word_tokenize require downloaded resources, whereas wordpunct_tokenize does not.

  • Deployment concerns: Evaluate multilingual text, domain shift, demographic bias, privacy, and sensitive training data explicitly.

  • Scaling: Sparse TF-IDF features do not require standardization; dense features used by k-NN, SVM, PCA, or gradient descent generally do.

Common follow-up questions include choosing stemming versus lemmatization, preventing preprocessing leakage, selecting metrics for imbalanced data, and comparing static embeddings with contextual transformers.

28

What are the benefits of using Hugging Face Transformers?

Hugging Face Transformers is a library and model ecosystem that provides standardized access to pretrained checkpoints, tokenizers, and task pipelines for NLP and other machine-learning tasks. It reduces engineering effort by enabling transfer learning and fine-tuning instead of implementing architectures and training models from scratch. The main trade-offs are compute cost, latency, and operational risks such as bias, privacy, licensing, and domain shift.

Read full answer

Applies toHugging Face Transformers 5.x (Python 3.12)

Why Transformers Are Useful

  • Large model collection: The Hugging Face Hub hosts pretrained models for classification, translation, summarization, question answering, token classification, and more.

  • Consistent APIs: Components such as pipeline, AutoTokenizer, and AutoModel provide common interfaces across compatible model architectures.

  • Transfer learning: Pretrained weights supply useful representations, so a smaller labeled dataset is often sufficient for fine-tuning.

  • End-to-end tooling: Tokenizers, training utilities, evaluation metrics, quantization, and deployment integrations support much of the model lifecycle.

  • PyTorch integration: Transformers 5.x is centered on PyTorch and integrates with common training, inference, and optimization workflows.

  • Task flexibility: The library supports many tasks, although each pretrained checkpoint still requires an appropriate architecture, task head, and compatible data format.

Classic and Modern NLP Pipelines

A classical NLP pipeline commonly performs tokenization, Unicode or whitespace normalization, optional stop-word removal, stemming or lemmatization, and feature extraction with bag-of-words, TF-IDF, or word n-grams.

Approach How it works Strengths Weaknesses Typical use or scale
Classical features Represent documents as sparse token or n-gram features Interpretable, inexpensive, effective on small datasets Weak context and synonym handling Sparse storage scales with nonzero features; linear models suit limited-data text
Static embeddings Map each word to one fixed vector, as in Word2Vec, GloVe, or fastText Capture some semantic similarity One vector cannot represent different word senses O(v*d) storage, where v is vocabulary size and d is embedding width; lookup is O(d)
Transformer models Learn contextual representations using attention and subword tokens Strong context modeling and broad transferability Greater compute, memory, latency, and opacity Self-attention is typically O(L^2*d) per layer, where L is sequence length and d is hidden width

Modern tokenizers commonly use Byte-Pair Encoding (BPE) or WordPiece, representing rare or unseen words through smaller units and reducing out-of-vocabulary problems. Transformer models produce contextual embeddings, so the same word can have different representations in different contexts. BERT-style models are commonly used as encoders for classification, retrieval, and extractive question answering, while GPT-style models support autoregressive generation for chat, summarization, and other generation tasks.

Supported Tasks and Evaluation

Task Typical output Suitable evaluation
Text classification Class label and probabilities Accuracy, precision, recall, F1, ROC-AUC, PR-AUC
Sentiment analysis Sentiment label Accuracy and macro F1
Named entity recognition Entity spans and labels Entity-level precision, recall, and F1
Part-of-speech tagging A label for each token Token-level accuracy and F1
Question answering Extracted span or generated answer Exact Match and F1 for extraction; semantic metrics and human review for generation
Summarization Generated summary ROUGE, BERTScore, and human evaluation
Machine translation Translated text BLEU, COMET, and human evaluation
Search Ranked relevant documents Recall@k, MRR, and nDCG

BLEU, ROUGE, and BERTScore are useful automated proxies, but they do not fully measure factual correctness, fluency, or usefulness. Human evaluation is often necessary for generation. Accuracy can also mislead on imbalanced classes because a model may obtain a high score while ignoring minority classes.

Advantages and Trade-offs

  • Fine-tuning is the default approach: When a suitable checkpoint and labeled data are available, fine-tuning is usually preferable to training from scratch.

  • Contextual representations: Attention can model dependencies that bag-of-words features miss.

  • Reduced implementation effort: Common APIs and pretrained components reduce architecture-specific engineering and maintenance.

  • Flexible deployment: GPU acceleration benefits large models and high-volume serving, while smaller or quantized models can run efficiently on CPUs.

  • Resource costs: Larger models generally require more memory, training time, and inference latency, although architecture and hardware also matter.

  • Operational risks: Pretrained models may inherit demographic or domain bias, perform poorly on multilingual or shifted inputs, or expose sensitive data when prompts and telemetry are handled insecurely.

  • Governance remains necessary: Review each model’s license, model card, training-data documentation, intended use, and provenance before deployment.

Common Pitfalls

  • Data leakage: For classical pipelines, fit TF-IDF, imputers, and feature selectors on training folds only and use a scikit-learn Pipeline. Reuse a pretrained tokenizer unchanged unless deliberately training a new tokenizer from a training-only corpus.

  • Wrong evaluation metric: Prefer macro F1 or PR-AUC for imbalanced classes rather than relying only on accuracy.

  • Weak validation design: Keep training, validation, and test sets separate, monitor validation loss for overfitting, and avoid tuning repeatedly against the test set.

  • Unrealistic evaluation: Test the target domain, language, user population, and time period, and handle malformed, missing, or non-string inputs before tokenization.

  • Irreproducible experiments: Record model and dataset revisions, dependency versions, seeds, generation settings, and hardware where relevant.

  • Missing governance: Assess bias, privacy, licensing, harmful outputs, and model provenance before production use.

29

How do PyTorch and TensorFlow help build NLP models?

PyTorch and TensorFlow are deep-learning frameworks that provide the building blocks for NLP models: tensors that run on GPUs and TPUs, automatic differentiation for computing gradients, ready-made layers (embeddings, LSTMs and GRUs, multi-head attention and full transformer encoder/decoder layers), loss functions, optimizers (Adam, AdamW), learning-rate schedules, data-loading pipelines and tools for saving, exporting and serving models. With them you define a model as a composition of layers, for example token IDs → embedding → transformer or LSTM → pooling → linear classifier, write or use a training loop that computes the loss, calls backpropagation and updates weights, and scale it with mixed precision, multiple GPUs or distributed training. PyTorch (with torch.nn, torch.utils.data and torch.compile) is the dominant choice in research and in most current NLP code, including Hugging Face Transformers, thanks to its Pythonic eager execution and easy debugging; TensorFlow with Keras offers a high-level model.fit API, the TextVectorization layer for in-model preprocessing, tf.data pipelines, and mature deployment through TensorFlow Serving and TensorFlow Lite for mobile, while Keras 3 can now also run on PyTorch and JAX backends. Both integrate with pretrained models (Hugging Face, TensorFlow Hub, torch.hub), ONNX export and production runtimes. The practical workflow today is rarely training from scratch: you load a pretrained transformer in your framework of choice, add a task head, and fine-tune.

Read full answer

Applies toPython 3.10+, PyTorch 2.x, TensorFlow 2.16+ / Keras 3

What the Frameworks Provide

Need PyTorch TensorFlow / Keras
Embeddings nn.Embedding, nn.EmbeddingBag layers.Embedding
Sequence layers nn.LSTM, nn.TransformerEncoder layers.LSTM, layers.MultiHeadAttention
Text preprocessing Tokenizers (Hugging Face) layers.TextVectorization
Training Custom loop or Lightning/Trainer model.fit
Data Dataset, DataLoader tf.data
Speed-ups torch.compile, AMP XLA, mixed precision
Deployment TorchScript/ONNX, TorchServe TF Serving, TF Lite

Code Example: PyTorch

import torch
from torch import nn

torch.manual_seed(42)
vocab = {"<pad>": 0, "good": 1, "great": 2, "bad": 3, "awful": 4, "movie": 5}
texts = [["good", "movie"], ["great", "movie"], ["bad", "movie"], ["awful", "movie"]]
X = torch.tensor([[vocab[w] for w in t] for t in texts])
y = torch.tensor([1, 1, 0, 0])

class BagClassifier(nn.Module):
    def __init__(self, n_vocab, dim=16):
        super().__init__()
        self.emb = nn.EmbeddingBag(n_vocab, dim, mode="mean")
        self.fc = nn.Linear(dim, 2)

    def forward(self, ids):
        return self.fc(self.emb(ids))

model = BagClassifier(len(vocab))
opt = torch.optim.AdamW(model.parameters(), lr=0.05)
loss_fn = nn.CrossEntropyLoss()
for epoch in range(100):
    opt.zero_grad()
    loss = loss_fn(model(X), y)
    loss.backward()                           # autograd computes all gradients
    opt.step()
print(round(loss.item(), 2) < 0.1, model(X).argmax(dim=1).tolist())
# True [1, 1, 0, 0]

Code Example: TensorFlow / Keras

import numpy as np
import tensorflow as tf
from tensorflow.keras import layers

texts = np.array(["good movie", "great movie", "bad movie", "awful movie"])
labels = np.array([1, 1, 0, 0])

vectorize = layers.TextVectorization(max_tokens=100, output_sequence_length=4)
vectorize.adapt(texts)                        # learns the vocabulary inside the model
model = tf.keras.Sequential([
    vectorize,
    layers.Embedding(100, 16),
    layers.GlobalAveragePooling1D(),
    layers.Dense(1, activation="sigmoid"),
])
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
model.fit(texts, labels, epochs=200, verbose=0)
print(model.predict(np.array(["great movie"]), verbose=0).shape)
# (1, 1)

Both toy models learn the four examples; real tasks start from pretrained embeddings or transformers.

Common Pitfalls

  • Training transformers from scratch instead of fine-tuning.

  • Forgetting model.eval() / torch.no_grad() at inference in PyTorch (dropout stays on).

  • Preprocessing mismatches between training and serving (Keras TextVectorization inside the model avoids this).

Interview Tips

  • Name the building blocks (tensors, autograd, layers, optimizers, data, deployment).

  • Compare PyTorch and TensorFlow/Keras strengths and mention the pretrained-model workflow.

30

How do modern NLP libraries process multilingual text?

Multilingual text processing combines Unicode-aware preprocessing with pretrained models that can encode, classify, retrieve, translate, or generate across languages. Modern workflows usually begin with a multilingual checkpoint and adapt it to the task, trading broad language coverage and transferability for higher compute and uneven quality across languages, scripts, and domains.

Read full answer

Applies toHugging Face Transformers 4.x with Python 3.12

Classic and Modern Pipelines

A classic NLP pipeline usually applies these steps:

  1. Tokenization: Split text into words, subwords, characters, or sentences using language-appropriate rules.

  2. Normalization: Optionally apply Unicode normalization, lowercasing, accent handling, or transliteration. Preserve case, punctuation, and meaningful accents.

  3. Stop-word removal: Remove common function words using language-specific lists. This can harm tasks where syntax matters.

  4. Stemming or lemmatization: Reduce inflected words to a base form. Stemming is fast but approximate; lemmatization is generally more accurate but needs language-specific resources.

  5. Feature extraction: Represent documents with bag-of-words, TF-IDF, word n-grams, or character n-grams. Character n-grams tolerate spelling variation and unknown words.

  6. Static embeddings: word2vec, GloVe, and fastText map words to vectors. fastText’s subword representations help with rare and noisy text, while standalone word2vec and GloVe resources are usually language-specific.

Modern systems use contextual encoders such as mBERT or XLM-R for understanding and GPT-style decoder models for generation. Subword schemes include BPE, WordPiece, and SentencePiece: mBERT uses WordPiece, while XLM-R uses SentencePiece. These schemes split rare or unseen words into known smaller units instead of treating them as out-of-vocabulary tokens. Contextual models also represent each occurrence of a word according to its context.

Models, Tasks, and Evaluation

Approach How It Works Strengths Weaknesses Typical Use Typical Complexity or Scale
Character n-gram TF-IDF Learns spelling patterns Simple and language-agnostic Misses deeper meaning Language identification or short-text classification Sparse features; memory grows with corpus and vocabulary
fastText Learns static word and subword vectors Handles rare words and noisy text Static contextual meaning Classification and similarity CPU-friendly and suitable for large corpora
mBERT/XLM-R Produces contextual token representations Broad language coverage and transfer learning Uneven performance across languages Classification, NER, retrieval, and embeddings Dense encoders; attention cost grows quadratically with sequence length, so GPUs can help
Translation models Convert or generate text in another language Direct cross-lingual processing Translation errors can alter meaning Translation and data augmentation Autoregressive generation cost grows with input and output lengths
Multilingual GPT-style models Generate text autoregressively Flexible generation and task adaptation High compute cost and hallucination risk Generation, assistants, and multilingual reasoning Memory and compute grow with context and generated output

Common task outputs should remain explicit:

  • Classification and sentiment analysis: A label and class probabilities; evaluate with accuracy, macro-F1, and per-language F1.

  • Named entity recognition: Labeled entity spans; evaluate with entity-level precision, recall, and F1.

  • Question answering: An answer span or generated answer.

  • Summarization and translation: Generated text; evaluate with ROUGE, BLEU, chrF, or BERTScore plus human review.

  • Search: Ranked documents or passages using multilingual dense retrieval or translated queries.

Evaluate important languages separately rather than relying only on an aggregate multilingual score. Learned metrics may correlate better with human judgments, but they can still miss culturally important differences.

Practical Strategy

  • Detect language when necessary: Use language identification, Unicode script analysis, or metadata, especially for code-switched documents.

  • Preserve Unicode: Prefer NFC normalization and do not indiscriminately strip accents.

  • Use pretrained models by default: Fine-tune multilingual checkpoints on representative target-language data; train from scratch only with sufficient domain-specific data.

  • Use language-specific resources when useful: Stop words, lemmatizers, tokenizers, and adapters can outperform one generic pipeline.

  • Prevent leakage: Fit learned preprocessing on training data only and apply identical normalization and tokenization during inference.

  • Account for deployment risks: Test for domain shift, audit bias, protect private data, and verify third-party retention and training policies.

Code Example

This example uses XLM-R’s multilingual SentencePiece tokenizer to show how several writing systems are split into subwords. The first execution downloads the tokenizer files from Hugging Face; later executions use the local cache.

from transformers import AutoTokenizer

# Load XLM-R's multilingual tokenizer; this downloads files on first use.
tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-base")

samples = {
    "Spanish": "¿Dónde está la estación de tren?",
    "Arabic": "أين توجد محطة القطار؟",
    "Japanese": "駅はどこですか?",
    "English": "Where is the train station?",
}

for language, text in samples.items():
    # Add boundary tokens and split the text into subword IDs.
    token_ids = tokenizer.encode(text, add_special_tokens=True)

    # Convert the IDs into their SentencePiece subword tokens.
    tokens = tokenizer.convert_ids_to_tokens(token_ids)

    print(f"\n{language}: {text}")
    print("Subword tokens:", tokens)
    print("Token count:", len(token_ids))

    # Decode the IDs back into readable text.
    print("Decoded text:", tokenizer.decode(token_ids, skip_special_tokens=True))

Common Pitfalls

  • Unequal evaluation: An aggregate score can hide poor low-resource-language performance. Report per-language metrics and sample sizes.

  • Wrong metric: Accuracy can mislead with imbalanced classes; use macro-F1 and per-class precision and recall where appropriate.

  • Insufficient generation evaluation: BLEU, ROUGE, and learned metrics can miss cultural nuance and domain terminology; include human review.

  • Destructive preprocessing: Removing accents or assuming one language per document can erase meaningful distinctions.

  • Fine-tuning without evaluation: Narrow fine-tuning can cause catastrophic forgetting, so retain representative validation sets and compare against the base checkpoint.

Interview follow-ups commonly cover language identification, BPE versus WordPiece, SentencePiece, per-language evaluation, catastrophic forgetting, translated versus shared representations, and privacy implications.

31

How do you write an NLTK text tokenization function in Python?

Tokenization splits raw text into smaller units such as words, subwords, or punctuation for downstream NLP processing. NLTK provides lightweight, rule-based tokenizers that are fast and easy to control, although they do not resolve meaning or handle unseen words as effectively as transformer-based subword tokenizers. The main trade-off is simplicity and speed versus contextual understanding and robustness.

Read full answer

Applies toNLTK 3.9.1 (Python 3.12)

How Tokenization Works

A tokenizer receives a string and returns a list of tokens. The following implementation uses TreebankWordTokenizer, which separates words and punctuation (including handling contractions) without requiring a downloaded NLTK dataset:

  • Case handling: Text is preserved by default because capitalization can carry meaning.

  • Lowercasing: Optional normalization is applied before tokenization.

  • Punctuation: Punctuation is returned as separate tokens, which is useful for sentiment analysis and linguistic processing.

  • Input validation: Non-string values produce a clear TypeError.

Classic and Modern NLP Pipelines

A classic NLP pipeline usually performs:

  1. Tokenization: Split text into tokens.

  2. Normalization: Optionally lowercase text and standardize formats.

  3. Stop-word removal: Remove common words such as “the” when they add little meaning.

  4. Stemming or lemmatization: Reduce inflected words to a root form; lemmatization uses linguistic rules.

  5. Feature extraction: Represent documents with bag-of-words, TF-IDF, or n-grams.

  6. Modeling: Train a classifier, regressor, or sequence model.

Modern systems usually begin with a pretrained transformer such as BERT or a GPT-style model. Their subword tokenizers use algorithms such as WordPiece or byte-pair encoding (BPE), splitting rare words into frequently observed pieces. This reduces out-of-vocabulary problems, supports multilingual text, and preserves useful spelling information.

Common Tasks and Evaluation

Task Typical output Suitable evaluation
Text classification Class label and class probabilities Accuracy, precision, recall, F1, ROC-AUC, PR-AUC
Sentiment analysis Positive, negative, or neutral sentiment Accuracy, precision, recall, F1
Named entity recognition Entity text, type, and character positions Entity-level precision, recall, and F1
Part-of-speech tagging A tag for each token Accuracy or macro-F1
Question answering Answer span or generated answer Exact match, F1, or human evaluation
Summarization Generated summary ROUGE, BERTScore, and human evaluation
Machine translation Translated text BLEU, chrF, COMET, and human evaluation
Information retrieval Ranked relevant documents Precision, recall, MRR, or NDCG

Accuracy can mislead for imbalanced classification because a model predicting the majority class may obtain a high score while failing on rare classes. Generation metrics also have limitations: BLEU emphasizes token overlap, ROUGE emphasizes recall or n-gram overlap, and BERTScore uses contextual similarity, so human evaluation remains important.

Tokenizer Choices

Tokenizer How it works Strengths Limitations Typical use
TreebankWordTokenizer Applies Treebank-style contraction rules with regex No downloaded data required; handles contractions consistently Still context-free General preprocessing, legacy pipelines
word_tokenize Performs sentence and word tokenization using Punkt More linguistically convenient Requires the punkt resource Sentence-aware processing
WordPunctTokenizer Splits on whitespace and punctuation Very fast, no dependencies Splits contractions aggressively Simple whitespace/punctuation splitting
Transformer WordPiece/BPE Produces vocabulary subword tokens Handles rare, noisy, and multilingual text More computational overhead Pretrained transformer models

Code Example

from nltk.tokenize import TreebankWordTokenizer

TOKENIZER = TreebankWordTokenizer()


def tokenize_text(text: str, *, lowercase: bool = False) -> list[str]:
    """Split text into word and punctuation tokens using NLTK's TreebankWordTokenizer."""
    if not isinstance(text, str):
        raise TypeError("text must be a string")

    if lowercase:
        text = text.lower()

    return TOKENIZER.tokenize(text)


texts = [
    "Text tokenization is the first NLP step!",
    "NLTK makes it easy to build a pipeline.",
]

for text in texts:
    print(tokenize_text(text))

# Deterministic output:
# ['Text', 'tokenization', 'is', 'the', 'first', 'NLP', 'step', '!']
# ['NLTK', 'makes', 'it', 'easy', 'to', 'build', 'a', 'pipeline', '.']

For sentence-aware tokenization, use nltk.word_tokenize after downloading the punkt resource once with nltk.download("punkt").

Common Pitfalls

  • Removing information too early: Lowercasing or removing punctuation can discard signals needed for sentiment, NER, or authorship analysis.

  • Confusing tokens with words: TreebankWordTokenizer may split contractions such as “don’t” into multiple tokens; choose a tokenizer that matches your intended representation.

  • Using a validation set during fitting: Fit tokenization-related transformations such as learned vocabularies, TF-IDF vectorizers, or fitted embeddings on training data only.

  • Ignoring domain shift: Generic tokenizers and pretrained models may behave differently on legal, medical, scientific, or multilingual text; evaluate on representative data.

  • Overlooking privacy and bias: Text may contain sensitive information, and pretrained models can reproduce demographic or language biases.

  • Forgetting evaluation requirements: Select task-appropriate metrics, preserve held-out test data, and include human evaluation for subjective generation tasks.

32

How do I build an n-gram language model in Python?

An n-gram language model estimates the probability of the next token from the previous n-1 tokens. It is simple to implement with counts and smoothing, but its limited context and sparse statistics make it less expressive than pretrained embeddings or transformer models.

Read full answer

Applies toPython 3.12 with NumPy 2.x

How an n-gram Language Model Works

An n-gram is a contiguous sequence of n tokens:

  • Unigram: n = 1

  • Bigram: n = 2

  • Trigram: n = 3

The model counts each n-gram and estimates:

P(token | context) = (count(context, token) + alpha) / (count(context) + alpha * V)

Here, count(context, token) is the number of observed occurrences, count(context) is the total number of times that context appeared, alpha is the smoothing strength, and V is the vocabulary size. Laplace smoothing prevents zero probabilities for unseen continuations of observed contexts.

The implementation adds beginning-of-sequence and end-of-sequence markers so the model can learn sentence boundaries and generate complete sentences.

Classic and Modern NLP Context

The classic NLP pipeline is tokenization, normalization, optional stop-word removal, stemming or lemmatization, and feature construction. Stemming aggressively removes suffixes, while lemmatization uses linguistic rules to map words to their base form.

  • Bag-of-words and TF-IDF represent documents as token counts or weighted frequencies and are useful for sentiment analysis and text classification.

  • Word n-grams preserve local order and can also be used as classification features.

  • Word2Vec, GloVe, and fastText learn static vector embeddings; fastText uses subword information and is more robust for rare or unseen words.

  • BERT produces contextual representations for classification, extraction, and embeddings.

  • GPT-style models use causal attention for text generation and are generally the default choice when high-quality generation is required.

Modern systems usually start with a pretrained model and fine-tune it when task-specific data is available.

Code Example

from __future__ import annotations

import math
import re
from collections import Counter

import numpy as np

TOKEN_PATTERN = re.compile(r"[a-z]+(?:'[a-z]+)?")
BOS = "<s>"
EOS = "</s>"


class NGramLanguageModel:
    def __init__(self, n: int = 3, alpha: float = 1.0):
        if n < 1:
            raise ValueError("n must be at least 1")
        if alpha <= 0:
            raise ValueError("alpha must be positive")

        self.n = n
        self.alpha = alpha
        self.counts: list[Counter] = []
        self.context_counts: list[Counter] = []
        self.vocabulary: list[str] = []

    def tokenize(self, text: str) -> list[str]:
        return TOKEN_PATTERN.findall(text.lower())

    def fit(self, corpus: list[str]) -> "NGramLanguageModel":
        if not corpus:
            raise ValueError("corpus cannot be empty")

        self.counts = [Counter() for _ in range(self.n)]
        self.context_counts = [Counter() for _ in range(self.n)]
        vocabulary = set()

        for text in corpus:
            sequence = (
                [BOS] * (self.n - 1)
                + self.tokenize(text)
                + [EOS]
            )
            vocabulary.update(token for token in sequence if token != BOS)

            for order in range(1, self.n + 1):
                for start in range(len(sequence) - order + 1):
                    context = tuple(sequence[start:start + order - 1])
                    target = sequence[start + order - 1]
                    self.counts[order - 1][(context, target)] += 1
                    self.context_counts[order - 1][context] += 1

        self.vocabulary = sorted(vocabulary)
        return self

    def probability(
        self, context: tuple[str, ...], target: str
    ) -> float:
        if len(context) != self.n - 1:
            raise ValueError(f"context must contain {self.n - 1} tokens")
        if target not in self.vocabulary:
            return 0.0

        count = self.counts[self.n - 1][(context, target)]
        context_count = self.context_counts[self.n - 1][context]
        vocabulary_size = len(self.vocabulary)

        return (count + self.alpha) / (
            context_count + self.alpha * vocabulary_size
        )

    def perplexity(self, texts: list[str]) -> float:
        total_log_probability = 0.0
        token_count = 0

        for text in texts:
            sequence = (
                [BOS] * (self.n - 1)
                + self.tokenize(text)
                + [EOS]
            )

            for index in range(self.n - 1, len(sequence)):
                context = tuple(
                    sequence[index - (self.n - 1):index]
                )
                target = sequence[index]
                probability = self.probability(context, target)

                if probability == 0.0:
                    return math.inf

                total_log_probability += math.log(probability)
                token_count += 1

        if token_count == 0:
            raise ValueError("no tokens to evaluate")

        return math.exp(-total_log_probability / token_count)

    def generate(self, max_tokens: int = 20, seed: int = 42) -> str:
        if not self.vocabulary:
            raise ValueError("the model must be fitted before generation")

        rng = np.random.default_rng(seed)
        context = (BOS,) * (self.n - 1)
        generated = []

        for _ in range(max_tokens):
            probabilities = np.array(
                [
                    self.probability(context, token)
                    for token in self.vocabulary
                ],
                dtype=float,
            )
            target = str(
                rng.choice(self.vocabulary, p=probabilities)
            )

            if target == EOS:
                break

            generated.append(target)
            if self.n == 1:
                context = ()
            else:
                context = (context + (target,))[-(self.n - 1):]

        return " ".join(generated)


train_corpus = [
    "the cat sat on the mat",
    "the dog sat on the rug",
    "birds fly over the blue sky",
    "machines learn patterns from data",
    "python code runs useful functions",
    "weather can be rainy or sunny today",
    "language models predict the next words",
    "children play together in the park",
    "people drink coffee in the morning",
    "the music sounds very good",
    "books contain useful stories",
    "data helps train smart models",
] * 3

test_corpus = [
    "the sunny weather sounds good today",
]

# Fit only on training data to avoid evaluation leakage.
model = NGramLanguageModel(n=3, alpha=1.0).fit(train_corpus)

print("Generated text:", model.generate(max_tokens=12, seed=42))
print("Held-out perplexity:", f"{model.perplexity(test_corpus):.3f}")

Evaluation and Alternatives

The implementation evaluates held-out text with perplexity:

PPL = exp(-(1 / N) * sum(log(P(w_i | c_i))))

Here, N is the number of evaluated target tokens, w_i is an actual token, c_i is its preceding context, and log is the natural logarithm. Lower perplexity generally indicates better predictions, although values are comparable only when the tokenization and evaluation data are consistent.

Approach How it works Strengths Weaknesses Typical use
N-gram model Counts local token sequences Simple, fast, interpretable Sparse; limited context Teaching, basic generation
TF-IDF Weights terms by corpus frequency Strong text-classification baseline Loses word order Classification, retrieval
Static embeddings Maps words to fixed vectors Captures similarity Weak contextual meaning Similarity, feature generation
Transformer Uses contextual attention Handles long-range relationships Computationally expensive High-quality NLP tasks

Common Pitfalls

  • Data leakage: Fit counts, tokenizers, and other learned preprocessing only on training data.

  • Blind preprocessing: Removing stop words or punctuation can destroy useful syntax; tokenization should match the application.

  • Overfitting: Larger n values create sparse counts and memorize training phrases. Compare several n values and use held-out perplexity.

  • Vocabulary gaps: Character and subword n-grams such as BPE or WordPiece are more robust for rare and out-of-vocabulary words.

  • Unlimited assumptions: Real text varies across domains, languages, and social groups. Check domain shift, demographic bias, and privacy requirements before deployment.

  • Efficiency: Training takes approximately O(T * n) time for T tokens and fixed n; generation takes O(L * V) for L generated tokens and vocabulary size V.

33

How do you extract email addresses with a Python regex?

A regular expression can extract email-like substrings by matching a local part, an @ symbol, and a domain name. The main trade-off is simple extraction versus complete validation: regexes work well as filters but do not prove that a mailbox exists or cover every address permitted by email standards.

Read full answer

Applies toPython 3.12 standard-library `re` module

How It Works

re.findall() scans a string and returns every non-overlapping match of the specified pattern. A practical common-address pattern separates an address into:

  1. A local part containing letters, digits, underscores, and symbols such as %, +, and -.

  2. An @ separator.

  3. A domain part containing dot-separated labels with letters, digits, and internal hyphens.

  4. A top-level domain containing at least two ASCII letters.

This is a conservative extraction pattern rather than a complete implementation of RFC 5322 or SMTP address validation.

Pattern Breakdown

  • (?<![A-Za-z0-9._%+-]): Prevents starting in the middle of a possible local part or immediately after a malformed leading dot.

  • [A-Za-z0-9_%-]: Matches the first non-empty local-part atom.

  • (?:\.[A-Za-z0-9_%-])*: Matches additional dot-separated atoms without allowing empty, repeated, or trailing dots.

  • @: Matches the literal @ separator.

  • [A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?: Matches one domain label without allowing a leading or trailing hyphen.

  • [A-Za-z]{2,}: Matches an alphabetic top-level domain of at least two letters.

  • The final \b: Ends the match at a word boundary while still allowing ordinary trailing punctuation.

Code Example

import re

text = """
Contact alice@example.com or bob.smith@company.co.uk.
The support address is support+help@service.org.
bad@localhost is not a conventional email address.
"""

pattern = (
    r"(?<![A-Za-z0-9._%+-])"
    r"[A-Za-z0-9_%+-]+(?:\.[A-Za-z0-9_%+-]+)*"
    r"@"
    r"(?:[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?\.)*"
    r"[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?\."
    r"[A-Za-z]{2,}\b"
)

emails = re.findall(pattern, text)

print(emails)

# Expected output:
# ['alice@example.com', 'bob.smith@company.co.uk', 'support+help@service.org']

For duplicate removal, lowercasing the entire address is appropriate only when the mail provider treats the local part case-insensitively. Otherwise, lowercase only the domain and preserve the local part’s case.

import re

text = "Alice@example.com, alice@example.com, bob@example.org."

pattern = (
    r"(?<![A-Za-z0-9._%+-])"
    r"[A-Za-z0-9_%+-]+(?:\.[A-Za-z0-9_%+-]+)*"
    r"@"
    r"(?:[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?\.)*"
    r"[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?\."
    r"[A-Za-z]{2,}\b"
)

emails = re.findall(pattern, text)
unique_emails = list(dict.fromkeys(email.lower() for email in emails))

print(unique_emails)

# Expected output:
# ['alice@example.com', 'bob@example.org']

Practical Considerations

  • Validation: Treat regex matching as extraction or a quick filter, not proof that an address exists or conforms to every standard.

  • Internationalized addresses: Unicode domains and internationalized email syntax require additional processing or a specialized validation library.

  • Valid but unsupported forms: Quoted local parts, comments, and other unusual but permitted forms are not covered.

  • Performance: Python regexes use backtracking. This pattern avoids highly ambiguous nested repetitions, but large or adversarial inputs should still be benchmarked.

  • Case sensitivity: Domain names are case-insensitive, while local-part sensitivity depends on the mail server or provider.

Common Pitfalls

  • Writing \. incorrectly as . causes the dot to match any character.

  • Using .*@.* can overmatch across spaces and unrelated text.

  • Forgetting to escape special regex characters causes unexpected behavior.

  • Assuming the regex verifies mailbox existence or accepts every valid address.

  • Lowercasing an entire address without checking the provider’s local-part policy.

  • Using overly long or ambiguous patterns without testing them against representative text.

Interview Tips

Interviewers may ask whether the regex supports unusual but valid addresses, how duplicates should be handled, or how internationalized addresses would be extracted. Explain that the proposed pattern is suitable for common-form extraction, while production systems should use normalization and a dedicated email-validation library when strict validation is required.

34

How do you implement cosine similarity between two text documents in Python?

Cosine similarity measures how closely the directions of two text vectors match, regardless of their lengths. Converting documents to TF-IDF vectors and comparing them with cosine similarity is a practical baseline for measuring textual overlap. Its main trade-off is that surface-level vocabulary similarity may miss semantic similarity when different words express the same idea.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

The classic NLP pipeline includes:

  1. Tokenization: splitting text into words or terms.

  2. Normalization: lowercasing and optionally removing accents.

  3. Stop-word removal: filtering common words such as “the” or “and”.

  4. N-gram construction: representing individual words or adjacent word combinations.

  5. Vectorization: assigning TF-IDF weights to the resulting terms.

  6. Similarity calculation: applying cosine similarity to the two vectors.

The cosine similarity formula is:

cosine_similarity(d1, d2) = (d1 · d2) / (||d1|| * ||d2||)

Here, d1 and d2 are the two document vectors, · is their dot product, and ||d1|| and ||d2|| are their Euclidean magnitudes. A value near 1 indicates similar term-weight patterns, 0 indicates little or no overlap, and values below 0 indicate opposing patterns.

TF-IDF gives each term a weight based on its frequency in a document and how widely it appears across the compared documents. Unlike raw counts, this reduces the influence of terms that appear everywhere.

Text Representation Choices

Representation How it works Strengths Weaknesses Typical use
Bag-of-words Counts each token Simple and fast Ignores order and meaning Baseline comparisons
TF-IDF Weights terms using frequency and inverse document frequency Reduces common-term influence Still relies on exact lexical overlap Similarity and classification
Word2Vec, GloVe, fastText Maps words to dense vectors Captures some semantic relationships A word has one fixed vector Legacy semantic search
BERT and other transformer embeddings Produces context-dependent token vectors Handles context and phrasing better More compute and model complexity Modern semantic search

Modern systems commonly use pretrained transformer embeddings, optionally followed by fine-tuning on domain-specific data. Word2Vec, GloVe, and fastText remain useful lightweight alternatives.

Code Example

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity


def text_cosine_similarity(
    document_a: str,
    document_b: str,
    *,
    ngram_range: tuple[int, int] = (1, 2),
) -> float:
    """Calculate TF-IDF cosine similarity between two documents."""
    if not document_a.strip() or not document_b.strip():
        raise ValueError("Both documents must contain non-whitespace text.")

    vectorizer = TfidfVectorizer(
        lowercase=True,
        strip_accents="unicode",
        stop_words="english",
        ngram_range=ngram_range,
    )

    try:
        # Fit one vocabulary and IDF model, then transform both documents.
        document_matrix = vectorizer.fit_transform([document_a, document_b])
    except ValueError as exc:
        raise ValueError(
            "The documents contain no tokens usable by the vectorizer."
        ) from exc

    # A document containing only stop words produces a zero vector.
    if np.any(document_matrix.getnnz(axis=1) == 0):
        raise ValueError(
            "Each document must contain at least one usable token."
        )

    similarity = cosine_similarity(
        document_matrix[0],
        document_matrix[1],
    )[0, 0]

    return float(similarity)


document_a = "Machine learning improves search ranking and data analysis."
document_b = "Search ranking benefits from machine learning."

similarity = text_cosine_similarity(document_a, document_b)
print(f"Cosine similarity: {similarity:.4f}")

The two documents are transformed with the same vectorizer. Calling fit_transform separately on each document would create potentially different vocabularies and make the comparison invalid.

Practical Considerations

  • Corpus-dependent IDF: The example learns IDF values from the two documents being compared. In production, fit the vectorizer on a representative background corpus, then use transform for each incoming document.

  • Negation and word order: TF-IDF may treat “good service” and “bad service” as similar because they share most terms. Bigrams help, while contextual embeddings handle these cases better.

  • Domain shift: A vectorizer trained on general text may assign poor weights to specialized terminology. Domain-specific corpora or fine-tuned models usually perform better.

  • Multilingual text: English stop-word lists are inappropriate for every language. Use language-appropriate preprocessing or multilingual transformer models.

  • Subword tokenization: BPE and WordPiece represent rare or unseen words through subword units, avoiding an entirely unknown-token representation.

  • Privacy and bias: Text may contain sensitive information, while pretrained models can reproduce biases present in their training data.

Common Pitfalls

  • Comparing vectors produced by independently fitted vectorizers.

  • Applying stemming or lemmatization without checking whether it improves the target task.

  • Interpreting similarity as semantic truth; human evaluation or contextual embeddings may be needed.

  • Reporting only overlap scores without checking whether the application requires precision, recall, or ranking metrics.

  • Fitting preprocessing on the complete dataset before cross-validation, which causes data leakage.

  • Forgetting that cosine similarity is undefined for an all-zero vector.

Interviewers commonly ask why TF-IDF is preferred over raw counts, how the score changes with different n-gram ranges, how to avoid data leakage, and when contextual embeddings would be preferable.

35

How do I implement a sentiment classifier with bag-of-words in scikit-learn?

A bag-of-words (BoW) model represents each document as a vector of token frequencies, allowing a classifier to learn sentiment patterns from labeled text. This implementation uses TF-IDF, a weighted BoW representation, with logistic regression, trading contextual understanding for fast training and interpretable features.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

  • Vectorization: TfidfVectorizer converts each document into a sparse TF-IDF matrix. A token receives more weight when it is common in that document but uncommon across the training corpus.

  • Classification: LogisticRegression learns a linear decision boundary over those features and predicts a sentiment label.

  • Pipeline: Pipeline ensures that the vocabulary and IDF weights are learned only from each training fold, preventing test-set information from leaking into training.

TF-IDF is still a bag-of-words model, although its features are weighted rather than raw counts. CountVectorizer is the raw-count alternative.

Classic NLP Pipeline

  • Tokenization: Split text into words or, with a suitable tokenizer, subwords.

  • Normalization: Lowercase text and optionally remove accents, punctuation, or other irrelevant variation.

  • Stop words: Remove common words such as “the” or “is.” Review the stop-word list first because sentiment-relevant negations can also be removed.

  • Stemming versus lemmatization: Stemming aggressively truncates words; lemmatization maps words to dictionary forms.

  • BoW and TF-IDF: BoW stores token counts; TF-IDF increases the weight of tokens that discriminate between documents.

  • N-grams: Unigrams ignore word order. Bigrams such as “not good” capture short phrases but enlarge the vocabulary and can overfit.

  • Subword tokenization: BPE and WordPiece split rare words into reusable subword units, allowing them to be represented even when the complete word was absent from the tokenizer vocabulary.

Modern NLP Context

Approach How it works Strengths Weaknesses When to use Typical scale
BoW/TF-IDF Sparse token features followed by a linear classifier Fast, small, interpretable Loses word order and context Small or medium labeled datasets and strong baselines Sparse storage grows with vocabulary
Static embeddings: word2vec, GloVe, fastText Assigns a mostly fixed dense vector to each word, then uses a downstream classifier Compact semantic features; fast inference A fixed vector cannot represent different contexts Limited compute or reusable features Storage grows roughly with vocabulary
Fine-tuned transformer Produces contextual token representations and adapts them to the task Usually stronger contextual accuracy More compute and data; less interpretable Quality-critical tasks with adequate resources Attention cost grows roughly quadratically with token count; GPUs are often useful

BERT-style encoders are commonly used for classification, while GPT-style models are primarily generators. Fine-tuning a pretrained transformer is the default for quality-critical systems when resources permit because it often outperforms BoW, but it also increases concerns about bias, privacy, and domain shift.

Common Tasks and Metrics

Task Typical output Evaluation
Sentiment classification Class label and probabilities Accuracy, precision, recall, macro-F1
Named entity recognition Entity spans and types Entity-level precision, recall, and F1
Part-of-speech tagging Tag for each token Accuracy or macro-F1
Question answering Extracted or generated answer Exact match, token F1, human review
Summarization Generated summary ROUGE, BERTScore, human evaluation
Machine translation Translated text BLEU, chrF, human evaluation
Semantic search Ranked relevant documents Recall@k, MRR, nDCG

Accuracy can mislead on imbalanced data, so inspect per-class precision and recall, macro-F1, the confusion matrix, ROC-AUC, and PR-AUC. BLEU and ROUGE rely heavily on lexical overlap and can penalize valid paraphrases; BERTScore uses contextual similarity but depends on its scoring model and references. Human evaluation checks usefulness and fluency, but it is costly and subjective.

Code Example

import numpy as np

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    classification_report,
    confusion_matrix,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_score,
    train_test_split,
)
from sklearn.pipeline import Pipeline

# Small labeled corpus: 1 = positive, 0 = negative.
corpus = [
    ("I absolutely loved this movie", 1),
    ("The plot was excellent and moving", 1),
    ("A wonderful performance with great acting", 1),
    ("This film was highly enjoyable", 1),
    ("I would recommend it to everyone", 1),
    ("An amazing and entertaining experience", 1),
    ("I really hated this movie", 0),
    ("The plot was boring and predictable", 0),
    ("Terrible acting ruined the entire story", 0),
    ("This film was deeply disappointing", 0),
    ("I would not recommend it to anyone", 0),
    ("A dull and forgettable experience", 0),
]

X = [text for text, _ in corpus]
y = np.array([label for _, label in corpus])

# Split before fitting the vectorizer or classifier.
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.25,
    stratify=y,
    random_state=42,
)

model = Pipeline([
    ("vectorizer", TfidfVectorizer(ngram_range=(1, 2))),
    ("classifier", LogisticRegression(max_iter=1000, random_state=42)),
])

# Each fold learns its vocabulary and IDF weights from training data only.
cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
cv_scores = cross_val_score(
    model,
    X_train,
    y_train,
    cv=cv,
    scoring="f1_macro",
)
print(f"Mean training CV macro-F1: {cv_scores.mean():.3f}")

# Fit the final pipeline on the training split and evaluate the holdout set.
model.fit(X_train, y_train)
predictions = model.predict(X_test)

print(f"Test accuracy: {accuracy_score(y_test, predictions):.3f}")
print(
    classification_report(
        y_test,
        predictions,
        labels=[0, 1],
        target_names=["negative", "positive"],
        zero_division=0,
    )
)
print(
    "Confusion matrix:\n",
    confusion_matrix(y_test, predictions, labels=[0, 1]),
)

# Inspect the tokens learned during training.
feature_names = model.named_steps["vectorizer"].get_feature_names_out()
print("Learned vocabulary:", list(feature_names))

Common Pitfalls

  • Data leakage: Never fit the vectorizer on the full dataset before splitting or cross-validation.

  • Small datasets: This example demonstrates the API but is not deployment-ready. Use representative data and stratified cross-validation.

  • Class imbalance: Do not rely on accuracy alone; inspect per-class metrics, the confusion matrix, ROC-AUC, and PR-AUC.

  • Lost context: BoW ignores word order. Add n-grams carefully or use contextual models when negation and phrasing matter.

  • Missing text: Handle null, empty, or malformed documents before vectorization.

  • Deployment differences: Evaluate on data resembling the target domain, handle multilingual text appropriately, and keep preprocessing consistent.

  • Reproducibility: Set random seeds for splitting, cross-validation, and stochastic estimators, and record package versions.

36

How do context-free grammars represent syntax and support parsing in NLP?

A context-free grammar (CFG) is a formal system for describing the syntactic structures of a language using rewriting rules. In NLP, CFGs support constituency parsing by determining how words form phrases and complete sentences according to a grammar. The main trade-off is that CFGs provide precise, interpretable structure but require manually designed rules and may struggle with ambiguity, unusual language, and domain-specific text.

Read full answer

Applies tocontext-free grammar and NLP parsing concepts (all versions)

Formal Definition

A CFG is commonly represented as:

CFG = (N, T, P, S)

  • N: the set of nonterminals, such as S, NP, and VP.

  • T: the set of terminals, which are actual words or tokens.

  • P: the set of production rules that specify how symbols can be rewritten.

  • S: the start symbol, usually representing an entire sentence.

A production has the form:

A → β

Here, A is one nonterminal and β is a sequence of terminals or nonterminals. For example:

  • S → NP VP means a sentence consists of a noun phrase followed by a verb phrase.

  • NP → Det N means a noun phrase consists of a determiner and a noun.

  • VP → Verb NP means a verb phrase consists of a verb followed by a noun phrase.

The grammar is “context-free” because each rule rewrites one nonterminal independently of the symbols surrounding it. Unlike context-sensitive grammars, a rule such as NP → Det N does not depend on the particular sentence in which NP appears.

How Parsing Works

Given a sentence and a CFG, a parser searches for a derivation and corresponding parse tree.

  1. Tokenize the input: Split the text into words or subword tokens.

  2. Match terminals: Identify tokens that correspond to grammar rules.

  3. Apply productions: Combine terminals into phrases and phrases into larger constituents.

  4. Construct a parse tree: Show the hierarchical structure of the sentence.

  5. Handle ambiguity: Select one interpretation or preserve multiple possible parses.

For example, consider the grammar S → NP VP, NP → Det N, and VP → Verb NP. The sentence “The student read the book” can be represented by a tree whose root is S, with NP and VP as its children.

Two important parsing approaches are:

  • Top-down parsing: Starts with S and tries to expand the start symbol into terminal tokens.

  • Bottom-up parsing: Starts with input tokens and combines them into grammatical constituents until reaching S.

Chart parsers such as Earley support both directions by recording which spans of a sentence can be derived from each grammar symbol. The CYK algorithm is a specific bottom-up chart parser that requires the grammar to be in Chomsky Normal Form. CFGs commonly support constituency parsing, while dependency grammars describe relationships between words instead of nested phrase structure.

Applications in NLP

CFGs have been used for:

  • Constituency parsing: Identifying noun phrases, verb phrases, subjects, and objects.

  • Syntactic analysis: Checking whether a token sequence conforms to a formal grammar.

  • Semantic parsing: Mapping sentences or logical forms to structured meaning representations.

  • Grammar-constrained generation: Restricting generated text so that it follows specified syntactic rules.

  • Language analysis and NLP tooling: Supporting grammar checkers, domain-specific parsers, and linguistically informed systems.

For example, a grammar could require an imperative sentence to contain a verb phrase. A generation system could then reject or repair candidates that do not satisfy that constraint.

Classic and Modern NLP Context

The classic NLP pipeline commonly uses tokenization, normalization, stop-word removal, stemming or lemmatization, and representations such as bag-of-words or TF-IDF. However, syntactic parsing generally requires word order, function words, and grammatical categories, so removing stop words or applying lemmatization indiscriminately can damage parse quality.

Earlier NLP systems also used static embeddings such as word2vec, GloVe, and fastText. Modern systems instead use contextual embeddings from models such as BERT, which represent a word differently depending on its surrounding context. GPT-style models are primarily used for generation.

CFGs remain valuable for formal language analysis and interpretable constraints, but modern parsers often combine learned neural models with grammar-inspired ideas. In production, using a pretrained model with task-specific fine-tuning is generally preferred over building a parser entirely from handcrafted rules.

Limitations and Practical Considerations

  • Rule-writing cost: Large, high-quality grammars require substantial linguistic effort.

  • Ambiguity: One sentence can have several valid constituency trees, making evaluation and rule design difficult.

  • Domain shift: A grammar written for news text may perform poorly on medical dialogue or informal social media.

  • Multilingual coverage: Grammars are often English-centric and may require language-specific resources.

  • Bias and privacy: Corpus-derived grammars or models can reproduce social bias; sensitive text also requires appropriate privacy controls.

  • Neural alternatives: Sequence tagging, dependency parsing, and transformer models can be more robust, but their outputs may be less explicitly interpretable as formal derivations.

Common Pitfalls

  • Confusing terminals and nonterminals: Terminals are words or tokens; nonterminals represent grammatical categories or constituents.

  • Treating CFGs as context-sensitive: The left-hand side of each production contains exactly one nonterminal.

  • Ignoring ambiguity: A parser may return multiple parses, and valid grammaticality does not imply a unique interpretation.

  • Removing words before parsing: Stop-word removal and lemmatization can erase information required for syntactic structure.

  • Treating grammar parsing as semantic understanding: A valid parse shows syntactic structure but does not necessarily capture meaning, truth, or real-world intent.

Useful follow-up questions include: “How would you resolve structural ambiguity?”, “What is the difference between constituency and dependency parsing?”, “Why are transformers not usually described as CFG parsers?”, and “How would you evaluate a parser across several domains?”

37

How does topic modeling analyze large document collections?

Topic modeling discovers latent themes by treating each document as a mixture of learned topics, enabling large collections to be organized without document-level topic labels. Its main trade-off is the transparent, word-based themes of classical models versus the richer semantics of contextual embeddings, which require more computation and interpretation.

Read full answer

Applies tomachine learning concepts (Python 3.12)

How Topic Modeling Works

Topic models infer themes by analyzing patterns in a document-term matrix. The resulting topic distributions can support exploration, retrieval, clustering, and downstream prediction.

Approach How it works Strengths Weaknesses When to use / scale
Latent Dirichlet Allocation (LDA) Models each document as a topic mixture and each topic as a distribution over words Interpretable and probabilistic Ignores word order and most context Large count-based corpora; iterative sparse-matrix inference
Non-negative Matrix Factorization (NMF) Factors a document-term matrix into document-topic and topic-term components Fast and effective for exploration Components are not probabilistic Large TF-IDF matrices; tune the topic count
Contextual embeddings and clustering Embeds documents and clusters their vectors Captures context-sensitive similarity Compute-intensive; labels need interpretation Complex or multilingual corpora; embedding storage grows with document count and vector width

LDA and NMF produce a document-topic matrix and a topic-term matrix. Inspecting the highest-weight terms helps label each topic, while each document’s topic proportions describe its thematic composition. Memory use depends on the sparse matrix, vocabulary size, topic count, and model representation.

NLP Representation Pipeline

  • Tokenization: Split text into words or subwords. Transformer tokenizers commonly use BPE or WordPiece, representing rare or unseen words through subword units rather than discarding them.

  • Normalization: Apply consistent lowercase conversion, accent handling, and whitespace cleanup while preserving meaningful domain terms.

  • Stop words: Removing frequent function words such as “the” and “is” can reduce noise, but aggressive removal may discard important context.

  • Stemming and lemmatization: Stemming reduces inflected forms to a stem, while lemmatization maps them to a dictionary form. Both can conflate words with different grammatical roles or senses.

  • Bag-of-words, TF-IDF, and n-grams: Bag-of-words uses term counts, frequencies, or binary presence. TF-IDF emphasizes terms distinctive to a document relative to the corpus, while n-grams capture local patterns such as “machine learning.”

  • Word and contextual embeddings: word2vec, GloVe, and fastText produce static word vectors; fastText can represent unseen words through character n-grams. BERT-style encoders produce contextual representations for semantic encoding. For downstream tasks, default to pretrained transformers, fine-tuning when labeled task data is available; GPT-style models are generators rather than direct topic estimators and require factual and human review.

Tasks and Outputs

Topic signals can support several NLP tasks, but they do not replace task-specific models:

  • Text classification and sentiment analysis: Output labels, scores, or probabilities; topic proportions can be auxiliary features.

  • Named entity recognition and part-of-speech tagging: Identify entities and grammatical roles; topics can provide interpretive context.

  • Question answering: Retrieve relevant passages and produce an answer span.

  • Summarization: Generate or select a concise thematic summary.

  • Machine translation: Produce translated text; topics can help analyze domain and terminology.

  • Semantic search: Rank documents by topic or embedding similarity.

For imbalanced classification, report precision, recall, F1, ROC-AUC, and PR-AUC because accuracy alone can be misleading. BLEU measures n-gram overlap and can underperform on paraphrases; ROUGE emphasizes lexical overlap and recall; BERTScore measures contextual similarity but does not establish factual correctness. Human evaluation remains important for generation.

Evaluating and Selecting Topics

  • Coherence: Measure whether each topic’s top terms meaningfully co-occur.

  • Diversity: Check whether different topics contain distinct vocabulary.

  • Stability: Compare topics across random seeds or document subsets.

  • Interpretability: Ask domain experts whether labels accurately summarize topic contents.

  • Downstream utility: Test whether topic features improve classification, retrieval, or clustering.

Fit vectorizers and topic models only on training folds, then transform held-out data. Select the topic count using validation or cross-validated downstream performance, coherence, diversity, and stability—not the test set or visual appeal alone.

Common Pitfalls

  • Duplicated or incoherent topics: Test several topic counts and inspect representative documents.

  • Word-order blindness: Classical models cannot distinguish reordered phrases; use n-grams or contextual embeddings when order matters.

  • Domain shift: General-news topics may not transfer to medical, legal, or scientific collections.

  • Multilingual data: Use language-aware processing or multilingual embeddings rather than mixing incomparable vocabularies.

  • Privacy leakage: Redact sensitive information before training; fitted vocabularies, embeddings, and topics may otherwise expose it.

  • Bias: Pretrained models and source corpora can reproduce social bias; review topic terms, document coverage, and potentially stigmatizing labels.

  • Scaling: Prune rare terms, preserve sparse matrices, batch embedding inference, and consider implementations that support online or distributed processing.

  • Interview follow-ups: Be ready to explain topic-count selection, stability, assignment of unseen documents, coherence metrics, and why contextual embeddings improve semantic matching.

38

How are Conditional Random Fields used in NLP sequence modeling?

A Conditional Random Field (CRF) assigns a label sequence by modeling dependencies between neighboring labels rather than classifying each token independently. It produces coherent outputs such as named entities or POS tags, with the main trade-off being stronger structural consistency at the cost of more training and inference complexity.

Read full answer

Applies toscikit-crfsuite 0.5.0 (Python 3.12)

How It Works

For an input sequence x of length L and a candidate label sequence y, a linear-chain CRF computes:

score(y, x) = sum(phi(y_i, x)) + sum(A(y_(i-1), y_i))

Here:

  • x is the token sequence.

  • y is the predicted label sequence.

  • L is the number of tokens.

  • phi(y_i, x) scores a token and its possible label using features or neural emissions.

  • A(y_(i-1), y_i) scores a transition between adjacent labels.

The CRF normalizes these scores:

P(y | x) = exp(score(y, x)) / sum(exp(score(y', x)))

The denominator sums over every legal label sequence y'. Training maximizes the probability of the correct sequence; inference uses the Viterbi algorithm to find the highest-scoring sequence.

For a standard BIO scheme, the CRF can learn transitions such as B-PER -> I-PER and discourage invalid transitions such as O -> I-PER. Forward computation and Viterbi decoding take approximately O(L * T²) time for T label types.

NLP Pipeline and Model Choices

In a classic CRF pipeline, text is tokenized and normalized, then features such as token identity, capitalization, word shape, prefixes, suffixes, neighboring words, and character n-grams are supplied to the CRF. Stop words are often retained because they provide context. Stemming or lemmatization is usually avoided for NER because it can damage entity forms. Bag-of-words, TF-IDF, and n-grams can provide features, although sparse global weighting is less effective than learned representations.

A modern pipeline commonly uses contextual encoders such as BERT or RoBERTa:

  1. WordPiece or BPE tokenization splits text into subword units, allowing rare words to be represented through shared subword pieces.

  2. BERT produces contextual token representations.

  3. A linear projection produces CRF emission scores.

  4. The CRF layer decodes the most likely globally consistent BIO sequence.

Word2Vec, GloVe, and fastText provide earlier static embeddings and can feed a BiLSTM-CRF, but they assign one vector to each word regardless of context. GPT-style autoregressive models are primarily used for generation and normally do not need a CRF output layer.

Comparisons

Approach Dependencies modeled Strengths Weaknesses Typical use
Independent softmax None between labels Simple and parallel Inconsistent sequences, such as I-PER without B-PER Simple token classification
HMM Generative label dependencies Interpretable, strong probabilistic assumptions Usually weaker for rich linguistic features Older sequence-labeling systems
BiLSTM-CRF Neural emissions and label transitions Context-sensitive and structured More complex training NER, POS, chunking
BERT/RoBERTa + CRF Contextual emissions and label transitions Strong modern baseline with valid global decoding More computation and model size Production NER when consistency matters
Transformer decoder Autoregressive token dependencies Natural fit for generation CRF decoding is usually unnecessary Text generation

Code Example

This offline example trains a classic feature-based CRF for NER. The data are split before fitting, and entity-level F1 is calculated over exact entity spans.

import numpy as np
from sklearn_crfsuite import CRF

rng = np.random.default_rng(42)

names = [("Alice", "Smith"), ("Bob", "Jones"), ("Carol", "White")]
cities = [("New", "York"), ("Paris", "France"), ("Berlin", "Germany")]
verbs = ["met", "called", "visited"]

records = []
for _ in range(80):
    first, second = rng.choice(len(names), 2, replace=False)
    city = rng.integers(len(cities))
    verb = rng.integers(len(verbs))

    tokens = [
        *names[first], verbs[verb], *names[second],
        "in", *cities[city], "."
    ]
    labels = [
        "B-PER", "I-PER", "O", "B-PER", "I-PER",
        "O", "B-LOC", "I-LOC", "O"
    ]
    records.append((tokens, labels))

indices = rng.permutation(len(records))
split = int(0.8 * len(records))
train = [records[i] for i in indices[:split]]
test = [records[i] for i in indices[split:]]

def token_features(tokens, index):
    token = tokens[index]
    shape = "".join("X" if char.isupper() else
                    "x" if char.islower() else "d" for char in token)
    return {
        "token": token.lower(),
        "shape": shape,
        "prefix3": token[:3].lower(),
        "suffix3": token[-3:].lower(),
        "previous": tokens[index - 1].lower() if index else "<BOS>",
        "next": tokens[index + 1].lower() if index + 1 < len(tokens) else "<EOS>",
    }

X_train = [[token_features(tokens, i) for i in range(len(tokens))]
           for tokens, _ in train]
y_train = [labels for _, labels in train]
X_test = [[token_features(tokens, i) for i in range(len(tokens))]
          for tokens, _ in test]

model = CRF(
    algorithm="lbfgs",
    c1=0.1,
    c2=0.1,
    max_iterations=100,
    all_possible_transitions=True,
)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

def entities(labels):
    found = set()
    kind = None
    start = None
    for index, label in enumerate([*labels, "O"]):
        if label.startswith("B-"):
            if kind is not None:
                found.add((kind, start, index))
            kind, start = label[2:], index
        elif label.startswith("I-") and kind == label[2:]:
            continue
        else:
            if kind is not None:
                found.add((kind, start, index))
            kind, start = None, None
    return found

gold_entities = set()
predicted_entities = set()
for (_, gold), pred in zip(test, y_pred):
    gold_entities |= entities(gold)
    predicted_entities |= entities(pred)

tp = len(gold_entities & predicted_entities)
precision = tp / len(predicted_entities) if predicted_entities else 0.0
recall = tp / len(gold_entities) if gold_entities else 0.0
f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0

print(f"Precision: {precision:.3f}")
print(f"Recall: {recall:.3f}")
print(f"Entity-level F1: {f1:.3f}")

Practical Guidance

  • Use CRFs when global label consistency matters: NER, POS tagging, chunking, and structured extraction are typical applications.

  • Prefer pretrained fine-tuning by default: BERT or RoBERTa plus a CRF is usually stronger than sparse hand-engineered features, although it requires more computation.

  • Evaluate NER with entity-level precision, recall, and F1: token accuracy can look high because most tokens usually have the O label.

  • Prevent leakage: split by document or entity group when the same entity can appear in multiple sentences.

  • Plan for production realities: monitor domain shift, multilingual tokenization, demographic bias, and whether sending text to a third-party or pretrained model violates privacy requirements.

Common Pitfalls

  • Invalid BIO sequences: explicitly mask forbidden transitions or use constrained decoding.

  • Weak evaluation: token-level accuracy hides entity-boundary errors; use exact-span entity F1.

  • Random sentence splits: repeated entities or documents can leak information across splits.

  • Over-tuning the test set: select hyperparameters with a separate validation set or cross-validation.

  • Unnecessary complexity: if sequence consistency does not affect the downstream task, independent token classifiers may be sufficient.

39

How do rule-based, statistical, and neural NLP approaches differ?

Rule-based NLP uses explicit rules and lexicons, statistical NLP learns patterns from data, and neural NLP learns contextual representations with neural networks. Their main trade-off is explicit, auditable behavior versus learned flexibility, with neural models generally requiring more compute and offering less interpretability.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12)

Core Differences

Approach How it works Strengths Weaknesses Best fit Typical complexity or scale
Rule-based Applies authored grammars, regular expressions, lexicons, or decision rules Predictable, interpretable, effective with limited data Brittle; maintenance grows with linguistic variation Normalization, validation, fixed-format extraction Cost grows with text length, rule count, and match complexity
Statistical Learns associations between features and outputs from data Handles variability well; often effective on structured tasks Requires representative data; features may omit context Classification, sentiment analysis, POS tagging, NER Sparse linear models scale well, but n-gram vocabularies can grow sharply
Neural Learns distributed representations and task-specific decision functions Captures complex, contextual patterns; supports end-to-end learning Large models require substantial data and compute; interpretation is difficult Classification, NER, translation, summarization, QA, generation Transformer attention is roughly O(n²d) per layer for sequence length n and dimension d

These categories overlap in practice: production systems commonly combine deterministic rules, statistical models, and neural models.

Representative Techniques and Pipelines

  • Rule-based NLP: A regular expression can recognize phone numbers, while a sentiment lexicon can classify expressions containing known positive or negative terms. Every decision is traceable, but synonyms, slang, negation, and ambiguity require growing rule coverage.

  • Statistical NLP: Classic models include logistic regression, SVMs, naive Bayes, Hidden Markov Models, and Conditional Random Fields. They commonly use tokenization, normalization, stop-word removal, bag-of-words, TF-IDF, and n-grams.

  • Word embeddings: Word2Vec, GloVe, and fastText represent words as dense vectors for similarity, clustering, and downstream prediction. They capture distributional similarity but assign one vector to each word regardless of context.

  • Neural NLP: Transformers learn contextual token representations through self-attention. BERT-style encoders are commonly used for classification, NER, and extractive question answering, while GPT-style decoders generate continuations for summarization, translation, dialogue, and generative QA.

  • Subword tokenization: BPE and WordPiece divide text into reusable units, allowing rare or previously unseen words to be represented partially rather than discarded.

A classic statistical pipeline usually:

  1. Tokenizes the text.

  2. Normalizes case, whitespace, or formatting.

  3. Removes or down-weights stop words when appropriate.

  4. Applies stemming or lemmatization. Stemming heuristically truncates a word to a stem, whereas lemmatization maps inflected forms to a dictionary or contextual base form.

  5. Builds bag-of-words, TF-IDF, or n-gram features.

  6. Trains and evaluates a statistical model.

A modern pipeline tokenizes text into subword units, processes it with a pretrained transformer, and commonly fine-tunes the model on representative target-task examples.

Task Outputs and Evaluation

  • Classification and sentiment analysis produce a label or class probability. Use accuracy, precision, recall, F1, ROC-AUC, and PR-AUC; accuracy can be misleading for imbalanced classes.

  • NER and POS tagging produce labels for tokens. NER should be evaluated with entity-level precision, recall, and F1 because exact boundaries matter.

  • Question answering produces an answer span or a generated answer.

  • Summarization and translation produce generated text. BLEU, ROUGE, and BERTScore provide complementary signals, but none reliably captures every factual error or meaning change, so human evaluation remains important.

  • Information retrieval produces ranked documents or passages. Relevant metrics include precision@k, recall@k, mean reciprocal rank, and NDCG.

Use stratified cross-validation for classification and time-series splits for temporal data. Keep the final test set separate from model and threshold selection.

Practical Guidance

Pretrained models with fine-tuning are a strong default when labeled task data is limited, although domain adaptation may still be necessary. Evaluate multilingual behavior explicitly, monitor domain shift between training and production data, test for demographic or social bias, and minimize exposure of private text. Keep deterministic rules for schema validation, safety constraints, and auditable post-processing. Compare systems using task quality, latency, memory use, privacy risk, and maintenance cost rather than a single benchmark score.

Common Pitfalls

  • Treating approaches as mutually exclusive: Strong systems often combine rules, statistical models, and neural models.

  • Using one metric for every task: Classification, sequence labeling, generation, and retrieval require different evaluations.

  • Ignoring data leakage: Fit TF-IDF and other learned preprocessing only on training data, preferably inside a scikit-learn Pipeline.

  • Overfitting the validation set: Repeated tuning eventually turns the validation split into training data; retain an untouched final test set.

  • Assuming larger models solve every problem: Domain shift, unsupported languages, biased data, retrieval quality, and prompt design can still dominate results.

  • Ignoring deployment costs: Offline accuracy gains may be insignificant compared with latency, memory, privacy, or operational complexity.

40

What metrics are used to evaluate machine translation accuracy?

Machine translation evaluation metrics are quantitative measures that compare a system’s output against reference translations to assess translation quality. They solve the problem of objectively ranking different MT systems or tracking improvements without manual inspection. The main trade-off is between automatic metrics, which are fast and scalable but may not capture fluency or meaning perfectly, and human evaluation, which is accurate but expensive and slow.

Read full answer

Applies tomachine translation evaluation concepts

Automatic Evaluation Metrics

  • BLEU (Bilingual Evaluation Understudy): computes n-gram precision between candidate and references, with a brevity penalty. The formula is BLEU = BP · exp(Σ w_n · log p_n), where BP = min(1, exp(1 − r/c)) is the brevity penalty, r is the reference length, c is the candidate length, p_n is the modified n-gram precision, and w_n = 1/N (uniform weights). Usually uses up to 4-grams. BLEU is fast and correlates well with human judgment for system-level evaluation, but it does not account for recall and can penalize synonyms.

  • ROUGE (Recall-Oriented Understudy for Gisting Evaluation): focuses on recall, measuring overlap of n-grams between candidate and reference. The common variant ROUGE-L uses longest common subsequence. Its formula is ROUGE-L = (1 + β²) · (P_L · R_L) / (P_L + β² · R_L), where P_L and R_L are precision and recall based on LCS, and β controls the weight of recall. ROUGE is more suitable for summarization but also used in MT.

  • METEOR: combines unigram precision and recall with a penalty for word order differences. It also includes stemming and synonymy (via WordNet). In the original formulation, METEOR = F_mean · (1 − penalty), where F_mean = 10 · P · R / (R + 9 · P) is a recall-weighted harmonic mean of unigram precision P and recall R, and penalty = 0.5 · (chunks / matched unigrams)³ grows when the matched words are split into many non-adjacent chunks. It correlates better with human judgments than BLEU.

  • chrF: character n-gram F-score, computed on character-level n-grams. It is robust to morphologically rich languages and works well for low-resource settings.

  • BERTScore: uses contextual embeddings from BERT to match candidate tokens with reference tokens by cosine similarity, reporting precision, recall, and F1. It needs a reference translation and captures semantic similarity better than surface-level metrics.

  • COMET: a neural metric that predicts human judgments directly using a trained model. It can be reference-based or reference-free and often outperforms traditional metrics.

Metric Type What it measures Strengths Weaknesses
BLEU n-gram precision lexical overlap fast, widely used ignores recall, synonymy
ROUGE n-gram recall overlap good for summarization less sensitive to fluency
METEOR unigram F-score + alignment lexical + some semantic better correlation slower, needs resources
chrF character n-gram F-score subword overlap robust to morphology still surface-level
BERTScore embedding similarity semantic similarity captures meaning computationally heavy
COMET neural regression predicted human score state-of-the-art requires training data

Human Evaluation

Human evaluation remains the gold standard for assessing translation quality, especially for nuanced aspects like fluency, adequacy, and cultural appropriateness. Common approaches:

  • Direct Assessment: raters score translation on a scale (e.g., 0–100) for adequacy and fluency.

  • Ranking: compare multiple systems’ outputs for the same source.

  • Error Analysis: categorize errors (e.g., mistranslation, omission, fluency).

Human evaluation is time-consuming, expensive, and suffers from inter-rater disagreement, but it provides insights that automatic metrics miss.

Trade-offs and Practical Guidance

  • Speed vs Accuracy: Automatic metrics allow rapid iteration and large-scale comparison; human evaluation is used for final validation.

  • Correlation: BLEU and chrF correlate moderately with human judgments; BERTScore and COMET show higher correlation.

  • Reference Quality: Metrics that rely on references are limited by reference quality; quality-estimation metrics such as COMET-QE (COMET-Kiwi) score a translation from the source sentence alone.

  • Task Dependence: For MT, BLEU is standard; for summarization, ROUGE; for low-resource languages, chrF or COMET may be better.

  • Pipeline: Always split data, use multiple references when possible, and report both automatic and human scores.

Common Pitfalls

  • Using a single metric: Relying solely on BLEU can be misleading; always complement with human evaluation or additional metrics.

  • Ignoring brevity penalty: Short translations may get high BLEU artificially; check length ratios.

  • Data leakage: When evaluating on test set, ensure no overlap with training data.

  • Overfitting to metric: Optimizing for BLEU may not improve actual translation quality; use human evaluation for final assessment.

  • Mismatched tokenization: Different tokenizers can produce different BLEU scores; use consistent tokenization (e.g., same as the metric expects).

41

How do you clean noisy social media text for NLP?

Noise handling is the process of converting inconsistent, abbreviated, or contaminated social media text into representations suitable for NLP models. It solves problems such as irrelevant URLs, mentions, malformed characters, duplicated content, misspellings, and label noise. The main trade-off is that aggressive cleaning can remove meaningful signals such as emojis, hashtags, slang, and negation, so normalization should be task-specific and evaluated rather than treated as universally correct.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Core Strategy

Start by defining which variation is noise and which carries predictive information:

  • Normalize safely: Decode HTML entities, standardize Unicode, normalize whitespace, and lowercase text when case does not matter.

  • Replace metadata: Convert URLs to URL and mentions to USER while preserving their positions in the sentence.

  • Preserve meaningful content: Keep hashtags, emojis, slang, and intensifiers when they indicate topics or sentiment.

  • Handle negation: Preserve words such as not and never; removing them can reverse a label.

  • Reduce accidental repetition: Carefully collapse repeated characters or punctuation when they clearly result from informal typing.

  • Detect duplicates: Remove exact duplicates and consider grouping near-duplicates before splitting data.

  • Address label noise: Correct or relabel examples using annotator agreement, adjudication, or confidence-based filtering.

Normalization is not sufficient by itself. Misspellings, domain-specific slang, multilingual content, and unfamiliar hashtags may require dictionaries, domain adaptation, or subword models.

Classic NLP Pipeline

The traditional text pipeline remains useful for small datasets and interpretable baselines:

  1. Tokenization: Split text into words, subwords, or characters.

  2. Normalization: Standardize casing, Unicode, URLs, mentions, and whitespace.

  3. Stop-word handling: Remove common words only when they add little to the task; avoid removing negation or task-critical words.

  4. Stemming or lemmatization: Stemming is faster but can produce nonstandard roots; lemmatization uses linguistic rules and generally produces dictionary words.

  5. Feature extraction: Bag-of-words records word presence and counts, while TF-IDF reduces the influence of words common across the corpus.

  6. N-grams: Unigrams capture individual words, while bigrams capture phrases such as not good and highly recommended.

  7. Modeling: Apply algorithms such as logistic regression, SVMs, or linear classifiers to sparse vectors.

Splitting data before fitting the vectorizer is essential. Otherwise, vocabulary and inverse document frequencies can leak test information into training.

Modern NLP Approach

Word embeddings represent words as dense vectors:

  • word2vec: Learns local co-occurrence patterns using skip-gram or continuous bag-of-words training.

  • GloVe: Uses global co-occurrence statistics.

  • fastText: Extends word embeddings to character-level subwords, helping with rare and misspelled words.

Contextual models produce different representations for the same word according to its context. BERT is primarily used for encoding and classification, while GPT-style models are designed for generation. Their tokenizers commonly use BPE or WordPiece, which split words into reusable subword units and reduce out-of-vocabulary problems.

For most real-world NLP projects, using a pretrained model with fine-tuning is a strong default. Fine-tuning should use task-specific training data, while validation data remains reserved for model selection and early stopping.

Evaluation

Choose metrics according to the output and class distribution. For classification, report precision, recall, and F1; for imbalanced data, also examine ROC-AUC and PR-AUC. Accuracy can be misleading when a rare positive class contributes little to the overall score.

Task Typical output Suitable evaluation
Text classification or sentiment Label and class probabilities Accuracy, precision, recall, macro/weighted F1
Named entity recognition Entity spans and types Entity-level precision, recall, F1
Part-of-speech tagging One tag per token Accuracy or macro F1
Question answering Answer span or generated answer Exact match, token F1, human review
Summarization or translation Generated text ROUGE, BLEU, BERTScore plus human evaluation
Information retrieval Ranked relevant documents MRR or nDCG

Automatic generation metrics are imperfect because they may not recognize factual errors, omissions, or meaning changes. Human evaluation remains important.

Code Example

This example trains a TF-IDF and logistic-regression classifier on noisy, inline social media text. The vectorizer is inside a pipeline so that it is fitted only on training data. The cleaner now also collapses repeated characters and preserves emojis in the text (though the TF-IDF token pattern used here does not capture emojis; for emoji-aware models, use a tokenizer that includes them).

import html
import re

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import (
    StratifiedKFold,
    cross_validate,
    train_test_split,
)
from sklearn.pipeline import Pipeline

texts = np.array([
    "I hate this game!!! 😡",
    "Worst match ever https://example.com",
    "This app is terrible #fail",
    "Awful service and broken UI",
    "Really disappointed @support",
    "Not good at all",
    "Too many bugs!!!",
    "The update is awful",
    "Great game!!! #NBA 😊",
    "Not bad at all",
    "This app works great",
    "Love the new update",
    "Really happy with the service",
    "Excellent and highly recommended",
    "The new feature is great",
    "Not terrible, actually quite good",
])

labels = np.array([0] * 8 + [1] * 8)

def clean_text(text: str) -> str:
    text = html.unescape(text)
    text = re.sub(r"https?://\S+|www\.\S+", " URL ", text, flags=re.IGNORECASE)
    text = re.sub(r"@\w+", " USER ", text)
    text = re.sub(r"#(\w+)", r" \1 ", text)
    # Collapse 3+ repeated characters to 2 (e.g., "!!!" -> "!!")
    text = re.sub(r"(.)\1{2,}", r"\1\1", text)
    text = re.sub(r"\s+", " ", text.casefold()).strip()
    return text

X_train, X_test, y_train, y_test = train_test_split(
    texts,
    labels,
    test_size=0.25,
    stratify=labels,
    random_state=42,
)

model = Pipeline([
    (
        "tfidf",
        TfidfVectorizer(
            preprocessor=clean_text,
            token_pattern=r"(?u)\b\w+\b",
            ngram_range=(1, 2),
            sublinear_tf=True,
        ),
    ),
    (
        "classifier",
        LogisticRegression(
            max_iter=1000,
            class_weight="balanced",
            solver="liblinear",
            random_state=42,
        ),
    ),
])

cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
cv_results = cross_validate(
    model,
    X_train,
    y_train,
    cv=cv,
    scoring="f1_macro",
)

model.fit(X_train, y_train)
predictions = model.predict(X_test)

print("Cross-validation macro F1:", round(cv_results["test_score"].mean(), 2))
print(classification_report(
    y_test,
    predictions,
    labels=[0, 1],
    target_names=["negative", "positive"],
    zero_division=0,
))
print("Confusion matrix:")
print(confusion_matrix(y_test, predictions, labels=[0, 1]))

Common Pitfalls

  • Over-cleaning: Removing every emoji, hashtag, or unusual token can discard useful sentiment or topic signals.

  • Negation loss: Stop-word lists and naive stemming can change the meaning of a sentence.

  • Data leakage: Fit TF-IDF, normalization dictionaries, and imputers only on training folds.

  • Duplicate leakage: Group duplicated or campaign-related posts before splitting; otherwise identical content can appear in both training and test sets.

  • Wrong metric: Accuracy may conceal severe minority-class failure; inspect per-class precision and recall.

  • Domain shift: A model trained on formal text may perform poorly on slang, abbreviations, or newly emerging topics.

  • Multilingual data: Identify language before applying language-specific stop words, tokenizers, or embeddings.

  • Bias and privacy: Pretrained models may reproduce social stereotypes, and raw social media can contain sensitive or personally identifiable information. Minimize retained data, apply appropriate access controls, and document filtering decisions.

Follow-up questions interviewers often ask: - How would you handle multilingual social media text in a single pipeline? - What are the trade-offs of using a rule-based cleaner versus a learned normalization model? - How do you evaluate the impact of cleaning choices on downstream task performance? - When would you choose character-level or subword tokenization over word-level for noisy text? - How do you prevent leakage when the cleaning step uses global statistics (e.g., a slang dictionary built from the whole corpus)?

42

How to handle slang and abbreviations in text processing

Dealing with slang and abbreviations is essential in text processing because these informal variants can cause vocabulary mismatch and hurt model performance. Effective strategies include rule-based normalization, using pre-trained embeddings that capture context, and fine-tuning transformers on domain-specific data. The main trade-off is between preserving the original nuance and reducing noise in the vocabulary.

Read full answer

Applies toscikit-learn 1.5+, Hugging Face Transformers 4.40+ (Python 3.12)

Understanding the Challenge

  • Vocabulary mismatch: slang and abbreviations are often out-of-vocabulary for models trained on standard text.

  • Ambiguity: abbreviations can have multiple expansions (e.g., “u” could be “you” or “U” as a letter).

  • Drift: slang evolves quickly, making static dictionaries outdated.

  • Domain specificity: informal language is more common in social media, forums, or chat logs.

Preprocessing Strategies

  • Dictionary mapping: Create a curated mapping from slang/abbreviations to their canonical forms. This is fast but requires manual maintenance.

  • Regular expressions: Use patterns to replace known abbreviations (e.g., replace “r” with “are” only when surrounded by word boundaries).

  • Subword tokenization: Algorithms like BPE (Byte-Pair Encoding) or WordPiece break unknown words into subword units, reducing the impact of unseen slang.

  • Data augmentation: Generate synthetic variations of training examples by injecting common slang, improving model robustness.

  • Leveraging external knowledge: Integrate Urban Dictionary or community-driven lexicons, but watch for noise.

Leveraging Pre-trained Models

  • Static embeddings (word2vec, GloVe, fastText) often lack vectors for rare slang; using subword embeddings (fastText) helps.

  • Contextual embeddings (ELMo, BERT, GPT) generate dynamic representations that can capture the meaning of unseen words from context.

  • Fine-tuning: Adapting a pre-trained transformer (e.g., BERT) on a domain-specific corpus that includes slang can significantly improve performance.

  • Zero‑shot prompting: Large language models like GPT‑4 can interpret slang without explicit training, but are not always reliable for structured tasks.

Evaluation and Practical Considerations

  • Validation: Use a validation set that contains a realistic mix of slang and abbreviations to gauge true performance.

  • Metrics: For classification, prefer F1-score over accuracy if slang instances are rare; for NER, use entity‑level F1.

  • Data leakage: If you use an external slang dictionary derived from the whole dataset, you must extract it from training data only.

  • Trade‑off: Aggressive normalization can lose nuance (e.g., “sad” vs “sadly”), while leaving slang untouched may hurt model interpretability.

Code Example

The following example demonstrates a simple rule‑based normalization pipeline using Python, scikit‑learn, and a small slang dictionary. It replaces common abbreviations before TF‑IDF vectorization.

import re
from sklearn.feature_extraction.text import TfidfVectorizer

# Synthetic corpus with slang
corpus = [
    "brb, gonna finish this task later",
    "lol that was funny",
    "u r my best friend",
    "pls help me with this",
    "thx for the answer",
    "I'm gonna use the laptop"
]

# Slang dictionary (slang -> formal)
slang_dict = {
    "brb": "be right back",
    "lol": "laugh out loud",
    "u": "you",
    "r": "are",
    "pls": "please",
    "thx": "thanks"
}

def normalize(text, slang_dict):
    # Sort keys by length descending to avoid partial replacements
    for slang, formal in sorted(slang_dict.items(), key=lambda x: -len(x[0])):
        # Use word boundaries to avoid replacing inside other words
        text = re.sub(rf"\b{re.escape(slang)}\b", formal, text)
    return text

# Apply normalization
normalized_corpus = [normalize(doc, slang_dict) for doc in corpus]

# Vectorize with TF-IDF
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(normalized_corpus)

print("Normalized sentences:")
for orig, norm in zip(corpus, normalized_corpus):
    print(f"  {orig}  ->  {norm}")

print("\nTF-IDF matrix shape:", X.shape)
print("Vocabulary size:", len(vectorizer.vocabulary_))
print("Feature names (first 10):", list(vectorizer.vocabulary_.keys())[:10])

# Expected output (deterministic):
# Normalized sentences:
#   brb, gonna finish this task later  ->  be right back, gonna finish this task later
#   lol that was funny  ->  laugh out loud that was funny
#   u r my best friend  ->  you are my best friend
#   pls help me with this  ->  please help me with this
#   thx for the answer  ->  thanks for the answer
#   I'm gonna use the laptop  ->  I'm gonna use the laptop
# TF-IDF matrix shape: (6, 29)
# Vocabulary size: 29
# Feature names (first 10): ['be', 'right', 'back', 'gonna', 'finish', 'this', 'task', 'later', 'laugh', 'out']

Common Pitfalls

  • Over‑replacement: Replacing substrings that are not actual slang (e.g., changing “r” in “word”).

  • Ignoring case: Slang may appear in uppercase; the regex should be case‑insensitive or handle variations.

  • Static dictionaries: Failing to update the dictionary leads to missing new slang.

  • Leakage: Using a slang list derived from the entire dataset before splitting causes data leakage.

  • Over‑reliance on rules: Rule‑based systems may not capture context; combine with statistical or neural methods for best results.

43

Why are domain-specific corpora important in NLP?

Domain-specific corpora are text collections representative of a field such as medicine, finance, law, or customer support. They expose specialized vocabulary, terminology, and contextual patterns missing from general resources, improving domain performance; annotated corpora also provide domain-specific labels. The trade-off is the cost of collecting, cleaning, annotating, and maintaining them, plus privacy, licensing, bias, and overfitting risks.

Read full answer

Applies toNLP concepts (all versions)

Why Domain-Specific Resources Matter

General-purpose models may know common words yet misunderstand specialist language. A medical system must distinguish symptoms from diagnoses, while a financial system must interpret “interest,” “principal,” and “charge” differently from ordinary usage.

Domain resources help by providing:

  • Relevant vocabulary: Industry terms, abbreviations, product names, acronyms, and colloquial expressions.

  • Correct labels: Classes, entities, intents, sentiment labels, and relationships defined for the field.

  • Contextual examples: Real sentence structures and documents rather than generic templates.

  • Evaluation data: Representative test cases for the intended environment.

  • Relevant patterns: Grammar, negation, measurements, dates, currencies, and terminology conventions.

Without domain adaptation, errors often arise from vocabulary mismatch, ambiguous words, incorrect assumptions, and differences between training and production data.

Resources Across the NLP Pipeline

Resource Role Domain-specific benefit
Annotated corpus Trains and evaluates models Labels follow industry definitions
Unlabeled corpus Learns vocabulary, embeddings, or language models Captures terminology and context
Lexicon Encodes meanings, sentiment, or relationships Handles industry-specific senses
Gazetteer Lists entities or domain terms Supports recognition of products, organizations, chemicals, or diseases
Ontology or taxonomy Represents concepts and relationships Supports extraction and knowledge graphs
Parallel corpus Aligns texts in two languages Supports specialized translation
Rules or grammar Validates syntax and terminology Covers cases statistical models may miss

Classic and Modern Approaches

The classic NLP pipeline usually performs:

  1. Tokenization: Splitting text into words, subwords, or other units.

  2. Normalization: Applying lowercasing, accent handling, contraction expansion, or domain-specific substitutions.

  3. Stop-word handling: Removing common words, with care because words such as “without” or “not” can carry medical meaning.

  4. Stemming and lemmatization: Stemming truncates words, while lemmatization maps them to dictionary forms; both may mishandle domain terms.

  5. Feature extraction: Using bag-of-words, TF-IDF, or n-grams, which can work well when labeled data is limited.

  6. Modeling and evaluation: Training task-specific classifiers, sequence taggers, or other models.

Modern NLP uses word embeddings such as word2vec, GloVe, and fastText, or contextual embeddings from transformers. BERT-style models produce contextual representations useful for encoding and classification, while GPT-style models support generation. BPE and WordPiece tokenizers represent rare or unseen words from smaller learned units, avoiding a hard out-of-vocabulary failure.

The usual modern approach is to start with a pretrained model and fine-tune it on carefully selected domain data, while comparing against a lightweight TF-IDF or domain-embedding baseline. Domain corpora can also train embeddings when the text is relevant and representative.

Task and Evaluation Requirements

Different NLP tasks need different resources and produce different outputs:

  • Text classification and sentiment analysis produce class labels and need labeled examples and domain lexicons.

  • Named entity recognition produces entity spans and benefits from annotated documents, gazetteers, and consistent guidelines.

  • Part-of-speech tagging produces token tags and depends on domain treebanks and tag-set definitions.

  • Question answering produces an answer or answer span and requires relevant passages, document structure, and answer annotations.

  • Summarization produces a summary and needs domain documents plus human or reference summaries.

  • Machine translation produces translated text and benefits from domain-specific parallel corpora.

  • Search or retrieval produces ranked results and requires a representative domain index and query set.

Evaluate classification with accuracy and F1. For imbalanced classes, also examine precision, recall, ROC-AUC, and especially PR-AUC; accuracy can hide poor minority-class performance. NER is commonly measured with entity-level F1. Generation may use BLEU, ROUGE, or BERTScore, but these metrics have limitations and should be supplemented with human evaluation.

Common Pitfalls and Interview Tips

  • Data leakage: Split documents before fitting TF-IDF, embeddings, feature selection, or classifiers, and keep near-duplicates in one split.

  • Generic evaluation data: A Wikipedia test set cannot establish performance on medical, legal, or industrial text.

  • Over-specialization: A narrow corpus can make a model brittle to new terminology, writing styles, or sources.

  • Ignoring imbalance: Use precision, recall, F1, PR-AUC, and class weighting when appropriate rather than accuracy alone.

  • Aggressive preprocessing: Stop-word removal or stemming can erase negations, measurements, or meaningful domain terms.

  • Domain shift: Production language changes, so monitor performance and refresh corpora, lexicons, and models.

  • Oversight risks: Domain corpora may contain protected-group bias, confidential information, or copyrighted material; apply privacy controls and document licensing.

  • Multilingual gaps: A monolingual resource may perform poorly on multilingual or code-switched text.

Common follow-ups ask how to select a corpus, prevent leakage, adapt pretrained models, measure domain shift, choose TF-IDF versus transformers, and make an NLP system auditable and privacy-safe.

44

What strategies address data scarcity in low-resource languages?

Data scarcity in NLP means having too few labeled examples to train a reliable model for a particular language, domain, or dialect. Common remedies include multilingual transfer learning, targeted data collection, weak supervision, semi-supervised learning, and carefully validated augmentation. The main trade-off is that these approaches can improve coverage while introducing domain mismatch, bias, translation artifacts, or leakage.

Read full answer

Applies tomachine learning concepts (all versions)

Strategies for Scarcity

  • Multilingual pretraining and transfer learning: Fine-tune a multilingual encoder such as XLM-R on the target task. This is usually the default modern approach because it transfers lexical, syntactic, and semantic knowledge from higher-resource languages.

  • Unlabeled data and weak supervision: Pretrain or adapt on domain text, pseudo-label confident examples, or train with weakly labeled data. Consistency regularization is a related semi-supervised option; active learning prioritizes examples for annotation.

  • Targeted data collection: Prefer examples from the intended domain and dialect, and use native speakers and domain experts to review labels. Treat machine translation as a source of bias and annotation noise, not a perfect substitute.

  • Careful augmentation: Back-translation, code-switching, transliteration, and synonym replacement can expand coverage only when task validity is preserved. Keep every variant of a source example in the same split.

  • Privacy and fairness governance: Record provenance, licenses, consent, and dialect coverage; remove unnecessary personal data and test performance across language and demographic groups.

Pipeline and Model Choices

The classic pipeline is Unicode normalization, language-aware tokenization, optional stop-word removal, stemming or lemmatization, and bag-of-words, TF-IDF, or n-gram representations. Stop-word lists, tokenizers, and lemmatizers must be appropriate for the target language rather than automatically transferred from English.

Modern alternatives include BPE and WordPiece, which reduce out-of-vocabulary problems by representing many rare words through subword units. Word2vec, GloVe, and fastText provide dense static representations, with fastText particularly useful for rare or misspelled words because it uses subword information. BERT-style models provide contextual encodings, while GPT-style decoders are generally preferred for generation.

Approach How it works Strengths Weaknesses Typical use Typical complexity or scale
Character n-gram TF-IDF + linear model Learns sparse character features for a classifier such as logistic regression Fast, interpretable, tolerant of unseen tokens Limited semantic generalization Very small labeled datasets Moderate compute; memory grows with the feature count
fastText Learns static word and subword representations Rare-word robustness, efficient CPU training Static context, limited transfer Modest corpora and quick baselines Efficient at modest scale
Multilingual transformer Fine-tunes a pretrained contextual encoder Strong transfer, contextual representations Compute-intensive, possible bias Default for many modern tasks High memory and compute; GPU often useful
GPT-style multilingual decoder Fine-tunes or prompts a generative model Flexible translation, summarization, and dialogue Hallucinations, costly evaluation, higher resource use Open-ended generation Highest resource use among these options; GPU generally useful for local fine-tuning

A useful baseline is to train a classifier head over a frozen pretrained encoder, then progressively unfreeze encoder layers if the dataset permits it. This tests whether full adaptation is necessary and can reduce overfitting.

Evaluation and Task Outputs

  • Classification and sentiment analysis produce labels and, when available, probability scores. Report accuracy and macro F1, plus per-class precision, recall, and confusion matrices; accuracy can conceal poor minority-class performance. For binary tasks with probability scores, ROC-AUC and PR-AUC can also be reported, with PR-AUC often more informative when positives are rare.

  • Named entity recognition produces entity spans and is evaluated with entity-level F1. Part-of-speech tagging produces one label per token and commonly uses token accuracy. Extractive question answering produces an answer span and uses exact match or token F1.

  • Semantic search produces ranked documents and should be evaluated with human judgments or retrieval metrics such as Recall@k, MRR, and nDCG.

  • Summarization and machine translation produce text. BLEU emphasizes n-gram precision and brevity, ROUGE uses n-gram overlap with precision, recall, and F1 variants, and BERTScore measures contextual embedding similarity. None guarantees factual correctness, so human evaluation remains important.

Evaluate each language and relevant dialect slice separately using native-speaker-reviewed test data. For small datasets, use stratified cross-validation, preserve an untouched test set, and document all model-selection experiments.

Common Pitfalls

  • Translating only from one high-resource language can create an unrealistic and biased target distribution.

  • Splitting before grouping augmented or near-duplicate documents can leak related examples into the test set.

  • Reporting only aggregate metrics can hide severe failures for particular languages or dialects.

  • Assuming a multilingual pretrained model is unbiased overlooks stereotypes and uneven training coverage.

  • Ignoring domain shift can make general-web models fail in healthcare, legal, or other specialized settings.

  • Repeatedly tuning on the test set invalidates the reported result; data collection and model iteration must also respect consent, licenses, and privacy requirements.

45

How can bias be reduced in NLP models?

NLP model bias is the systematic tendency of a model to produce worse outcomes for certain groups, dialects, languages, or topics than for others. It can arise from training data, annotation practices, tokenization, pretrained models, or deployment conditions. The main trade-off is that fairness improvements may reduce aggregate accuracy or require additional data, computation, and human review.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12)

Understanding Bias

Bias may affect representation, performance, and harm in different ways:

  • Representation bias: Some languages, identities, regions, or viewpoints are underrepresented in the corpus.

  • Annotation bias: Annotators apply inconsistent stereotypes or interpret ambiguous text differently.

  • Measurement bias: A label or metric treats different linguistic expressions as equivalent when they are not.

  • Algorithmic bias: The model learns correlations between identity terms and unwanted predictions.

  • Deployment bias: User behavior, domain shift, or feedback loops differ from the training environment.

Start by defining the intended use, affected groups, acceptable harms, and fairness criteria. Demographic parity, equalized odds, calibration, and equal group error rates represent different goals and cannot always be satisfied simultaneously.

Data and Annotation Practices

  • Audit the corpus: Examine language varieties, named entities, topics, sentiment, code-switching, and demographic coverage. A balanced word count does not necessarily produce balanced or representative examples.

  • Improve collection: Use multiple sources, domain experts, and users from the communities affected by the system.

  • Respect consent and privacy: Collect only necessary data, protect sensitive attributes, and avoid using protected characteristics without a clear and lawful purpose.

  • Revise annotation guidelines: Include diverse examples, counter-stereotypical cases, and explicit rules for ambiguous or identity-dependent language.

  • Use multiple annotators: Blind annotators to irrelevant identity information, measure agreement, preserve disagreement as uncertainty, and use adjudication where necessary.

  • Balance carefully: Oversampling, undersampling, class weighting, and targeted augmentation can help, but they should not manufacture stereotypes or treat groups as interchangeable.

  • Audit downstream data: Historical records may contain discriminatory decisions. Removing protected names alone does not remove proxies such as ZIP codes, schools, or topics.

Classic NLP Pipeline

Bias controls should be applied throughout the traditional text pipeline rather than only during model training.

Pipeline component Bias risk Mitigation
Tokenization Splits names, contractions, URLs, emojis, or non-Latin scripts poorly Test domain-specific cases and preserve meaningful punctuation and boundaries
Normalization Lowercasing, accent removal, or spelling correction can erase identity signals Normalize selectively and evaluate transformations separately
Stop-word removal Generic lists can disproportionately remove words meaningful in dialects or negation Use context-aware or minimal stop-word lists and inspect removed terms
Stemming and lemmatization Aggressive normalization can conflate different identities or meanings Compare alternatives with task-specific evaluations
Bag-of-words and TF-IDF Sparse features expose direct correlations between identity terms and labels Inspect coefficients, use regularization, and test counterfactuals
N-grams Larger n-grams increase dimensionality and may capture demographic or contextual proxies Tune n-gram ranges using validation data and check subgroup performance

Before fitting a vectorizer, tokenizer, encoder, or other learned preprocessing step, split the data into training and test sets. Learn those transformations only from the training portion to prevent data leakage.

Embeddings and Modern Models

  • word2vec, GloVe, and fastText inherit associations from their training corpora. fastText can represent out-of-vocabulary words through subword information, but that does not make its embeddings bias-free.

  • BERT-style encoder models and GPT-style generative models capture context better than static embeddings, but their pretraining data can still contain stereotypes, toxic associations, privacy risks, and uneven language coverage.

  • Use pretrained models as the default starting point, then fine-tune them on carefully audited, representative data rather than training from scratch unnecessarily.

  • Test whether performance depends on names, pronouns, dialect, translation artifacts, or counterfactual substitutions.

  • Post-processing and embedding-debiasing techniques can help with particular associations, but they are partial solutions and may remove useful information.

  • For multilingual systems, evaluate every language separately and include lower-resource languages. Translation-based augmentation should be checked for meaning changes and added stereotypes.

  • Fine-tuning can overfit a small group if validation data is narrow. Use group-aware splits and cross-validation where the dataset permits.

Evaluation and Deployment

Evaluate more than overall accuracy, which can hide poor subgroup performance.

  • Classification: Report precision, recall, F1, and confusion matrices for each relevant group. Use ROC-AUC and PR-AUC for imbalanced tasks.

  • Regression: Report MAE, RMSE, and R² by subgroup.

  • Generation: Use human evaluation alongside BLEU, ROUGE, or BERTScore; these metrics do not reliably detect harmful stereotypes.

  • Fairness metrics: Compare error rates, calibration, false-positive rates, and false-negative rates. Explain which fairness definition the chosen metric represents.

  • Counterfactual testing: Replace names, locations, identity terms, or grammatical forms while preserving the intended meaning. A large prediction change may indicate inappropriate reliance on identity signals.

  • Human evaluation: Use qualified reviewers from affected communities, structured rubrics, and blinding where possible.

  • Production monitoring: Track subgroup performance, data drift, user feedback, and emerging harmful outputs. Provide appeal or review mechanisms for consequential decisions.

Common Pitfalls

  • Removing protected attributes does not eliminate correlated proxies.

  • A larger or more diverse dataset is not automatically unbiased; it may simply contain different stereotypes.

  • Rebalancing classes can reduce one fairness metric while worsening another.

  • Oversampling before splitting data causes leakage; resample only within training folds.

  • Comparing groups with very few examples produces unstable estimates. Report uncertainty and consider qualitative review.

  • Testing only English, standard spelling, and one demographic group creates a false sense of fairness.

  • A model can have similar aggregate error rates while performing substantially worse for intersectional groups.

  • Constant retraining does not replace audits, documentation, monitoring, and accountability.

Interviewers commonly ask which fairness metric was chosen, how annotation disagreement was handled, whether preprocessing was fitted inside each training fold, how counterfactual tests were constructed, and how fairness was monitored after deployment.

46

How do you build an NLP chatbot?

A chatbot uses natural language processing (NLP) to interpret a user’s message, determine its intent and context, and produce an appropriate response. A classic implementation combines tokenization, TF-IDF features, intent classification, and rule-based responses, while modern systems often use transformer models for contextual understanding and generation. The main trade-off is control and predictability versus conversational fluency, adaptability, and development cost.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Core NLP Pipeline

A traditional NLP pipeline usually performs these steps:

  1. Tokenization: Split text into words, subwords, or sentences. Modern tokenizers use BPE or WordPiece, allowing rare or unknown words to be represented as subword units rather than discarded.

  2. Normalization: Optionally lowercase text, remove punctuation, and normalize contractions or Unicode characters. Preserve case when it carries meaning, such as in named entity recognition.

  3. Stop-word handling: Remove common words such as “the” or “is.” This can reduce noise, although transformer tokenizers often benefit from retaining them.

  4. Stemming and lemmatization: Stemming produces approximate root forms; lemmatization maps words to dictionary forms. Lemmatization is usually more accurate but requires more linguistic resources.

  5. Feature extraction: Bag-of-words represents document frequency, while TF-IDF reduces the influence of common terms. Unigram features capture individual words; n-grams capture phrases such as “reset password.”

  6. Prediction: Classify the intent, retrieve relevant information, or generate a response.

Chatbot Architecture

A production chatbot commonly separates these responsibilities:

  • Intent classification: Produces a label such as track_order and optionally a confidence score.

  • Entity extraction: Extracts values such as order numbers, dates, products, and locations.

  • Dialogue management: Maintains session state and handles follow-up questions such as “What about tomorrow?”

  • Knowledge retrieval: Searches a database or document collection for grounded information.

  • Response generation: Selects a template response or asks a language model to generate one.

  • Safety and orchestration: Applies authentication, rate limits, content policies, tool controls, logging, and escalation to a human.

A retrieval-augmented generation (RAG) pipeline retrieves current or domain-specific information before generation, reducing unsupported answers.

Models and Evaluation

Approach How it works Strengths Weaknesses Typical use Typical complexity/scale
TF-IDF plus linear classifier Sparse word or n-gram features feed a linear model Fast, explainable, inexpensive Limited context and semantics Small, fixed intent sets Training O(n·d) for vectorization + O(n·d) for linear model; prediction O(d). Scales to large datasets with sparse features.
Word2Vec, GloVe, fastText Maps words to dense vectors; fastText uses subwords Efficient semantic features Static vectors and limited context Classification and similarity Training O(n·d·window) for Word2Vec/fastText; GloVe O(n·d) with co-occurrence matrix. Prediction O(1) lookup. Scales to billions of tokens.
BERT-style encoder Produces contextual representations using bidirectional attention Strong classification, extraction, and retrieval Usually does not generate conversational text directly Encoders and RAG Training O(n·d²) per layer; fine-tuning O(n·d²). Inference O(d²) per token. Typically fine-tuned on GPU for domain tasks.
GPT-style decoder Generates text autoregressively Fluent, flexible responses Hallucinations, cost, and weaker factuality Open-ended conversation Training O(n·d²) per layer; inference O(d²) per token generated. Requires significant GPU memory for large models.

Common NLP tasks have different outputs and metrics:

  • Classification and sentiment analysis: A label or score; use accuracy, precision, recall, and F1, especially for imbalanced data.

  • NER and POS tagging: Entities or grammatical labels; use entity-level F1 or token accuracy.

  • Question answering and summarization: An answer or summary; combine task metrics such as F1, ROUGE, or BERTScore with human evaluation.

  • Machine translation: Translated text; use BLEU or related metrics alongside human judgments.

  • Information retrieval/search: Ranked documents; use precision, recall, MRR, or NDCG.

BLEU, ROUGE, and BERTScore have limited semantic coverage, so chatbot quality should also be judged through human ratings, task success, groundedness, latency, and safety.

Code Example

This offline example predicts chatbot intent using TF-IDF, unigrams, bigrams, and logistic regression. The pipeline ensures that vocabulary is learned only from the training split, preventing data leakage.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report, f1_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

texts = [
    "hello", "hi there", "good morning", "hey",
    "I need help", "support please", "can you guide me", "how do I reset my password",
    "where is my order", "track my package", "order status", "shipment location",
    "bye", "goodbye", "see you later", "end chat",
]

labels = [
    "greeting", "greeting", "greeting", "greeting",
    "help", "help", "help", "help",
    "order_status", "order_status", "order_status", "order_status",
    "goodbye", "goodbye", "goodbye", "goodbye",
]

X_train, X_test, y_train, y_test = train_test_split(
    texts,
    labels,
    test_size=0.25,
    random_state=42,
    stratify=labels,
)

model = Pipeline([
    ("tfidf", TfidfVectorizer(
        lowercase=True,
        stop_words="english",
        ngram_range=(1, 2),
        sublinear_tf=True,
    )),
    ("classifier", LogisticRegression(
        max_iter=1000,
        random_state=42,
    )),
])

model.fit(X_train, y_train)
predictions = model.predict(X_test)

print(f"Accuracy: {accuracy_score(y_test, predictions):.3f}")
print(f"Weighted F1: {f1_score(y_test, predictions, average='weighted'):.3f}")
print(classification_report(y_test, predictions))

Best Practices

  • Start with a pretrained transformer and fine-tune it when appropriate; this usually outperforms training a language model from scratch.

  • Keep intent classification, entity extraction, retrieval, and generation independently testable.

  • Use strict separation between training, validation, and test data. Do not repeatedly tune against the test set.

  • Record anonymized conversations and apply retention, access-control, and deletion policies for sensitive data.

  • Evaluate multilingual performance, domain shift, demographic bias, prompt injection, and unsafe outputs separately.

  • Use GPUs for large model fine-tuning or generation; CPU inference may be sufficient for small classifiers and retrieval systems.

Common Pitfalls

  • Data leakage: Fitting TF-IDF, encoders, or scalers on the full dataset before splitting.

  • Wrong metric: Reporting accuracy for a highly imbalanced intent classifier instead of macro F1 or per-class recall.

  • Overconfident routing: Sending every message to a generative model instead of handling fixed intents with predictable templates.

  • Ignoring evaluation limits: Treating BLEU, ROUGE, or LLM-generated evaluations as substitutes for human judgment.

  • Missing context and entities: Recognizing “track my order” without extracting the relevant order number.

  • Uncontrolled generation: Allowing unsupported answers, private data retrieval, or tool calls without validation and guardrails.

47

How do you build a textual recommendation system?

A text-based recommendation system ranks items by matching textual content with a user’s inferred interests. It works especially well in cold-start situations, but content-only matching can miss complementary, collaborative, and serendipitous preferences.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+, and NumPy 2.x (Python 3.12)

Architecture

  1. Prepare item text: Combine titles, descriptions, categories, attributes, and reviews; normalize Unicode, HTML, casing, and whitespace; detect language and remove PII.

  2. Represent items: Convert text into TF-IDF vectors or embedding vectors suitable for similarity search.

  3. Build user profiles: Aggregate positively interacted items, weighting recent and strong interactions more heavily.

  4. Retrieve candidates: Find similar items with vector search, then apply availability, geography, price, safety, and diversity constraints.

  5. Rank candidates: Combine textual relevance with popularity, quality, freshness, and behavioral signals. Use collaborative or hybrid ranking when interaction data is sufficient.

  6. Evaluate and operate: Measure offline ranking quality, run online experiments, monitor latency and feedback loops, and retrain when new interactions arrive.

Text Pipeline and Model Choices

The classic pipeline tokenizes text, normalizes it, optionally removes stop words, and applies stemming or lemmatization. BoW represents term counts, TF-IDF downweights common terms, and n-grams preserve phrases such as “noise cancelling.” Aggressive stop-word removal can erase negation, so validate preprocessing for the domain.

Modern NLP uses BPE or WordPiece subword tokenization, representing rare words as several known pieces. Word2Vec, GloVe, and fastText produce static embeddings; BERT-style encoders produce contextual embeddings, while GPT-style models generate text. Fine-tuning a pretrained encoder is generally preferable to training from scratch.

Common outputs include classification labels and probabilities, sentiment polarity, entity spans, part-of-speech tags, answer spans, summaries, translations, and ranked semantic search results. Validate these outputs for domain relevance and bias.

Approach How it works Strengths Weaknesses When to use and scale
BoW/TF-IDF Sparse term weights Fast and interpretable Loses context and order Small or medium catalogs
Static embeddings One vector per word or item Fast semantic lookup One fixed meaning per word Medium-scale ANN retrieval
Contextual encoder Context-dependent vectors Strong semantic matching Compute-heavy and potentially biased GPU inference; attention is roughly O(L²)
Two-tower model Separate user and item encoders Efficient personalized retrieval Requires behavioral labels Large catalogs with ANN indexes

Evaluation and Operations

Use Recall@K, NDCG@K, MAP@K, and MRR; NDCG rewards relevant items near the top, while MAP averages precision at each relevant result. Evaluate cold-start users and long-tail items separately, and use user-held-out or temporal folds to avoid leakage. Ordinary accuracy is misleading because almost every candidate is irrelevant. Online A/B tests should measure clicks, saves, purchases, dwell time, and abandonment. For generated explanations, combine ROUGE or BERTScore with human evaluation; BLEU is more suitable for translation.

Code Example

This offline baseline holds out one interaction per user and fits TF-IDF and logistic regression without using the held-out item.

import numpy as np
import pandas as pd

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import ndcg_score
from sklearn.pipeline import Pipeline

items = pd.DataFrame([
    ("i1", "wireless headphones noise cancellation music"),
    ("i2", "wireless earbuds Bluetooth music"),
    ("i3", "noise cancelling over ear headphones travel"),
    ("i4", "ergonomic laptop stand office"),
    ("i5", "lightweight running shoes fitness"),
    ("i6", "exercise yoga mat fitness"),
    ("i7", "portable Bluetooth speaker music"),
    ("i8", "mechanical keyboard office productivity"),
    ("i9", "waterproof hiking backpack travel"),
    ("i10", "waterproof rain jacket hiking travel"),
], columns=["item_id", "text"])

tastes = {
    "u1": ["i1", "i2", "i3", "i7"],
    "u2": ["i2", "i1", "i3", "i7"],
    "u3": ["i5", "i6", "i9", "i10"],
    "u4": ["i5", "i6", "i9", "i10"],
    "u5": ["i4", "i8", "i1", "i2"],
}
interactions = pd.DataFrame(
    [(user_id, item_id)
     for user_id, item_ids in tastes.items()
     for item_id in item_ids],
    columns=["user_id", "item_id"],
)

text = items.set_index("item_id")["text"]
rng = np.random.default_rng(42)
k = 3
metrics = {
    f"HitRate@{k}": [],
    f"Precision@{k}": [],
    f"Recall@{k}": [],
    f"NDCG@{k}": [],
}

for user_id, group in interactions.groupby("user_id"):
    # Hold out one positive interaction before fitting anything.
    test_id = rng.choice(group["item_id"].to_numpy())
    train_ids = group.loc[
        group["item_id"] != test_id, "item_id"
    ].tolist()

    excluded = set(train_ids) | {test_id}
    negatives = [
        item_id for item_id in items["item_id"]
        if item_id not in excluded
    ]

    # The held-out item is excluded from vectorizer and model fitting.
    fit_ids = train_ids + negatives
    labels = [1] * len(train_ids) + [0] * len(negatives)
    model = Pipeline([
        ("tfidf", TfidfVectorizer(ngram_range=(1, 2))),
        ("logreg", LogisticRegression(
            max_iter=1000,
            class_weight="balanced",
            random_state=42,
        )),
    ])
    model.fit([text[item_id] for item_id in fit_ids], labels)

    candidate_ids = negatives + [test_id]
    positive_column = model.named_steps[
        "logreg"
    ].classes_.tolist().index(1)
    scores = model.predict_proba(
        [text[item_id] for item_id in candidate_ids]
    )[:, positive_column]

    ranking = np.argsort(-scores)
    top_ids = [candidate_ids[index] for index in ranking[:k]]

    relevant = {test_id}
    hit_count = len(relevant.intersection(top_ids))
    metrics[f"HitRate@{k}"].append(float(hit_count > 0))
    metrics[f"Precision@{k}"].append(hit_count / k)
    metrics[f"Recall@{k}"].append(hit_count / len(relevant))

    relevance = [int(item_id in relevant) for item_id in candidate_ids]
    metrics[f"NDCG@{k}"].append(
        ndcg_score([relevance], [scores], k=k)
    )

for name, values in metrics.items():
    print(f"{name}: {np.mean(values):.2f}")

Common Pitfalls

  • Data leakage: Fit vectorizers and learned models only on training data.

  • Poor preprocessing: Removing stop words can erase negation or domain-critical terms.

  • Wrong metrics: Use ranking metrics rather than ordinary accuracy.

  • Ignoring scale: Exhaustive pairwise comparison becomes expensive; use ANN indexes for large catalogs.

  • Overfitting evaluation: Tune on validation folds, not the final test set.

  • Operational blind spots: Monitor multilingual quality, domain shift, bias, privacy, popularity concentration, and GPU-versus-CPU latency.

48

How do you detect fake news articles with NLP?

Fake-news detection uses NLP to estimate whether an article or claim is likely to be misinformation, usually as a supervised classification task. It can flag suspicious content for review, but false positives waste moderation effort and false negatives allow harmful stories to spread, especially under domain shift or deliberate manipulation.

Read full answer

Applies toPython 3.12 with scikit-learn 1.5+ (PyTorch 2.x for optional BERT-style models)

Detection Pipeline

  1. Collect and label data: Use credible fact-checking labels such as reliable and misinformation. Retain article, publisher, date, topic, and story-event identifiers so related coverage can be grouped.

  2. Preprocess text: Tokenize, normalize Unicode, optionally lowercase, and remove stop words cautiously because words such as “not” affect meaning. Handle missing or empty documents explicitly.

  3. Extract features: Compare sparse TF-IDF features with contextual transformer representations.

  4. Train and evaluate: Tune the decision threshold on validation data, preserve a final test set, and deploy the model as a triage aid rather than a truth oracle.

  5. Combine evidence: Publisher reputation, metadata, duplicate reporting, and trusted claim databases can complement text signals.

Related NLP outputs include sentiment polarity, NER entity spans, POS token labels, question-answering answers, summarization abstracts, machine-translation text, and semantic search ranked documents.

Representations and Models

  • Bag of words and TF-IDF: Bag of words counts tokens, while TF-IDF emphasizes discriminative terms. Word unigrams capture topics, and n-grams capture phrases such as “election fraud.”

  • Stemming and lemmatization: Stemming applies crude suffix rules; lemmatization maps words to dictionary forms. Aggressive normalization can remove useful distinctions.

  • Static embeddings: Word2Vec, GloVe, and fastText represent words as dense vectors. fastText also uses character subwords, helping with rare and morphologically complex words.

  • Contextual embeddings: BERT-style encoders represent a word according to its context. GPT-style decoders are better suited to generation, explanation, and zero-shot assistance.

  • Subword tokenization: BPE and WordPiece reuse smaller units, reducing out-of-vocabulary problems. Multilingual support requires a suitably trained vocabulary and model.

  • Default approach: Fine-tune a pretrained encoder for classification when sufficient labeled data and compute are available.

Model How It Works Strengths Weaknesses When to Use Scale
TF-IDF + Logistic Regression Sparse n-grams feed a linear classifier Fast and interpretable Misses context and long-range meaning Baseline or limited data Sparse; cost scales with samples, nonzero features, and iterations
Fine-tuned BERT-style encoder An encoder produces document representations Handles context and negation Requires more compute and data Production classification Attention is approximately O(L²); GPUs help at scale
GPT-style model A decoder generates or scores text Supports explanations and claim analysis Higher latency and hallucination risk Verification assistance Autoregressive inference with higher memory cost

Evaluation Strategy

  • Split by article and story event to prevent duplicate coverage from leaking across sets. Use StratifiedKFold, grouped folds for related stories, and TimeSeriesSplit or chronological splits for temporal evaluation.

  • Report precision, recall, F1, ROC-AUC, and PR-AUC. PR-AUC is important for imbalanced classes, while accuracy can look high when one class dominates.

  • Select thresholds according to moderation costs: high recall catches more suspicious stories, while high precision reduces unnecessary reviews.

  • For generated explanations or summaries, use ROUGE, BLEU, or BERTScore alongside human evaluation because these metrics do not establish factual correctness.

Code Example

This offline baseline fits TF-IDF and logistic regression inside a pipeline so preprocessing is learned only from the training split.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    auc,
    classification_report,
    precision_recall_curve,
    roc_auc_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

texts = [
    "Audited data show a gradual decline in reported infections.",
    "The central bank published its quarterly inflation report.",
    "Researchers published the study methodology and results.",
    "City council approved the budget after a public hearing.",
    "SHOCKING government plot exposed, insiders claim.",
    "Scientists reveal astonishing cure doctors cannot explain.",
    "You will not believe this ridiculous social media rumor.",
    "BREAKING election scandal proved by fabricated screenshots.",
]
labels = [0, 0, 0, 0, 1, 1, 1, 1]

X_train, X_test, y_train, y_test = train_test_split(
    texts,
    labels,
    test_size=0.25,
    stratify=labels,
    random_state=42,
)

model = Pipeline([
    ("tfidf", TfidfVectorizer(
        ngram_range=(1, 2),
        sublinear_tf=True,
        strip_accents="unicode",
    )),
    ("classifier", LogisticRegression(
        max_iter=1000,
        class_weight="balanced",
        random_state=42,
    )),
])

# The vectorizer is fitted only on the training texts.
model.fit(X_train, y_train)

predictions = model.predict(X_test)
probabilities = model.predict_proba(X_test)[:, 1]
precision, recall, _ = precision_recall_curve(y_test, probabilities)

print(f"Accuracy: {accuracy_score(y_test, predictions):.3f}")
print(f"PR-AUC: {auc(recall, precision):.3f}")
print(f"ROC-AUC: {roc_auc_score(y_test, probabilities):.3f}")
print(classification_report(
    y_test,
    predictions,
    target_names=["reliable", "misinformation"],
    zero_division=0,
))

Common Pitfalls

  • Data leakage: Duplicates, syndicated stories, or preprocessing fitted on all data inflate results.

  • Missing or overcleaned text: Handle empty documents and preserve negation, capitalization, and other potentially meaningful signals.

  • Class imbalance: Use stratification, suitable class weights, and PR metrics rather than accuracy alone.

  • Domain shift and multilingual text: Test separate topics, publishers, time periods, and languages because performance may not transfer.

  • Bias and privacy: Audit publisher and demographic bias, restrict access, and avoid retaining unnecessary personal information.

  • Invalid evaluation: Do not repeatedly tune against the test set; preserve reproducibility with fixed seeds and versioned data and models. Text alone cannot establish truth, so interviewers may ask about label quality, adversarial manipulation, calibration, multilingual coverage, and threshold selection.

49

How can long documents be summarized automatically?

Extractive summarization selects important sentences from the source, while abstractive summarization generates new text that captures the document’s meaning. Extraction is easier to verify and preserves factual wording; abstraction is more fluent and compact but risks hallucinating unsupported information. A hybrid pipeline often extracts relevant material before rewriting it.

Read full answer

Applies toscikit-learn 1.5+ and Hugging Face Transformers 4.x (Python 3.12)

Common Approaches

Approach How it works Strengths Weaknesses Typical use
Extractive Scores and selects source sentences Faithful, simple, easy to audit May produce awkward transitions and limited compression News, legal, and internal documents
Abstractive A Transformer generates a new summary Fluent, concise, can reorganize information May invent facts; requires careful evaluation Reports, articles, and long-form content
Hybrid Extracts content, then a model rewrites it Balances grounding and readability More pipeline complexity High-quality production systems

For long documents, a Transformer’s context limit may require chunking. Each chunk can be summarized independently, followed by a second pass that combines and deduplicates the chunk summaries. Overlapping chunks or shared context can preserve information near boundaries.

Classic NLP Pipeline

  1. Segment the document into paragraphs and sentences. Sentence boundaries are important because summaries are usually assembled from complete sentences.

  2. Normalize the text through lowercasing, punctuation handling, and optionally removing stop words. Stop-word removal should be cautious because words such as “not” can change sentiment or meaning.

  3. Represent the text using bag-of-words, TF-IDF, or n-grams. TF-IDF reduces the influence of common words, while character or word n-grams capture phrases such as “risk assessment.”

  4. Apply morphological processing when useful. Stemming is fast but aggressive; lemmatization produces more meaningful dictionary-based forms.

  5. Score candidate sentences using frequency, graph centrality, supervised classification, or a pretrained embedding model. Redundancy penalties can prevent several nearly identical sentences from being selected.

  6. Assemble the summary in source order, remove duplicates, and enforce the desired length.

Word embeddings such as word2vec, GloVe, and fastText represent semantic similarity, but each word generally has one fixed vector. Contextual models address this limitation by producing different representations for the same word in different contexts.

Modern Transformer Approach

A common modern pipeline tokenizes text into subword units using BPE, WordPiece, or a similar algorithm. Subword tokenization represents unseen, rare, multilingual, and compound words without requiring a separate vocabulary entry for every possible word.

BERT-style encoders are useful for document and sentence representation, including extractive ranking and semantic search. GPT-style decoder models are better suited to abstractive generation because they produce text sequentially. The usual production strategy is to start with a pretrained model and fine-tune it on domain-specific document-summary pairs.

Prompts can also instruct a general model to summarize, but output length, factual consistency, and formatting are less predictable. Long-document systems should also account for multilingual quality, domain shift, demographic bias, confidential data, and the cost of running large models.

Evaluation

  • ROUGE measures overlap with reference summaries, especially recall of reference n-grams. It does not detect factual contradictions well.

  • BLEU emphasizes n-gram precision and is primarily associated with machine translation; it is less informative when several valid summaries exist.

  • BERTScore compares contextual token embeddings and captures semantic similarity better than exact n-gram overlap.

  • Human evaluation should assess factual consistency, relevance, information coverage, fluency, and appropriate emphasis.

  • For a supervised extractive system, precision, recall, and F1 can measure whether summary-worthy sentences are identified.

No single metric fully captures summarization quality, so generated summaries should be reviewed for unsupported claims before use.

Code Example

The following offline example trains a sentence classifier to recognize summary-worthy material. In a real system, the labels would come from many documents paired with human-written summaries.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split

training_sentences = [
    "Trained models make predictions from input data.",
    "The city council approved a new bicycle lane.",
    "Representative training data improves generalization.",
    "Summer temperatures reached a record high.",
    "Validation data helps estimate model performance.",
    "The museum opened a modern art exhibition.",
    "Training and validation data must remain separate.",
    "Regularization can reduce model overfitting.",
    "Cross-validation evaluates performance across folds.",
    "The football team won its opening match.",
    "Feature pipelines prepare data before modeling.",
    "Weather forecasts predict atmospheric conditions.",
]
labels = [1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0]

X_train, X_test, y_train, y_test = train_test_split(
    training_sentences,
    labels,
    test_size=0.25,
    random_state=42,
    stratify=labels,
)

# Fit preprocessing only on the training split to prevent leakage.
vectorizer = TfidfVectorizer(
    stop_words="english",
    ngram_range=(1, 2),
)
X_train_tfidf = vectorizer.fit_transform(X_train)
X_test_tfidf = vectorizer.transform(X_test)

classifier = LogisticRegression(max_iter=1000, random_state=42)
classifier.fit(X_train_tfidf, y_train)
predictions = classifier.predict(X_test_tfidf)

print(classification_report(y_test, predictions, zero_division=0))

document_sentences = [
    "Machine learning models learn patterns from training data.",
    "The exhibition features paintings from several countries.",
    "Representative data improves model generalization.",
    "A separate test set measures performance on unseen examples.",
    "Feature pipelines prepare inputs before training.",
    "The railway timetable changes next Monday.",
]

candidate_matrix = vectorizer.transform(document_sentences)
importance = classifier.predict_proba(candidate_matrix)[:, 1]

# Select the three highest-scoring sentences, then restore source order.
selected_indices = sorted(importance.argsort()[-3:].tolist())
summary = " ".join(document_sentences[i] for i in selected_indices)

print("Summary:", summary)

Common Pitfalls

  • Data leakage: Fit TF-IDF, tokenizers, and other learned preprocessing on training data only.

  • Factual errors: Abstractive models can contradict the source; compare claims against the original document.

  • Repetition: Similar high-scoring sentences may all be selected unless redundancy is controlled.

  • Position bias: Extractive systems may overvalue introductory or concluding sentences.

  • Weak evaluation: ROUGE alone can reward copied wording while missing incorrect or omitted facts.

  • Domain shift: A pretrained model may perform poorly on specialist terminology or changing topics.

  • Privacy: Documents sent to external model services may expose confidential information.

Interviewers commonly ask how you would select a length limit, remove redundant summaries, evaluate factual consistency, handle documents longer than the model context, and detect whether the summary introduces information absent from the source.

50

How do you design an NLP-powered voice assistant like Siri or Alexa?

A voice assistant pipeline converts speech into text, interprets the user’s intent, performs the requested action, and converts the response back into speech. NLP is central to understanding requests, while ASR, dialogue management, retrieval or generation, and TTS complete the system. The main trade-off is balancing understanding accuracy and response quality against latency, compute cost, reliability, and privacy.

Read full answer

Applies toPython 3.12 with scikit-learn 1.5+ for the NLP example

System Architecture

  1. Wake-word detection: Runs continuously, often locally, to avoid recording unrelated audio.

  2. Speech recognition: Streaming ASR produces candidate transcripts and confidence scores.

  3. NLU layer: Classifies intent, extracts entities or slots, and detects sentiment, urgency, and language.

  4. Dialogue and task layer: Resolves context, asks clarifying questions, calls APIs, and maintains short-term state.

  5. Response layer: Selects a template, retrieves grounded information, or generates a response with a suitable language model.

  6. Text-to-speech: Converts the response into natural speech and manages interruption or barge-in.

For an assistant such as Alexa, intent detection might recognize play_music, while entity extraction identifies the artist. The execution layer can then call a music API rather than allowing a generative model to invent the result.

NLP Approaches

The classic pipeline performs tokenization, normalization, optional stop-word removal, and either stemming or lemmatization. It then represents text with bag-of-words, TF-IDF, or n-grams.

Modern systems commonly use pretrained transformers, with BERT-style encoders for classification, embeddings, and extractive tasks, and GPT-style decoders for generation. Fine-tuning a pretrained model for the assistant’s domain is usually more practical than training from scratch. BPE and WordPiece tokenization split rare words into subword units, reducing out-of-vocabulary problems.

Approach How it works Strengths Weaknesses and typical scale
TF-IDF and linear models Sparse token or n-gram weights feed a linear classifier Fast, interpretable, inexpensive Limited semantics; linear training is roughly O(epochs * n * d)
word2vec, GloVe, fastText Represent words or subwords as fixed vectors Efficient semantic similarity; fastText handles rare words One static vector per word; weaker for context and domain shift
BERT-style encoders Produce contextual token representations Strong classification, NER, and retrieval Quadratic attention in sequence length; usually GPU-oriented
GPT-style decoders Generate tokens autoregressively Flexible conversation and synthesis Higher latency and hallucination risk; requires careful grounding

Tasks and Outputs

  • Text classification and sentiment analysis: Output an intent or sentiment label with confidence scores.

  • Named entity recognition: Identify spans such as people, locations, dates, and products.

  • Part-of-speech tagging: Assign a grammatical role to each token.

  • Question answering: Return an extracted answer or a generated answer grounded in retrieved documents.

  • Summarization: Produce an abstractive or extractive summary.

  • Machine translation: Produce translated text while preserving meaning and terminology.

  • Semantic search: Rank passages using contextual or static embeddings, BM25, or a hybrid retriever.

Code Example

This example classifies assistant intents using TF-IDF features and LogisticRegression. The Pipeline ensures that vocabulary selection and IDF fitting happen only on training data.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_score,
    train_test_split,
)
from sklearn.pipeline import Pipeline

texts = [
    "set a timer for ten minutes",
    "start a timer for five minutes",
    "cancel the countdown timer",
    "how much time is left on my timer",
    "play some jazz music",
    "put on songs by Miles Davis",
    "stop the music",
    "play my jazz playlist",
    "what is the latest news today",
    "give me today's headlines",
    "read the morning news",
    "tell me what happened today",
]
labels = [
    "set_timer", "set_timer", "set_timer", "set_timer",
    "play_music", "play_music", "play_music", "play_music",
    "get_news", "get_news", "get_news", "get_news",
]

X_train, X_test, y_train, y_test = train_test_split(
    texts,
    labels,
    test_size=0.25,
    stratify=labels,
    random_state=42,
)

model = Pipeline([
    (
        "tfidf",
        TfidfVectorizer(
            lowercase=True,
            stop_words="english",
            ngram_range=(1, 2),
            sublinear_tf=True,
        ),
    ),
    (
        "classifier",
        LogisticRegression(max_iter=1000, random_state=42),
    ),
])

model.fit(X_train, y_train)
predictions = model.predict(X_test)

print("Holdout accuracy:", accuracy_score(y_test, predictions))
print(classification_report(y_test, predictions, zero_division=0))

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_scores = cross_val_score(model, texts, labels, cv=cv)
print("Mean cross-validation accuracy:", cv_scores.mean())

Evaluation and Common Pitfalls

  • Measure each stage separately: Use WER for ASR, accuracy and macro-F1 for intent classification, entity-level F1 for NER, and recall or nDCG for semantic retrieval.

  • Evaluate generation carefully: BLEU, ROUGE, and BERTScore help, but they do not establish factuality; human evaluation and task-specific checks are still needed.

  • Prevent leakage: Fit tokenizers, vectorizers, and encoders only on training folds. Do not repeatedly tune against the test set.

  • Handle imbalance: Macro-F1, precision, recall, and PR-AUC are often more informative than accuracy.

  • Address language and domain shift: Evaluate multilingual queries and domain-specific vocabulary separately; consider multilingual encoders and domain fine-tuning.

  • Protect privacy: Prefer on-device wake-word detection, minimize audio retention, encrypt data, and obtain consent for cloud processing.

  • Audit bias: Test performance across dialects, accents, genders, and demographic groups; pretrained language models can reproduce social bias.

  • Ground generated answers: Use retrieved, authorized sources and tool results rather than relying on model memory alone.

  • Plan for scale: Run small quantized models on CPU or mobile devices; reserve GPUs for larger encoders and generation when latency requires it.

  • Use appropriate feature handling: Sparse TF-IDF vectors should not be standardized. If k-NN is used over dense embeddings, consider scaling or normalized similarity based on the feature distribution.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 50 NLP answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Machine Learning & Data Science subjects