Skip to content
Machine Learning & Data Science

TensorFlow interview questions and answers

70 questions, 69 with code. About 3 h 52 min to read them all.

By TestLab · Updated

Plan for an interview
70 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What is TensorFlow and who developed it?

TensorFlow is an open-source library for numerical computing and machine learning, developed by the Google Brain team and released in November 2015. It provides tensors, automatic differentiation, hardware acceleration on CPUs, GPUs and TPUs, and a large ecosystem for training and deploying models, with Keras as its high-level API. Its strength is production deployment across servers, mobile and browsers; the trade-off is a larger, more complex ecosystem than some research-focused alternatives.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Key Facts

  • Origin: successor to Google’s internal system DistBelief; now maintained by Google with a large open-source community under the Apache 2.0 license.

  • TensorFlow 2.0 (2019): made eager execution the default and Keras the main API.

  • TensorFlow 2.16 (2024): Keras 3 became the default Keras version.

What It Provides

Component Purpose
Tensors and ops N-dimensional arrays and math on CPU/GPU/TPU
tf.GradientTape Automatic differentiation
tf.function Compiles Python code into optimized graphs
Keras High-level API to build and train models
tf.data Fast, scalable input pipelines
tf.distribute Multi-GPU, multi-machine and TPU training
TensorBoard Visualization of training metrics, graphs and profiles
TensorFlow Serving Serving models over REST/gRPC
TensorFlow Lite (LiteRT) Mobile and edge inference
TensorFlow.js Models in the browser and Node.js
TFX End-to-end production ML pipelines

Code Example

import keras
import numpy as np
import tensorflow as tf

keras.utils.set_random_seed(42)
print(tf.__version__, "| GPUs:", tf.config.list_physical_devices("GPU"))

x = np.random.rand(1000, 4).astype("float32")
y = (x.sum(axis=1) > 2).astype("float32")

model = keras.Sequential([keras.Input(shape=(4,)), keras.layers.Dense(16, activation="relu"),
                          keras.layers.Dense(1, activation="sigmoid")])
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
history = model.fit(x, y, epochs=10, batch_size=32, validation_split=0.2, verbose=0)
print("validation accuracy:", round(history.history["val_accuracy"][-1], 3))

Common Uses

  • Image classification, detection and segmentation.

  • NLP and recommendation systems.

  • Time series forecasting.

  • On-device ML in mobile apps.

Interview Tips

  • Mention both Google Brain (origin) and the ecosystem (Keras, TFX, Lite, Serving, TensorBoard).

  • Be ready to compare with PyTorch: TensorFlow is strong in deployment tooling and TPUs; PyTorch is dominant in research.

2

What are the main features of TensorFlow?

TensorFlow’s main features are tensor computation with hardware acceleration, automatic differentiation, a high-level Keras API, graph compilation for speed, scalable data pipelines, distributed training, and a complete deployment ecosystem covering servers, mobile, browsers and production pipelines. Together they let one framework cover research prototypes through to production. The trade-off is a broad, sometimes complex ecosystem with older (1.x) material still around that no longer applies.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Feature Overview

Feature What it gives you
Eager execution (default) Operations run immediately, like NumPy; easy debugging
tf.function Turns Python functions into graphs for speed and export
Automatic differentiation tf.GradientTape records operations and computes gradients
Keras API Sequential, functional and subclassing models; compile and fit
tf.data Efficient pipelines: map, shuffle, batch, prefetch
Hardware support CPUs, NVIDIA GPUs, Google TPUs
Distributed training tf.distribute.MirroredStrategy, MultiWorkerMirroredStrategy, TPUStrategy
Mixed precision keras.mixed_precision.set_global_policy("mixed_float16")
TensorBoard Metrics, histograms, graphs, profiler
SavedModel format Language-neutral export for serving
TensorFlow Serving High-performance model server (REST and gRPC)
TensorFlow Lite (LiteRT) Quantized, small models for phones and microcontrollers
TensorFlow.js Training and inference in JavaScript
TFX Data validation, transformation, training, evaluation and deployment pipelines
Ecosystem libraries TensorFlow Probability, TF-Agents (RL), TensorFlow Datasets, KerasHub

Code Example

import tensorflow as tf

w = tf.Variable(3.0)                              # trainable state

@tf.function                                      # compiled into a graph on first call
def loss_fn(x):
    return (w * x - 6.0) ** 2

with tf.GradientTape() as tape:                   # automatic differentiation
    loss = loss_fn(tf.constant(1.0))
print("loss:", loss.numpy(), "gradient:", tape.gradient(loss, w).numpy())   # 9.0 and -6.0

ds = tf.data.Dataset.range(10).map(lambda i: i * 2).batch(4).prefetch(tf.data.AUTOTUNE)
print([b.numpy().tolist() for b in ds])           # [[0, 2, 4, 6], [8, 10, 12, 14], [16, 18]]

The gradient: loss = (3x - 6)^2 at x = 1 is 9, and d(loss)/dw = 2(3 - 6)(1) = -6.

Common Pitfalls

  • Following TensorFlow 1.x tutorials (tf.Session, tf.placeholder), which do not apply to TF 2.

  • Python side effects (like print or appending to lists) inside tf.function, which run only during tracing.

  • Assuming every feature works the same with Keras 3 multi-backend code; TensorFlow-specific APIs need the TensorFlow backend.

3

What is a computation graph in TensorFlow?

A computation graph is a data structure in which nodes are operations (such as matrix multiplication or ReLU) and edges are the tensors flowing between them. TensorFlow uses graphs to optimize, parallelize and export computations independently of Python. In TensorFlow 2 you write normal eager Python, and tf.function traces it into a graph when you want graph benefits; the trade-off is that graph code is harder to debug and Python side effects behave differently.

Read full answer

Applies toTensorFlow 2.16+ (Python 3.12)

Why Graphs Help

  • Optimization: Grappler rewrites the graph (constant folding, removing unused nodes, fusing operations); XLA can compile it further.

  • Speed: less Python overhead, especially for many small operations.

  • Portability: graphs are saved in SavedModel files and run in TensorFlow Serving, TensorFlow Lite or other languages without Python.

  • Parallelism and placement: independent operations can run concurrently on different devices.

Graphs in TensorFlow 1.x Versus 2.x

Aspect TensorFlow 1.x TensorFlow 2.x
How graphs are built Explicitly, then run in a tf.Session Automatically by tracing tf.function code
Default mode Graph mode Eager execution
Feeding data tf.placeholder + feed_dict Function arguments

Tracing

  • On the first call, tf.function runs the Python code once, records TensorFlow operations into a ConcreteFunction (a graph), and reuses it for later calls with compatible input shapes and types.

  • New input shapes or Python values cause retracing (building a new graph).

Code Example

import tensorflow as tf

@tf.function
def dense_relu(x, w, b):
    return tf.nn.relu(tf.matmul(x, w) + b)

x = tf.random.normal([2, 3], seed=42)
w = tf.random.normal([3, 4], seed=42)
b = tf.zeros([4])

print(dense_relu(x, w, b).shape)                     # (2, 4), runs the traced graph

concrete = dense_relu.get_concrete_function(x, w, b)
ops = [op.type for op in concrete.graph.get_operations()]
print(ops)                                           # includes 'MatMul', 'AddV2', 'Relu'

Common Pitfalls

  • Python print and list appends run only during tracing; use tf.print for values inside graphs.

  • Calling a tf.function with many different Python scalars or shapes causes retracing and slowdowns; pass tensors or set an input_signature.

  • Using Python if or for on tensor values: AutoGraph converts many cases to tf.cond and tf.while_loop, but not all Python constructs are supported.

4

What are tensors in TensorFlow?

Tensors are TensorFlow’s core data structure: multi-dimensional arrays with a single data type (dtype) and a shape, similar to NumPy arrays but able to live on GPUs and TPUs and to take part in automatic differentiation. Scalars, vectors, matrices and higher-dimensional arrays (such as image batches) are all tensors. Regular tf.Tensor objects are immutable; trainable, changeable state is held in tf.Variable.

Read full answer

Applies toTensorFlow 2.16+ (Python 3.12)

Key Properties

Property Meaning Example
Rank Number of dimensions 0 scalar, 1 vector, 2 matrix, 4 image batch
Shape Size of each dimension (32, 224, 224, 3) for 32 RGB images
dtype Element type float32, int64, bool, string
Device Where it is stored /CPU:0, /GPU:0

Kinds of Tensors

  • tf.Tensor: immutable values from tf.constant or operations.

  • tf.Variable: mutable, trainable state (weights).

  • tf.RaggedTensor: rows of different lengths (e.g. sentences).

  • tf.SparseTensor: mostly-zero data stored efficiently.

  • String tensors: hold byte strings for text pipelines.

Code Example

import numpy as np
import tensorflow as tf

scalar = tf.constant(7)
matrix = tf.constant([[1.0, 2.0], [3.0, 4.0]])
images = tf.zeros([8, 28, 28, 1])                    # batch of 8 grayscale images

print(scalar.ndim, matrix.shape, images.shape, matrix.dtype)   # 0 (2, 2) (8, 28, 28, 1) float32

print(tf.matmul(matrix, matrix).numpy())             # [[ 7. 10.] [15. 22.]]
print((matrix * 2 + 1).numpy())                      # element-wise with broadcasting
print(tf.reshape(matrix, [4]).numpy())               # [1. 2. 3. 4.]
print(tf.cast(matrix, tf.int32).dtype)               # int32

as_numpy = matrix.numpy()                            # tensor -> NumPy
back = tf.convert_to_tensor(np.arange(3))            # NumPy -> tensor (int64)

ragged = tf.ragged.constant([[1, 2, 3], [4]])        # rows of different lengths
print(ragged.shape)                                  # (2, None)

Common Pitfalls

  • dtype mismatches: tf.constant(1) + tf.constant(1.0) fails; TensorFlow does not auto-promote types. Use tf.cast.

  • Trying to modify a tensor in place: tensors are immutable; use tf.Variable or tf.tensor_scatter_nd_update.

  • Shape errors: check .shape and use tf.debugging.assert_shapes when building models.

  • NumPy float64 defaults: NumPy arrays are often float64; cast to float32 for most models.

5

How does TensorFlow differ from other machine learning libraries?

TensorFlow differs from other machine learning libraries mainly in scope: it is an end-to-end deep learning platform with strong production deployment tools (Serving, Lite, TensorFlow.js, TFX) and first-class TPU support, whereas scikit-learn focuses on classical machine learning and PyTorch is the leading choice for research with a very Python-native feel. In practice, TensorFlow and PyTorch overlap heavily for building neural networks; the choice depends on deployment targets, team experience and ecosystem.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3, compared with PyTorch 2.x and scikit-learn 1.5+

Comparison

Aspect TensorFlow (+ Keras) PyTorch scikit-learn JAX
Main focus Deep learning, production Deep learning, research and production Classical ML (trees, linear, clustering) High-performance numerical computing, research
Execution Eager by default, graphs via tf.function Eager by default, torch.compile for speed NumPy-based Functional, compiled with jit (XLA)
High-level API Keras (compile/fit) Plain Python loops, Lightning fit/predict estimators Flax, Keras 3
Deployment TF Serving, TF Lite (LiteRT), TF.js, TFX TorchServe, ExecuTorch, ONNX export Pickle/joblib, ONNX Export via XLA, TF
Hardware CPU, GPU, strong TPU support CPU, GPU (TPU via XLA) CPU CPU, GPU, TPU
Research adoption Smaller share than before Dominant n/a Growing in research labs
Debugging Easy in eager mode; harder inside graphs Very Python-native Simple Requires functional style

Where TensorFlow Stands Out

  • Deployment breadth: the same model can serve in a datacenter, a phone app and a web page.

  • TPUs: mature support via TPUStrategy.

  • Production pipelines: TFX covers data validation, transformation, training, evaluation and pushing models.

  • Keras 3: the same Keras code can also run on JAX or PyTorch backends.

Where Others Stand Out

  • PyTorch: most new research code and pretrained models; intuitive debugging.

  • scikit-learn: best for tabular data with classical models and preprocessing pipelines.

  • XGBoost/LightGBM: usually better than neural networks on tabular data.

Interview Tips

  • Avoid claiming one framework is strictly better; explain trade-offs honestly.

  • Mention that ONNX allows moving models between frameworks for inference.

  • Point out that TensorFlow 1.x criticisms (sessions, static graphs) no longer apply to TensorFlow 2.

6

Which devices can TensorFlow run computations on?

TensorFlow runs computations on CPUs, NVIDIA GPUs (through CUDA and cuDNN), Google TPUs, and, through TensorFlow Lite (LiteRT), on mobile CPUs, mobile GPUs, DSPs/NPUs and microcontrollers. TensorFlow.js adds browsers (WebGL, WebGPU, WebAssembly) and Node.js. Device placement is mostly automatic: operations run on a GPU when one is available, and you can control placement manually. The trade-off is that each hardware target has its own setup and supported operations.

Read full answer

Applies toTensorFlow 2.16+ (Python 3.12)

Supported Hardware

Device How Notes
CPU Default x86 and ARM; oneDNN optimizations on Intel
NVIDIA GPU pip install "tensorflow[and-cuda]" on Linux Needs compatible CUDA/cuDNN; native Windows GPU support ended after TF 2.10 (use WSL2)
Apple silicon GPU tensorflow-metal plugin Community/Apple-maintained plugin
Google TPU tf.distribute.TPUStrategy on Cloud TPU or Colab Very fast for large matrix workloads
Mobile and edge TensorFlow Lite (LiteRT) with delegates (GPU, NNAPI, Core ML, Hexagon) Quantized models; Android, iOS, embedded Linux
Microcontrollers TensorFlow Lite for Microcontrollers Kilobytes of memory
Browser / Node.js TensorFlow.js WebGL, WebGPU, WASM backends

Device Placement

  • By default, TensorFlow places an operation on the first GPU if it has a GPU kernel.

  • tf.device("/CPU:0") or "/GPU:1" forces placement.

  • tf.debugging.set_log_device_placement(True) logs where each operation runs.

  • tf.config.experimental.set_memory_growth(gpu, True) stops TensorFlow from reserving all GPU memory at start-up.

Code Example

import tensorflow as tf

print("CPUs:", tf.config.list_physical_devices("CPU"))
gpus = tf.config.list_physical_devices("GPU")
print("GPUs:", gpus)

for gpu in gpus:                                   # must run before any GPU is initialized
    tf.config.experimental.set_memory_growth(gpu, True)

with tf.device("/CPU:0"):
    a = tf.random.normal([1000, 1000], seed=42)

device = "/GPU:0" if gpus else "/CPU:0"
with tf.device(device):
    b = tf.matmul(a, a)
print("result computed on:", b.device)

Common Pitfalls

  • CUDA/cuDNN version mismatches, which make TensorFlow silently fall back to the CPU; check list_physical_devices("GPU").

  • Setting memory growth after the GPU has been initialized (raises an error).

  • Operations without a GPU kernel, which run on the CPU and cause slow host-device copies.

  • Expecting every model to convert to TensorFlow Lite; unsupported ops need SELECT_TF_OPS or redesign.

7

What is a Session in TensorFlow 1.x and what replaced it?

A Session in TensorFlow 1.x was the object that executed a computation graph: you first built a static graph of operations, then created a tf.Session and called session.run(...) to evaluate tensors, feeding inputs through placeholders. It managed resources such as variable values and device allocation. TensorFlow 2 removed sessions from the main API: eager execution runs operations immediately, and tf.function provides graph execution when needed.

Read full answer

Applies toTensorFlow 1.x (historical) and TensorFlow 2.16+ (Python 3.12)

Role of a Session in TF 1.x

  • Execute graphs: sess.run(fetches, feed_dict=...) computed only the parts of the graph needed for the requested outputs.

  • Hold state: variable values lived inside the session; they had to be initialized with tf.global_variables_initializer().

  • Manage devices: placement on CPUs/GPUs and distributed execution.

  • Lifecycle: opened and closed explicitly, or used as a context manager.

TF 1.x Versus TF 2.x

Task TensorFlow 1.x TensorFlow 2.x
Run an operation sess.run(c) c is computed immediately (eager)
Feed inputs tf.placeholder + feed_dict Ordinary function arguments
Initialize variables sess.run(tf.global_variables_initializer()) Automatic when a tf.Variable is created
Graph speed Always graph mode @tf.function
Save models tf.train.Saver model.save("model.keras"), tf.saved_model.save, tf.train.Checkpoint

Code Example

import tensorflow as tf

# TensorFlow 2: eager execution, no session
a = tf.constant(2.0)
b = tf.constant(3.0)
print((a * b).numpy())                          # 6.0, computed immediately

@tf.function                                    # graph execution when you want it
def multiply(x, y):
    return x * y
print(multiply(a, b).numpy())                   # 6.0

# Legacy TF 1.x style, only through the compatibility module (for old code)
g = tf.Graph()
with g.as_default():
    x = tf.compat.v1.placeholder(tf.float32, shape=())
    y = x * 3.0
with tf.compat.v1.Session(graph=g) as sess:
    print(sess.run(y, feed_dict={x: 2.0}))      # 6.0

Migration Tips

  • Replace sess.run with direct calls or tf.function-decorated functions.

  • Replace placeholders with function parameters; use input_signature with tf.TensorSpec to fix shapes.

  • Use tf_upgrade_v2 to convert old scripts automatically as a first step.

Common Pitfalls

  • Copying TF 1.x tutorials into TF 2 code.

  • Mixing tf.compat.v1 graph code with Keras 3 models; keep legacy code isolated or migrate it fully.

8

What are the differences between TensorFlow 1.x and TensorFlow 2.x?

TensorFlow 2.x redesigned the framework around ease of use: eager execution is the default, Keras is the standard high-level API, and graphs are created automatically with tf.function instead of being built and run through sessions. TensorFlow 1.x required building a static graph first, feeding data through placeholders and running it in a tf.Session. TF 2 code is shorter and easier to debug, while keeping graph performance and deployment through tf.function and SavedModel.

Read full answer

Applies toTensorFlow 1.15 (last 1.x release) and TensorFlow 2.16+ (Python 3.12)

Key Differences

Area TensorFlow 1.x TensorFlow 2.x
Execution Static graph + tf.Session Eager by default; tf.function for graphs
Inputs tf.placeholder, feed_dict Function arguments, tf.data
High-level API Many (tf.layers, Estimators, tf.contrib, Keras) Keras (Keras 3 since TF 2.16)
Variables Global collections, tf.get_variable, scopes Python objects (tf.Variable), tracked by Keras layers
Initialization global_variables_initializer() Automatic
Gradients tf.gradients, optimizers’ minimize on graphs tf.GradientTape
Control flow tf.cond, tf.while_loop by hand Python if/for converted by AutoGraph
Contrib modules tf.contrib Removed; moved to separate projects or core
Saving tf.train.Saver, frozen graphs SavedModel, .keras files, tf.train.Checkpoint
Estimators Recommended Deprecated, removed in TF 2.16
Compatibility — tf.compat.v1 for legacy code; tf_upgrade_v2 script

Code Comparison

import tensorflow as tf

# TF 2.x: train a linear model y = 2x + 1 with eager execution and GradientTape
x = tf.constant([[1.0], [2.0], [3.0], [4.0]])
y = 2 * x + 1
w, b = tf.Variable(0.0), tf.Variable(0.0)
opt = tf.keras.optimizers.SGD(learning_rate=0.05)

for step in range(500):
    with tf.GradientTape() as tape:
        loss = tf.reduce_mean((w * x + b - y) ** 2)
    opt.apply_gradients(zip(tape.gradient(loss, [w, b]), [w, b]))

print(round(float(w.numpy()), 2), round(float(b.numpy()), 2))   # close to 2.0 and 1.0

In TF 1.x, the same model needed placeholders for x and y, a minimize operation, a session, variable initialization and sess.run(train_op, feed_dict=...) inside the loop.

Migration Advice

  • Run tf_upgrade_v2 to rename APIs, then refactor sessions and placeholders into functions.

  • Replace tf.contrib usage with core APIs or add-ons.

  • Rewrite Estimator code with Keras models.

Common Pitfalls

  • Mixing tf.compat.v1 graph code with eager code in the same program.

  • Assuming TF 2 is slower: wrap hot code in tf.function to get graph performance.

9

How does TensorFlow handle automatic differentiation?

TensorFlow handles automatic differentiation with tf.GradientTape: while the tape is active, it records the operations applied to watched tensors (trainable tf.Variables are watched automatically), and tape.gradient(target, sources) then replays that record backwards using the chain rule (reverse-mode differentiation) to compute exact gradients. This powers every optimizer step in Keras and in custom training loops. The trade-off is memory: the tape keeps intermediate values until the gradient is computed.

Read full answer

Applies toTensorFlow 2.16+ (Python 3.12)

How It Works

  1. Operations inside with tf.GradientTape() as tape: are recorded.

  2. Each TensorFlow operation has a registered gradient function.

  3. tape.gradient(loss, variables) walks the recorded operations in reverse, multiplying local gradients (chain rule).

  4. The tape is released after one gradient call unless persistent=True.

Reverse mode is efficient for machine learning because there is one scalar loss and many parameters: one backward pass gives all partial derivatives.

Key Features

Feature Use
Automatic watching Trainable tf.Variables are tracked; call tape.watch(t) for constant tensors
persistent=True Compute several gradients from one tape
Nested tapes Higher-order derivatives (e.g. second derivatives, gradient penalties)
tape.jacobian, tape.batch_jacobian Full Jacobian matrices
tf.stop_gradient Treat a value as constant
@tf.custom_gradient Define your own gradient (e.g. for numerical stability)

Code Example

import tensorflow as tf

# 1. Gradient of a simple function: f(x) = x^3 + 2x, f'(x) = 3x^2 + 2
x = tf.constant(2.0)
with tf.GradientTape() as tape:
    tape.watch(x)                       # constants are not watched automatically
    y = x ** 3 + 2 * x
print("dy/dx:", tape.gradient(y, x).numpy())          # 14.0

# 2. Second derivative with nested tapes: f''(x) = 6x
with tf.GradientTape() as outer:
    outer.watch(x)
    with tf.GradientTape() as inner:
        inner.watch(x)
        y = x ** 3 + 2 * x
    dy = inner.gradient(y, x)
print("d2y/dx2:", outer.gradient(dy, x).numpy())       # 12.0

# 3. Gradients for model weights
w = tf.Variable([[1.0], [2.0]])
b = tf.Variable(0.5)
inputs = tf.constant([[1.0, 3.0]])
with tf.GradientTape() as tape:
    loss = tf.reduce_sum((tf.matmul(inputs, w) + b - 10.0) ** 2)
dw, db = tape.gradient(loss, [w, b])
print(dw.numpy().ravel(), db.numpy())                  # [-5. -15.] -5.0

For the last case: prediction = 11 + 32 + 0.5 = 7.5, error = -2.5, so dL/db = 2 * (-2.5) = -5 and dL/dw = -5 times the inputs, giving [-5, -15].

Common Pitfalls

  • None gradients: the variable was not used in the computation, the path went through NumPy (which breaks the tape), or a non-differentiable op such as tf.argmax or an integer cast was used.

  • Computing outside the tape: the loss must be calculated inside the with block.

  • Integer tensors: gradients only flow through floating-point tensors.

  • Memory: persistent tapes keep resources until deleted (del tape).

10

What is a placeholder in TensorFlow and what replaced it?

A placeholder (tf.placeholder) was a TensorFlow 1.x graph node that stood in for input data to be supplied later: you declared its type and shape when building the static graph and fed real values through feed_dict when running it in a tf.Session. Placeholders no longer exist in the TensorFlow 2 API: data is passed as ordinary function arguments, tf.function input signatures (tf.TensorSpec) fix shapes and types when needed, and keras.Input defines model inputs.

Read full answer

Applies toTensorFlow 1.x (historical) and TensorFlow 2.16+ (Python 3.12)

How Placeholders Worked in TF 1.x

  • Declared with a dtype and a (possibly partially unknown) shape, e.g. shape=(None, 784) for any batch size.

  • Fed at run time: sess.run(output, feed_dict={x: batch}).

  • Separated graph definition from data, which made graphs reusable but code verbose.

TensorFlow 2 Replacements

TF 1.x TF 2.x
tf.placeholder as a graph input Python function argument
Fixed shape/dtype of an input @tf.function(input_signature=[tf.TensorSpec(...)])
Placeholder for model inputs keras.Input(shape=...)
feed_dict batches tf.data.Dataset pipelines

Code Example

import keras
import numpy as np
import tensorflow as tf

# TF 2: an input signature plays the role of a placeholder's shape and dtype
@tf.function(input_signature=[tf.TensorSpec(shape=[None, 3], dtype=tf.float32)])
def normalize(batch):
    return (batch - tf.reduce_mean(batch, axis=0)) / (tf.math.reduce_std(batch, axis=0) + 1e-8)

print(normalize(tf.random.normal([4, 3], seed=42)).shape)    # (4, 3); any batch size works

# Keras models declare inputs with keras.Input
inputs = keras.Input(shape=(3,), name="features")             # batch dimension is implicit
model = keras.Model(inputs, keras.layers.Dense(1)(inputs))
print(model.predict(np.ones((2, 3), dtype="float32"), verbose=0).shape)   # (2, 1)

# Legacy TF 1.x style, only through tf.compat.v1 for old code
g = tf.Graph()
with g.as_default():
    x = tf.compat.v1.placeholder(tf.float32, shape=(None, 3))
    total = tf.reduce_sum(x)
with tf.compat.v1.Session(graph=g) as sess:
    print(sess.run(total, feed_dict={x: np.ones((2, 3))}))    # 6.0

Common Pitfalls

  • Using tf.placeholder directly in TF 2 raises AttributeError; it only exists under tf.compat.v1.

  • Over-constraining input_signature shapes; use None for dimensions that vary (batch size, sequence length).

  • Converting old code mechanically while keeping session-based logic; refactor to functions and tf.data.

11

What is TensorFlow Lite and where is it used?

TensorFlow Lite (now branded LiteRT) is TensorFlow’s runtime for running machine learning models on mobile phones, embedded devices and microcontrollers. A trained model is converted to a compact .tflite flatbuffer, often quantized to 8-bit integers, and executed by a small interpreter that can use hardware accelerators through delegates. It enables fast, private, offline inference on device; the trade-off is that not every TensorFlow operation is supported and quantization can reduce accuracy slightly.

Read full answer

Applies toTensorFlow 2.16+ and TensorFlow Lite / LiteRT (Python 3.12)

Workflow

  1. Train a model in TensorFlow/Keras.

  2. Convert with tf.lite.TFLiteConverter.

  3. Optimize: post-training quantization (dynamic range, float16, full integer) or quantization-aware training.

  4. Deploy the .tflite file in an Android, iOS, embedded Linux or microcontroller app.

  5. Run with the interpreter, optionally with delegates (GPU, NNAPI on Android, Core ML on iOS, Hexagon DSP, Edge TPU).

Where It Is Used

Area Examples
Mobile apps Image classification, object detection, OCR, smart replies, on-device translation
Camera and AR Real-time segmentation, pose estimation, face effects
IoT and edge devices Raspberry Pi, Coral Edge TPU, industrial sensors
Microcontrollers Keyword spotting (“wake words”), gesture detection (TensorFlow Lite for Microcontrollers)
Privacy-sensitive features Health, keyboard prediction, where data stays on the device

Code Example

import keras
import numpy as np
import tensorflow as tf

model = keras.Sequential([keras.Input(shape=(28, 28)), keras.layers.Flatten(),
                          keras.layers.Dense(64, activation="relu"), keras.layers.Dense(10)])

converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]       # dynamic-range quantization
tflite_bytes = converter.convert()
with open("model.tflite", "wb") as f:
    f.write(tflite_bytes)
print(f"TFLite model size: {len(tflite_bytes) / 1024:.1f} KB")

interpreter = tf.lite.Interpreter(model_content=tflite_bytes)
interpreter.allocate_tensors()
inp, out = interpreter.get_input_details()[0], interpreter.get_output_details()[0]
interpreter.set_tensor(inp["index"], np.random.rand(1, 28, 28).astype(np.float32))
interpreter.invoke()
print("output shape:", interpreter.get_tensor(out["index"]).shape)   # (1, 10)

Benefits and Limitations

  • Benefits: low latency, works offline, keeps data private, low power use, small binary.

  • Limitations: subset of operations (use SELECT_TF_OPS for others at a size cost), limited on-device training, accuracy loss from aggressive quantization.

Common Pitfalls

  • Not testing the converted model’s accuracy against the original.

  • Full-integer quantization without a representative dataset for calibration.

  • Assuming a delegate supports every layer; unsupported ops fall back to the CPU.

12

What is a tf.Variable in TensorFlow and why is it important?

A tf.Variable is a tensor whose value can change during a program and persists across calls, which makes it the way TensorFlow stores model state such as weights, biases and optimizer statistics. Unlike an immutable tf.Tensor, a variable can be updated in place with assign, assign_add or assign_sub, is watched automatically by tf.GradientTape, and is saved in checkpoints. Without variables, models could not learn, because training is the repeated updating of these values.

Read full answer

Applies toTensorFlow 2.16+ (Python 3.12)

Key Properties

Property Description
Mutable Updated in place with assign* methods
Persistent Keeps its value between function calls and tf.function executions
trainable If True (default), tapes watch it and Keras optimizers update it
Device placement Stored on CPU or GPU like tensors
Checkpointing Saved and restored with tf.train.Checkpoint and Keras saving
Shape and dtype Fixed when created (unless shape=tf.TensorShape(None) is given)

Variable Versus Tensor

Aspect tf.Tensor tf.Variable
Mutability Immutable Mutable
Typical use Inputs, intermediate results Weights, counters, state
Gradient tracking Only if watched Automatic when trainable
Saved in checkpoints No Yes

Code Example

import tensorflow as tf

w = tf.Variable(tf.random.normal([3, 2], seed=42), name="weights")
b = tf.Variable(tf.zeros([2]), name="bias")
step = tf.Variable(0, trainable=False, dtype=tf.int64)    # non-trainable state

x = tf.constant([[1.0, 2.0, 3.0]])
target = tf.constant([[1.0, 0.0]])
optimizer = tf.keras.optimizers.SGD(learning_rate=0.1)

for _ in range(3):
    with tf.GradientTape() as tape:
        loss = tf.reduce_mean((tf.matmul(x, w) + b - target) ** 2)
    grads = tape.gradient(loss, [w, b])                   # trainable variables are watched
    optimizer.apply_gradients(zip(grads, [w, b]))         # updates variables in place
    step.assign_add(1)
    print(f"step {step.numpy()}: loss={loss.numpy():.4f}")

ckpt = tf.train.Checkpoint(w=w, b=b, step=step)
path = ckpt.save("./ckpt/model")                          # variables persist to disk
print("saved to", path)

In Keras, layers create their variables automatically (layer.kernel, layer.bias), and model.trainable_variables lists them.

Common Pitfalls

  • Creating variables inside a tf.function on every call; create them once outside (or only on the first call).

  • Replacing a variable with a new tensor (w = w + 1) instead of w.assign_add(1); the first creates a new tensor and breaks tracking.

  • Forgetting trainable=False for counters or statistics that should not receive gradients.

13

Which data types does TensorFlow support?

TensorFlow supports a range of data types (dtypes) for tensors: floating-point (float16, bfloat16, float32, float64), signed and unsigned integers (int8 to int64, uint8 to uint64), bool, string (byte strings), complex numbers (complex64, complex128) and quantized types used in model compression. Every tensor has exactly one dtype, and TensorFlow does not automatically mix types in operations, so you convert with tf.cast. Choosing the right dtype affects memory, speed and numerical stability.

Read full answer

Applies toTensorFlow 2.16+ (Python 3.12)

Common Data Types

Category dtypes Typical use
Floating point tf.float32 (default for floats) Model weights and activations
tf.float16, tf.bfloat16 Mixed-precision training on GPUs/TPUs
tf.float64 High-precision scientific computing
Signed integers tf.int8, tf.int16, tf.int32 (default for Python ints), tf.int64 Indices, counts, labels, quantized weights
Unsigned integers tf.uint8, tf.uint16, tf.uint32, tf.uint64 Image pixels (uint8)
Boolean tf.bool Masks and conditions
String tf.string Raw text, file paths, serialized records (variable-length bytes)
Complex tf.complex64, tf.complex128 Signal processing, FFTs
Quantized tf.qint8, tf.quint8, tf.qint32 Low-level quantized operations

Code Example

import numpy as np
import tensorflow as tf

print(tf.constant(1).dtype, tf.constant(1.0).dtype)          # int32 float32
print(tf.constant(np.array([1.0])).dtype)                    # float64: NumPy's default is kept

pixels = tf.constant([[0, 128, 255]], dtype=tf.uint8)
scaled = tf.cast(pixels, tf.float32) / 255.0                 # convert before math
print(scaled.numpy())                                        # [[0. 0.50196 1.]]

text = tf.constant(["hello", "tensorflow"])
print(text.dtype, tf.strings.length(text).numpy())           # <dtype: 'string'> [ 5 10]

mask = tf.constant([True, False, True])
print(tf.boolean_mask(tf.constant([1, 2, 3]), mask).numpy()) # [1 3]

half = tf.constant([65504.0], dtype=tf.float16)
print(half * 2)                                              # inf: float16 overflows above 65504

try:
    tf.constant(1) + tf.constant(1.0)                        # int32 + float32
except tf.errors.InvalidArgumentError as e:
    print("dtype mismatch:", type(e).__name__)

Choosing dtypes

  • Models: float32 by default; mixed precision (float16 or bfloat16 compute with float32 weights) for speed on modern accelerators.

  • Labels: int32 or int64 for sparse categorical losses.

  • Images: store as uint8, convert to float32 when training.

  • Text: tf.string in input pipelines, converted to integer token IDs.

Common Pitfalls

  • NumPy float64 arrays creating float64 tensors and mismatches with float32 weights.

  • Integer division surprises; cast to float first.

  • float16 overflow and underflow; use loss scaling in mixed precision (Keras does it automatically).

14

How do you build a neural network in TensorFlow?

Building a neural network in TensorFlow is usually done with Keras, its high-level API: you define the layers (with the Sequential API, the functional API or by subclassing keras.Model), compile the model with an optimizer, a loss and metrics, train it with fit on your data, and evaluate it on held-out data. For full control you can also write a custom training loop with tf.GradientTape. Keras keeps simple models short while still allowing complex architectures.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Steps

  1. Prepare data: split into train, validation and test sets; scale features; build a tf.data pipeline for large data.

  2. Define the architecture: input shape, hidden layers with activations, output layer matching the task.

  3. Compile: choose optimizer (e.g. Adam), loss (e.g. sparse categorical cross-entropy) and metrics.

  4. Train: fit with validation data and callbacks such as EarlyStopping.

  5. Evaluate and predict: evaluate on the test set; predict for new data.

  6. Save: model.save("model.keras").

Choosing the Output Layer

Task Output layer Loss
Binary classification Dense(1, activation="sigmoid") binary_crossentropy
Multi-class (integer labels) Dense(k) with softmax, or logits + from_logits=True sparse_categorical_crossentropy
Regression Dense(1) (linear) mse or mae

Code Example

import keras
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

keras.utils.set_random_seed(42)
X, y = load_digits(return_X_y=True)
X = (X / 16.0).astype("float32")                       # scale pixel values to [0, 1]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

model = keras.Sequential([
    keras.Input(shape=(64,)),
    keras.layers.Dense(128, activation="relu"),
    keras.layers.Dropout(0.2),
    keras.layers.Dense(64, activation="relu"),
    keras.layers.Dense(10, activation="softmax"),
])
model.compile(optimizer=keras.optimizers.Adam(1e-3),
              loss="sparse_categorical_crossentropy", metrics=["accuracy"])

model.fit(X_train, y_train, validation_split=0.1, epochs=30, batch_size=32, verbose=0,
          callbacks=[keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True)])
loss, acc = model.evaluate(X_test, y_test, verbose=0)
print(f"test accuracy: {acc:.3f}")                      # around 0.97
model.summary()

Common Pitfalls

  • Mismatched loss and labels (one-hot labels with a sparse loss, or the reverse).

  • Softmax output combined with from_logits=True, which applies softmax twice.

  • Unscaled inputs, which slow or prevent learning.

  • Evaluating on the training data, which hides overfitting.

15

What does compiling a model mean in TensorFlow Keras?

Compiling a Keras model with model.compile(...) configures how it will be trained: which optimizer updates the weights, which loss function is minimized, which metrics are reported, and options such as JIT compilation or the number of steps per execution. It does not train the model or change its weights. Compilation must happen before fit or evaluate, and you must compile again after changing which layers are trainable.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

What compile Sets

Argument Purpose Examples
optimizer Algorithm that updates the weights "adam", keras.optimizers.SGD(0.01, momentum=0.9)
loss Quantity to minimize "binary_crossentropy", "sparse_categorical_crossentropy", "mse", custom function
metrics Values to monitor (not optimized) ["accuracy"], keras.metrics.AUC(), keras.metrics.Precision()
loss_weights Weights for multiple outputs {"price": 1.0, "category": 0.5}
jit_compile XLA compilation for speed True, False or "auto" (default in Keras 3)
steps_per_execution Batches per compiled call; reduces overhead 32
run_eagerly Disable graph compilation for debugging True

What Happens After Compile

  • fit builds a training step (forward pass, loss, gradients, optimizer update) and compiles it as a tf.function graph (or XLA program).

  • Optimizer state (e.g. Adam’s moment estimates) is created when training starts.

Code Example

import keras
import numpy as np

keras.utils.set_random_seed(42)
model = keras.Sequential([keras.Input(shape=(8,)), keras.layers.Dense(16, activation="relu"),
                          keras.layers.Dense(1)])          # outputs logits (no sigmoid)

model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=1e-3),
    loss=keras.losses.BinaryCrossentropy(from_logits=True),   # matches the logit output
    metrics=[keras.metrics.BinaryAccuracy(threshold=0.0),     # threshold 0 on logits = 0.5 on probabilities
             keras.metrics.AUC(from_logits=True)],
)

X = np.random.rand(500, 8).astype("float32")
y = (X[:, 0] + X[:, 1] > 1).astype("float32")
history = model.fit(X, y, epochs=5, verbose=0)
print({k: round(v[-1], 3) for k, v in history.history.items()})

model.compile(optimizer="adam", loss=keras.losses.BinaryCrossentropy(from_logits=True),
              run_eagerly=True)                             # recompile to debug step by step

Common Pitfalls

  • Mismatched output activation and loss (from_logits set wrongly).

  • Forgetting to recompile after setting layer.trainable = False/True; the change is not applied to training.

  • Expecting metrics to influence training; only the loss does.

  • Using run_eagerly=True in production; it is much slower.

16

How do TensorFlow optimizers work, and which ones are common?

Optimizers in TensorFlow decide how model weights are updated from their gradients to reduce the loss. Each step they take the gradients computed by tf.GradientTape (or by fit), apply an update rule that may use momentum or adaptive per-parameter learning rates, and write the new values into the model’s variables. Common choices are SGD with momentum, Adam, AdamW, RMSprop and Adagrad; the choice affects training speed, stability and how well the model generalizes.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Common Optimizers

Optimizer Update idea Typical use
SGD w = w - lr * g Simple; with momentum often generalizes well (vision)
SGD + momentum / Nesterov Accumulates a velocity of past gradients CNN training with learning rate schedules
RMSprop Divides by a moving average of squared gradients RNNs, noisy problems
Adagrad Per-parameter rates shrink with accumulated squared gradients Sparse features
Adam Momentum (first moment) + RMSprop-style scaling (second moment), with bias correction Default for many problems
AdamW Adam with decoupled weight decay Transformers, modern deep nets
Nadam, Adamax, Lion, Adafactor Variants with different trade-offs Specialized cases; Adafactor saves memory

Adam’s Update

With gradient g_t, learning rate lr, decay rates beta_1 (about 0.9), beta_2 (about 0.999) and small epsilon:

  • m_t = beta_1 * m_{t-1} + (1 - beta_1) * g_t (moving average of gradients)

  • v_t = beta_2 * v_{t-1} + (1 - beta_2) * g_t^2 (moving average of squared gradients)

  • m_hat = m_t / (1 - beta_1^t), v_hat = v_t / (1 - beta_2^t) (bias correction)

  • w = w - lr * m_hat / (sqrt(v_hat) + epsilon)

Code Example

import keras
import numpy as np
import tensorflow as tf

keras.utils.set_random_seed(42)
X = np.random.rand(1000, 10).astype("float32")
y = (X @ np.arange(10, dtype="float32") + np.random.normal(0, 0.1, 1000)).astype("float32")

schedule = keras.optimizers.schedules.CosineDecay(initial_learning_rate=1e-2, decay_steps=1000)
optimizers = {
    "sgd_momentum": keras.optimizers.SGD(learning_rate=1e-2, momentum=0.9),
    "adam": keras.optimizers.Adam(learning_rate=1e-2),
    "adamw_cosine": keras.optimizers.AdamW(learning_rate=schedule, weight_decay=1e-4),
}
for name, opt in optimizers.items():
    model = keras.Sequential([keras.Input(shape=(10,)), keras.layers.Dense(32, activation="relu"),
                              keras.layers.Dense(1)])
    model.compile(optimizer=opt, loss="mse")
    hist = model.fit(X, y, epochs=20, batch_size=32, verbose=0)
    print(f"{name}: final training MSE = {hist.history['loss'][-1]:.4f}")

# Manual use in a custom loop
w = tf.Variable(5.0)
opt = keras.optimizers.Adam(learning_rate=0.1)
for _ in range(100):
    with tf.GradientTape() as tape:
        loss = (w - 2.0) ** 2
    opt.apply_gradients([(tape.gradient(loss, w), w)])
print("w after Adam:", round(float(w.numpy()), 3))        # close to 2.0

Common Pitfalls

  • A learning rate that is too high (divergence, NaN loss) or too low (very slow training); the learning rate matters more than the optimizer choice.

  • Using Adam’s L2 regularization when you want decoupled weight decay (use AdamW).

  • Reusing one optimizer instance for two different models; each model needs its own optimizer state.

17

How do you do batch processing in TensorFlow?

Batch processing in TensorFlow means grouping several examples into one tensor so the model processes them together, which uses the GPU efficiently and gives stable gradient estimates. You create batches with tf.data.Dataset.batch (or pass batch_size to model.fit), usually after shuffling and before prefetching. The batch size is a trade-off: larger batches are faster per example but use more memory and can generalize slightly worse without learning-rate adjustments.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Key Tools

Tool Purpose
Dataset.batch(n, drop_remainder=False) Stack n consecutive elements into one batch
Dataset.padded_batch(n) Batch variable-length sequences by padding
Dataset.ragged_batch(n) Batch into ragged tensors without padding
Dataset.shuffle(buffer) Randomize order before batching
Dataset.prefetch(tf.data.AUTOTUNE) Prepare the next batch while the current one trains
Dataset.unbatch() / rebatch() Split or re-group existing batches
model.fit(x, y, batch_size=32) Batching for in-memory NumPy data
model.predict(x, batch_size=256) Batched inference

Code Example

import numpy as np
import tensorflow as tf

tf.random.set_seed(42)
features = np.random.rand(1000, 8).astype("float32")
labels = np.random.randint(0, 2, size=1000)

ds = (tf.data.Dataset.from_tensor_slices((features, labels))
      .shuffle(buffer_size=1000, seed=42)       # shuffle individual examples first
      .batch(64)                                # then group into batches
      .prefetch(tf.data.AUTOTUNE))

for x_batch, y_batch in ds.take(1):
    print(x_batch.shape, y_batch.shape)         # (64, 8) (64,)
print("batches per epoch:", ds.cardinality().numpy())   # 16 (the last batch has 40 examples)

# Variable-length sequences: pad within each batch
seqs = tf.data.Dataset.from_generator(lambda: ([1] * n for n in [3, 5, 2, 4]),
                                      output_signature=tf.TensorSpec([None], tf.int32))
for padded in seqs.padded_batch(2):
    print(padded.numpy())                       # [[1 1 1 0 0] [1 1 1 1 1]], then [[1 1 0 0] [1 1 1 1]]

Choosing a Batch Size

  • Start with 32-256 for most models; increase until GPU memory is nearly full if speed matters.

  • With much larger batches, scale the learning rate up and use warm-up.

  • For multi-GPU training, the global batch is split across replicas.

  • Use drop_remainder=True when every batch must have the same shape (TPUs, some custom layers).

Common Pitfalls

  • Batching before shuffling, which shuffles only whole batches.

  • Batch normalization with very small batches, which gives noisy statistics.

  • Forgetting batch dimensions when predicting a single example (x[None, ...] or tf.expand_dims).

18

What role do loss functions play in TensorFlow, and which are common?

A loss function measures how far a model’s predictions are from the true targets and turns that into a single number that training minimizes. Its gradients drive every weight update, so the loss defines what “better” means for the model. TensorFlow/Keras provides standard losses for regression, classification and other tasks, and you can write custom ones; choosing a loss that matches the output layer and the business goal is essential.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Common Losses

Task Loss Formula / idea
Regression MeanSquaredError mean((y - y_hat)^2); penalizes large errors
Regression (robust) MeanAbsoluteError, Huber Less sensitive to outliers
Binary classification BinaryCrossentropy -[y log p + (1 - y) log(1 - p)]
Multi-class, one-hot labels CategoricalCrossentropy -sum_k y_k log p_k
Multi-class, integer labels SparseCategoricalCrossentropy Same, without one-hot encoding
Imbalanced classification BinaryFocalCrossentropy Down-weights easy examples
Distribution matching KLDivergence Distillation, VAEs
Similarity learning CosineSimilarity Embedding alignment
Quantiles / intervals Custom pinball loss Probabilistic forecasting

Logits Versus Probabilities

  • If the last layer has no activation (it outputs logits), set from_logits=True; this is more numerically stable.

  • If the last layer uses sigmoid or softmax, keep from_logits=False (the default).

Code Example

import keras
import numpy as np
import tensorflow as tf

y_true = tf.constant([0, 2, 1])
logits = tf.constant([[2.0, 0.5, 0.1], [0.2, 0.3, 3.0], [0.1, 2.5, 0.3]])
loss_fn = keras.losses.SparseCategoricalCrossentropy(from_logits=True)
print("cross-entropy:", round(float(loss_fn(y_true, logits)), 4))

print("MSE:", float(keras.losses.MeanSquaredError()([1.0, 2.0], [1.5, 2.5])))   # 0.25
print("Huber:", float(keras.losses.Huber(delta=1.0)([0.0], [3.0])))              # 2.5 (linear beyond delta)

def pinball_loss(q):
    """Quantile loss for predicting the q-th quantile."""
    def loss(y, y_hat):
        e = y - y_hat
        return tf.reduce_mean(tf.maximum(q * e, (q - 1) * e))
    return loss

model = keras.Sequential([keras.Input(shape=(4,)), keras.layers.Dense(1)])
model.compile(optimizer="adam", loss=pinball_loss(0.9))      # custom loss
model.fit(np.random.rand(64, 4), np.random.rand(64, 1), epochs=1, verbose=0)

Huber with delta = 1 and error 3: delta * (|e| - delta / 2) = 1 * (3 - 0.5) = 2.5.

Common Pitfalls

  • Using CategoricalCrossentropy with integer labels (or Sparse... with one-hot labels).

  • Double softmax: softmax output plus from_logits=True.

  • Optimizing MSE when the business cares about relative or asymmetric errors.

  • Custom losses that use NumPy or Python control flow on tensor values, which breaks gradients or graph tracing.

19

How do you implement a custom layer in TensorFlow Keras?

A custom layer in Keras is a subclass of keras.layers.Layer that defines its own weights and computation. You create weights in build (called once the input shape is known) with self.add_weight, implement the forward pass in call, and add get_config so the layer can be saved and reloaded. Custom layers let you add operations Keras does not provide while keeping the benefits of Keras models: tracking of weights, saving, fit and serialization.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

The Structure

Method Purpose
__init__ Store configuration (units, activation); create sublayers
build(input_shape) Create weights with self.add_weight once the input size is known
call(inputs, training=None) Forward computation; training switches behaviour such as dropout
get_config() Return constructor arguments for saving
compute_output_shape (optional) Needed only in some cases for shape inference

Code Example

import keras
import numpy as np
from keras import ops


@keras.saving.register_keras_serializable(package="custom")
class ScaledDense(keras.layers.Layer):
    """Dense layer with a learnable output scale per unit and optional activation."""

    def __init__(self, units, activation=None, **kwargs):
        super().__init__(**kwargs)
        self.units = units
        self.activation = keras.activations.get(activation)

    def build(self, input_shape):
        self.kernel = self.add_weight(shape=(input_shape[-1], self.units),
                                      initializer="glorot_uniform", name="kernel")
        self.bias = self.add_weight(shape=(self.units,), initializer="zeros", name="bias")
        self.scale = self.add_weight(shape=(self.units,), initializer="ones", name="scale")

    def call(self, inputs):
        out = (ops.matmul(inputs, self.kernel) + self.bias) * self.scale
        return self.activation(out)

    def get_config(self):
        config = super().get_config()
        config.update({"units": self.units, "activation": keras.activations.serialize(self.activation)})
        return config


keras.utils.set_random_seed(42)
model = keras.Sequential([keras.Input(shape=(4,)), ScaledDense(8, activation="relu"), keras.layers.Dense(1)])
model.compile(optimizer="adam", loss="mse")
model.fit(np.random.rand(128, 4), np.random.rand(128, 1), epochs=2, verbose=0)
print([w.name for w in model.layers[0].weights])        # ['kernel', 'bias', 'scale']

model.save("custom_layer_model.keras")
reloaded = keras.models.load_model("custom_layer_model.keras")   # works thanks to registration and get_config
print(np.allclose(model.predict(np.ones((1, 4)), verbose=0), reloaded.predict(np.ones((1, 4)), verbose=0)))

Using keras.ops keeps the layer backend-agnostic in Keras 3; with the TensorFlow backend you could also use tf operations.

Common Pitfalls

  • Creating weights in call, which creates new weights on every call.

  • Forgetting get_config or registration, which makes saving or loading fail.

  • Ignoring the training argument for layers that behave differently during training (dropout, noise).

  • Using NumPy inside call, which breaks gradients and graph compilation.

20

What is the difference between the Sequential and Functional APIs in Keras?

The Sequential API builds a model as a simple linear stack of layers, each feeding the next, while the Functional API builds a model as a graph by calling layers on tensors, so it supports multiple inputs and outputs, shared layers, skip connections and branches. Sequential is the quickest way to define straightforward models; the Functional API is needed as soon as the architecture is not a single straight line. A third option, subclassing keras.Model, gives full Python flexibility.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Comparison

Aspect Sequential Functional Subclassing
Structure Linear stack Directed acyclic graph of layers Arbitrary Python code in call
Multiple inputs/outputs No Yes Yes
Shared layers, skip connections No Yes Yes
Model summary and plotting Yes Yes Limited (graph not known in advance)
Shape checks when building Yes Yes At call time
Serialization Easy Easy Needs get_config
Best for Simple feed-forward or CNN stacks Most real architectures (ResNets, multi-task models) Research models with dynamic behaviour

Code Example

import keras

# Sequential: a straight stack
seq = keras.Sequential([
    keras.Input(shape=(32,)),
    keras.layers.Dense(64, activation="relu"),
    keras.layers.Dense(1, activation="sigmoid"),
])

# Functional: two inputs, a skip connection and two outputs
numeric = keras.Input(shape=(32,), name="numeric")
text_ids = keras.Input(shape=(20,), dtype="int32", name="text_ids")

x = keras.layers.Dense(64, activation="relu")(numeric)
x = keras.layers.Add()([x, keras.layers.Dense(64)(numeric)])            # skip connection
t = keras.layers.Embedding(input_dim=5000, output_dim=16)(text_ids)
t = keras.layers.GlobalAveragePooling1D()(t)
merged = keras.layers.Concatenate()([x, t])

churn = keras.layers.Dense(1, activation="sigmoid", name="churn")(merged)
spend = keras.layers.Dense(1, name="spend")(merged)
func = keras.Model(inputs=[numeric, text_ids], outputs=[churn, spend])

func.compile(optimizer="adam",
             loss={"churn": "binary_crossentropy", "spend": "mse"},
             loss_weights={"churn": 1.0, "spend": 0.2})
print(len(seq.layers), "layers in Sequential |", len(func.inputs), "inputs,", len(func.outputs), "outputs in Functional")

When to Use Which

  • Sequential: prototypes, MLPs, simple CNNs.

  • Functional: anything with branches, multiple inputs/outputs, residual connections, or reused layers; also convenient for extracting intermediate outputs (keras.Model(model.input, model.get_layer("x").output)).

  • Subclassing: custom training logic or control flow that depends on inputs.

Common Pitfalls

  • Trying to add a second input to a Sequential model.

  • Reusing a layer instance by accident, which shares its weights.

  • Forgetting the Input layer, so the model is not built until it sees data.

21

How does TensorFlow support regularization to prevent overfitting?

TensorFlow (through Keras) supports regularization with weight penalties (L1, L2) attached to layers, dropout layers, batch and layer normalization, data augmentation layers, early stopping, label smoothing and optimizers with decoupled weight decay. These techniques limit how closely the model can fit noise in the training data. Each adds a hyperparameter to tune, and too much regularization causes underfitting.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Techniques

Technique Keras API Effect
L2 weight penalty kernel_regularizer=keras.regularizers.L2(1e-4) Adds lambda * sum(w^2) to the loss; smaller weights
L1 weight penalty keras.regularizers.L1(1e-5) Adds ‘lambda * sum( w )‘; drives weights to zero
Decoupled weight decay keras.optimizers.AdamW(weight_decay=1e-4) Shrinks weights each step, independent of adaptive rates
Dropout keras.layers.Dropout(0.3) Randomly zeroes activations in training
Spatial dropout SpatialDropout1D/2D Drops whole feature maps in CNNs
Batch / layer normalization BatchNormalization, LayerNormalization Stabilizes training; mild regularizing effect
Data augmentation RandomFlip, RandomRotation, RandomZoom More varied training data
Early stopping keras.callbacks.EarlyStopping Stops when validation loss stops improving
Label smoothing CategoricalCrossentropy(label_smoothing=0.1) Prevents over-confident predictions
Gaussian noise keras.layers.GaussianNoise(0.1) Noise on inputs or activations

Code Example

import keras
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

keras.utils.set_random_seed(42)
X, y = make_classification(n_samples=600, n_features=50, n_informative=8, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X.astype("float32"), y, test_size=0.3, random_state=42)

def build(regularized):
    reg = keras.regularizers.L2(1e-3) if regularized else None
    layers = [keras.Input(shape=(50,)),
              keras.layers.Dense(256, activation="relu", kernel_regularizer=reg)]
    if regularized:
        layers.append(keras.layers.Dropout(0.4))
    layers += [keras.layers.Dense(256, activation="relu", kernel_regularizer=reg),
               keras.layers.Dense(1, activation="sigmoid")]
    model = keras.Sequential(layers)
    model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
    return model

for regularized in (False, True):
    callbacks = [keras.callbacks.EarlyStopping(patience=10, restore_best_weights=True)] if regularized else []
    hist = build(regularized).fit(X_train, y_train, validation_data=(X_val, y_val), epochs=100,
                                  batch_size=32, verbose=0, callbacks=callbacks)
    print(f"regularized={regularized}: train acc={hist.history['accuracy'][-1]:.3f}, "
          f"best val acc={max(hist.history['val_accuracy']):.3f}")

Common Pitfalls

  • Applying dropout or augmentation at inference: Keras disables them automatically in predict, but custom code must respect the training flag.

  • Combining L2 regularization with Adam and expecting it to equal weight decay; use AdamW for decoupled decay.

  • Tuning regularization on the test set.

22

How do you save and restore models in TensorFlow?

Saving and restoring models in TensorFlow can be done at three levels: the native Keras format (model.save("model.keras"), which stores architecture, weights and optimizer state in one file), weights only (model.save_weights and load_weights), and the SavedModel format (model.export(...) or tf.saved_model.save), which stores a language-neutral graph for TensorFlow Serving, TensorFlow Lite and other runtimes. Training checkpoints with tf.train.Checkpoint or the ModelCheckpoint callback let you resume interrupted training.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Options

Method What is saved Use for
model.save("m.keras") / keras.models.load_model Architecture, weights, optimizer state, compile config Resuming training, sharing Keras models
model.save_weights("m.weights.h5") / load_weights Weights only Same architecture defined in code
model.export("dir") SavedModel for inference (serving signature) TensorFlow Serving, conversion to TF Lite
tf.saved_model.save(obj, "dir") SavedModel of any trackable object / tf.functions Custom exports
keras.callbacks.ModelCheckpoint Best or periodic .keras files or weights During training
tf.train.Checkpoint + CheckpointManager Variables of any objects Custom loops, keeping the last N checkpoints

In Keras 3, model.save requires a .keras (or legacy .h5) path; exporting a SavedModel directory uses model.export.

Code Example

import keras
import numpy as np
import tensorflow as tf

keras.utils.set_random_seed(42)
X, y = np.random.rand(200, 5).astype("float32"), np.random.rand(200, 1).astype("float32")
model = keras.Sequential([keras.Input(shape=(5,)), keras.layers.Dense(16, activation="relu"), keras.layers.Dense(1)])
model.compile(optimizer="adam", loss="mse")
model.fit(X, y, epochs=3, verbose=0,
          callbacks=[keras.callbacks.ModelCheckpoint("best.keras", monitor="loss", save_best_only=True)])

# 1. Full Keras model
model.save("model.keras")
restored = keras.models.load_model("model.keras")
print("same predictions:", np.allclose(model.predict(X[:3], verbose=0), restored.predict(X[:3], verbose=0)))

# 2. Weights only
model.save_weights("model.weights.h5")
clone = keras.models.clone_model(model)
clone.load_weights("model.weights.h5")

# 3. SavedModel for serving
model.export("exported_model")                          # directory with saved_model.pb + variables/
served = tf.saved_model.load("exported_model")
print(served.serve(tf.constant(X[:2])).shape)           # (2, 1)

Common Pitfalls

  • Custom layers or losses without get_config and registration, which fail to load.

  • Saving only weights and later changing the architecture code.

  • Using pickle for Keras models; use the built-in formats.

  • Loading a model saved with a much newer TensorFlow/Keras version in an older one.

23

How do you use callbacks in TensorFlow Keras?

Callbacks are objects passed to fit, evaluate or predict that run code at specific points during training, such as the end of each epoch or batch. Keras includes callbacks for early stopping, saving checkpoints, adjusting the learning rate, logging to TensorBoard and stopping on NaN, and you can write your own by subclassing keras.callbacks.Callback. They automate training control without writing a custom loop.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Built-In Callbacks

Callback Purpose
EarlyStopping(monitor="val_loss", patience=5, restore_best_weights=True) Stop when the monitored metric stops improving
ModelCheckpoint("best.keras", save_best_only=True) Save the best (or every) model
ReduceLROnPlateau(factor=0.5, patience=3) Lower the learning rate when progress stalls
LearningRateScheduler(fn) Set the learning rate per epoch with a function
TensorBoard(log_dir="logs") Log metrics, histograms and profiles
CSVLogger("log.csv") Write metrics to a CSV file
TerminateOnNaN() Stop if the loss becomes NaN
BackupAndRestore("backup") Resume after an interruption
LambdaCallback(on_epoch_end=...) Quick custom hooks

Hook Methods for Custom Callbacks

on_train_begin/end, on_epoch_begin/end, on_train_batch_begin/end, on_test_batch_*, on_predict_batch_*. Inside a callback, self.model gives access to the model and logs contains current metrics.

Code Example

import keras
import numpy as np

class GapMonitor(keras.callbacks.Callback):
    """Warn when validation loss is much higher than training loss (overfitting)."""
    def __init__(self, ratio=1.5):
        super().__init__()
        self.ratio = ratio

    def on_epoch_end(self, epoch, logs=None):
        if logs["val_loss"] > self.ratio * logs["loss"]:
            print(f"epoch {epoch + 1}: val_loss is {logs['val_loss'] / logs['loss']:.1f}x the training loss")

keras.utils.set_random_seed(42)
X = np.random.rand(500, 10).astype("float32")
y = (X.sum(axis=1) + np.random.normal(0, 0.5, 500)).astype("float32")

model = keras.Sequential([keras.Input(shape=(10,)), keras.layers.Dense(64, activation="relu"),
                          keras.layers.Dense(1)])
model.compile(optimizer="adam", loss="mse")

callbacks = [
    keras.callbacks.EarlyStopping(monitor="val_loss", patience=8, restore_best_weights=True),
    keras.callbacks.ReduceLROnPlateau(monitor="val_loss", factor=0.5, patience=3, min_lr=1e-5),
    keras.callbacks.ModelCheckpoint("best_model.keras", monitor="val_loss", save_best_only=True),
    keras.callbacks.CSVLogger("training_log.csv"),
    GapMonitor(),
]
history = model.fit(X, y, validation_split=0.2, epochs=200, batch_size=32, verbose=0, callbacks=callbacks)
print("stopped after", len(history.history["loss"]), "epochs")

Common Pitfalls

  • Monitoring a metric that does not exist (e.g. val_accuracy without validation data or that metric); Keras warns and the callback does nothing.

  • EarlyStopping without restore_best_weights=True, which keeps the weights from the last (worse) epoch.

  • Heavy work in batch-level hooks, which slows training.

24

Which strategies does TensorFlow offer to handle overfitting during training?

Overfitting happens when a model learns the training data, including its noise, so well that it performs worse on new data. TensorFlow and Keras offer strategies at every level: more and augmented data, simpler architectures, weight penalties and weight decay, dropout, normalization, early stopping, learning-rate schedules and careful validation. The key is to detect overfitting first, by watching the gap between training and validation metrics, and then apply the least intrusive fix.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Detecting Overfitting

  • Training loss keeps falling while validation loss rises.

  • Large gap between training and validation accuracy.

  • Plot history.history["loss"] and ["val_loss"], or watch them in TensorBoard.

Strategies

Category Strategy Keras tool
Data More data, augmentation keras.layers.RandomFlip/RandomRotation/RandomZoom, text augmentation
Model capacity Fewer layers or units; transfer learning with a frozen backbone Smaller Dense/Conv layers, base.trainable = False
Weight penalties L1/L2, weight decay kernel_regularizer, AdamW(weight_decay=...)
Noise Dropout, Gaussian noise Dropout, SpatialDropout2D, GaussianNoise
Normalization Batch/layer normalization BatchNormalization, LayerNormalization
Training control Early stopping, learning-rate reduction EarlyStopping, ReduceLROnPlateau
Targets Label smoothing label_smoothing=0.1 in cross-entropy losses
Evaluation Proper validation and cross-validation validation_data, k-fold splits outside Keras
Ensembles Average several models Multiple models or snapshot ensembles

Code Example: Augmentation + Dropout + Early Stopping for Images

import keras
import numpy as np

keras.utils.set_random_seed(42)
(x_train, y_train), (x_test, y_test) = keras.datasets.cifar10.load_data()   # downloads the data
x_train, y_train = x_train[:5000], y_train[:5000]                           # small subset: easy to overfit

augment = keras.Sequential([keras.layers.RandomFlip("horizontal"),
                            keras.layers.RandomTranslation(0.1, 0.1), keras.layers.RandomZoom(0.1)])
model = keras.Sequential([
    keras.Input(shape=(32, 32, 3)),
    augment,                                              # active only during training
    keras.layers.Rescaling(1.0 / 255),
    keras.layers.Conv2D(32, 3, activation="relu", padding="same"),
    keras.layers.MaxPooling2D(),
    keras.layers.Conv2D(64, 3, activation="relu", padding="same"),
    keras.layers.MaxPooling2D(),
    keras.layers.GlobalAveragePooling2D(),               # fewer parameters than Flatten + Dense
    keras.layers.Dropout(0.3),
    keras.layers.Dense(10, activation="softmax"),
])
model.compile(optimizer=keras.optimizers.AdamW(1e-3, weight_decay=1e-4),
              loss="sparse_categorical_crossentropy", metrics=["accuracy"])
model.fit(x_train, y_train, validation_split=0.2, epochs=30, batch_size=64, verbose=0,
          callbacks=[keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True)])
print("test accuracy:", round(model.evaluate(x_test, y_test, verbose=0)[1], 3))

Common Pitfalls

  • Adding every technique at once; change one thing at a time and measure validation performance.

  • Data leakage between training and validation sets, which hides overfitting.

  • Using the test set for early stopping or tuning.

25

How does TensorFlow integrate with Keras?

Keras is TensorFlow’s high-level API for building, training and saving models, while TensorFlow supplies the tensors, automatic differentiation, compiled graphs, input pipelines and deployment tools underneath. Since TensorFlow 2.16, installing TensorFlow gives you Keras 3, a multi-backend library that can run on TensorFlow, JAX or PyTorch; with the TensorFlow backend, Keras models train with tf.function graphs, read tf.data datasets and export to SavedModel for TensorFlow Serving and LiteRT.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

How the Pieces Fit

Layer Provided by Examples
Model definition Keras keras.Sequential, Functional API, keras.layers.*
Training loop Keras (fit, evaluate, predict) Callbacks, metrics, losses, optimizers
Execution TensorFlow backend Ops, tf.function graphs, XLA via jit_compile
Gradients TensorFlow tf.GradientTape (used inside Keras’s training step)
Input pipeline TensorFlow tf.data.Dataset passed straight to fit
Scaling TensorFlow tf.distribute.MirroredStrategy and other strategies
Deployment TensorFlow model.export → SavedModel → TF Serving, LiteRT, TF.js

Keras 3 Versus tf.keras

  • import keras is the recommended import. In TF 2.16+, tf.keras also points to Keras 3 when it is installed.

  • Code written for Keras 2 that breaks under Keras 3 can use the tf_keras package with the environment variable TF_USE_LEGACY_KERAS=1 during migration.

  • The backend is chosen with KERAS_BACKEND (tensorflow by default when TensorFlow is installed). keras.ops functions work on every backend.

Code Example: Keras Model, TensorFlow Pipeline and Custom Training Step

import os
os.environ["KERAS_BACKEND"] = "tensorflow"        # must be set before importing keras

import keras
import numpy as np
import tensorflow as tf

keras.utils.set_random_seed(42)
X = np.random.rand(1000, 20).astype("float32")
y = (X[:, 0] + X[:, 1] > 1).astype("float32")
ds = tf.data.Dataset.from_tensor_slices((X, y)).shuffle(1000, seed=42).batch(32).prefetch(tf.data.AUTOTUNE)


class GradClipModel(keras.Model):
    """Keras model whose training step uses TensorFlow's GradientTape directly."""
    def train_step(self, data):
        x, y = data
        with tf.GradientTape() as tape:
            y_pred = self(x, training=True)
            loss = self.compute_loss(x=x, y=y, y_pred=y_pred)
        grads = tape.gradient(loss, self.trainable_variables)
        grads = [tf.clip_by_norm(g, 1.0) for g in grads]
        self.optimizer.apply_gradients(zip(grads, self.trainable_variables))
        for metric in self.metrics:
            if metric.name == "loss":
                metric.update_state(loss)
            else:
                metric.update_state(y, y_pred)
        return {m.name: m.result() for m in self.metrics}


inputs = keras.Input(shape=(20,))
h = keras.layers.Dense(32, activation="relu")(inputs)
outputs = keras.layers.Dense(1, activation="sigmoid")(h)
model = GradClipModel(inputs, outputs)
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"], jit_compile=True)
model.fit(ds, epochs=3, verbose=0)

model.export("exported_model")                   # SavedModel for TF Serving / LiteRT conversion
print(tf.saved_model.load("exported_model").serve(tf.constant(X[:2])).shape)   # (2, 1)

Interview Tips

  • Explain that Keras is the API and TensorFlow is the engine; mention the Keras 3 multi-backend change in TF 2.16.

  • Know the three ways to build models (Sequential, Functional, subclassing) and how to drop to tf.GradientTape when fit is not flexible enough.

  • Mention saving formats: .keras for Keras models, model.export for a SavedModel used in deployment.

26

What is eager execution in TensorFlow?

Eager execution is TensorFlow’s default mode since 2.0: operations run immediately when Python calls them and return concrete values, just like NumPy. This makes code easy to write and debug with print statements, breakpoints and normal Python control flow. For speed and deployment you wrap functions in tf.function, which traces them once into a graph that TensorFlow can optimize, so modern TensorFlow combines eager development with graph execution.

Read full answer

Applies toTensorFlow 2.16+ (Python 3.12)

Eager Versus Graph Execution

Aspect Eager Graph (tf.function)
When ops run Immediately After tracing, as one compiled graph
Results Concrete EagerTensor values (.numpy() works) Symbolic tensors during tracing
Debugging Print, pdb, Python exceptions Harder; use tf.print and run_functions_eagerly
Python control flow Normal Python Converted to tf.cond / tf.while_loop by AutoGraph
Performance Per-op Python overhead Op fusion, constant folding, optional XLA
Export Not directly Required for SavedModel signatures

Code Example

import tensorflow as tf

print(tf.executing_eagerly())              # True

a = tf.constant([[1.0, 2.0], [3.0, 4.0]])
b = tf.matmul(a, a)                        # computed right away
print(b.numpy())                           # [[ 7. 10.] [15. 22.]]

# Gradients in eager mode
x = tf.Variable(3.0)
with tf.GradientTape() as tape:
    y = x ** 2 + 2 * x
print(tape.gradient(y, x).numpy())         # dy/dx = 2x + 2 = 8.0


@tf.function
def scaled_sum(t, factor):
    print("tracing")                       # Python side effect: runs only when tracing
    tf.print("running")                    # graph op: runs on every call
    if tf.reduce_sum(t) > 0:               # AutoGraph converts this to tf.cond
        return tf.reduce_sum(t) * factor
    return tf.constant(0.0)

scaled_sum(a, 2.0)                         # prints "tracing" and "running"
scaled_sum(a, 2.0)                         # prints only "running": the graph is reused
scaled_sum(a, 3.0)                         # new Python float value -> retraces

# Debug a tf.function eagerly
tf.config.run_functions_eagerly(True)
scaled_sum(a, 2.0)
tf.config.run_functions_eagerly(False)

Why It Matters

  • Faster development: errors point at the line that failed, and intermediate values can be inspected.

  • Keras uses tf.function for fit by default, so you get eager convenience for model code and graph speed for training. Pass run_eagerly=True to compile when debugging.

Common Pitfalls

  • Passing Python numbers that change on each call to a tf.function, which causes retracing; pass tensors instead or fix an input_signature.

  • Relying on print or list appends inside a tf.function; they run only during tracing.

  • Calling .numpy() inside a tf.function, which fails because tensors are symbolic there.

27

What role does tf.data play in TensorFlow?

tf.data is TensorFlow’s API for building input pipelines: it reads data from memory, files or generators, transforms it, and feeds batches to the model efficiently. Pipelines are chains of Dataset transformations (map, shuffle, batch, prefetch) that run in parallel C++ threads, so data loading overlaps with training and the accelerator does not wait. It handles datasets larger than memory and plugs directly into model.fit and custom training loops.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Main Building Blocks

Category API Purpose
Sources from_tensor_slices, TFRecordDataset, TextLineDataset, from_generator, Dataset.list_files Create a dataset
Keras helpers keras.utils.image_dataset_from_directory, text_dataset_from_directory Datasets from folders
Transformations map, filter, flat_map, interleave, window Parse, preprocess, combine
Ordering shuffle, repeat Randomize and loop
Grouping batch, padded_batch, ragged_batch, bucket_by_sequence_length Build batches
Performance prefetch, cache, num_parallel_calls=tf.data.AUTOTUNE Overlap and parallelize work
Persistence Dataset.save / Dataset.load Store a processed dataset

Typical Pipeline Order

  1. List or read files (use interleave to read several at once).

  2. map parsing and cheap deterministic preprocessing in parallel.

  3. cache (in memory or to a file) if the result fits and is reused every epoch.

  4. shuffle with a reasonably large buffer.

  5. map random augmentation (after cache, so it changes each epoch).

  6. batch, then prefetch(tf.data.AUTOTUNE).

Code Example: TFRecord Pipeline

import numpy as np
import tensorflow as tf

rng = np.random.default_rng(42)

# Write a small TFRecord file
with tf.io.TFRecordWriter("data.tfrecord") as writer:
    for _ in range(500):
        features = rng.random(4).astype("float32")
        label = int(features.sum() > 2)
        example = tf.train.Example(features=tf.train.Features(feature={
            "x": tf.train.Feature(float_list=tf.train.FloatList(value=features)),
            "y": tf.train.Feature(int64_list=tf.train.Int64List(value=[label])),
        }))
        writer.write(example.SerializeToString())

spec = {"x": tf.io.FixedLenFeature([4], tf.float32), "y": tf.io.FixedLenFeature([], tf.int64)}

def parse(record):
    ex = tf.io.parse_single_example(record, spec)
    return ex["x"], ex["y"]

def augment(x, y):
    return x + tf.random.normal(tf.shape(x), stddev=0.01), y

ds = (tf.data.TFRecordDataset(["data.tfrecord"])
      .map(parse, num_parallel_calls=tf.data.AUTOTUNE)
      .cache()
      .shuffle(500, seed=42)
      .map(augment, num_parallel_calls=tf.data.AUTOTUNE)
      .batch(32)
      .prefetch(tf.data.AUTOTUNE))

for xb, yb in ds.take(1):
    print(xb.shape, yb.shape)          # (32, 4) (32,)

Pass ds directly to model.fit(ds, epochs=...). Do not also pass batch_size, because the dataset is already batched.

Common Pitfalls

  • Doing preprocessing in Python (from_generator or tf.py_function), which holds the GIL and slows the pipeline.

  • Putting random augmentation before cache, which freezes it.

  • A tiny shuffle buffer on data sorted by label.

  • Forgetting prefetch. Use the TensorBoard profiler’s input-pipeline analysis to find bottlenecks.

28

How do you use Transformers in TensorFlow for sequence modeling?

Transformers model sequences with self-attention: each position computes weighted combinations of every other position, so the model captures long-range dependencies without recurrence and trains in parallel. In TensorFlow you can build them from Keras layers (MultiHeadAttention, LayerNormalization, Dense) or use KerasHub (formerly KerasNLP), which provides ready-made Transformer encoder and decoder blocks and pretrained models such as BERT and Gemma. Use an encoder for classification or tagging, and a decoder with a causal mask for generation or forecasting.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3, KerasHub 0.18+ (Python 3.12)

Building Blocks

Component Keras / KerasHub Purpose
Token + position embedding keras_hub.layers.TokenAndPositionEmbedding or two Embedding layers Word identity plus order information
Self-attention keras.layers.MultiHeadAttention Each token attends to all others
Causal mask use_causal_mask=True in the attention call Stop decoder tokens from seeing the future
Feed-forward block Two Dense layers Per-position transformation
Residual + normalization Add, LayerNormalization Stable deep training
Encoder / decoder blocks keras_hub.layers.TransformerEncoder, TransformerDecoder Prebuilt blocks
Pretrained models keras_hub.models.BertClassifier.from_preset(...) Fine-tuning

Attention in words: for each token, compare its query with every key (dot product divided by the square root of the key size), turn the scores into weights with softmax, and take the weighted average of the values.

Code Example: Transformer Encoder for Sequence Classification

import keras
from keras import layers

keras.utils.set_random_seed(42)
VOCAB, SEQ_LEN, DIM, HEADS = 20000, 200, 64, 4


class PositionalEmbedding(layers.Layer):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.tokens = layers.Embedding(VOCAB, DIM, mask_zero=True)
        self.positions = layers.Embedding(SEQ_LEN, DIM)

    def call(self, x):
        positions = keras.ops.arange(keras.ops.shape(x)[-1])
        return self.tokens(x) + self.positions(positions)

    def compute_mask(self, x, mask=None):
        return self.tokens.compute_mask(x)


def encoder_block(x):
    attn = layers.MultiHeadAttention(num_heads=HEADS, key_dim=DIM // HEADS, dropout=0.1)(x, x)
    x = layers.LayerNormalization()(layers.Add()([x, attn]))
    ff = layers.Dense(DIM * 4, activation="gelu")(x)
    ff = layers.Dropout(0.1)(layers.Dense(DIM)(ff))
    return layers.LayerNormalization()(layers.Add()([x, ff]))


inputs = keras.Input(shape=(SEQ_LEN,), dtype="int32")
x = PositionalEmbedding()(inputs)
for _ in range(2):
    x = encoder_block(x)
x = layers.GlobalAveragePooling1D()(x)
outputs = layers.Dense(1, activation="sigmoid")(layers.Dropout(0.1)(x))
model = keras.Model(inputs, outputs)
model.compile(optimizer=keras.optimizers.AdamW(3e-4), loss="binary_crossentropy", metrics=["accuracy"])

(x_train, y_train), _ = keras.datasets.imdb.load_data(num_words=VOCAB)        # downloads the data
x_train = keras.utils.pad_sequences(x_train, maxlen=SEQ_LEN)
model.fit(x_train, y_train, validation_split=0.2, epochs=2, batch_size=64, verbose=0)

With KerasHub, fine-tuning a pretrained encoder takes a few lines: keras_hub.models.BertClassifier.from_preset("bert_base_en_uncased", num_classes=2), then fit on raw strings, because the preset includes its tokenizer and preprocessor.

Common Pitfalls

  • Forgetting positional information, which makes attention order-blind.

  • Missing padding masks, so the model attends to padding tokens.

  • Training large Transformers from scratch on small data; fine-tune a pretrained model instead.

  • Attention cost grows with the square of the sequence length, so long sequences need truncation, chunking or efficient attention.

29

What are TensorFlow distribution strategies, and when should you use them?

tf.distribute.Strategy is TensorFlow’s API for spreading training across several GPUs, several machines or TPUs with few code changes. You create a strategy, build and compile the model inside strategy.scope(), and Keras’s fit handles splitting each batch across devices and combining gradients. Use it when training on one device is too slow or the data is too large, and pick the strategy that matches your hardware.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Strategies

Strategy Hardware How it works
MirroredStrategy Several GPUs on one machine Copies the model on each GPU; synchronous all-reduce of gradients
MultiWorkerMirroredStrategy GPUs on several machines Same idea across workers; configured with TF_CONFIG
TPUStrategy TPU pods / Cloud TPU Synchronous training on TPU cores
ParameterServerStrategy Many CPU/GPU workers + parameter servers Asynchronous updates; large embeddings, many workers
OneDeviceStrategy A single chosen device Testing code paths before scaling
Default strategy No distribution What runs when no strategy is set

Key Concepts

  • Data parallelism: every replica holds a full copy of the model and processes a slice of the global batch.

  • Global batch size = per-replica batch size × strategy.num_replicas_in_sync. Scale the learning rate roughly in proportion, often with warm-up.

  • Synchronous vs asynchronous: mirrored strategies keep replicas identical; parameter-server training lets workers progress independently.

Code Example

import keras
import numpy as np
import tensorflow as tf

keras.utils.set_random_seed(42)
strategy = tf.distribute.MirroredStrategy()          # uses all visible GPUs (or the CPU if none)
print("replicas:", strategy.num_replicas_in_sync)

per_replica_batch = 64
global_batch = per_replica_batch * strategy.num_replicas_in_sync

X = np.random.rand(10000, 32).astype("float32")
y = (X.sum(axis=1) > 16).astype("float32")
ds = (tf.data.Dataset.from_tensor_slices((X, y))
      .shuffle(10000, seed=42).batch(global_batch).prefetch(tf.data.AUTOTUNE))

with strategy.scope():                               # variables are created on every replica
    model = keras.Sequential([
        keras.Input(shape=(32,)),
        keras.layers.Dense(128, activation="relu"),
        keras.layers.Dense(1, activation="sigmoid"),
    ])
    model.compile(optimizer=keras.optimizers.Adam(1e-3 * strategy.num_replicas_in_sync),
                  loss="binary_crossentropy", metrics=["accuracy"])

model.fit(ds, epochs=3, verbose=0)                  # fit distributes each batch automatically

For multi-worker training, each machine runs the same script with a TF_CONFIG environment variable that lists the cluster and its own task index; add keras.callbacks.BackupAndRestore so training can resume if a worker fails.

When to Use It

  • Training takes hours or days on one GPU and the model fits in a single GPU’s memory (data parallelism).

  • You have TPUs (TPUStrategy is required).

  • Do not use it when the input pipeline is the bottleneck; more GPUs will just wait for data.

Common Pitfalls

  • Creating the model or optimizer outside strategy.scope().

  • Keeping the per-replica batch size as the global batch size, which shrinks each GPU’s batch.

  • In custom loops, averaging the loss per replica instead of over the global batch (tf.nn.compute_average_loss).

30

What is TensorFlow Extended (TFX), and what are its main components?

TensorFlow Extended (TFX) is an end-to-end platform for production ML pipelines. It chains standard components that ingest data, validate it, transform features, train and evaluate models, and push approved models to serving, while recording every artifact and run in ML Metadata (MLMD) for lineage and reproducibility. Pipelines run locally for development and on orchestrators such as Kubeflow Pipelines, Vertex AI Pipelines, Apache Airflow or Apache Beam in production.

Read full answer

Applies toTFX 1.x with TensorFlow 2.x (Python 3.10-3.11; TFX releases pin specific TensorFlow versions, so check compatibility)

Standard Components

Component Role Built on
ExampleGen Ingests data (CSV, TFRecord, BigQuery) and splits it into train/eval Apache Beam
StatisticsGen Computes dataset statistics TensorFlow Data Validation (TFDV)
SchemaGen Infers a schema (types, ranges, vocabularies) TFDV
ExampleValidator Detects anomalies, missing values and drift against the schema TFDV
Transform Feature engineering applied identically in training and serving TensorFlow Transform (TFT)
Trainer Trains the model from a user module file (run_fn) TensorFlow / Keras
Tuner Hyperparameter search KerasTuner
Evaluator Sliced metrics; “blesses” the model if it beats the baseline TensorFlow Model Analysis (TFMA)
InfraValidator Checks the model loads and serves in a sandbox TF Serving
Pusher Deploys blessed models to a serving location Filesystem, TF Serving, Vertex AI

Why Transform Matters

TensorFlow Transform computes full-pass statistics (means, vocabularies) once and bakes them into the exported graph, so serving uses exactly the same preprocessing as training. This removes training/serving skew, a frequent source of silent production errors.

Code Example: Minimal Pipeline

from tfx import v1 as tfx

DATA_ROOT, MODULE_FILE = "data/", "trainer_module.py"     # module defines run_fn(fn_args)
PIPELINE_ROOT, SERVING_DIR, METADATA = "pipeline/", "serving_model/", "metadata.db"

example_gen = tfx.components.CsvExampleGen(input_base=DATA_ROOT)
statistics_gen = tfx.components.StatisticsGen(examples=example_gen.outputs["examples"])
schema_gen = tfx.components.SchemaGen(statistics=statistics_gen.outputs["statistics"])
validator = tfx.components.ExampleValidator(statistics=statistics_gen.outputs["statistics"],
                                            schema=schema_gen.outputs["schema"])
trainer = tfx.components.Trainer(module_file=MODULE_FILE,
                                 examples=example_gen.outputs["examples"],
                                 schema=schema_gen.outputs["schema"],
                                 train_args=tfx.proto.TrainArgs(num_steps=1000),
                                 eval_args=tfx.proto.EvalArgs(num_steps=100))
pusher = tfx.components.Pusher(
    model=trainer.outputs["model"],
    push_destination=tfx.proto.PushDestination(
        filesystem=tfx.proto.PushDestination.Filesystem(base_directory=SERVING_DIR)))

pipeline = tfx.dsl.Pipeline(
    pipeline_name="churn_pipeline", pipeline_root=PIPELINE_ROOT,
    metadata_connection_config=tfx.orchestration.metadata.sqlite_metadata_connection_config(METADATA),
    components=[example_gen, statistics_gen, schema_gen, validator, trainer, pusher])
tfx.orchestration.LocalDagRunner().run(pipeline)

In production, add Transform and Evaluator, and give Pusher the Evaluator’s blessing so only models that pass the quality checks are deployed.

Interview Tips

  • Explain the flow: ingest → validate → transform → train → evaluate → push, with MLMD tracking lineage.

  • Mention training/serving skew and how TFT prevents it.

  • Know the trade-off: TFX is powerful but heavy and TensorFlow-centric; lighter teams often use MLflow, Kubeflow or Vertex AI Pipelines with custom steps.

31

How can TensorFlow be used for reinforcement learning?

Reinforcement learning (RL) trains an agent to choose actions that maximize cumulative reward through trial and error in an environment. TensorFlow supports RL in two ways: you can write algorithms directly with Keras networks and tf.GradientTape (policy gradients, DQN, actor-critic), or use TF-Agents, a library with tested implementations of DQN, PPO, SAC, DDPG and others, plus replay buffers, drivers and environment wrappers. The neural network approximates the policy, the value function, or both.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3, TF-Agents (Python 3.12)

Main Approaches

Family Learns Examples Typical use
Value-based Q-values Q(s, a) DQN, Double DQN Discrete actions (games)
Policy gradient Policy ‘pi(a s)‘ directly REINFORCE, PPO Discrete or continuous actions
Actor-critic Policy + value baseline A2C, PPO, SAC, DDPG Robotics, control
Model-based Environment dynamics World models Expensive environments

Code Example: REINFORCE on a Small Environment

The agent starts at position 0 on a line of 5 cells and must reach cell 4. Each step costs a small penalty, so shorter paths earn more reward.

import keras
import numpy as np
import tensorflow as tf

keras.utils.set_random_seed(42)
rng = np.random.default_rng(42)
N_STATES, N_ACTIONS, GAMMA = 5, 2, 0.99           # actions: 0 = left, 1 = right

def step(state, action):
    nxt = min(max(state + (1 if action == 1 else -1), 0), N_STATES - 1)
    done = nxt == N_STATES - 1
    return nxt, (1.0 if done else -0.05), done

policy = keras.Sequential([keras.Input(shape=(N_STATES,)),
                           keras.layers.Dense(16, activation="relu"),
                           keras.layers.Dense(N_ACTIONS)])          # logits
optimizer = keras.optimizers.Adam(0.01)

for episode in range(300):
    state, states, actions, rewards = 0, [], [], []
    for _ in range(20):
        logits = policy(tf.one_hot([state], N_STATES))
        probs = tf.nn.softmax(logits)[0].numpy()
        action = rng.choice(N_ACTIONS, p=probs)
        nxt, reward, done = step(state, action)
        states.append(state); actions.append(action); rewards.append(reward)
        state = nxt
        if done:
            break

    # Discounted return from each step: G_t = r_t + gamma * G_(t+1)
    returns, g = [], 0.0
    for r in reversed(rewards):
        g = r + GAMMA * g
        returns.insert(0, g)
    returns = np.array(returns, dtype="float32")
    returns = (returns - returns.mean()) / (returns.std() + 1e-8)   # baseline-like normalization

    with tf.GradientTape() as tape:
        logits = policy(tf.one_hot(states, N_STATES))
        log_probs = tf.nn.log_softmax(logits)
        chosen = tf.reduce_sum(log_probs * tf.one_hot(actions, N_ACTIONS), axis=1)
        loss = -tf.reduce_sum(chosen * returns)         # raise probability of high-return actions
    optimizer.apply_gradients(zip(tape.gradient(loss, policy.trainable_variables),
                                  policy.trainable_variables))

print(tf.nn.softmax(policy(tf.one_hot(range(4), N_STATES))).numpy().round(2))   # "right" should dominate

The policy-gradient loss in words: increase the log-probability of each action in proportion to how good the return after it was.

Using TF-Agents

TF-Agents wraps environments (suite_gym, custom PyEnvironment), provides agents such as DqnAgent and PPOAgent, replay buffers and drivers that collect experience. It saves you from implementing target networks, epsilon schedules or clipping by hand. Check the release notes for supported TensorFlow versions before you start.

Common Pitfalls

  • High-variance gradients in REINFORCE; use baselines or actor-critic methods.

  • Forgetting target networks and replay buffers for DQN, which makes training unstable.

  • Judging progress from one seed; RL results vary a lot, so average over several seeds.

32

What is TensorFlow Serving, and how does it help deploy models?

TensorFlow Serving is a high-performance server for deploying TensorFlow models in production. It loads models saved in the SavedModel format, exposes them through REST and gRPC APIs, and handles model versioning, hot-swapping new versions without downtime, and request batching for GPU efficiency. It separates the model from application code, so data scientists can publish a new version by copying it into a directory the server watches.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3, TensorFlow Serving 2.x (Docker image `tensorflow/serving`)

Key Features

Feature Benefit
SavedModel loading Language-neutral model format with serving signatures
Versioned directories (/models/name/1, /2, …) Automatic rollout of the newest version; rollback by removing a version
REST API (port 8501) Easy JSON calls from any language
gRPC API (port 8500) Lower latency, binary tensors
Server-side batching (--enable_batching) Groups requests to use the GPU efficiently
Model config file Serve several models or pin specific versions; labels such as stable and canary
Monitoring Prometheus metrics endpoint

Deployment Workflow

  1. Train and export: model.export("models/churn/1") creates a SavedModel with a serving_default signature.

  2. Start the server:

docker run -p 8501:8501 -p 8500:8500 \
  --mount type=bind,source="$(pwd)/models/churn",target=/models/churn \
  -e MODEL_NAME=churn -t tensorflow/serving
  1. Send requests:

import json
import keras
import numpy as np
import requests

keras.utils.set_random_seed(42)
model = keras.Sequential([keras.Input(shape=(4,)), keras.layers.Dense(8, activation="relu"),
                          keras.layers.Dense(1, activation="sigmoid")])
model.compile(optimizer="adam", loss="binary_crossentropy")
model.fit(np.random.rand(200, 4), np.random.randint(0, 2, 200), epochs=2, verbose=0)
model.export("models/churn/1")                     # version 1 of the "churn" model

# After the Docker container is running:
payload = {"signature_name": "serving_default", "instances": np.random.rand(2, 4).round(3).tolist()}
resp = requests.post("http://localhost:8501/v1/models/churn:predict", data=json.dumps(payload), timeout=5)
print(resp.json())                                 # {"predictions": [[...], [...]]}
print(requests.get("http://localhost:8501/v1/models/churn", timeout=5).json())   # version status

Inspect a SavedModel’s inputs and outputs with saved_model_cli show --dir models/churn/1 --all.

Updating Models

Export the next version to models/churn/2. The server detects it, loads it, and switches traffic once it is ready; by default only the latest version is served. Use a model config file to keep several versions available for A/B tests or canary releases.

Common Pitfalls

  • Exporting without a version subdirectory, so the server finds no model.

  • Preprocessing done in client code that differs from training; put preprocessing layers inside the exported model.

  • Sending inputs whose names or shapes do not match the signature.

  • Using REST for very high throughput where gRPC would be faster.

33

How do you use TensorFlow's Estimator API, and what replaces it?

The Estimator API (tf.estimator) was TensorFlow 1.x’s high-level training interface: you wrote input functions and a model_fn, and the Estimator handled training, evaluation, checkpointing and export. It was deprecated in TensorFlow 2 and removed in TensorFlow 2.16, so new code should use Keras (model.fit) with tf.data, tf.distribute and model.export. Interviewers still ask about it because older production systems use it, so you should know how it worked and how to migrate.

Read full answer

Applies toTensorFlow 2.15 and earlier for `tf.estimator`; TensorFlow 2.16+ with Keras 3 for the replacement (Python 3.12)

How Estimators Worked

Concept Estimator (TF 1.x / ≤2.15) Modern replacement
Model definition model_fn(features, labels, mode, params) returning EstimatorSpec Keras Sequential / Functional model
Premade models DNNClassifier, LinearRegressor, BoostedTreesClassifier Keras models; TF Decision Forests or XGBoost for trees
Input input_fn returning a tf.data.Dataset Pass the dataset to fit
Features tf.feature_column Keras preprocessing layers (Normalization, StringLookup, Discretization, FeatureSpace)
Train / evaluate estimator.train, evaluate, train_and_evaluate fit, evaluate, callbacks
Checkpoints Automatic in model_dir ModelCheckpoint, BackupAndRestore
Distribution RunConfig(train_distribute=...) strategy.scope()
Export export_saved_model with a serving input receiver model.export

Legacy Usage (for Recognition Only)

# Runs only on TensorFlow 2.15 or earlier
import tensorflow as tf

feature_cols = [tf.feature_column.numeric_column("x", shape=[4])]
estimator = tf.estimator.DNNClassifier(hidden_units=[16, 8], feature_columns=feature_cols,
                                       n_classes=2, model_dir="est_model")

def input_fn(features, labels, training=True):
    ds = tf.data.Dataset.from_tensor_slices(({"x": features}, labels))
    return ds.shuffle(1000, seed=42).repeat().batch(32) if training else ds.batch(32)

# estimator.train(lambda: input_fn(X_train, y_train), steps=1000)

Modern Equivalent

import keras
import numpy as np

keras.utils.set_random_seed(42)
X = np.random.rand(1000, 4).astype("float32")
y = (X.sum(axis=1) > 2).astype("int32")

norm = keras.layers.Normalization()
norm.adapt(X)                                      # replaces numeric feature-column normalization
model = keras.Sequential([keras.Input(shape=(4,)), norm,
                          keras.layers.Dense(16, activation="relu"),
                          keras.layers.Dense(8, activation="relu"),
                          keras.layers.Dense(1, activation="sigmoid")])
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
model.fit(X, y, validation_split=0.2, epochs=5, batch_size=32, verbose=0,
          callbacks=[keras.callbacks.ModelCheckpoint("best.keras", save_best_only=True)])
model.export("exported_model")                    # replaces export_saved_model

Migration Tips

  • Move feature columns to Keras preprocessing layers inside the model, so serving gets the same preprocessing.

  • Replace train_and_evaluate with fit(..., validation_data=...) plus callbacks.

  • Pin TensorFlow 2.15 only as a temporary measure while you migrate.

34

How do you approach optimizing TensorFlow model performance?

Optimizing TensorFlow performance starts with measuring: profile to find whether time goes to the input pipeline, the device computation or host-device transfers, then fix the biggest bottleneck first. Common wins are a parallel, prefetched tf.data pipeline, compiling with tf.function and XLA, mixed precision on modern GPUs, larger batches, and multi-GPU strategies. For inference, add quantization, pruning or a smaller architecture. “Performance” can also mean accuracy, so be clear about which target you are optimizing.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Step 1: Profile

  • TensorBoard profiler: keras.callbacks.TensorBoard(log_dir="logs", profile_batch=(10, 20)).

  • Check the input-bound percentage in the overview page. If the GPU is idle waiting for data, fix the pipeline before touching the model.

  • nvidia-smi shows GPU utilization and memory at a glance.

Step 2: Apply the Matching Fix

Bottleneck Techniques
Input pipeline num_parallel_calls=AUTOTUNE, prefetch, cache, interleave for files, TFRecords instead of many small files
Python overhead tf.function, steps_per_execution in compile, avoid .numpy() in loops
Device compute Mixed precision (mixed_float16 or mixed_bfloat16), XLA (jit_compile=True), channel sizes divisible by 8
Small batches Increase batch size; adjust the learning rate
Single device limit MirroredStrategy, MultiWorkerMirroredStrategy, TPUs
Memory Gradient checkpointing, smaller batches with accumulation, memory growth
Inference latency/size Quantization and conversion with LiteRT, pruning, distillation, TF Serving batching

Code Example

import keras
import numpy as np
import tensorflow as tf

keras.utils.set_random_seed(42)
keras.mixed_precision.set_global_policy("mixed_float16")   # faster on GPUs with Tensor Cores

X = np.random.rand(20000, 64).astype("float32")
y = np.random.randint(0, 10, size=20000)
ds = (tf.data.Dataset.from_tensor_slices((X, y))
      .cache()
      .shuffle(20000, seed=42)
      .batch(256)
      .prefetch(tf.data.AUTOTUNE))

model = keras.Sequential([
    keras.Input(shape=(64,)),
    keras.layers.Dense(512, activation="relu"),
    keras.layers.Dense(512, activation="relu"),
    keras.layers.Dense(10),
    keras.layers.Activation("softmax", dtype="float32"),    # keep outputs in float32 for stability
])
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"],
              jit_compile=True,            # XLA compilation
              steps_per_execution=32)      # fewer Python round trips per epoch
model.fit(ds, epochs=2, verbose=0,
          callbacks=[keras.callbacks.TensorBoard(log_dir="logs", profile_batch=(5, 15))])

Measure before and after each change (time per epoch, examples per second) instead of assuming a gain; on CPUs, mixed float16 can be slower.

Common Pitfalls

  • Optimizing the model while the GPU sits idle waiting for data.

  • Retracing tf.function on every call because of changing Python arguments or shapes.

  • Turning on mixed precision without keeping the final softmax in float32.

  • Comparing speeds including the first epoch, which includes tracing and compilation.

35

What techniques does TensorFlow use for graph optimization?

Graph optimization happens after tf.function traces Python code into a graph: TensorFlow’s Grappler optimizer rewrites the graph to remove waste, and the optional XLA compiler fuses operations into efficient device kernels. Because the whole computation is known in advance, TensorFlow can fold constants, drop unused nodes, fuse operations, reorder layouts and plan memory, which is not possible when running op by op in eager mode. Deployment tools add further optimizations such as quantization when converting models.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Grappler Optimizations

Optimizer What it does
Constant folding Computes parts of the graph that depend only on constants once
Arithmetic simplification Removes redundant operations (e.g. x * 1, repeated transposes)
Common subexpression elimination Reuses identical computations
Dependency / pruning Removes nodes that do not affect outputs
Layout optimizer Chooses NHWC or NCHW layouts for faster GPU convolutions
Remapper Fuses patterns such as Conv2D + BiasAdd + ReLU into one kernel
Memory optimizer Swaps or recomputes tensors to reduce peak memory
Loop optimizer Hoists loop-invariant work out of loops
Auto mixed precision Optional: rewrites eligible ops to float16

XLA (Accelerated Linear Algebra)

XLA compiles a whole function into fused kernels specific to the device, cutting memory traffic and launch overhead. Enable it with @tf.function(jit_compile=True) or model.compile(..., jit_compile=True). It works best with static shapes; changing shapes trigger recompilation.

Code Example

import time
import tensorflow as tf

tf.random.set_seed(42)
x = tf.random.normal([2048, 2048])

def block(t):
    for _ in range(10):
        t = tf.nn.relu(t @ tf.transpose(tf.transpose(t)) * 1.0 + 0.0)   # redundant ops Grappler removes
        t = t / (tf.norm(t) + 1e-6)
    return t

eager_fn = block
graph_fn = tf.function(block)
xla_fn = tf.function(block, jit_compile=True)

for name, fn in [("eager", eager_fn), ("graph", graph_fn), ("xla", xla_fn)]:
    fn(x)                                   # warm-up: tracing and compilation
    start = time.perf_counter()
    for _ in range(5):
        fn(x).numpy()
    print(f"{name}: {(time.perf_counter() - start) / 5 * 1000:.1f} ms per call")

# Inspect Grappler options and turn individual passes on or off
print(tf.config.optimizer.get_experimental_options())
tf.config.optimizer.set_experimental_options({"constant_folding": True, "remapping": True})

Timings depend on hardware; on small CPU workloads the differences may be minor.

Deployment-Time Optimizations

  • LiteRT (formerly TensorFlow Lite) conversion applies operator fusion and optional quantization.

  • TensorRT integration (TF-TRT) builds optimized NVIDIA engines from SavedModels.

  • Freezing variables into constants enables more constant folding for inference graphs.

Common Pitfalls

  • Expecting speedups from eager code: optimizations apply only inside tf.function.

  • Dynamic shapes that force XLA to recompile repeatedly.

  • Python side effects in traced functions, which run once at trace time.

36

How do you optimize memory usage in TensorFlow?

Memory optimization in TensorFlow keeps models and data within GPU (or host) memory so training does not crash with out-of-memory (OOM) errors and so you can use larger models or batches. Memory is used by weights, optimizer state, activations saved for backpropagation, and input batches; activations usually dominate during training. The main levers are batch size, numeric precision, recomputation of activations, efficient input pipelines and controlling how TensorFlow allocates GPU memory.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Techniques

Technique How Saves
Smaller batch + gradient accumulation Accumulate gradients over k steps before applying (custom loop or gradient_accumulation_steps in Keras optimizers) Activation memory, same effective batch
Mixed precision keras.mixed_precision.set_global_policy("mixed_float16") About half the activation memory
Gradient checkpointing tf.recompute_grad on blocks Recomputes activations instead of storing them
Smaller optimizer state SGD with momentum instead of Adam (Adam stores two extra values per weight) Optimizer memory
Memory growth tf.config.experimental.set_memory_growth(gpu, True) Avoids grabbing all GPU memory at start
Memory limit tf.config.set_logical_device_configuration(gpu, [LogicalDeviceConfiguration(memory_limit=4096)]) Shares a GPU between processes
Streaming input tf.data from files instead of loading everything with NumPy Host memory
Efficient architecture Global pooling instead of large Flatten + Dense; depthwise convolutions Weights and activations
Free resources keras.backend.clear_session() between experiments Leaked graphs and models

Code Example

import keras
import tensorflow as tf

for gpu in tf.config.list_physical_devices("GPU"):
    tf.config.experimental.set_memory_growth(gpu, True)     # must run before GPUs are initialized

keras.utils.set_random_seed(42)
keras.mixed_precision.set_global_policy("mixed_float16")


class CheckpointedBlock(keras.layers.Layer):
    """Dense block whose activations are recomputed in the backward pass."""
    def __init__(self, units, **kwargs):
        super().__init__(**kwargs)
        self.units = units
        self.d1 = keras.layers.Dense(units, activation="relu")
        self.d2 = keras.layers.Dense(units, activation="relu")

    def build(self, input_shape):
        # Create the weights here, not inside recompute_grad
        self.d1.build(input_shape)
        self.d2.build((*input_shape[:-1], self.units))

    def call(self, x):
        return tf.recompute_grad(lambda t: self.d2(self.d1(t)))(x)


model = keras.Sequential([keras.Input(shape=(1024,)), CheckpointedBlock(2048), CheckpointedBlock(2048),
                          keras.layers.Dense(10), keras.layers.Activation("softmax", dtype="float32")])
model.compile(optimizer=keras.optimizers.Adam(1e-3, gradient_accumulation_steps=4),   # effective batch x4
              loss="sparse_categorical_crossentropy")

if tf.config.list_physical_devices("GPU"):
    print(tf.config.experimental.get_memory_info("GPU:0"))   # current and peak bytes

Estimating Memory

Weights in float32 take 4 bytes per parameter; Adam adds two more copies, so roughly 16 bytes per parameter for weights + gradients + optimizer state, before activations. A 100M-parameter model therefore needs about 1.6 GB before counting activations.

Common Pitfalls

  • Calling set_memory_growth after the GPU has already been used, which raises an error.

  • Accumulating tensors in Python lists across steps (e.g. storing every loss tensor), which keeps memory alive.

  • Building new models in a loop without clear_session.

  • Loading a whole image dataset into a NumPy array instead of streaming it.

37

What is quantization in TensorFlow, and when should you use it?

Quantization stores and computes a model’s weights (and often activations) with lower-precision numbers, such as 8-bit integers instead of 32-bit floats. This makes models about four times smaller and usually faster on CPUs, mobile devices and edge accelerators, at the cost of a small accuracy loss. In TensorFlow you apply it while converting to LiteRT (formerly TensorFlow Lite), either after training (post-training quantization) or by simulating it during training (quantization-aware training) when accuracy must be preserved.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3, LiteRT converter `tf.lite.TFLiteConverter` (Python 3.12)

How It Works

A float value is mapped to an integer with a scale and zero point: q = round(x / scale) + zero_point, and recovered approximately as x ≈ scale * (q - zero_point). In words, the range of real values is divided into 256 steps, and each value is stored as the index of its step.

Types of Quantization

Method Weights Activations Needs Size Typical accuracy cost
Dynamic range int8 float (quantized on the fly) Nothing ~4x smaller Small
Full integer int8 int8 Representative dataset (~100-500 samples) ~4x smaller Small; required for some accelerators (Edge TPU, microcontrollers)
Float16 float16 float16 Nothing ~2x smaller Negligible; good for GPU delegates
Quantization-aware training (QAT) int8 int8 Retraining with fake-quant ops (TensorFlow Model Optimization Toolkit) ~4x smaller Lowest

Code Example: Post-Training Full Integer Quantization

import keras
import numpy as np
import tensorflow as tf

keras.utils.set_random_seed(42)
X = np.random.rand(1000, 16).astype("float32")
y = (X.sum(axis=1) > 8).astype("float32")
model = keras.Sequential([keras.Input(shape=(16,)), keras.layers.Dense(64, activation="relu"),
                          keras.layers.Dense(1, activation="sigmoid")])
model.compile(optimizer="adam", loss="binary_crossentropy")
model.fit(X, y, epochs=5, verbose=0)

def representative_data():
    for i in range(200):
        yield [X[i:i + 1]]                     # real-looking inputs to calibrate activation ranges

converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_data
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8
tflite_model = converter.convert()

with open("model_int8.tflite", "wb") as f:
    f.write(tflite_model)
print("quantized size (KB):", round(len(tflite_model) / 1024, 1))

Always evaluate the quantized model on held-out data with the LiteRT interpreter and compare it with the float model before shipping.

When to Use It

  • Deploying to phones, browsers, microcontrollers or edge TPUs.

  • CPU inference where latency or cost matters.

  • Not usually needed for GPU training, where mixed precision is the right tool.

Common Pitfalls

  • A representative dataset that does not match real inputs, which gives poor calibration.

  • Forgetting to quantize inputs on the client when using int8 input types.

  • Assuming the accuracy is unchanged without measuring it, especially for small models and regression outputs.

38

How do you use mixed-precision training in TensorFlow?

Mixed-precision training runs most computations in 16-bit floats (float16 or bfloat16) while keeping the weights and numerically sensitive operations in float32. On GPUs with Tensor Cores (NVIDIA Volta and newer) and on TPUs, this can speed training up significantly and roughly halve activation memory, usually with no loss in accuracy. In Keras you enable it with one global policy; float16 also needs loss scaling to stop small gradients from underflowing to zero, which fit handles automatically.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Policies

Policy Compute Weights Use on
mixed_float16 float16 float32 NVIDIA GPUs (compute capability 7.0+)
mixed_bfloat16 bfloat16 float32 TPUs and NVIDIA Ampere+ GPUs; no loss scaling needed
float32 float32 float32 Default; CPUs

bfloat16 keeps float32’s exponent range with less precision, so gradients rarely underflow; float16 has a narrow range, which is why loss scaling multiplies the loss by a large factor before backpropagation and divides the gradients by it afterwards.

Code Example with fit

import keras
import numpy as np

keras.utils.set_random_seed(42)
keras.mixed_precision.set_global_policy("mixed_float16")

inputs = keras.Input(shape=(784,))
x = keras.layers.Dense(1024, activation="relu")(inputs)
x = keras.layers.Dense(1024, activation="relu")(x)
logits = keras.layers.Dense(10)(x)
outputs = keras.layers.Activation("softmax", dtype="float32")(logits)   # final output in float32
model = keras.Model(inputs, outputs)

print(model.layers[1].compute_dtype, model.layers[1].variable_dtype)    # float16 float32
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"])
# With mixed_float16, compile wraps the optimizer in a LossScaleOptimizer automatically.
model.fit(np.random.rand(2048, 784).astype("float32"), np.random.randint(0, 10, 2048),
          batch_size=256, epochs=1, verbose=0)

Custom Training Loop

import keras
import tensorflow as tf

optimizer = keras.optimizers.LossScaleOptimizer(keras.optimizers.Adam())
loss_fn = keras.losses.SparseCategoricalCrossentropy()

@tf.function
def train_step(model, x, y):
    with tf.GradientTape() as tape:
        loss = loss_fn(y, model(x, training=True))
        scaled_loss = optimizer.scale_loss(loss)          # multiply by the current loss scale
    grads = tape.gradient(scaled_loss, model.trainable_variables)
    optimizer.apply(grads, model.trainable_variables)     # unscales, skips steps with inf/NaN
    return loss

Tips

  • Make layer sizes and batch sizes multiples of 8 so Tensor Cores are used.

  • Keep softmax, losses and normalization statistics in float32 (Keras does this for its built-in layers).

  • Measure: on CPUs or older GPUs, mixed precision can be slower.

Common Pitfalls

  • Leaving the output layer in float16, which can make the loss unstable or overflow.

  • Using float16 in a custom loop without loss scaling, so small gradients become zero.

  • Setting the policy after building the model; set it first.

39

How does TensorFlow support multi-GPU and distributed training?

TensorFlow supports multi-GPU and distributed training through the tf.distribute API. A strategy object decides how variables and computation are placed across GPUs, machines or TPUs, and Keras’s fit uses it automatically when the model is built inside strategy.scope(). Most setups use synchronous data parallelism: each device holds a full copy of the model, processes part of every batch, and gradients are averaged with an all-reduce before every replica applies the same update.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Options

Setup Strategy Notes
One machine, several GPUs MirroredStrategy NCCL all-reduce; easiest to adopt
Several machines MultiWorkerMirroredStrategy Needs TF_CONFIG on each worker; collective ops over the network
TPUs TPUStrategy Connect with TPUClusterResolver
Many workers, async updates ParameterServerStrategy Variables live on parameter servers; tolerant of slow workers

Multi-Worker Setup

Each worker runs the same script with its own TF_CONFIG, for example:

import json
import os

os.environ["TF_CONFIG"] = json.dumps({
    "cluster": {"worker": ["10.0.0.1:12345", "10.0.0.2:12345"]},
    "task": {"type": "worker", "index": 0},           # index 1 on the second machine
})

Worker 0 is the chief and usually writes checkpoints and TensorBoard logs.

Code Example: Custom Loop on Multiple GPUs

import keras
import numpy as np
import tensorflow as tf

keras.utils.set_random_seed(42)
strategy = tf.distribute.MirroredStrategy()
GLOBAL_BATCH = 64 * strategy.num_replicas_in_sync

X = np.random.rand(4096, 32).astype("float32")
y = np.random.randint(0, 2, size=4096).astype("float32")
dataset = tf.data.Dataset.from_tensor_slices((X, y)).shuffle(4096, seed=42).batch(GLOBAL_BATCH)
dist_dataset = strategy.experimental_distribute_dataset(dataset)

with strategy.scope():
    model = keras.Sequential([keras.Input(shape=(32,)), keras.layers.Dense(64, activation="relu"),
                              keras.layers.Dense(1)])
    optimizer = keras.optimizers.Adam(1e-3)
    loss_obj = keras.losses.BinaryCrossentropy(from_logits=True, reduction="none")

def train_step(inputs):
    x, labels = inputs
    with tf.GradientTape() as tape:
        per_example = loss_obj(labels[:, None], model(x, training=True))
        loss = tf.nn.compute_average_loss(per_example, global_batch_size=GLOBAL_BATCH)
    grads = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(grads, model.trainable_variables))
    return loss

@tf.function
def distributed_step(inputs):
    per_replica = strategy.run(train_step, args=(inputs,))
    return strategy.reduce(tf.distribute.ReduceOp.SUM, per_replica, axis=None)

for epoch in range(2):
    total = sum(float(distributed_step(batch)) for batch in dist_dataset)
    print(f"epoch {epoch + 1}: summed loss {total:.3f}")

With fit, all of this is automatic: build and compile inside strategy.scope() and call model.fit(dataset).

Practical Tips

  • Scale the global batch size with the number of replicas and adjust the learning rate (often with warm-up).

  • Make sure the input pipeline can feed every GPU; use prefetch and sharded files.

  • Add BackupAndRestore for fault tolerance in multi-worker jobs.

  • For models too large for one GPU, data parallelism is not enough; consider model parallelism (for example the Keras 3 distribution API on the JAX backend) or smaller models.

Common Pitfalls

  • Averaging the loss over the per-replica batch instead of the global batch, which scales gradients wrongly.

  • Every worker writing checkpoints to the same path.

  • Expecting linear speed-ups when communication or input loading is the bottleneck.

40

What are best practices for writing efficient TensorFlow code?

Efficient TensorFlow code keeps the accelerator busy and avoids Python overhead: express work as vectorized tensor operations, compile hot paths with tf.function, feed data with a parallel tf.data pipeline, and keep data on the device instead of converting to NumPy in loops. It also means writing code that is easy to profile and debug: develop eagerly, then compile, and measure every change with the profiler rather than guessing.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Best Practices

Area Do Avoid
Computation Vectorized ops (tf.reduce_*, broadcasting, tf.vectorized_map) Python loops over tensor elements
Compilation tf.function, jit_compile=True for stable shapes Retracing from changing Python arguments
Data tf.data with map(num_parallel_calls=AUTOTUNE), prefetch, cache from_generator with heavy Python work
Host-device Keep tensors on the GPU .numpy() or float(tensor) inside training loops
Precision Mixed precision on supported GPUs/TPUs float64 unless needed
Training API fit with callbacks; steps_per_execution Hand-written loops without tf.function
Shapes Static shapes, input_signature for serving Unbounded variable shapes causing recompiles
Debugging tf.debugging.assert_*, run_functions_eagerly(True) while developing Leaving eager mode on in production
Reproducibility keras.utils.set_random_seed(42) Unseeded experiments

Code Example: Loop Versus Vectorized

import time
import tensorflow as tf

tf.random.set_seed(42)
points = tf.random.normal([2000, 3])
centers = tf.random.normal([50, 3])

def nearest_center_loop(p, c):
    out = []
    for i in range(p.shape[0]):                                 # Python loop: slow
        d = tf.reduce_sum((c - p[i]) ** 2, axis=1)
        out.append(int(tf.argmin(d)))
    return out

@tf.function(input_signature=[tf.TensorSpec([None, 3]), tf.TensorSpec([None, 3])])
def nearest_center_vectorized(p, c):
    # Pairwise squared distances via broadcasting: (N, 1, 3) - (1, K, 3) -> (N, K)
    d = tf.reduce_sum((p[:, None, :] - c[None, :, :]) ** 2, axis=-1)
    return tf.argmin(d, axis=1)

start = time.perf_counter()
loop_result = nearest_center_loop(points, centers)
loop_time = time.perf_counter() - start

nearest_center_vectorized(points, centers)                      # trace once
start = time.perf_counter()
vec_result = nearest_center_vectorized(points, centers)
vec_time = time.perf_counter() - start

print("same result:", loop_result == vec_result.numpy().tolist())
print(f"loop {loop_time:.3f}s vs vectorized {vec_time:.4f}s")

The fixed input_signature lets the function accept any number of points without retracing.

More Habits

  • Use Keras preprocessing layers inside the model so the same logic runs in training and serving.

  • Profile with TensorBoard before optimizing, and compare examples per second rather than wall time of the first epoch.

  • Keep functions pure in tf.function: no hidden Python state, no printing for logic (use tf.print).

Common Pitfalls

  • Creating variables or models inside a tf.function that is called repeatedly.

  • Mixing NumPy and TensorFlow in hot loops, which copies data back and forth.

  • Building large Python lists of tensors and calling tf.stack at the end of each step; use tf.TensorArray inside graphs.

41

How do you handle image data in TensorFlow?

Handling image data in TensorFlow means loading image files into tensors, resizing and scaling them consistently, augmenting them during training, and feeding them in efficient batches. Keras provides image_dataset_from_directory for folder-organized data and preprocessing layers (Resizing, Rescaling, RandomFlip) that can live inside the model, while tf.io and tf.image handle custom decoding and transformations. Images are represented as tensors of shape (batch, height, width, channels) by default.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Tools

Task API
Load from class folders keras.utils.image_dataset_from_directory(path, image_size=(224, 224), batch_size=32)
Read/decode files manually tf.io.read_file, tf.io.decode_image / decode_jpeg / decode_png
Resize, crop, pad keras.layers.Resizing, tf.image.resize, tf.image.resize_with_crop_or_pad
Scale pixels keras.layers.Rescaling(1/255) or a pretrained model’s preprocess_input
Augment RandomFlip, RandomRotation, RandomZoom, RandomContrast, RandomCrop
Large datasets TFRecord files with serialized images, tf.data with parallel map
Ready-made datasets keras.datasets (MNIST, CIFAR-10), TensorFlow Datasets (tfds.load)

Code Example

import keras
import numpy as np
import tensorflow as tf
from pathlib import Path

keras.utils.set_random_seed(42)

# Create a tiny folder dataset so the example is self-contained: data/cats, data/dogs
rng = np.random.default_rng(42)
for label in ["cats", "dogs"]:
    folder = Path("data") / label
    folder.mkdir(parents=True, exist_ok=True)
    for i in range(20):
        img = (rng.random((64, 64, 3)) * 255).astype("uint8")
        keras.utils.save_img(folder / f"{i}.png", img)

train_ds, val_ds = keras.utils.image_dataset_from_directory(
    "data", validation_split=0.2, subset="both", seed=42,
    image_size=(128, 128), batch_size=8, label_mode="binary")
print(train_ds.class_names)                         # ['cats', 'dogs']

augment = keras.Sequential([keras.layers.RandomFlip("horizontal"), keras.layers.RandomRotation(0.1)])
train_ds = train_ds.prefetch(tf.data.AUTOTUNE)
val_ds = val_ds.cache().prefetch(tf.data.AUTOTUNE)

model = keras.Sequential([
    keras.Input(shape=(128, 128, 3)),
    augment,                                        # runs only in training
    keras.layers.Rescaling(1.0 / 255),              # same scaling at inference
    keras.layers.Conv2D(16, 3, activation="relu"),
    keras.layers.MaxPooling2D(),
    keras.layers.GlobalAveragePooling2D(),
    keras.layers.Dense(1, activation="sigmoid"),
])
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
model.fit(train_ds, validation_data=val_ds, epochs=2, verbose=0)

# Manual decoding for a single file
raw = tf.io.read_file("data/cats/0.png")
img = tf.image.resize(tf.io.decode_png(raw, channels=3), (128, 128))
print(img.shape, img.dtype)                         # (128, 128, 3) float32

Common Pitfalls

  • Different preprocessing in training and serving; keep resizing and scaling inside the model.

  • Using the wrong preprocess_input for a pretrained backbone (some expect [-1, 1], others [0, 255]).

  • Applying augmentation to validation or test data.

  • Loading thousands of images into one NumPy array instead of streaming with tf.data.

42

What is the text preprocessing process in TensorFlow?

Text preprocessing in TensorFlow turns raw strings into numeric tensors a model can use: standardize the text (lowercase, strip punctuation), split it into tokens, map tokens to integer IDs from a vocabulary, and pad or truncate sequences to a fixed length. The main tool is the Keras TextVectorization layer, which learns its vocabulary with adapt() on training data and can be placed inside the model so raw strings work at inference. For pretrained Transformers, use the matching subword tokenizer from KerasHub instead.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3, KerasHub (Python 3.12)

Pipeline Steps

Step What happens Tool
Standardize Lowercase, remove punctuation/HTML TextVectorization(standardize=...), tf.strings.regex_replace
Tokenize Split into words, n-grams or subwords TextVectorization(split="whitespace", ngrams=2), KerasHub tokenizers
Build vocabulary Keep the most frequent tokens; reserve padding (0) and out-of-vocabulary (1) adapt() on training text only
Encode Tokens → integer IDs, multi-hot, counts or TF-IDF output_mode="int", "multi_hot", "count", "tf_idf"
Pad/truncate Fixed sequence length output_sequence_length=...
Embed IDs → dense vectors Embedding(mask_zero=True)

Code Example

import re
import string

import keras
import numpy as np
import tensorflow as tf

keras.utils.set_random_seed(42)
train_texts = np.array(["Great product, works well!", "Terrible. Broke after a day.",
                        "<br>Really great value", "Not good, would not buy again"])
train_labels = np.array([1, 0, 1, 0])

def custom_standardize(text):
    text = tf.strings.lower(text)
    text = tf.strings.regex_replace(text, "<br>", " ")
    return tf.strings.regex_replace(text, f"[{re.escape(string.punctuation)}]", "")

vectorizer = keras.layers.TextVectorization(max_tokens=10000, output_mode="int",
                                            output_sequence_length=8, standardize=custom_standardize)
vectorizer.adapt(train_texts)                      # vocabulary from training data only
print(vectorizer.get_vocabulary()[:6])             # ['', '[UNK]', ...]
print(vectorizer(["great value, unknownword"]).numpy())

model = keras.Sequential([
    keras.Input(shape=(1,), dtype="string"),
    vectorizer,                                     # raw strings in, IDs out
    keras.layers.Embedding(10000, 16, mask_zero=True),
    keras.layers.GlobalAveragePooling1D(),
    keras.layers.Dense(1, activation="sigmoid"),
])
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
model.fit(train_texts[:, None], train_labels, epochs=5, verbose=0)
print(model.predict(np.array([["great product"]]), verbose=0))

For bag-of-words models, use output_mode="tf_idf" with ngrams=2 and a Dense classifier; it is a strong, fast baseline.

Pretrained Tokenizers

Transformer models need their own tokenizer, for example keras_hub.models.BertTokenizer.from_preset("bert_base_en_uncased"). Mixing a custom vocabulary with pretrained embeddings breaks the token-to-vector mapping.

Common Pitfalls

  • Calling adapt on the full dataset including test data, which leaks vocabulary information.

  • Removing words that carry meaning for the task (e.g. “not” in sentiment analysis).

  • Truncating long texts so much that the useful part is lost; check the length distribution first.

43

How would you approach time-series forecasting with TensorFlow?

Time-series forecasting with TensorFlow frames the problem as supervised learning on sliding windows: the model sees the last N time steps and predicts the next value or the next H values. The workflow is a chronological train/validation/test split, scaling with statistics from the training period only, window creation with tf.data or keras.utils.timeseries_dataset_from_array, a simple baseline, then models of increasing complexity (dense, LSTM/GRU, 1D CNN, Transformer). Always compare against a naive forecast, because deep models often fail to beat it on small or noisy series.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Approach

Step Details
Split By time: e.g. 70% train, 15% validation, 15% test; never shuffle across the split
Scale Fit mean/std on the training period only
Features Lags, calendar features (day of week, holidays), known future inputs
Window Input width (lookback), forecast horizon, stride
Baselines Last value (naive), seasonal naive (value one season ago)
Models Dense on flattened windows, LSTM/GRU, Conv1D, Transformer encoder
Multi-step output Single-shot (Dense of size H) or autoregressive (feed predictions back)
Metrics MAE, RMSE, MASE (error relative to naive)

Code Example

import keras
import numpy as np

keras.utils.set_random_seed(42)
rng = np.random.default_rng(42)
t = np.arange(2000)
series = (10 + 0.01 * t + 3 * np.sin(2 * np.pi * t / 50) + rng.normal(0, 0.5, t.size)).astype("float32")

LOOKBACK, HORIZON = 100, 10
n_train, n_val = 1400, 300
mean, std = series[:n_train].mean(), series[:n_train].std()     # training statistics only
scaled = (series - mean) / std

def make_ds(data, start, end, shuffle):
    # inputs: data[i : i + LOOKBACK]; targets: the next HORIZON values
    segment = data[start:end]
    targets = np.stack([segment[i + LOOKBACK:i + LOOKBACK + HORIZON]
                        for i in range(len(segment) - LOOKBACK - HORIZON + 1)])
    return keras.utils.timeseries_dataset_from_array(
        segment[:len(targets) + LOOKBACK - 1, None], targets, sequence_length=LOOKBACK,
        batch_size=32, shuffle=shuffle, seed=42)

train_ds = make_ds(scaled, 0, n_train, shuffle=True)        # shuffling windows within train is fine
val_ds = make_ds(scaled, n_train - LOOKBACK, n_train + n_val, shuffle=False)
test_ds = make_ds(scaled, n_train + n_val - LOOKBACK, len(scaled), shuffle=False)

model = keras.Sequential([
    keras.Input(shape=(LOOKBACK, 1)),
    keras.layers.Conv1D(32, 5, activation="relu"),
    keras.layers.LSTM(32),
    keras.layers.Dense(HORIZON),                            # single-shot multi-step forecast
])
model.compile(optimizer="adam", loss="mse", metrics=["mae"])
model.fit(train_ds, validation_data=val_ds, epochs=30, verbose=0,
          callbacks=[keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True)])

# Compare with a naive forecast (repeat the last observed value) on the test set
model_mae, naive_mae = [], []
for x, y in test_ds:
    pred = model.predict(x, verbose=0)
    model_mae.append(np.abs(pred - y.numpy()).mean())
    naive_mae.append(np.abs(x.numpy()[:, -1, :] - y.numpy()).mean())
print(f"model MAE: {np.mean(model_mae) * std:.3f} | naive MAE: {np.mean(naive_mae) * std:.3f}")

Validation and test windows start LOOKBACK steps early so their inputs use past values, but their targets never overlap the training targets.

Common Pitfalls

  • Shuffling before splitting or scaling with full-series statistics, which leaks the future.

  • Skipping the naive baseline, which hides models that learned nothing.

  • Using features at prediction time that would not be known then (e.g. future weather actuals).

  • Evaluating with a single split; use rolling-origin (walk-forward) validation when possible.

44

What support does TensorFlow offer for transfer learning?

TensorFlow supports transfer learning through pretrained models you can load with one line and adapt to a new task: Keras Applications (ResNet, EfficientNet, MobileNet, ConvNeXt and more, with ImageNet weights), KerasHub (pretrained text and multimodal models such as BERT, Gemma and vision backbones via from_preset), and Kaggle Models / TensorFlow Hub. The usual workflow is feature extraction with a frozen backbone and a new head, followed by fine-tuning some top layers at a low learning rate.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3, KerasHub (Python 3.12)

Sources of Pretrained Models

Source Examples How to load
Keras Applications ResNet50, EfficientNetV2, MobileNetV3, ConvNeXt keras.applications.EfficientNetV2B0(include_top=False, weights="imagenet")
KerasHub BERT, DistilBERT, Gemma, ResNet, ViT keras_hub.models.TextClassifier.from_preset(...)
Kaggle Models / TF Hub SavedModels for images, text, audio Load the SavedModel and wrap it, or use KerasHub presets

Two-Phase Workflow

Phase Backbone Learning rate Purpose
1. Feature extraction Frozen (trainable = False) Normal (e.g. 1e-3) Train the new head without destroying pretrained features
2. Fine-tuning Top layers unfrozen Low (e.g. 1e-5) Adapt features to the new data

Code Example

import keras

keras.utils.set_random_seed(42)
NUM_CLASSES = 5

base = keras.applications.EfficientNetV2B0(include_top=False, weights="imagenet",
                                           input_shape=(224, 224, 3))    # downloads weights
base.trainable = False

inputs = keras.Input(shape=(224, 224, 3))
x = keras.layers.RandomFlip("horizontal")(inputs)
x = base(x, training=False)                 # keep BatchNorm in inference mode
x = keras.layers.GlobalAveragePooling2D()(x)
x = keras.layers.Dropout(0.2)(x)
outputs = keras.layers.Dense(NUM_CLASSES, activation="softmax")(x)
model = keras.Model(inputs, outputs)

model.compile(optimizer=keras.optimizers.Adam(1e-3), loss="sparse_categorical_crossentropy",
              metrics=["accuracy"])
# model.fit(train_ds, validation_data=val_ds, epochs=10)        # phase 1

base.trainable = True
for layer in base.layers[:-30]:             # unfreeze only the last 30 layers
    layer.trainable = False
model.compile(optimizer=keras.optimizers.Adam(1e-5), loss="sparse_categorical_crossentropy",
              metrics=["accuracy"])         # recompile after changing trainable flags
# model.fit(train_ds, validation_data=val_ds, epochs=5)         # phase 2
print(len(model.trainable_weights), "trainable weight tensors")

EfficientNetV2 in Keras includes its own input rescaling, so it takes raw pixel values in [0, 255]; other backbones need their preprocess_input function.

Common Pitfalls

  • Not calling the frozen base with training=False, so BatchNormalization statistics change during fine-tuning.

  • Fine-tuning with a high learning rate, which destroys pretrained features.

  • Forgetting to recompile after changing trainable.

  • Using a backbone whose pretraining domain is very different (natural images for X-rays) without enough fine-tuning data.

45

How is TensorFlow deployed on mobile and edge devices?

TensorFlow models are deployed to mobile and edge devices with LiteRT (formerly TensorFlow Lite): you convert a trained model to the compact .tflite flatbuffer format, optionally quantize it, and run it with the LiteRT interpreter on Android, iOS, embedded Linux or microcontrollers. Hardware delegates offload work to GPUs, DSPs and NPUs for speed. For browsers there is TensorFlow.js. On-device inference gives low latency, offline operation and better privacy, at the cost of model size and accuracy constraints.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3, LiteRT (Python 3.12)

Deployment Options

Target Runtime Notes
Android LiteRT (Java/Kotlin API, Google Play services runtime) GPU delegate; vendor NPU delegates
iOS LiteRT (Swift/Objective-C) Metal GPU delegate, Core ML delegate
Embedded Linux (Raspberry Pi, Jetson, Coral) LiteRT Python/C++; Edge TPU runtime Edge TPU needs full int8 quantization
Microcontrollers LiteRT for Microcontrollers (C++) Kilobytes of memory; int8 models only
Browser / Node.js TensorFlow.js WebGL/WebGPU/WASM backends

Workflow

  1. Train in Keras and keep preprocessing inside the model where possible.

  2. Convert with tf.lite.TFLiteConverter.

  3. Optimize: post-training quantization (dynamic, float16, full int8), pruning, or a smaller architecture such as MobileNet.

  4. Check accuracy and latency of the converted model on real devices (use the LiteRT benchmark tool).

  5. Bundle the .tflite file with the app or download it at runtime.

Code Example: Convert and Test

import keras
import numpy as np
import tensorflow as tf

keras.utils.set_random_seed(42)
X = np.random.rand(500, 10).astype("float32")
y = (X.mean(axis=1) > 0.5).astype("float32")
model = keras.Sequential([keras.Input(shape=(10,)), keras.layers.Dense(32, activation="relu"),
                          keras.layers.Dense(1, activation="sigmoid")])
model.compile(optimizer="adam", loss="binary_crossentropy")
model.fit(X, y, epochs=5, verbose=0)

converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]          # dynamic-range quantization
tflite_bytes = converter.convert()
with open("model.tflite", "wb") as f:
    f.write(tflite_bytes)

# Run it the way a device would (the standalone ai-edge-litert package offers the same Interpreter API)
interpreter = tf.lite.Interpreter(model_path="model.tflite")
interpreter.allocate_tensors()
inp, out = interpreter.get_input_details()[0], interpreter.get_output_details()[0]
interpreter.set_tensor(inp["index"], X[:1])
interpreter.invoke()
print("LiteRT:", interpreter.get_tensor(out["index"]), "| Keras:", model.predict(X[:1], verbose=0))

Design Considerations

  • Model size and memory: app-store limits, RAM on low-end phones.

  • Latency and battery: prefer smaller architectures and quantization.

  • Supported operations: some TensorFlow ops are not in LiteRT’s builtin set; SELECT_TF_OPS adds them but increases binary size.

  • Updates: download new models remotely; monitor on-device accuracy where possible.

Common Pitfalls

  • Preprocessing in the app that differs from training (resizing, normalization, color order).

  • Skipping on-device benchmarking; desktop timing is not representative.

  • Using unsupported custom ops that fail at conversion time.

46

How is TensorFlow used in healthcare? Give an example.

TensorFlow is used in healthcare for medical image analysis (detecting diabetic retinopathy in retinal photos, finding lung nodules in CT scans, classifying skin lesions), clinical risk prediction from electronic health records, analysis of ECG and other signals, and genomics. A well-known example is Google’s research on deep learning for diabetic retinopathy screening from retinal fundus images, built with TensorFlow models. Healthcare ML differs from other domains because errors have clinical consequences, data is private and regulated, and models must be validated on data from the hospitals where they will be used.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3, scikit-learn 1.5+ (Python 3.12)

Typical Applications

Area Data Model type
Radiology, ophthalmology, dermatology X-rays, CT, MRI, retinal photos, skin images CNNs / vision transformers with transfer learning
Pathology Gigapixel slide images Patch-based CNNs, multiple-instance learning
Clinical risk scores EHR tabular and time-series data Dense networks, RNNs, gradient-boosted trees
Cardiology ECG signals 1D CNNs
Clinical text Notes, reports Transformers

Code Example: Malignant vs Benign Tumor Classifier

This uses the Wisconsin breast cancer dataset in scikit-learn (features computed from cell-nucleus images).

import keras
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import classification_report, roc_auc_score
from sklearn.model_selection import train_test_split

keras.utils.set_random_seed(42)
data = load_breast_cancer()
X, y = data.data.astype("float32"), (data.target == 0).astype("float32")   # 1 = malignant
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

norm = keras.layers.Normalization()
norm.adapt(X_train[: int(len(X_train) * 0.8)])      # only the rows fit() trains on (validation_split takes the last 20%)

model = keras.Sequential([
    keras.Input(shape=(X.shape[1],)), norm,
    keras.layers.Dense(32, activation="relu"), keras.layers.Dropout(0.2),
    keras.layers.Dense(1, activation="sigmoid"),
])
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=[keras.metrics.AUC(name="auc"),
                                                                     keras.metrics.Recall(name="recall")])
model.fit(X_train, y_train, validation_split=0.2, epochs=100, batch_size=32, verbose=0,
          callbacks=[keras.callbacks.EarlyStopping(monitor="val_auc", mode="max", patience=10,
                                                   restore_best_weights=True)])

proba = model.predict(X_test, verbose=0).ravel()
threshold = 0.3                                      # lower threshold: missing a malignancy costs more
print("ROC AUC:", round(roc_auc_score(y_test, proba), 3))
print(classification_report(y_test, (proba >= threshold).astype(int), target_names=["benign", "malignant"]))

The threshold should be chosen on validation data with clinicians, based on the acceptable trade-off between missed cancers (false negatives) and unnecessary follow-ups (false positives).

What Makes Healthcare Different

  • Metrics: sensitivity (recall), specificity, AUROC, calibration; accuracy is misleading with rare diseases.

  • Validation: external validation on other hospitals, devices and populations; prospective studies before clinical use.

  • Privacy and regulation: HIPAA/GDPR, de-identification, federated learning; medical-device regulation (e.g. FDA) for diagnostic software.

  • Fairness: check performance across age, sex and ethnicity groups.

  • Explainability: saliency maps or feature attributions help clinicians trust and audit predictions.

Common Pitfalls

  • Splitting by image instead of by patient, so the same patient is in train and test.

  • Shortcut learning, such as a model detecting the scanner type or hospital markers instead of disease.

  • Reporting only accuracy on imbalanced data.

47

Write TensorFlow code to multiply two tensors element-wise.

Element-wise multiplication multiplies values at the same positions in two tensors, producing a tensor of the same shape. In TensorFlow you use tf.multiply(a, b) or the * operator, which also supports broadcasting: when shapes differ, dimensions of size 1 are stretched to match. It is different from matrix multiplication (tf.matmul or @), which combines rows and columns.

Read full answer

Applies toTensorFlow 2.16+ (Python 3.12)

Code Example

import tensorflow as tf

a = tf.constant([[1, 2, 3],
                 [4, 5, 6]], dtype=tf.float32)
b = tf.constant([[10, 20, 30],
                 [40, 50, 60]], dtype=tf.float32)

product = tf.multiply(a, b)            # same as a * b
print(product.numpy())
# [[ 10.  40.  90.]
#  [160. 250. 360.]]

# Broadcasting: shape (2, 3) * shape (3,) multiplies each row by the vector
scale = tf.constant([1.0, 0.5, 2.0])
print((a * scale).numpy())
# [[ 1.   1.   6. ]
#  [ 4.   2.5 12. ]]

# Broadcasting with a column: shape (2, 3) * shape (2, 1)
col = tf.constant([[2.0], [10.0]])
print((a * col).numpy())
# [[ 2.  4.  6.]
#  [40. 50. 60.]]

# Contrast with matrix multiplication: (2, 3) @ (3, 2) -> (2, 2)
print(tf.matmul(a, tf.transpose(b)).numpy())
# [[ 140.  320.]
#  [ 320.  770.]]

The matrix product entry [0, 0] is 1*10 + 2*20 + 3*30 = 140, the sum of the element-wise products of row 0 of a and row 0 of b.

Rules to Remember

Situation Result
Same shapes Element-wise product, same shape
Shapes compatible by broadcasting (align from the right; sizes equal or 1) Expanded result shape
Incompatible shapes, e.g. (2, 3) and (2,) InvalidArgumentError
Different dtypes, e.g. int32 and float32 Error: TensorFlow does not auto-cast; use tf.cast

Related Operations

  • tf.math.multiply_no_nan(a, b) returns 0 where b is 0 even if a is NaN or inf.

  • tf.einsum("ij,ij->ij", a, b) expresses the same product with index notation.

  • tf.reduce_sum(a * b) gives the sum of all element-wise products (a dot product when both are vectors).

Common Pitfalls

  • Mixing integer and float tensors without casting.

  • Expecting * to perform matrix multiplication.

  • Unintended broadcasting, for example shape (3,) with (3, 1) producing a (3, 3) result; check shapes with tf.debugging.assert_shapes.

48

Implement logistic regression using TensorFlow.

Logistic regression predicts the probability of a binary outcome by passing a weighted sum of the features through the sigmoid function, p = 1 / (1 + exp(-(w·x + b))). It is trained by minimizing binary cross-entropy (log loss), which penalizes confident wrong predictions heavily. In TensorFlow you can implement it from scratch with variables and tf.GradientTape, or as a Keras model with a single Dense(1, activation="sigmoid") layer; both are shown below.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3, scikit-learn 1.5+ (Python 3.12)

The Model in Words

  • Linear score: z = w·x + b, a weighted sum of the features plus a bias.

  • Probability: the sigmoid squashes z into (0, 1).

  • Loss: for each example, -(y log p + (1 - y) log(1 - p)), averaged over the batch.

  • Update: gradient descent moves w and b against the gradient of the loss; for logistic regression the gradient is (p - y) * x.

Code Example: From Scratch

import numpy as np
import tensorflow as tf
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import accuracy_score, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

tf.random.set_seed(42)
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
scaler = StandardScaler().fit(X_train)                     # fit on training data only
X_train = scaler.transform(X_train).astype("float32")
X_test = scaler.transform(X_test).astype("float32")
y_train = y_train.astype("float32")

w = tf.Variable(tf.zeros([X_train.shape[1], 1]))
b = tf.Variable(0.0)
learning_rate, l2 = 0.1, 1e-3

@tf.function
def train_step(x, y):
    with tf.GradientTape() as tape:
        logits = tf.squeeze(x @ w + b, axis=1)
        loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels=y, logits=logits))
        loss += l2 * tf.reduce_sum(w ** 2)                 # L2 regularization
    dw, db = tape.gradient(loss, [w, b])
    w.assign_sub(learning_rate * dw)                       # plain gradient descent step
    b.assign_sub(learning_rate * db)
    return loss

ds = tf.data.Dataset.from_tensor_slices((X_train, y_train)).shuffle(1000, seed=42).batch(64)
for epoch in range(100):
    for xb, yb in ds:
        loss = train_step(xb, yb)

proba = tf.sigmoid(tf.squeeze(X_test @ w + b, axis=1)).numpy()
print("accuracy:", round(accuracy_score(y_test, proba >= 0.5), 3),
      "| ROC AUC:", round(roc_auc_score(y_test, proba), 3))

sigmoid_cross_entropy_with_logits combines the sigmoid and the log loss in a numerically stable way, which avoids log(0) when probabilities saturate.

Code Example: Keras Version

import keras

keras.utils.set_random_seed(42)
model = keras.Sequential([keras.Input(shape=(X_train.shape[1],)),
                          keras.layers.Dense(1, activation="sigmoid",
                                             kernel_regularizer=keras.regularizers.L2(1e-3))])
model.compile(optimizer=keras.optimizers.SGD(0.1), loss="binary_crossentropy", metrics=["accuracy"])
model.fit(X_train, y_train, epochs=100, batch_size=64, verbose=0)
print(model.evaluate(X_test, y_test.astype("float32"), verbose=0))

Interview Tips

  • Scale features; gradient descent converges much faster and regularization treats features fairly.

  • Explain why log loss is used rather than MSE: with a sigmoid, MSE is non-convex in the weights and gives weak gradients for confident mistakes.

  • For multi-class problems, replace the sigmoid with softmax and use categorical cross-entropy (multinomial logistic regression).

49

Build a simple CNN in TensorFlow for image classification.

A convolutional neural network (CNN) classifies images by learning small filters that detect local patterns (edges, textures, shapes), stacking them so deeper layers see larger and more abstract features, and finishing with a classifier. A simple CNN for small images alternates Conv2D and pooling layers, then uses global average pooling and a dense softmax output. Below is a complete Keras example on CIFAR-10 (60,000 32×32 color images in 10 classes).

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Architecture

Layer Output shape Role
Input 32×32×3 RGB image
Rescaling + augmentation 32×32×3 Scale to [0, 1]; random flips/shifts in training
Conv block 1 (2 × Conv2D 32, BatchNorm, MaxPool) 16×16×32 Edges and colors
Conv block 2 (2 × Conv2D 64, MaxPool) 8×8×64 Textures and parts
Conv block 3 (2 × Conv2D 128, MaxPool) 4×4×128 Object parts
GlobalAveragePooling2D 128 One value per feature map
Dropout + Dense(10, softmax) 10 Class probabilities

Code Example

import keras
from keras import layers

keras.utils.set_random_seed(42)
(x_train, y_train), (x_test, y_test) = keras.datasets.cifar10.load_data()   # downloads the data
class_names = ["airplane", "automobile", "bird", "cat", "deer", "dog", "frog", "horse", "ship", "truck"]


def conv_block(x, filters):
    for _ in range(2):
        x = layers.Conv2D(filters, 3, padding="same", use_bias=False)(x)
        x = layers.BatchNormalization()(x)
        x = layers.Activation("relu")(x)
    return layers.MaxPooling2D()(x)


inputs = keras.Input(shape=(32, 32, 3))
x = layers.Rescaling(1.0 / 255)(inputs)
x = layers.RandomFlip("horizontal")(x)
x = layers.RandomTranslation(0.1, 0.1)(x)
for filters in (32, 64, 128):
    x = conv_block(x, filters)
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dropout(0.3)(x)
outputs = layers.Dense(10, activation="softmax")(x)
model = keras.Model(inputs, outputs)
model.summary()

model.compile(optimizer=keras.optimizers.Adam(1e-3), loss="sparse_categorical_crossentropy",
              metrics=["accuracy"])
model.fit(x_train, y_train, validation_split=0.1, epochs=30, batch_size=128, verbose=2,
          callbacks=[keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True),
                     keras.callbacks.ReduceLROnPlateau(factor=0.5, patience=2)])

test_loss, test_acc = model.evaluate(x_test, y_test, verbose=0)
print(f"test accuracy: {test_acc:.3f}")
pred = model.predict(x_test[:1], verbose=0).argmax()
print("first test image predicted as:", class_names[pred], "| true:", class_names[y_test[0][0]])

Training on a GPU takes a few minutes; on a CPU, reduce the epochs.

Design Choices Explained

  • 3×3 kernels with padding="same" keep spatial size inside a block; pooling halves it.

  • Doubling filters as resolution halves keeps computation balanced.

  • BatchNormalization stabilizes and speeds up training.

  • Global average pooling instead of Flatten + large Dense reduces parameters and overfitting.

  • Augmentation layers inside the model are active only during training.

Common Pitfalls

  • Forgetting to scale pixel values, which slows or breaks training.

  • Using categorical_crossentropy with integer labels (use the sparse version).

  • Evaluating on the validation split used for early stopping and reporting it as test accuracy.

50

Write a TensorFlow script to normalize a dataset's features.

Feature normalization rescales input features so they have comparable ranges, which helps gradient-based training converge faster and stops large-valued features from dominating. The most common form is standardization: subtract each feature’s mean and divide by its standard deviation, giving mean 0 and standard deviation 1. In TensorFlow, the Keras Normalization layer learns these statistics with adapt() on the training data and applies them inside the model, so the same transformation is used at inference.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3, scikit-learn 1.5+ (Python 3.12)

Normalization Methods

Method Formula in words Keras / TF tool Use when
Standardization (z-score) (value - mean) / standard deviation keras.layers.Normalization Default for neural networks
Min-max scaling (value - min) / (max - min), into [0, 1] Manual with tf.reduce_min/max or Rescaling for known ranges Bounded features, pixel values
Fixed rescaling Multiply by a constant keras.layers.Rescaling(1/255) Images
Log transform then standardize log(1 + value), then z-score tf.math.log1p + Normalization Skewed, positive features (income, counts)
L2 normalization per sample Divide each row by its length keras.layers.UnitNormalization, tf.math.l2_normalize Embeddings, cosine similarity

Code Example

import keras
import numpy as np
import tensorflow as tf
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split

keras.utils.set_random_seed(42)
X, y = fetch_california_housing(return_X_y=True)             # downloads the data
X = X.astype("float32")
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# 1. Keras Normalization layer (z-score), statistics from training data only
norm = keras.layers.Normalization(axis=-1)
norm.adapt(X_train)
print("means:", norm.mean.numpy().round(2))
scaled_train = norm(X_train)
print("after: mean", tf.reduce_mean(scaled_train, axis=0).numpy().round(2),
      "std", tf.math.reduce_std(scaled_train, axis=0).numpy().round(2))

# 2. Manual min-max scaling with TensorFlow ops
x_min = tf.reduce_min(X_train, axis=0)
x_max = tf.reduce_max(X_train, axis=0)
minmax_test = (X_test - x_min) / (x_max - x_min + 1e-8)       # test can fall slightly outside [0, 1]

# 3. Normalization inside the model: raw features in, predictions out
model = keras.Sequential([
    keras.Input(shape=(X.shape[1],)),
    norm,
    keras.layers.Dense(64, activation="relu"),
    keras.layers.Dense(1),
])
model.compile(optimizer="adam", loss="mse", metrics=["mae"])
model.fit(X_train, y_train, validation_split=0.2, epochs=10, verbose=0)
print("test MAE:", round(model.evaluate(X_test, y_test, verbose=0)[1], 3))

Common Pitfalls

  • Computing statistics on the whole dataset (including test data), which leaks information.

  • Normalizing training data in a script but forgetting to apply the same statistics at serving time; putting the layer inside the model prevents this.

  • Dividing by zero for constant features; add a small epsilon or drop the feature.

  • Normalizing categorical IDs or one-hot columns as if they were continuous.

51

Create a recurrent neural network in TensorFlow for sequential data.

A recurrent neural network (RNN) processes a sequence one step at a time while carrying a hidden state that summarizes what it has seen, so the output can depend on order and earlier context. In practice you use gated variants, LSTM or GRU, because plain RNNs struggle to learn long-range dependencies due to vanishing gradients. Keras provides SimpleRNN, LSTM, GRU and the Bidirectional wrapper; the example below classifies movie reviews as positive or negative.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Layer Choices

Layer Idea Notes
SimpleRNN New state = tanh(weights × input + weights × previous state) Fast, short memory
LSTM Cell state with input, forget and output gates Long dependencies; most common
GRU Update and reset gates, no separate cell state Fewer parameters, often similar accuracy
Bidirectional(LSTM(...)) Reads the sequence forwards and backwards When the whole sequence is available (not streaming)
return_sequences=True Output at every step Needed to stack RNN layers or for per-step tagging

Code Example: Sentiment Classification

import keras
from keras import layers

keras.utils.set_random_seed(42)
VOCAB, MAXLEN = 20000, 200
(x_train, y_train), (x_test, y_test) = keras.datasets.imdb.load_data(num_words=VOCAB)   # downloads data
x_train = keras.utils.pad_sequences(x_train, maxlen=MAXLEN)     # pads/truncates at the start by default
x_test = keras.utils.pad_sequences(x_test, maxlen=MAXLEN)

model = keras.Sequential([
    keras.Input(shape=(MAXLEN,), dtype="int32"),
    layers.Embedding(VOCAB, 64, mask_zero=True),                # mask tells the RNN to skip padding
    layers.Bidirectional(layers.LSTM(64, return_sequences=True)),
    layers.Bidirectional(layers.LSTM(32)),                      # last layer returns only the final state
    layers.Dropout(0.3),
    layers.Dense(1, activation="sigmoid"),
])
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
model.fit(x_train, y_train, validation_split=0.2, epochs=5, batch_size=64, verbose=2,
          callbacks=[keras.callbacks.EarlyStopping(patience=2, restore_best_weights=True)])
print("test accuracy:", round(model.evaluate(x_test, y_test, verbose=0)[1], 3))

Sequence-to-Value on Numeric Data

For numeric sequences such as sensor readings, the input shape is (timesteps, features):

import keras
import numpy as np

keras.utils.set_random_seed(42)
X = np.random.rand(500, 30, 3).astype("float32")             # 500 sequences, 30 steps, 3 features
y = X[:, -5:, 0].mean(axis=1)                                  # target depends on recent steps
model = keras.Sequential([keras.Input(shape=(30, 3)), keras.layers.GRU(32), keras.layers.Dense(1)])
model.compile(optimizer="adam", loss="mse")
model.fit(X, y, epochs=10, verbose=0)

Common Pitfalls

  • Forgetting return_sequences=True on every RNN layer except the last when stacking.

  • Not masking padding, so the RNN processes long runs of zeros.

  • Very long sequences: training is slow and memory-heavy; consider truncation, 1D CNNs or Transformers.

  • Keras uses the fast cuDNN kernel for LSTM/GRU only with default settings (tanh activation, no recurrent dropout); changing them can make training much slower.

52

Write a Python function using TensorFlow to compute a function's gradient.

Computing a gradient in TensorFlow uses automatic differentiation with tf.GradientTape: operations run inside the tape are recorded, and tape.gradient(target, sources) applies the chain rule backwards to return the derivative of the target with respect to each source. Trainable tf.Variables are watched automatically; plain tensors need tape.watch. The same mechanism powers neural network training, but it works for any differentiable function you write with TensorFlow operations.

Read full answer

Applies toTensorFlow 2.16+ (Python 3.12)

Code Example

import tensorflow as tf


def compute_gradient(fn, x):
    """Return fn(x) and the gradient of fn at x (x can be a scalar or a tensor)."""
    x = tf.convert_to_tensor(x, dtype=tf.float32)
    with tf.GradientTape() as tape:
        tape.watch(x)                      # needed because x is a tensor, not a Variable
        y = fn(x)
    return y, tape.gradient(y, x)


# 1. Scalar function: f(x) = x^3 + 2x, so f'(x) = 3x^2 + 2
y, grad = compute_gradient(lambda x: x ** 3 + 2 * x, 2.0)
print(y.numpy(), grad.numpy())            # 12.0 14.0

# 2. Multivariate function: f(x, y) = x^2 * y + sin(y), gradient = [2xy, x^2 + cos(y)]
def f(v):
    return v[0] ** 2 * v[1] + tf.sin(v[1])

y, grad = compute_gradient(f, [1.0, 0.0])
print(grad.numpy())                       # [0. 2.]  (2*1*0 = 0, 1 + cos(0) = 2)

# 3. Second derivative with nested tapes: f''(x) = 6x for f(x) = x^3 + 2x
x = tf.Variable(2.0)
with tf.GradientTape() as outer:
    with tf.GradientTape() as inner:
        y = x ** 3 + 2 * x
    dy_dx = inner.gradient(y, x)
d2y_dx2 = outer.gradient(dy_dx, x)
print(dy_dx.numpy(), d2y_dx2.numpy())     # 14.0 12.0

# 4. Jacobian of a vector-valued function
x = tf.Variable([1.0, 2.0])
with tf.GradientTape() as tape:
    out = tf.stack([x[0] * x[1], x[0] + x[1]])
print(tape.jacobian(out, x).numpy())      # [[2. 1.] [1. 1.]]

# 5. Check against a numerical (finite-difference) estimate
eps = 1e-3
numeric = ((2.0 + eps) ** 3 + 2 * (2.0 + eps) - ((2.0 - eps) ** 3 + 2 * (2.0 - eps))) / (2 * eps)
print(round(numeric, 3))                  # close to 14.0

Useful Tape Options

Option Purpose
tape.watch(t) Track a constant tensor
GradientTape(persistent=True) Call gradient more than once (delete the tape afterwards)
watch_accessed_variables=False Track only what you explicitly watch
tape.jacobian, batch_jacobian Full Jacobian matrices
unconnected_gradients=tf.UnconnectedGradients.ZERO Return zeros instead of None for unused sources
tf.stop_gradient(t) Treat a value as a constant

Common Pitfalls

  • Getting None because the computation used NumPy or Python math instead of TensorFlow ops, or the source was not watched.

  • Integer tensors: gradients are only defined for floating-point types.

  • Calling gradient twice on a non-persistent tape.

  • Non-differentiable operations (tf.argmax, tf.round) give no useful gradient.

53

Write code to save and load a trained TensorFlow model.

Saving and loading a trained model lets you reuse it without retraining, resume training later, or deploy it. In TensorFlow with Keras 3, use model.save("name.keras") and keras.models.load_model("name.keras") to store the architecture, weights, optimizer state and compile settings in one file; custom layers must be registered so they can be rebuilt. For serving, export a SavedModel with model.export(...), and for long training jobs use checkpoints to resume.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Code Example

import keras
import numpy as np
import tensorflow as tf

keras.utils.set_random_seed(42)
X = np.random.rand(500, 6).astype("float32")
y = (X @ np.arange(1, 7, dtype="float32") + 0.1).reshape(-1, 1)


@keras.saving.register_keras_serializable(package="demo")
class Scale(keras.layers.Layer):
    """Multiplies inputs by a constant factor (a custom layer to show serialization)."""
    def __init__(self, factor=1.0, **kwargs):
        super().__init__(**kwargs)
        self.factor = factor

    def call(self, x):
        return x * self.factor

    def get_config(self):
        return {**super().get_config(), "factor": self.factor}


model = keras.Sequential([keras.Input(shape=(6,)), Scale(0.5),
                          keras.layers.Dense(32, activation="relu"), keras.layers.Dense(1)])
model.compile(optimizer="adam", loss="mse")
model.fit(X, y, epochs=20, verbose=0)

# --- Save and load the whole model ---
model.save("regressor.keras")
loaded = keras.models.load_model("regressor.keras")
assert np.allclose(model.predict(X[:5], verbose=0), loaded.predict(X[:5], verbose=0))
loaded.fit(X, y, epochs=5, verbose=0)          # optimizer state was restored, so training continues

# --- Weights only (architecture comes from code) ---
model.save_weights("regressor.weights.h5")
same_arch = keras.Sequential([keras.Input(shape=(6,)), Scale(0.5),
                              keras.layers.Dense(32, activation="relu"), keras.layers.Dense(1)])
same_arch.load_weights("regressor.weights.h5")

# --- SavedModel for TF Serving / LiteRT ---
model.export("regressor_savedmodel")
reloaded = tf.saved_model.load("regressor_savedmodel")
print(reloaded.serve(tf.constant(X[:2])).numpy())   # inference only; no Keras API

# --- Checkpoint to resume an interrupted training run ---
ckpt = tf.train.Checkpoint(model=model, optimizer=model.optimizer)
manager = tf.train.CheckpointManager(ckpt, "ckpts", max_to_keep=3)
manager.save()
ckpt.restore(manager.latest_checkpoint)
print("restored from", manager.latest_checkpoint)

Which Format to Use

Goal Format
Share or reload a Keras model, continue training .keras
Same code, just new weights .weights.h5
Production serving, other languages, LiteRT conversion SavedModel via model.export
Fault-tolerant training ModelCheckpoint, BackupAndRestore, or tf.train.CheckpointManager

Common Pitfalls

  • Custom objects not registered (or not passed via custom_objects), which makes load_model fail.

  • Saving with model.save("dir") without an extension, which Keras 3 rejects; use .keras or model.export.

  • Loading weights into a model whose layers differ from the saved one.

  • Loading .keras files from untrusted sources: keep safe_mode=True (the default), which blocks arbitrary code in Lambda layers.

54

Implement a custom training loop in TensorFlow for a basic neural network.

A custom training loop replaces model.fit with your own code: for each batch, run the forward pass under tf.GradientTape, compute the loss, get gradients, and apply them with an optimizer, while updating metrics and running validation yourself. It gives full control for things fit does not support easily, such as GANs, multiple optimizers, unusual losses or gradient manipulation. Wrapping the step in tf.function keeps it as fast as fit.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3, scikit-learn 1.5+ (Python 3.12)

Code Example

import keras
import tensorflow as tf
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

keras.utils.set_random_seed(42)
X, y = make_classification(n_samples=3000, n_features=20, n_informative=10, n_classes=3, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
scaler = StandardScaler().fit(X_train)
X_train, X_val = scaler.transform(X_train).astype("float32"), scaler.transform(X_val).astype("float32")

train_ds = tf.data.Dataset.from_tensor_slices((X_train, y_train)).shuffle(len(X_train), seed=42).batch(64)
val_ds = tf.data.Dataset.from_tensor_slices((X_val, y_val)).batch(256)

model = keras.Sequential([
    keras.Input(shape=(20,)),
    keras.layers.Dense(64, activation="relu"),
    keras.layers.Dropout(0.2),
    keras.layers.Dense(3),                               # logits
])
loss_fn = keras.losses.SparseCategoricalCrossentropy(from_logits=True)
optimizer = keras.optimizers.Adam(1e-3)
train_loss = keras.metrics.Mean()
train_acc = keras.metrics.SparseCategoricalAccuracy()
val_acc = keras.metrics.SparseCategoricalAccuracy()


@tf.function
def train_step(x, y):
    with tf.GradientTape() as tape:
        logits = model(x, training=True)                 # dropout active
        loss = loss_fn(y, logits) + sum(model.losses)    # include any regularization losses
    grads = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(grads, model.trainable_variables))
    train_loss.update_state(loss)
    train_acc.update_state(y, logits)


@tf.function
def val_step(x, y):
    val_acc.update_state(y, model(x, training=False))  # dropout off


best_acc, patience, wait = 0.0, 5, 0
for epoch in range(50):
    for metric in (train_loss, train_acc, val_acc):
        metric.reset_state()
    for x, y in train_ds:
        train_step(x, y)
    for x, y in val_ds:
        val_step(x, y)

    acc = float(val_acc.result())
    print(f"epoch {epoch + 1}: loss {float(train_loss.result()):.4f} "
          f"train acc {float(train_acc.result()):.3f} val acc {acc:.3f}")
    if acc > best_acc:                                   # simple early stopping
        best_acc, wait = acc, 0
        model.save_weights("best.weights.h5")
    else:
        wait += 1
        if wait >= patience:
            break

model.load_weights("best.weights.h5")

What fit Did for You

Responsibility Where it is now
Forward pass and loss train_step
Gradients and update GradientTape + optimizer.apply_gradients
Training vs inference behaviour training=True/False
Metrics keras.metrics objects, reset each epoch
Validation, early stopping, checkpoints Epoch loop
Speed @tf.function

Common Pitfalls

  • Forgetting training=True, so dropout and batch normalization behave as at inference.

  • Not resetting metrics each epoch, which averages over all epochs.

  • Leaving out model.losses, which silently drops L1/L2 regularization.

  • Creating the optimizer or variables inside the tf.function.

55

Write a TensorFlow program using dataset shuffle, repeat and batch.

shuffle, repeat and batch are the three core tf.data transformations for training input: shuffle(buffer_size) randomizes example order using a buffer, repeat(count) loops over the data for several epochs, and batch(size) groups consecutive examples into tensors. Their order matters: shuffling before batching shuffles individual examples, and putting repeat before or after shuffle changes whether epoch boundaries are blurred. The usual training order is shuffle → batch → prefetch, letting model.fit handle epochs.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Code Example

import keras
import numpy as np
import tensorflow as tf

ds = tf.data.Dataset.range(10)                          # 0, 1, ..., 9

# 1. shuffle -> batch: examples are mixed, then grouped
print("shuffle then batch:")
for batch in ds.shuffle(buffer_size=10, seed=42).batch(4):
    print(batch.numpy())                               # three batches: 4, 4, 2 elements

# 2. batch -> shuffle: only whole batches move; contents stay [0-3], [4-7], [8-9]
print("batch then shuffle:")
for batch in ds.batch(4).shuffle(buffer_size=3, seed=42):
    print(batch.numpy())

# 3. shuffle -> repeat: each epoch is a full pass, reshuffled every epoch
epochs = ds.shuffle(10, seed=42, reshuffle_each_iteration=True).repeat(2).batch(5)
print("shuffle then repeat (2 epochs):", [b.numpy().tolist() for b in epochs])

# 4. repeat -> shuffle: epoch boundaries blur; an element may appear twice before another appears once
blurred = ds.repeat(2).shuffle(20, seed=42).batch(5)
print("repeat then shuffle:", [b.numpy().tolist() for b in blurred])

# 5. A training pipeline used with fit
keras.utils.set_random_seed(42)
X = np.random.rand(1000, 8).astype("float32")
y = (X.sum(axis=1) > 4).astype("float32")
train = (tf.data.Dataset.from_tensor_slices((X, y))
         .shuffle(buffer_size=1000, seed=42)          # buffer = dataset size: a full shuffle
         .batch(32, drop_remainder=True)
         .prefetch(tf.data.AUTOTUNE))

model = keras.Sequential([keras.Input(shape=(8,)), keras.layers.Dense(16, activation="relu"),
                          keras.layers.Dense(1, activation="sigmoid")])
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
model.fit(train, epochs=5, verbose=0)                # fit iterates the dataset once per epoch

# With an infinite dataset (repeat()), tell fit how many steps make an epoch
model.fit(train.repeat(), epochs=2, steps_per_epoch=1000 // 32, verbose=0)

The exact shuffled orders depend on the seed and TensorFlow version, so they are not listed here.

How Each Transformation Behaves

Transformation Key arguments Notes
shuffle buffer_size, seed, reshuffle_each_iteration Fills a buffer and samples from it; a buffer smaller than the dataset gives only a partial shuffle
repeat count (None = forever) Needed for custom loops counting steps, or with steps_per_epoch
batch batch_size, drop_remainder drop_remainder=True for fixed shapes (TPUs)

Common Pitfalls

  • A small shuffle buffer on data sorted by label, so batches contain one class.

  • repeat() without steps_per_epoch in fit, which never ends an epoch.

  • Shuffling validation or test data needlessly (harmless for metrics, but it makes outputs harder to match to rows).

56

Write a TensorFlow function for image dataset augmentation.

Data augmentation creates modified versions of training images (flips, rotations, crops, brightness and contrast changes) so the model sees more variety and generalizes better, especially with small datasets. In TensorFlow you can augment with Keras preprocessing layers, which can run inside the model or in a tf.data pipeline, or with tf.image functions in a map step. Augmentation is applied only to training data, and each epoch sees different random variations.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Code Example

import keras
import tensorflow as tf

keras.utils.set_random_seed(42)
IMG_SIZE = 64

# Option 1: Keras preprocessing layers
keras_augment = keras.Sequential([
    keras.layers.RandomFlip("horizontal"),
    keras.layers.RandomRotation(0.1),           # up to ±10% of a full turn (±36 degrees)
    keras.layers.RandomZoom(0.1),
    keras.layers.RandomContrast(0.2),
], name="augment")


# Option 2: a tf.image function for use in Dataset.map
def augment_image(image, label):
    """Random crop, flip, brightness and saturation on one image in [0, 1]."""
    image = tf.image.resize_with_crop_or_pad(image, IMG_SIZE + 8, IMG_SIZE + 8)   # pad by 4 px per side
    image = tf.image.random_crop(image, size=[IMG_SIZE, IMG_SIZE, 3])
    image = tf.image.random_flip_left_right(image)
    image = tf.image.random_brightness(image, max_delta=0.1)
    image = tf.image.random_saturation(image, lower=0.8, upper=1.2)
    return tf.clip_by_value(image, 0.0, 1.0), label


# Example data: random "images" so the snippet runs anywhere
images = tf.random.uniform([100, IMG_SIZE, IMG_SIZE, 3], seed=42)
labels = tf.random.uniform([100], maxval=5, dtype=tf.int32, seed=42)

train_ds = (tf.data.Dataset.from_tensor_slices((images, labels))
            .shuffle(100, seed=42)
            .map(augment_image, num_parallel_calls=tf.data.AUTOTUNE)    # per-image tf.image ops
            .batch(32)
            .map(lambda x, y: (keras_augment(x, training=True), y),     # batched Keras layers
                 num_parallel_calls=tf.data.AUTOTUNE)
            .prefetch(tf.data.AUTOTUNE))

for x, y in train_ds.take(1):
    print(x.shape, float(tf.reduce_min(x)), float(tf.reduce_max(x)))

# Option 3: augmentation inside the model (active only during fit)
model = keras.Sequential([
    keras.Input(shape=(IMG_SIZE, IMG_SIZE, 3)),
    keras_augment,
    keras.layers.Conv2D(16, 3, activation="relu"),
    keras.layers.GlobalAveragePooling2D(),
    keras.layers.Dense(5, activation="softmax"),
])

Pipeline Versus In-Model

Placement Pros Cons
In tf.data (CPU) Runs in parallel with GPU training Must remember to skip it for validation
Inside the model (GPU) Automatically off at inference; travels with the model Uses GPU time; exported model contains inactive layers

Choosing Augmentations

  • Use transformations that keep the label valid: horizontal flips suit cats and dogs, but not digits or text, and medical images may need domain-specific choices.

  • Start mild and increase strength if the model still overfits.

  • For detection or segmentation, transform boxes or masks together with the image.

Common Pitfalls

  • Augmenting validation or test data.

  • Augmenting before cache(), which freezes one random version.

  • Colour augmentations producing values outside the expected range; clip afterwards.

57

How would you debug a TensorFlow model that isn't learning?

A model that isn’t learning shows a training loss that stays flat, decreases very slowly, oscillates or becomes NaN. Debug it systematically from the data to the model to the optimizer: confirm the data and labels are correct, check that the loss and output layer match the task, make the model overfit a tiny batch, then tune the learning rate and inspect gradients. Most failures come from data bugs, a wrong loss or activation, a bad learning rate, or preprocessing errors, not from the architecture.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Checklist

Area What to check
Data Inputs and labels aligned after shuffling; labels in the expected format; no NaNs; features scaled
Baseline Loss at initialization matches theory (e.g. log(num_classes) for balanced softmax); compare with predicting the majority class
Output/loss pairing Sigmoid + binary cross-entropy, softmax + categorical, from_logits set correctly; regression output with no activation
Overfit a tiny batch A correct model drives the loss near zero on ~32 examples; if not, there is a bug
Learning rate Too high: loss oscillates or explodes; too low: barely moves. Try 1e-4 to 1e-2 or a learning-rate range test
Gradients Zero (dead ReLUs, frozen layers, trainable=False), vanishing or exploding norms
Architecture Too small to fit, or too deep without normalization/residual connections
Training flags training=True in custom loops; BatchNorm with tiny batches

Code Example: Overfit a Small Batch and Inspect Gradients

import keras
import numpy as np
import tensorflow as tf

keras.utils.set_random_seed(42)
X = np.random.rand(32, 20).astype("float32")
y = np.random.randint(0, 5, size=32)

model = keras.Sequential([keras.Input(shape=(20,)), keras.layers.Dense(64, activation="relu"),
                          keras.layers.Dense(5)])                     # logits
loss_fn = keras.losses.SparseCategoricalCrossentropy(from_logits=True)

# 1. Initial loss should be close to log(5) = 1.609 for 5 balanced classes
print("initial loss:", float(loss_fn(y, model(X))), "expected about", round(np.log(5), 3))

# 2. Gradient norms per layer
with tf.GradientTape() as tape:
    loss = loss_fn(y, model(X, training=True))
for var, grad in zip(model.trainable_variables, tape.gradient(loss, model.trainable_variables)):
    print(f"{var.path:30s} grad norm {float(tf.norm(grad)):.4f}")

# 3. A correct model should memorize 32 random examples
model.compile(optimizer=keras.optimizers.Adam(1e-2), loss=loss_fn, metrics=["accuracy"])
hist = model.fit(X, y, epochs=200, verbose=0)
print("final loss on the tiny batch:", round(hist.history["loss"][-1], 4))

If the tiny-batch loss does not approach zero, look for a bug in labels, loss configuration or frozen layers before trying anything else.

Tools

  • TensorBoard: loss curves, weight and gradient histograms (histogram_freq=1).

  • tf.debugging.enable_check_numerics() for NaN losses.

  • keras.callbacks.TerminateOnNaN.

Common Pitfalls

  • Softmax in the last layer combined with from_logits=True.

  • Labels shuffled separately from features.

  • Unscaled inputs (e.g. raw pixel values or features in the thousands).

  • Changing many things at once, which makes it impossible to know what fixed the problem.

58

What are common TensorFlow errors, and how do you resolve them?

Common TensorFlow errors fall into a few groups: shape and dtype mismatches, problems with tf.function tracing, GPU setup and memory errors, version and API changes (notably the move to Keras 3 in TensorFlow 2.16), and numerical issues such as NaN losses. Most error messages name the operation and the offending values, so reading the first and last lines of the traceback carefully usually points to the fix.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Errors and Fixes

Error (typical message) Cause Fix
InvalidArgumentError: Incompatible shapes / Matrix size-incompatible Shape mismatch Print shapes; fix input, reshape or label format
cannot compute AddV2 as input #1 was expected to be a float tensor but is a int32 tensor Mixed dtypes tf.cast explicitly; NumPy float64 arrays should be astype("float32")
ResourceExhaustedError: OOM when allocating tensor GPU memory exhausted Smaller batch, mixed precision, memory growth, smaller model
Could not load dynamic library 'libcudnn...' or no GPU found CUDA/cuDNN mismatch Install tensorflow[and-cuda] on Linux; native Windows GPU support ended after TF 2.10 (use WSL2)
OperatorNotAllowedInGraphError: using a tf.Tensor as a Python bool Python if on a tensor inside graph code Use tensor ops / tf.cond, or let AutoGraph convert inside tf.function
tf.function retracing warnings New Python values or shapes on each call Pass tensors; use input_signature with None dimensions
ValueError: ... variable creation ... tf.function Creating variables on every call Create variables once, outside the function or in build
AttributeError: module 'tensorflow' has no attribute 'Session' / placeholder TF1 code Rewrite with eager execution and tf.function; tf.compat.v1 only as a temporary bridge
ValueError: Invalid filepath extension for saving Keras 3 model.save without .keras Use model.save("m.keras") or model.export(dir)
Unknown layer / Could not locate class on load Custom object not registered @keras.saving.register_keras_serializable() or custom_objects=
ModuleNotFoundError: tensorflow.estimator Estimator API removed in 2.16 Migrate to Keras
Loss becomes nan High learning rate, log(0), bad data Lower the learning rate, from_logits=True, check inputs, enable_check_numerics
Input 0 of layer is incompatible with the layer Wrong input rank Add batch/channel dimension; check Input(shape=...)

Code Example: Two Frequent Fixes

import numpy as np
import tensorflow as tf

a = tf.constant([1, 2, 3])                          # int32
b = tf.constant([0.5, 0.5, 0.5])                    # float32
try:
    a + b
except tf.errors.InvalidArgumentError as err:
    print("dtype error:", type(err).__name__)
print((tf.cast(a, tf.float32) + b).numpy())         # [1.5 2.5 3.5]

x64 = np.random.default_rng(42).random((4, 3))     # NumPy defaults to float64
print(tf.constant(x64).dtype, "->", tf.constant(x64, dtype=tf.float32).dtype)

@tf.function
def relu_like(x):
    if tf.reduce_sum(x) > 0:                        # fine: AutoGraph converts this to tf.cond
        return x
    return tf.zeros_like(x)

print(relu_like(tf.constant([1.0, -0.5])).numpy())

Debugging Habits

  • Pin package versions and check the TensorFlow, Keras, CUDA and Python compatibility table when upgrading.

  • Reproduce with a tiny batch in eager mode (run_eagerly=True), then switch back.

  • Search for the operation name in the error; it often identifies the exact layer.

59

How does TensorFlow's tf.debugging package help with debugging?

tf.debugging is TensorFlow’s module of runtime checks and diagnostic tools. Its assertions (assert_shapes, assert_equal, assert_non_negative, check_numerics) stop execution with a clear message the moment a tensor has the wrong shape, type or value, instead of letting a bad value spread and fail far away. It also provides global switches: enable_check_numerics finds the first operation that produces NaN or infinity, and experimental.enable_dump_debug_info records tensors for inspection in TensorBoard’s Debugger V2.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Main Tools

Tool What it checks
assert_shapes([(x, ("N", "D")), (w, ("D", "K"))]) Shapes and consistent symbolic dimensions across tensors
assert_rank, assert_type Rank and dtype
assert_equal, assert_less, assert_near Value relationships
assert_non_negative, assert_positive, assert_all_finite Value ranges
check_numerics(t, message) NaN/inf in one tensor
enable_check_numerics() NaN/inf in every operation, with the op and Python stack
experimental.enable_dump_debug_info(logdir) Dumps tensors and graphs for Debugger V2 in TensorBoard
set_log_device_placement(True) Shows which device runs each op

Related helpers: tf.config.run_functions_eagerly(True) to step through tf.function code, and tf.print for printing inside graphs.

Code Example

import tensorflow as tf

tf.random.set_seed(42)

def dense_layer(x, w, b):
    tf.debugging.assert_shapes([(x, ("N", "D")), (w, ("D", "K")), (b, ("K",))])
    tf.debugging.assert_all_finite(x, "input contains NaN or inf")
    return x @ w + b

x = tf.random.normal([8, 4])
w = tf.random.normal([4, 3])
b = tf.zeros([3])
print(dense_layer(x, w, b).shape)                  # (8, 3)

try:
    dense_layer(x, tf.random.normal([5, 3]), b)     # D mismatch: 4 vs 5
except (ValueError, tf.errors.InvalidArgumentError) as err:
    print("shape check failed:", type(err).__name__)

def safe_log_loss(p):
    tf.debugging.assert_non_negative(p, message="probabilities must be >= 0")
    return tf.debugging.check_numerics(-tf.math.log(p), "log produced inf")

try:
    safe_log_loss(tf.constant([0.5, 0.0]))          # log(0) = -inf
except tf.errors.InvalidArgumentError as err:
    print("numeric check failed:", err.message.splitlines()[0])

# Global NaN/inf detection: reports the first op that produces a bad value
tf.debugging.enable_check_numerics()
try:
    tf.sqrt(tf.constant(-1.0))                      # NaN
except tf.errors.InvalidArgumentError:
    print("enable_check_numerics caught the NaN in Sqrt")
tf.debugging.disable_check_numerics()

Workflow for NaN Losses

  1. Turn on tf.debugging.enable_check_numerics() and rerun a few steps.

  2. Read the reported op and stack trace (often Log, Div, Exp or Sqrt).

  3. Fix the cause: use from_logits=True, add epsilons, clip values, lower the learning rate, or check the input data.

  4. For deeper inspection, record with enable_dump_debug_info("logs/debug", tensor_debug_mode="FULL_HEALTH") and open the Debugger V2 tab in TensorBoard.

Common Pitfalls

  • Leaving global debugging switches on in production; they slow execution considerably.

  • Using Python assert inside a tf.function: it runs only while tracing, whereas tf.debugging.assert_* ops run on every call.

  • Checking shapes with Python if on dynamic dimensions; use assert_shapes, which handles unknown sizes.

60

How would you investigate and fix a shape mismatch error in TensorFlow?

A shape mismatch error means an operation received tensors whose dimensions do not fit its rules, for example a matrix multiply with incompatible inner sizes or a model fed inputs with the wrong number of features. The fix starts with reading the error message, which names the operation and both shapes, then tracing back to the layer or data step that produced the unexpected shape. Most cases come from a missing batch dimension, wrong label format, a flatten or reshape mistake, or data and model definitions that drifted apart.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Step-by-Step Investigation

  1. Read the full error: it states the op (MatMul, Add, ConcatV2) or layer and the incompatible shapes, e.g. “expected axis -1 of input shape to have value 20, but received input with shape (None, 18)”.

  2. Print shapes at each stage: model.summary() for layer outputs, and inspect a batch with for x, y in ds.take(1): print(x.shape, y.shape).

  3. Check the data side: number of features, image size and channels, label shape and encoding.

  4. Check the model side: Input shape, Flatten/Reshape, return_sequences, output units.

  5. Add assertions with tf.debugging.assert_shapes in custom code, and run eagerly (run_eagerly=True in compile) for clearer tracebacks.

  6. Fix at the source rather than reshaping blindly to make the error disappear.

Common Causes and Fixes

Symptom Cause Fix
Expected (None, 20), got (None, 18) Feature columns changed after preprocessing Rebuild features consistently; keep preprocessing inside the model
Expected 4 dims, got 3 (images) Missing batch or channel dimension x[None, ...] for one sample; x[..., None] for grayscale
Loss error with labels (32, 1) vs logits (32, 10) One-hot vs integer label mismatch sparse_categorical_crossentropy for integers, categorical_crossentropy for one-hot
Logits (32, 50, 10) vs labels (32,) RNN with return_sequences=True before the classifier Set return_sequences=False on the last RNN layer or add pooling
MatMul incompatible inner dims Wrong transpose or weight shape Check orders; use tf.einsum with named axes
Concatenation shape error Branch outputs differ on non-concat axes Align sizes with pooling or Resizing
Shape changes per batch in tf.function Variable lengths padded_batch, None in input_signature, or drop_remainder=True

Code Example

import keras
import numpy as np
import tensorflow as tf

keras.utils.set_random_seed(42)
model = keras.Sequential([keras.Input(shape=(28, 28, 1)), keras.layers.Conv2D(8, 3, activation="relu"),
                          keras.layers.Flatten(), keras.layers.Dense(10, activation="softmax")])

image = np.random.rand(28, 28).astype("float32")          # a single grayscale image without batch/channel
try:
    model.predict(image, verbose=0)
except (ValueError, tf.errors.InvalidArgumentError) as err:
    print("error:", str(err).splitlines()[0][:100])

fixed = image[None, ..., None]                              # add batch and channel dimensions
print("input shape:", fixed.shape)                          # (1, 28, 28, 1)
print("output shape:", model.predict(fixed, verbose=0).shape)   # (1, 10)

# Inspect shapes layer by layer
for layer in model.layers:
    print(f"{layer.name:12s} -> {layer.output.shape}")

# Catch problems early in custom code
labels = tf.constant([3, 1, 4])
tf.debugging.assert_shapes([(labels, ("N",))])

Interview Tips

  • Mention None as the batch dimension in Keras shapes and why it is flexible.

  • Distinguish static shapes (x.shape, known when tracing) from dynamic shapes (tf.shape(x), known at runtime).

  • Emphasize fixing the root cause and adding a shape assertion so the error cannot silently come back.

61

How can TensorBoard be used to debug TensorFlow programs?

TensorBoard is TensorFlow’s visualization tool. For debugging it shows training and validation curves to spot overfitting, divergence or a stalled loss; histograms of weights and gradients to find dead or exploding layers; the model graph; images and text to check what the model actually receives; the Profiler to find performance bottlenecks; and Debugger V2 to trace where NaN or infinite values first appear. Logging is a one-line Keras callback, and custom values can be written with tf.summary.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3, TensorBoard 2.16+ (Python 3.12)

Dashboards and What They Reveal

Dashboard Use for debugging
Scalars Loss/metric curves: overfitting (gap), divergence (spikes), learning-rate effects
Histograms / Distributions Weights not changing, dead ReLUs, exploding activations or gradients
Graphs Model structure, unexpected ops, tf.function graphs
Images Verify inputs and augmentations look right; view predictions
Text Check tokenization or generated outputs
HParams Compare runs across hyperparameter settings
Profiler Input-bound steps, slow ops, GPU idle time, memory use
Debugger V2 Find the first op producing NaN/inf, with its stack trace
Projector Inspect embeddings

Code Example

import datetime

import keras
import numpy as np
import tensorflow as tf

keras.utils.set_random_seed(42)
(x_train, y_train), _ = keras.datasets.mnist.load_data()          # downloads the data
x_train = x_train[:5000, ..., None].astype("float32") / 255
y_train = y_train[:5000]
log_dir = "logs/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")

model = keras.Sequential([keras.Input(shape=(28, 28, 1)), keras.layers.Conv2D(16, 3, activation="relu"),
                          keras.layers.Flatten(), keras.layers.Dense(10, activation="softmax")])
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"])

# Log a few input images to confirm preprocessing
writer = tf.summary.create_file_writer(log_dir + "/inputs")
with writer.as_default():
    tf.summary.image("training samples", x_train[:9], max_outputs=9, step=0)

tensorboard_cb = keras.callbacks.TensorBoard(
    log_dir=log_dir,
    histogram_freq=1,          # weight histograms every epoch
    profile_batch=(10, 20),    # profile batches 10-20
)
model.fit(x_train, y_train, validation_split=0.2, epochs=3, verbose=0, callbacks=[tensorboard_cb])

# Custom scalar from your own code
with tf.summary.create_file_writer(log_dir + "/custom").as_default():
    for step in range(3):
        tf.summary.scalar("custom/learning_rate", 1e-3 * 0.5 ** step, step=step)

# Optional: record tensors for Debugger V2 before training
# tf.debugging.experimental.enable_dump_debug_info(log_dir + "/debug", tensor_debug_mode="FULL_HEALTH")

Start it with tensorboard --logdir logs and open http://localhost:6006, or use %load_ext tensorboard and %tensorboard --logdir logs in a notebook.

Reading the Signals

  • Validation loss rising while training loss falls: overfitting.

  • Loss flat from the start: data, loss or learning-rate bug.

  • Histograms frozen across epochs: layer not training (frozen or zero gradients).

  • Profiler shows a high “input” share: fix the tf.data pipeline.

Common Pitfalls

  • Writing all runs into the same folder, which mixes curves; use one subfolder per run.

  • Heavy histogram or image logging every batch, which slows training.

  • Leaving Debugger V2 dumping on for long runs; it uses a lot of disk.

62

How does GradientTape work in TensorFlow?

tf.GradientTape records the operations executed inside its context so TensorFlow can compute gradients with reverse-mode automatic differentiation. During the forward pass, the tape stores each operation and the intermediate values it needs; when you call tape.gradient(target, sources), it walks the recorded operations backwards, applying the chain rule to get the derivative of the target with respect to each source. It is the engine behind Keras training and custom training loops.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

How It Works

  1. Entering with tf.GradientTape() as tape: starts recording.

  2. Trainable tf.Variables accessed inside are watched automatically; constant tensors need tape.watch(t).

  3. Each op is recorded with references to its inputs and outputs.

  4. tape.gradient(loss, vars) starts with d(loss)/d(loss) = 1 and propagates backwards: each op’s registered gradient function turns the gradient of its output into gradients of its inputs.

  5. By default the tape releases its resources after one gradient call.

Reverse mode is efficient when there are many inputs (weights) and one scalar output (loss), which is exactly the neural network case.

Code Example

import tensorflow as tf

tf.random.set_seed(42)

# Basic: d/dw of (w * x - y)^2
w = tf.Variable(2.0)
x, y = tf.constant(3.0), tf.constant(5.0)
with tf.GradientTape() as tape:
    loss = (w * x - y) ** 2                 # (6 - 5)^2 = 1
print(tape.gradient(loss, w).numpy())       # 2 * (w*x - y) * x = 2 * 1 * 3 = 6.0

# Persistent tape: several gradients from one recording
a = tf.Variable(1.0)
b = tf.Variable(2.0)
with tf.GradientTape(persistent=True) as tape:
    u = a * b
    v = u + b ** 2
print(tape.gradient(u, a).numpy(), tape.gradient(v, b).numpy())   # 2.0 and a + 2b = 5.0
del tape                                                          # free resources

# Stopping gradients and unwatched values
c = tf.Variable(3.0)
with tf.GradientTape() as tape:
    z = c * tf.stop_gradient(c)             # second factor treated as a constant
print(tape.gradient(z, c).numpy())          # 3.0 instead of 6.0

# Gradient of a model's loss with respect to its weights
dense = tf.keras.layers.Dense(1)
inputs = tf.random.normal([4, 3])
with tf.GradientTape() as tape:
    out = dense(inputs)
    loss = tf.reduce_mean(out ** 2)
grads = tape.gradient(loss, dense.trainable_variables)
print([g.shape for g in grads])             # [TensorShape([3, 1]), TensorShape([1])]

# Gradient with respect to the input (saliency maps, adversarial examples)
with tf.GradientTape() as tape:
    tape.watch(inputs)
    score = tf.reduce_sum(dense(inputs))
print(tape.gradient(score, inputs).shape)   # (4, 3)

Key Behaviours

Behaviour Detail
Returns None Source not used, not watched, or computation left TensorFlow (NumPy, .numpy(), Python math)
Integer tensors Not differentiable; cast to float
Higher-order derivatives Nest tapes
Control flow Only the branch actually executed is recorded
Custom gradients @tf.custom_gradient defines your own backward function

Common Pitfalls

  • Calling gradient twice on a non-persistent tape.

  • Computing the loss outside the with block, so nothing is recorded.

  • Converting to NumPy in the middle of the computation, which breaks the chain.

63

How would you implement attention mechanisms in TensorFlow?

An attention mechanism lets a model decide, for each output position, how much to focus on each input position: it scores how relevant every input (key) is to the current query, turns the scores into weights with softmax, and returns the weighted average of the inputs’ values. In TensorFlow you can use the built-in Keras layers (MultiHeadAttention, Attention, AdditiveAttention) or write scaled dot-product attention yourself in a few lines, which is a common interview exercise.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Scaled Dot-Product Attention in Words

  1. Compare each query with each key by a dot product.

  2. Divide by the square root of the key size so large dimensions don’t push softmax into extreme values.

  3. Optionally mask positions (padding, or future tokens in a decoder) by setting their scores to a large negative number.

  4. Apply softmax over the keys to get weights that sum to 1.

  5. Multiply the weights by the values to get the output.

Keras Layers

Layer Type Typical use
keras.layers.Attention Dot-product (Luong-style) Seq2seq decoder attending to encoder outputs
keras.layers.AdditiveAttention Additive (Bahdanau-style) Same, with a small learned scoring network
keras.layers.MultiHeadAttention Several scaled dot-product heads in parallel Transformers, self- and cross-attention

Code Example

import keras
import tensorflow as tf

tf.random.set_seed(42)


def scaled_dot_product_attention(q, k, v, mask=None):
    """q: (..., Tq, d), k: (..., Tk, d), v: (..., Tk, dv); mask: 1 = keep, 0 = block."""
    scores = tf.matmul(q, k, transpose_b=True)                 # (..., Tq, Tk)
    scores /= tf.math.sqrt(tf.cast(tf.shape(k)[-1], tf.float32))
    if mask is not None:
        scores += (1.0 - tf.cast(mask, tf.float32)) * -1e9
    weights = tf.nn.softmax(scores, axis=-1)
    return tf.matmul(weights, v), weights


batch, seq_len, dim = 2, 5, 8
x = tf.random.normal([batch, seq_len, dim])

# Causal mask: position i may only attend to positions <= i
causal = tf.linalg.band_part(tf.ones([seq_len, seq_len]), -1, 0)
out, w = scaled_dot_product_attention(x, x, x, mask=causal)
print(out.shape, w.shape)                     # (2, 5, 8) (2, 5, 5)
print(w[0, 0].numpy().round(3))               # first position attends only to itself: [1. 0. 0. 0. 0.]

# Built-in multi-head self-attention with a causal mask
mha = keras.layers.MultiHeadAttention(num_heads=2, key_dim=4)
y, scores = mha(x, x, use_causal_mask=True, return_attention_scores=True)
print(y.shape, scores.shape)                  # (2, 5, 8) (2, 2, 5, 5): one weight matrix per head

# Attention pooling for classification: learn which time steps matter
inputs = keras.Input(shape=(20, 16))
h = keras.layers.LSTM(32, return_sequences=True)(inputs)
query = keras.layers.GlobalAveragePooling1D(keepdims=True)(h)      # (batch, 1, 32)
context = keras.layers.Attention()([query, h])                      # (batch, 1, 32)
outputs = keras.layers.Dense(1, activation="sigmoid")(keras.layers.Flatten()(context))
model = keras.Model(inputs, outputs)
model.summary()

Self-Attention Versus Cross-Attention

  • Self-attention: queries, keys and values come from the same sequence (encoder layers).

  • Cross-attention: queries come from the decoder, keys and values from the encoder output (translation, captioning).

  • Multi-head: several attention heads with different learned projections capture different relationships, then their outputs are concatenated.

Common Pitfalls

  • Forgetting the scaling factor, which makes softmax too peaked and gradients small.

  • Missing padding masks, so the model attends to padding tokens.

  • Memory grows with the square of the sequence length; long inputs need truncation, chunking or efficient attention variants.

64

What is the difference between graph mode and eager mode in TensorFlow?

Eager mode runs each TensorFlow operation immediately from Python and returns real values, while graph mode first builds a dataflow graph of the whole computation and then executes it as a unit. TensorFlow 2 is eager by default; you get graph mode by decorating a function with tf.function, which traces the Python code once per input signature into a graph that can be optimized, run without the Python interpreter, and exported. The usual pattern is to develop eagerly and run training and serving in graph mode.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12)

Comparison

Aspect Eager mode Graph mode (tf.function)
Execution Op by op, from Python Whole graph after tracing
Values Concrete (.numpy() works) Symbolic during tracing
Python code Runs every call Runs only while tracing
Control flow Python if/for AutoGraph converts tensor-dependent if/while to graph ops
Optimization None across ops Grappler rewrites, optional XLA fusion
Portability Needs Python Saved as SavedModel; runs in TF Serving, LiteRT, C++
Debugging Easy Harder; tf.print, run_functions_eagerly(True)
Speed Python overhead per op Faster for many small ops; similar for a few large ops

Tracing and Polymorphism

A tf.function creates one concrete function (graph) per distinct input signature: tensor shapes and dtypes, plus the values of Python arguments. Calling it with a new shape or a new Python value triggers a retrace. Fix the signature to control this.

Code Example

import tensorflow as tf

@tf.function
def square_sum(x):
    print("Tracing with", x.shape)          # Python: runs only while tracing
    return tf.reduce_sum(x ** 2)

square_sum(tf.constant([1.0, 2.0]))         # traces for shape (2,)
square_sum(tf.constant([3.0, 4.0]))         # reuses the graph: no print
square_sum(tf.constant([1.0, 2.0, 3.0]))    # new shape (3,) -> traces again

@tf.function(input_signature=[tf.TensorSpec(shape=[None], dtype=tf.float32)])
def square_sum_any(x):
    print("Tracing once")
    return tf.reduce_sum(x ** 2)

for n in (2, 3, 10):
    square_sum_any(tf.range(n, dtype=tf.float32))    # one trace for any length

# Inspect the graph behind a concrete function
concrete = square_sum_any.get_concrete_function()
print([op.type for op in concrete.graph.get_operations()][:6])

# Keras uses graph mode in fit by default; switch to eager for debugging
# model.compile(..., run_eagerly=True)

When to Use Which

  • Eager: exploring data, writing and debugging model code, small one-off computations.

  • Graph: training loops, inference servers, export to SavedModel, mobile conversion, anything performance-critical.

Common Pitfalls

  • Expecting Python side effects (prints, list appends, counters) to happen on every call of a tf.function.

  • Retracing on every step because a Python scalar such as the step number is passed instead of a tensor.

  • Creating tf.Variables inside a tf.function on each call.

65

How is TensorFlow used in natural language processing?

TensorFlow is used in NLP to build and deploy models for text classification, sentiment analysis, named-entity recognition, translation, summarization, question answering and text generation. The toolkit covers the whole pipeline: the TextVectorization layer and TensorFlow Text for tokenization, Keras layers (Embedding, LSTM, MultiHeadAttention) for models, KerasHub for pretrained Transformers such as BERT and Gemma, and TF Serving or LiteRT for production. Today most NLP tasks start by fine-tuning a pretrained Transformer rather than training from scratch.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3, KerasHub 0.18+ (Python 3.12)

Tasks and Approaches

Task Model approach Tools
Text classification / sentiment TF-IDF + dense baseline; fine-tuned BERT-style encoder TextVectorization, keras_hub.models.BertClassifier
Named-entity recognition Token classification with an encoder Encoder backbone + per-token Dense
Translation, summarization Encoder-decoder (seq2seq) Transformer keras_hub.layers.TransformerEncoder/Decoder
Text generation, chat Decoder-only LLM keras_hub.models.GemmaCausalLM
Semantic search Sentence embeddings + nearest neighbours Encoder backbones, ScaNN
On-device text tasks Small quantized models LiteRT, MediaPipe Text tasks

Code Example 1: Fast Baseline

import keras
import numpy as np

keras.utils.set_random_seed(42)
texts = np.array(["loved it, brilliant acting", "boring and far too long", "a great watch",
                  "terrible plot, awful ending", "wonderful soundtrack", "worst film this year"])
labels = np.array([1, 0, 1, 0, 1, 0])

vectorizer = keras.layers.TextVectorization(max_tokens=20000, output_mode="tf_idf", ngrams=2)
vectorizer.adapt(texts)                           # adapt on training texts only

model = keras.Sequential([keras.Input(shape=(1,), dtype="string"), vectorizer,
                          keras.layers.Dense(16, activation="relu"), keras.layers.Dropout(0.3),
                          keras.layers.Dense(1, activation="sigmoid")])
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
model.fit(texts[:, None], labels, epochs=20, verbose=0)
print(model.predict(np.array([["a brilliant, great film"]]), verbose=0))

Code Example 2: Fine-Tuning a Pretrained Transformer with KerasHub

import keras
import keras_hub

keras.utils.set_random_seed(42)
texts = ["loved it, brilliant acting", "boring and far too long", "a great watch",
         "terrible plot, awful ending"]
labels = [1, 0, 1, 0]

# The preset bundles the tokenizer and preprocessing, so raw strings go straight in
classifier = keras_hub.models.BertClassifier.from_preset("bert_tiny_en_uncased", num_classes=2)
classifier.compile(optimizer=keras.optimizers.AdamW(5e-5),
                   loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
                   metrics=["accuracy"])
classifier.fit(x=texts, y=labels, batch_size=2, epochs=3, verbose=0)
print(classifier.predict(["what a brilliant film"], verbose=0))   # logits for the two classes

With real data, evaluate on a held-out test set using accuracy and F1, and compare against the TF-IDF baseline; a Transformer must beat it to justify its cost.

Common Pitfalls

  • Fitting the vocabulary or tokenizer on test data.

  • Training large models from scratch on small datasets.

  • Truncating long documents without checking where the useful text is.

  • Using a pretrained model with a different tokenizer from the one it was trained with.

66

Which recent TensorFlow features and changes are gaining traction?

The biggest recent changes in the TensorFlow ecosystem are Keras 3 becoming the default high-level API (multi-backend, from TensorFlow 2.16), the renaming of TensorFlow Lite to LiteRT for on-device inference, and KerasHub as the home for pretrained models, including large language models. Alongside these, XLA compilation through OpenXLA, simpler GPU installation with tensorflow[and-cuda], and NumPy 2 support have matured. TensorFlow releases now focus on stability and deployment, while much new Google research uses JAX, which Keras 3 can also target.

Read full answer

Applies toTensorFlow 2.16-2.18+ ecosystem (Python 3.12); check release notes for the latest versions

Notable Changes

Change What it means
Keras 3 as default (TF 2.16) One Keras API running on TensorFlow, JAX or PyTorch; .keras save format; model.export for SavedModel; legacy code via tf_keras
KerasHub (formerly KerasNLP and KerasCV) Pretrained presets such as Gemma, Llama, Mistral, BERT and vision backbones; LoRA fine-tuning with enable_lora
LiteRT (renamed TF Lite) On-device runtime for TensorFlow, JAX and PyTorch models; standalone ai-edge-litert package; GPU/NPU acceleration
AI Edge Torch / conversion tools Convert PyTorch models to LiteRT for mobile
OpenXLA Shared compiler stack; jit_compile=True speedups on GPUs and TPUs
tensorflow[and-cuda] pip installs matching CUDA libraries on Linux; native Windows GPU support ended after TF 2.10 (use WSL2)
NumPy 2.x support (TF 2.18) Compatibility with the modern scientific Python stack
Estimator API removed (TF 2.16) Everything moves to Keras and tf.function
YDF (Yggdrasil Decision Forests) Successor library to TensorFlow Decision Forests for tree models
Keras distribution API Data and model parallelism (most complete on the JAX backend)

Code Example: Using Several of These Together

import os
os.environ["KERAS_BACKEND"] = "tensorflow"         # or "jax" / "torch" with the same Keras code

import keras
import numpy as np
import tensorflow as tf

keras.utils.set_random_seed(42)
model = keras.Sequential([keras.Input(shape=(16,)), keras.layers.Dense(32, activation="relu"),
                          keras.layers.Dense(1, activation="sigmoid")])
model.compile(optimizer="adam", loss="binary_crossentropy", jit_compile=True)    # XLA
model.fit(np.random.rand(256, 16), np.random.randint(0, 2, 256), epochs=2, verbose=0)

model.save("model.keras")                          # Keras 3 native format
model.export("saved_model_dir")                    # SavedModel for serving

converter = tf.lite.TFLiteConverter.from_saved_model("saved_model_dir")
converter.optimizations = [tf.lite.Optimize.DEFAULT]
open("model.tflite", "wb").write(converter.convert())     # runs on the LiteRT runtime

# Pretrained LLM from KerasHub (needs keras-hub, model access on Kaggle, and a GPU)
# import keras_hub
# gemma = keras_hub.models.GemmaCausalLM.from_preset("gemma2_instruct_2b_en")
# gemma.backbone.enable_lora(rank=4)              # parameter-efficient fine-tuning

Interview Tips

  • Show you know the Keras 3 migration issues: model.save needs .keras, tf.keras maps to Keras 3, and tf_keras keeps Keras 2 code running.

  • Mention the deployment story (SavedModel → TF Serving; LiteRT on device) as TensorFlow’s main strength today.

  • Be honest about trends: PyTorch dominates research, JAX is popular at Google, and Keras 3 lets teams move between them.

68

What steps would you take to develop a chatbot using TensorFlow?

Developing a chatbot with TensorFlow starts with defining what it must do, then choosing an architecture: a retrieval or intent-based bot that classifies the user’s message and returns or triggers a predefined response, or a generative bot built on a pretrained large language model, usually grounded in your documents with retrieval-augmented generation (RAG). TensorFlow and KerasHub cover intent classifiers, embedding models and LLM fine-tuning, and TF Serving or LiteRT handles deployment; the rest is data, evaluation and safety work.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3, KerasHub (Python 3.12)

Steps

Step Details
1. Scope Users, tasks (FAQ, bookings, support), channels, languages, what must be escalated to a human
2. Data Conversation logs, FAQs, documentation; label intents and entities; remove personal data
3. Architecture Intent + entity models with dialogue rules; retrieval of answers; or LLM + RAG
4. Models Intent classifier (TextVectorization or a fine-tuned encoder); embeddings for retrieval; LLM such as Gemma via KerasHub
5. Dialogue management Track state (slots filled, previous turns), fallbacks for low-confidence predictions
6. Evaluation Intent accuracy/F1, retrieval recall, answer correctness reviewed by people, task completion rate
7. Safety Input/output filtering, refusal of out-of-scope requests, prompt-injection defences, rate limits
8. Deployment TF Serving or a Python API; latency budget; logging and feedback loop for retraining

Code Example: Intent-Based Bot

import keras
import numpy as np

keras.utils.set_random_seed(42)
training = {
    "greeting": ["hi", "hello there", "good morning", "hey"],
    "hours": ["when are you open", "what are your opening hours", "are you open on sunday"],
    "refund": ["i want my money back", "how do i get a refund", "return an item"],
}
responses = {
    "greeting": "Hello! How can I help you today?",
    "hours": "We are open 9am to 6pm, Monday to Saturday.",
    "refund": "You can request a refund within 30 days from your Orders page.",
}
intents = list(training)
texts = np.array([t for intent in intents for t in training[intent]])
labels = np.array([intents.index(intent) for intent in intents for _ in training[intent]])

vectorizer = keras.layers.TextVectorization(output_mode="tf_idf", ngrams=2)
vectorizer.adapt(texts)
model = keras.Sequential([keras.Input(shape=(1,), dtype="string"), vectorizer,
                          keras.layers.Dense(16, activation="relu"),
                          keras.layers.Dense(len(intents), activation="softmax")])
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy")
model.fit(texts[:, None], labels, epochs=150, verbose=0)


def reply(message, threshold=0.6):
    probs = model.predict(np.array([[message.lower()]]), verbose=0)[0]
    if probs.max() < threshold:                     # low confidence: don't guess
        return "Sorry, I didn't understand. Let me connect you to a person."
    return responses[intents[int(probs.argmax())]]


print(reply("hello"))
print(reply("what time do you open"))

A real bot needs far more examples per intent, a held-out evaluation set and an “out of scope” class.

Generative Option

Load a pretrained instruction-tuned LLM, for example keras_hub.models.GemmaCausalLM.from_preset("gemma2_instruct_2b_en"), fine-tune it with LoRA on your support conversations if needed, and add RAG: embed your documents, retrieve the most relevant passages for each question, and put them in the prompt so answers are grounded and up to date.

Common Pitfalls

  • Letting a generative bot answer without grounding, which produces confident wrong answers.

  • No fallback or human hand-off.

  • Logging conversations without consent or redaction.

69

How would you use TensorFlow in an autonomous vehicle perception system?

An autonomous vehicle perception system turns raw sensor data from cameras, LiDAR and radar into a structured view of the world: detected and tracked objects (cars, pedestrians, cyclists), drivable area and lanes, traffic signs and lights, and distances. TensorFlow would be used to train the deep learning models for these tasks, mainly detection, segmentation and depth estimation, and to export them as optimized graphs for in-vehicle hardware. The hard parts are real-time latency, robustness in rare conditions and safety validation, not just model accuracy.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 (Python 3.12); in-vehicle inference typically via TensorRT or vendor runtimes

Perception Tasks

Task Input Model type
2D object detection Camera images Single-stage detectors (YOLO-style, EfficientDet, RetinaNet)
3D object detection LiDAR point clouds, camera + LiDAR PointPillars-style, BEV (bird’s-eye-view) fusion networks
Semantic / panoptic segmentation Camera Encoder-decoder (U-Net, DeepLab)
Depth estimation Stereo or mono camera Depth networks, or from LiDAR
Lane and drivable area Camera Segmentation or lane-curve regression
Tracking Detections over time Kalman filters + learned association
Sensor fusion All sensors Early (raw data) or late (per-sensor detections) fusion

System Design

  1. Data: large, diverse driving datasets (e.g. Waymo Open Dataset, nuScenes) plus your own fleet data; careful labelling; mining of rare cases (night, rain, unusual objects).

  2. Training: multi-task models sharing a backbone to save compute; tf.data pipelines over TFRecords; multi-GPU training with tf.distribute.

  3. Optimization: mixed precision, pruning, int8 quantization; export a SavedModel and convert with TF-TRT or TensorRT for the vehicle’s GPU.

  4. Runtime: fixed latency budget per frame (tens of milliseconds), redundancy across sensors, timestamps synchronized.

  5. Validation: metrics per class, distance and condition; simulation and closed-course tests; safety standards such as ISO 26262 and ISO 21448 (SOTIF).

Code Example: Lightweight Drivable-Area Segmentation Model

import time

import keras
import numpy as np
from keras import layers

keras.utils.set_random_seed(42)
H, W, NUM_CLASSES = 128, 256, 3           # classes: background, road, lane marking


def conv_bn(x, filters, stride=1):
    x = layers.SeparableConv2D(filters, 3, strides=stride, padding="same", use_bias=False)(x)
    x = layers.BatchNormalization()(x)
    return layers.ReLU()(x)


inputs = keras.Input(shape=(H, W, 3))
x = layers.Rescaling(1.0 / 255)(inputs)
e1 = conv_bn(x, 16, stride=2)              # 1/2 resolution
e2 = conv_bn(e1, 32, stride=2)             # 1/4
e3 = conv_bn(e2, 64, stride=2)             # 1/8
d2 = layers.Concatenate()([layers.UpSampling2D()(e3), e2])
d2 = conv_bn(d2, 32)
d1 = layers.Concatenate()([layers.UpSampling2D()(d2), e1])
d1 = conv_bn(d1, 16)
logits = layers.Conv2D(NUM_CLASSES, 1)(layers.UpSampling2D()(d1))
model = keras.Model(inputs, logits)
model.compile(optimizer="adam", loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=[keras.metrics.MeanIoU(num_classes=NUM_CLASSES, sparse_y_pred=False)])

frame = np.random.randint(0, 255, (1, H, W, 3)).astype("float32")
model.predict(frame, verbose=0)            # warm-up
start = time.perf_counter()
for _ in range(20):
    model.predict_on_batch(frame)
print(f"{(time.perf_counter() - start) / 20 * 1000:.1f} ms per frame on this machine")

Measure latency on the target hardware after conversion; desktop numbers are only indicative.

Common Pitfalls

  • Optimizing average accuracy while missing rare but critical cases (children, unusual vehicles, glare).

  • Ignoring calibration of confidence scores, which downstream planning relies on.

  • Validating only on data from the same cities and weather as training.

70

What approach would you use for real-time object detection with TensorFlow?

Real-time object detection means locating and classifying objects in each video frame fast enough to keep up with the stream, typically 15-30+ frames per second. The practical approach with TensorFlow is to use a single-stage detector designed for speed (SSD MobileNet, EfficientDet-Lite, YOLO-style models), run it with an optimized runtime (LiteRT on CPUs, phones and edge devices, or TensorRT on NVIDIA GPUs), and fine-tune it on your own classes. Speed comes from model choice, input resolution, quantization and an efficient capture-to-inference pipeline.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3, LiteRT, MediaPipe Tasks (Python 3.12)

Choosing a Detector

Model family Speed Accuracy Where it runs well
SSD MobileNet (V2/V3) Very fast Moderate Phones, Raspberry Pi, Edge TPU
EfficientDet-Lite0-4 Fast to moderate Good, tunable by size LiteRT on mobile and edge
YOLO-style single-stage models Fast Good GPUs; also exportable to LiteRT
Two-stage (Faster R-CNN) Slow High Offline analysis, not real time

Pipeline

  1. Capture frames (OpenCV, camera API) in a separate thread so capture never waits for inference.

  2. Resize to the model’s input size (e.g. 320×320); convert BGR to RGB.

  3. Run inference with a quantized LiteRT model or a TensorRT engine.

  4. Post-process: score threshold and non-maximum suppression (often built into the model).

  5. Optionally track objects across frames to smooth boxes and skip detection on some frames.

Code Example: Webcam Detection with a LiteRT Model via MediaPipe

MediaPipe’s Object Detector runs LiteRT (.tflite) detection models such as EfficientDet-Lite0 and handles pre- and post-processing.

import time

import cv2
import mediapipe as mp
from mediapipe.tasks import python as mp_python
from mediapipe.tasks.python import vision

options = vision.ObjectDetectorOptions(
    base_options=mp_python.BaseOptions(model_asset_path="efficientdet_lite0.tflite"),   # downloaded model
    running_mode=vision.RunningMode.VIDEO,
    max_results=5,
    score_threshold=0.5,
)
detector = vision.ObjectDetector.create_from_options(options)

cap = cv2.VideoCapture(0)
start = time.perf_counter()
frames = 0
while cap.isOpened():
    ok, frame = cap.read()
    if not ok:
        break
    rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    timestamp_ms = int((time.perf_counter() - start) * 1000)
    result = detector.detect_for_video(mp.Image(image_format=mp.ImageFormat.SRGB, data=rgb), timestamp_ms)

    for det in result.detections:
        box, cat = det.bounding_box, det.categories[0]
        cv2.rectangle(frame, (box.origin_x, box.origin_y),
                      (box.origin_x + box.width, box.origin_y + box.height), (0, 255, 0), 2)
        cv2.putText(frame, f"{cat.category_name} {cat.score:.2f}", (box.origin_x, box.origin_y - 5),
                    cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 1)

    frames += 1
    fps = frames / (time.perf_counter() - start)
    cv2.putText(frame, f"FPS: {fps:.1f}", (10, 20), cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 0, 255), 2)
    cv2.imshow("detections", frame)
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

cap.release()
cv2.destroyAllWindows()

Custom Classes

Fine-tune a pretrained detector on your own labelled images (bounding boxes in COCO or Pascal VOC format) with MediaPipe Model Maker or the TensorFlow Model Garden, export to LiteRT with int8 quantization, and evaluate with mean average precision (mAP) on a held-out set before measuring FPS on the target device.

Common Pitfalls

  • Measuring speed on a desktop GPU and deploying to a much slower device.

  • Too high an input resolution for small gains in accuracy, or too low for small, distant objects.

  • Capture, drawing or display code being the real bottleneck rather than the model.

  • Choosing the score threshold without checking the precision/recall trade-off on validation data.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 70 TensorFlow answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Machine Learning & Data Science subjects