Skip to content
Machine Learning & Data Science

Scala interview questions and answers

70 questions, 70 with code. About 5 h 6 min to read them all.

By TestLab · Updated

Plan for an interview
70 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What is Scala, and why does it matter for machine learning?

Scala (“scalable language”) is a statically typed programming language that combines object-oriented and functional programming and runs on the Java Virtual Machine; it was created by Martin Odersky at EPFL, first released in 2004, with Scala 3 released in 2021. It matters for machine learning mainly through data engineering at scale: Apache Spark, the most widely used engine for distributed data processing and MLlib, is written in Scala, and Scala is common in the systems around ML (Kafka-based streaming, feature pipelines, model-serving services on the JVM). Scala lets teams write type-safe, concise and fast feature pipelines, custom Spark transformations and production services, and it interoperates with Java ML libraries. Model research and deep-learning training are usually done in Python; Scala’s strength is turning data into features and running models reliably in production.

Read full answer

Applies toScala 3 (general); Scala 2.13 for Spark 3.5 / 4.x

Key Facts

Aspect Detail
Paradigm Object-oriented + functional
Runtime JVM (also Scala.js and Scala Native)
Typing Static, with type inference
Current versions Scala 3.x; Scala 2.13 still common for Spark
Build tools sbt, Mill, scala-cli, Maven
Notable software Apache Spark, Apache Kafka (core), Akka/Pekko, Play, Flink (older Scala API)

Where Scala Fits in an ML System

Stage Scala role
Ingestion / streaming Kafka consumers, Spark Structured Streaming, Akka/Pekko Streams
Data cleaning and features Spark DataFrames/Datasets, JVM UDFs
Training at scale Spark MLlib, XGBoost4J-Spark, LightGBM on Spark
Serving Akka HTTP / http4s / Play services with ONNX Runtime or DJL
Research / deep learning Usually Python

Code Example (Scala 3): Typed Feature Building

//> using scala 3.5

final case class Txn(user: String, amount: Double)
final case class UserFeatures(user: String, count: Int, avgAmount: Double)

def features(txns: Seq[Txn]): List[UserFeatures] =
  txns.groupBy(_.user).toList.sortBy(_._1).map { (user, ts) =>
    UserFeatures(user, ts.size, ts.map(_.amount).sum / ts.size)
  }

@main def scalaForMl(): Unit =
  val txns = List(Txn("a", 10), Txn("a", 30), Txn("b", 5))
  features(txns).foreach(println)
  // UserFeatures(a,2,20.0)
  // UserFeatures(b,1,5.0)

The same logic on a cluster is a typed Spark job:

// Spark, Scala 2.13
val feats = txns.as[Txn].groupByKey(_.user).mapGroups { (user, it) =>
  val amounts = it.map(_.amount).toVector
  UserFeatures(user, amounts.size, amounts.sum / amounts.size)
}

Why Teams Choose Scala for ML Infrastructure

  • Spark-native: full APIs, no Python–JVM serialization for UDFs, typed Datasets.

  • Reliability: compile-time checks catch schema and logic errors before long cluster jobs.

  • Performance: JVM JIT compilation; good for high-throughput serving and streaming.

  • Concise code: functional collections and case classes keep pipelines short.

Common Pitfalls

  • Using Scala where the ML team works entirely in Python; the language boundary then costs more than it saves.

  • Mismatched Scala and Spark versions (_2.12 vs _2.13).

  • Ignoring the learning curve of advanced features (implicits, type-level code) for new team members.

Interview Tips

  • Define Scala in one sentence (OOP + FP on the JVM, statically typed).

  • Tie its importance to Spark and production data pipelines.

  • Be balanced: Python for modelling, Scala for scale and production.

2

What is the difference between var and val in Scala?

val declares an immutable reference: it is assigned once and can never be reassigned, while var declares a mutable reference that can be reassigned any number of times. Note that val controls the name, not the object: val buf = ArrayBuffer(1, 2) cannot point to a different buffer, but the buffer’s contents can still change. Idiomatic Scala uses val by default (safer, easier to reason about, safe to capture in closures and share between threads) and var only for local, short-lived state such as loop counters or accumulators in performance-critical code. Related forms are lazy val (computed once, on first access) and def (re-evaluated on every call). Both val and var have the same runtime cost.

Read full answer

Applies toScala 3 and Scala 2.13

Comparison

val var lazy val def
Reassignable No Yes No Not a variable
Evaluated Once, at definition At each assignment Once, on first use On every call
Thread-safe to share Yes (if the object is immutable) No Yes Depends
Typical use Default Loop counters, local accumulators Expensive setup Computed values

Code Example (Scala 3)

//> using scala 3.5
import scala.collection.mutable.ArrayBuffer

class Counter(val name: String, var count: Int)   // val field: read-only; var field: getter and setter

@main def valVarDemo(): Unit =
  val rate = 0.01
  // rate = 0.02                     // does not compile: Reassignment to val rate

  var epoch = 0
  while epoch < 3 do epoch += 1
  println(epoch)                     // 3

  // val reference to a mutable object: contents can change, reference cannot
  val buf = ArrayBuffer(1, 2)
  buf += 3
  println(buf)                       // ArrayBuffer(1, 2, 3)
  // buf = ArrayBuffer()             // does not compile

  // var reference to an immutable object: reassign to a new value
  var xs = List(1, 2)
  xs = xs :+ 3
  println(xs)                        // List(1, 2, 3)

  // Prefer expressions over var accumulators
  val total = (1 to 3).sum
  println(total)                     // 6

  // val and var class parameters
  val c = Counter("clicks", 0)
  c.count += 5
  println(s"${c.name} = ${c.count}") // clicks = 5
  // c.name = "views"                // does not compile: name is a val

  // lazy val vs def
  lazy val loaded = { println("loading once"); 42 }
  def computed = { println("computing"); 42 }
  println(loaded + loaded)
  // loading once
  // 84
  println(computed + computed)
  // computing
  // computing
  // 84

The Four Combinations

Declaration Reassign? Change contents?
val xs = List(1, 2) No No
var xs = List(1, 2) Yes No
val buf = ArrayBuffer(1, 2) No Yes
var buf = ArrayBuffer(1, 2) Yes Yes (avoid)

Why Prefer val

  • The value a name refers to never changes, so code is easier to read and debug.

  • Closures that capture a val behave predictably; a captured var is shared and can change.

  • Safe to share across threads (when the object is immutable).

  • In Spark, updating a driver-side var inside a closure does nothing useful: each task updates its own copy.

Common Pitfalls

  • Assuming val makes the object immutable.

  • Using var with mutable collections, giving two kinds of mutation to track.

  • Reassigning a var captured by a closure or a Future, creating race conditions.

  • Using var fields in case classes, which breaks equality and hashing over time.

Interview Tips

  • Say “val = immutable reference, var = mutable reference” and then add the object-vs-reference distinction.

  • Show the four combinations table.

  • Mention lazy val and def to show you know the evaluation differences.

3

Which Scala features make it suitable for data science and ML?

Scala suits data and machine-learning work mainly because of five features: it runs on the JVM (fast, mature runtime and access to Java libraries), it is the native language of Apache Spark (and of Kafka’s core and parts of the big-data stack), it has a strong static type system with type inference (concise code with compile-time checks of data schemas via case classes), it is functional (immutable collections and higher-order functions such as map, filter and groupMapReduce that match how data pipelines are written and parallelized), and it has good concurrency tools (Futures, actors, parallel collections). Pattern matching and case classes model records and events cleanly, and libraries such as Spark MLlib, Breeze, Smile, Spark NLP and XGBoost4J cover the ML side. Its main limitation is the smaller ecosystem for deep learning and research compared with Python.

Read full answer

Applies toScala 3; Scala 2.13 for Spark

Features and Why They Matter

Feature Why it helps in data/ML work
JVM runtime JIT-compiled speed, mature GC, Java libraries (JDBC, Kafka clients, Lucene)
Spark’s native language Full API, typed Dataset[T], JVM UDFs with no Python serialization
Static typing + inference Schema and logic errors at compile time with little boilerplate
Case classes + pattern matching Concise, immutable records and exhaustive handling of variants
Functional collections map/filter/fold/groupBy mirror distributed transformations
Immutability Safe parallelism and reproducible pipelines
Concurrency Future, Akka/Pekko, parallel collections for services and streaming
Expressive DSLs Readable pipeline and query code (Spark, Quill, Breeze operators)
Notebooks Almond (Jupyter), Databricks and Zeppelin notebooks

Code Example (Scala 3)

//> using scala 3.5

final case class Sale(region: String, amount: Double)

@main def dataFeatures(): Unit =
  val sales = List(Sale("north", 120), Sale("south", 80), Sale("north", 60), Sale("east", 40))

  // Group and aggregate in one step
  val byRegion = sales.groupMapReduce(_.region)(_.amount)(_ + _)
  println(byRegion.toList.sortBy(-_._2))     // List((north,180.0), (south,80.0), (east,40.0))

  // Typed records and safe statistics
  val top = sales.maxBy(_.amount)
  println(top)                               // Sale(north,120.0)
  val mean = sales.map(_.amount).sum / sales.size
  println(mean)                              // 75.0

  // Pattern matching for feature buckets
  val buckets = sales.map {
    case Sale(_, a) if a >= 100 => "high"
    case Sale(_, a) if a >= 50  => "mid"
    case _                      => "low"
  }
  println(buckets)                           // List(high, mid, mid, low)

The same groupBy → aggregate shape becomes df.groupBy("region").agg(sum("amount")) in Spark when the data no longer fits on one machine.

Scala vs Python for Data Work

Aspect Scala Python
Big-data processing Native Spark, strong typing PySpark (same engine)
Custom UDF speed in Spark JVM speed Slower unless pandas UDFs
Classic ML libraries Spark MLlib, Smile scikit-learn (broader)
Deep learning DJL, ONNX Runtime for serving PyTorch, JAX, TensorFlow (dominant)
Exploration and plotting Fewer tools pandas, Jupyter, matplotlib
Production services Strong (JVM, typed) Good, more ops variation

Common Pitfalls

  • Choosing Scala for research or deep-learning-heavy work where the team and libraries are in Python.

  • Scala/Spark binary-version mismatches (_2.12 vs _2.13 artifacts).

  • Writing Scala like Java (mutable loops, nulls) and losing the benefits.

  • Slow compile times on large projects with heavy implicits or macros.

Interview Tips

  • Lead with JVM + Spark + type safety + functional style.

  • Back each point with a concrete benefit (compile-time schemas, JVM UDFs, safe parallelism).

  • Acknowledge Python’s lead in modelling, and say where you would use each.

4

Can you describe Scala's type hierarchy?

Scala has a single unified type hierarchy with Any at the top: every value, including numbers and booleans, is an object of some type under Any. Any has two main branches: AnyVal for value types (Int, Long, Double, Float, Short, Byte, Char, Boolean, Unit), which compile to JVM primitives where possible, and AnyRef (an alias of java.lang.Object) for all reference types (String, List, Java classes and every user-defined class). At the bottom are Null, the type of null, a subtype of all reference types (but not of value types), and Nothing, a subtype of every type, with no values, used for expressions that never return normally (throw, ???) and for empty collections (List[Nothing], None). Scala 3 adds Matchable between Any and both branches to mark types that are safe to pattern match on.

Read full answer

Applies toScala 3 (notes for Scala 2.13)

The Hierarchy

Type Position Examples / notes
Any Top Defines ==, !=, equals, hashCode, toString
Matchable (Scala 3) Below Any Parent of AnyVal and AnyRef; supports pattern matching
AnyVal Value types Int, Long, Double, Float, Short, Byte, Char, Boolean, Unit, value classes
AnyRef Reference types java.lang.Object: String, List, case classes, Java classes
Null Below all AnyRef types Only value: null (Scala 3 “explicit nulls” option removes this)
Nothing Bottom of everything No values; type of throw, ???

Code Example (Scala 3)

//> using scala 3.5

def kind(x: Any): String = x match
  case null       => "Null"
  case _: Int     => "Int (AnyVal)"
  case _: Double  => "Double (AnyVal)"
  case _: Boolean => "Boolean (AnyVal)"
  case _: Unit    => "Unit (AnyVal)"
  case _: String  => "String (AnyRef)"
  case _: List[?] => "List (AnyRef)"
  case _          => "other"

def fail(msg: String): Nothing = throw IllegalStateException(msg)

def safeRatio(a: Double, b: Double): Double =
  if b != 0 then a / b else fail("division by zero")   // Nothing fits where Double is expected

@main def hierarchyDemo(): Unit =
  val values: List[Any] = List(1, "two", 3.0, true, (), null, List(1))
  values.map(kind).foreach(println)
  // Int (AnyVal)
  // String (AnyRef)
  // Double (AnyVal)
  // Boolean (AnyVal)
  // Unit (AnyVal)
  // Null
  // List (AnyRef)

  println(safeRatio(6, 3))          // 2.0

  val empty = List()                // List[Nothing]: fits any List[A]
  val names: List[String] = empty
  println(names)                    // List()

  val s: String = null              // allowed: Null <: String
  // val i: Int = null              // does not compile: Null is not a subtype of Int
  println(Option(s))                // None

  println(List(1, 2.5))             // List(1.0, 2.5)   numeric literals harmonized to Double

Value Types vs Reference Types

AnyVal AnyRef
Runtime JVM primitives (int, double) when possible; boxed in generic code Heap objects
null allowed No Yes (unless explicit nulls are on)
Equality Value equality == calls equals; eq compares references
User-defined Value classes (extends AnyVal), opaque types Any class

Numeric Widening Is Not Subtyping

Int is not a subtype of Long or Double; Scala converts Byte → Short → Int → Long → Float → Double (and Char → Int) automatically where needed. That is why val d: Double = 3 compiles, but a List[Int] is not a List[Double].

Why It Matters in Practice

  • Nothing lets throw and ??? appear anywhere and lets Nil/None work for any element type thanks to covariance.

  • Generic code over Any boxes AnyVal values (performance cost in numeric code).

  • Spark rows with nullable columns must map to Option[Int], not Int, because Int cannot hold null.

Common Pitfalls

  • Using Any as a parameter type to accept “anything”, losing type safety; use generics or type classes.

  • Confusing Nothing (no values), Null (only null), Unit (one value, ()) and None (empty Option).

  • Expecting null to work for Int fields.

  • Getting List[Any] from mixing types by accident, then needing casts.

Interview Tips

  • Draw it top-down: Any → (Matchable) → AnyVal / AnyRef → Null → Nothing.

  • Explain Nothing with throw and Nil.

  • Mention that value types are JVM primitives but boxed in generic code.

5

How does Scala interoperate with Java?

Scala compiles to the same JVM bytecode as Java, so the two interoperate directly: Scala code can create Java objects, call Java methods, implement Java interfaces and extend Java classes with no wrappers, and Java code can call most Scala classes and methods. The friction points are the type systems around them: Java collections need conversion with scala.jdk.CollectionConverters (asScala/asJava), Java’s nulls should be wrapped with Option(...), Scala lambdas convert to Java functional interfaces (SAM conversion), checked exceptions do not exist in Scala (add @throws for Java callers), and Scala-specific features need help to look natural in Java (@BeanProperty getters/setters, @varargs, static forwarders for object methods). This is why Scala can use the whole Java ecosystem (JDBC, Kafka clients, Hadoop, Lucene, Jackson) and why Spark offers a Java API.

Read full answer

Applies toScala 3 and Scala 2.13 on JDK 17 / 21

Interop Cheatsheet

Need Scala solution
Use a Java class import java.time.LocalDate; call it normally
Java List/Map → Scala jlist.asScala, jmap.asScala (import scala.jdk.CollectionConverters.*)
Scala collection → Java xs.asJava
Nullable Java result Option(javaCall())
Java functional interface Pass a Scala lambda (SAM conversion)
Java Optional scala.jdk.OptionConverters.* (.toScala, .toJava)
Java streams / futures scala.jdk.StreamConverters, scala.jdk.FutureConverters
Getters/setters for Java frameworks @BeanProperty var x
Java varargs call into Scala @varargs def f(xs: Double*)
Checked exceptions for Java callers @throws[IOException]

Code Example (Scala 3)

//> using scala 3.5
import java.time.LocalDate
import java.util.{ArrayList, Collections}
import java.util.function.{Function => JFunction}
import scala.jdk.CollectionConverters.*
import scala.beans.BeanProperty
import scala.annotation.varargs

// Scala code designed to be called from Java
class ModelConfig:
  @BeanProperty var learningRate: Double = 0.01     // Java: getLearningRate() / setLearningRate(..)

object Stats:
  @varargs def mean(xs: Double*): Double = xs.sum / xs.size   // Java: Stats.mean(1.0, 2.0)

@main def interopDemo(): Unit =
  // Using Java classes directly
  println(LocalDate.of(2026, 11, 11).plusDays(30))  // 2026-12-11

  // Java collections <-> Scala collections
  val jlist = ArrayList[String]()
  jlist.add("b")
  jlist.add("a")
  println(Collections.max(jlist))                   // b
  println(jlist.asScala.toList.sorted)              // List(a, b)

  val jmap: java.util.Map[String, Integer] = Map("x" -> Integer.valueOf(1)).asJava
  println(jmap.get("x"))                            // 1
  println(Option(jmap.get("missing")))              // None   (Java returned null)

  // Scala lambda as a Java functional interface
  val twice: JFunction[Integer, Integer] = x => x * 2
  println(twice.apply(21))                          // 42

  val cfg = ModelConfig()
  cfg.setLearningRate(0.1)
  println(cfg.getLearningRate())                    // 0.1
  println(Stats.mean(1.0, 2.0, 6.0))                // 3.0

Calling Scala from Java

// Java
ModelConfig cfg = new ModelConfig();
cfg.setLearningRate(0.05);
double m = Stats.mean(1.0, 2.0);            // static forwarder generated for the object method
scala.collection.immutable.List<Integer> xs = ...;   // Scala types are usable but awkward in Java

Expose Java-friendly APIs (Java collections, Optional, no implicit parameters, no default arguments) at the boundary if Java teams will call your Scala code.

Mixed Projects

sbt and Maven (with the scala-maven-plugin) compile Java and Scala sources in the same module; Scala traits with concrete methods compile to Java interfaces with default methods (since Scala 2.12), so Java classes can implement them.

Common Pitfalls

  • Forgetting that Java methods can return null; wrap with Option(...) at the boundary.

  • Using the deprecated JavaConverters/JavaConversions (implicit, surprising); use scala.jdk.CollectionConverters.

  • Exposing implicit/using parameters, default arguments or Scala collections in an API meant for Java.

  • Boxing surprises: Java Integer can be null, while Scala Int cannot.

  • Version mismatch between the JDK target and libraries compiled for a newer JDK.

Interview Tips

  • Start with “same bytecode, direct calls both ways”.

  • Mention collection converters, Option(null) and SAM conversion as the everyday tools.

  • Give the Spark example: written in Scala, with Java and Python APIs on top.

6

What are case classes, and how do they help pattern matching?

A case class is a class declared with case class for which the compiler generates the boilerplate of an immutable data type: constructor parameters become public vals, and you get apply (create without new), unapply (an extractor for pattern matching), structural equals/hashCode, a readable toString, copy for modified versions, and Product methods. For pattern matching, the generated extractor lets you deconstruct values by their fields (case Purchase(user, item, amount) =>), match nested structures, add guards (if amount >= 100), and bind the whole value with @. Combined with a sealed parent trait, the compiler knows every possible case and warns when a match is not exhaustive, which is how Scala models events, messages, ASTs and records safely.

Read full answer

Applies toScala 3 and Scala 2.13

What the Compiler Generates

Feature Example
apply Click("asha", "book") without new
unapply case Click(u, i) =>
Public immutable fields click.user
Structural equality Click("a", "b") == Click("a", "b") is true
toString Click(a,b)
copy click.copy(item = "pen")
Product productArity, productElementNames

Code Example (Scala 3)

//> using scala 3.5

sealed trait Event
final case class Click(user: String, item: String) extends Event
final case class Purchase(user: String, item: String, amount: Double) extends Event
final case class Refund(orderId: Long, amount: Double) extends Event

def describe(e: Event): String = e match
  case Click(u, i)                       => s"$u viewed $i"
  case Purchase(u, _, amt) if amt >= 100 => s"$u made a large purchase"
  case p @ Purchase(u, i, _)             => s"$u bought $i (${p.amount})"
  case Refund(id, amt)                   => f"refund #$id: $amt%.2f"
  // removing any case gives a "match may not be exhaustive" warning

@main def caseClassDemo(): Unit =
  val events = List(
    Click("asha", "book"),
    Purchase("ben", "laptop", 120.0),
    Purchase("chen", "pen", 3.5),
    Refund(7L, 3.5)
  )
  events.map(describe).foreach(println)
  // asha viewed book
  // ben made a large purchase
  // chen bought pen (3.5)
  // refund #7: 3.50

  val c = Click("asha", "book")
  println(c == Click("asha", "book"))         // true    structural equality
  println(c.copy(item = "pen"))               // Click(asha,pen)
  println(Set(c, Click("asha", "book")).size) // 1       usable as set elements and map keys
  println(c.productElementNames.toList)       // List(user, item)

  // Nested patterns
  val pair: (Event, Event) = (Click("ben", "laptop"), Purchase("ben", "laptop", 120.0))
  pair match
    case (Click(u1, i1), Purchase(u2, i2, _)) if u1 == u2 && i1 == i2 => println("converted click")
    case _                                                            => println("other")
  // converted click

Case Classes in Spark (Scala 2.13)

case class Purchase(user: String, item: String, amount: Double)

val ds = df.as[Purchase]                              // typed Dataset from a DataFrame
val big = ds.filter(_.amount >= 100).map(_.user)      // field names checked at compile time

Spark derives the schema and the encoder from the case class fields. Sealed hierarchies of case classes are not directly supported as Dataset row types; flatten them into one case class with a kind field for Spark.

Case Class vs Regular Class

case class class
Equality By field values By reference
Pattern matching Built-in extractor Needs a custom unapply
Fields Public vals Private unless declared val
Intended use Immutable data Behaviour and state

Common Pitfalls

  • Mutable fields (var, Array) in case classes: equality and hash codes change or compare arrays by reference.

  • Extending one case class from another (prohibited); use a sealed trait parent.

  • Forgetting sealed, which loses exhaustiveness checking.

  • Case classes with very many fields becoming slow to compile and awkward to pattern match; match with case p: Purchase => and use field access instead.

Interview Tips

  • List the generated members: apply, unapply, equals/hashCode, toString, copy.

  • Show a sealed trait with case classes and an exhaustive match with a guard.

  • Mention Spark Dataset[CaseClass] as a practical use.

7

What is an object in Scala, and how is it used?

In Scala, the object keyword defines a singleton: a class with exactly one instance, created lazily the first time it is accessed. Objects replace Java’s static members and are used for utility functions and constants, companion objects (same name as a class, holding factories, extractors and given instances), program entry points (object Main { def main(...) }, or @main methods in Scala 3), case objects and enum cases in algebraic data types, singleton implementations of traits (a stateless Euclidean extends Distance), and per-JVM holders for expensive resources. An object is a real value: it has a type (Defaults.type), can extend classes and traits, and can be passed around.

Read full answer

Applies toScala 3 and Scala 2.13

Uses of object

Use Example
Utilities and constants object Defaults { val Seed = 42 }
Companion object object User { def apply(...) } next to class User
Entry point object Main { def main(args: Array[String]): Unit = ... }
ADT values case object Train extends Split
Trait implementation object Euclidean extends Distance
Per-JVM cache object ModelHolder { lazy val model = load() }

Code Example (Scala 3)

//> using scala 3.5

object Defaults:
  val Seed = 42
  val TestFraction = 0.2
  def split(n: Int): (Int, Int) =
    val test = (n * TestFraction).round.toInt
    (n - test, test)

trait Distance:
  def apply(a: Vector[Double], b: Vector[Double]): Double

object Manhattan extends Distance:                  // stateless singleton implementation
  def apply(a: Vector[Double], b: Vector[Double]): Double =
    a.lazyZip(b).map((x, y) => math.abs(x - y)).sum

sealed trait Split
case object Train extends Split
case object Test extends Split

object Heavy:
  println("initializing Heavy")                     // runs once, on first access
  val squares: Map[Int, Int] = (1 to 3).map(i => i -> i * i).toMap

@main def objectDemo(): Unit =
  println(Defaults.split(1000))                     // (800,200)
  println(Manhattan(Vector(1.0, 2.0), Vector(4.0, 0.0)))   // 5.0
  val splits: List[Split] = List(Train, Test, Train)
  println(splits.count(_ == Train))                 // 2

  println("start")
  println(Heavy.squares(3))
  println(Heavy.squares(2))
  // start
  // initializing Heavy
  // 9
  // 4

Objects in Spark Jobs

An object is initialized once per JVM, so on a cluster each executor creates its own instance on first use. This is a common pattern for expensive, non-serializable resources:

object ModelHolder {
  lazy val model: Model = Model.load("/models/churn")   // loaded once per executor, not shipped
}

df.as[Features].map(f => ModelHolder.model.score(f))

Mutable state in such an object is per executor, not shared across the cluster, so do not use it to collect results.

object vs class vs case object

class object case object
Instances Many (new) Exactly one Exactly one
Constructor parameters Yes No No
toString ClassName@hash Name$@hash Name
Pattern matching Via type Via identity Via identity; serializable

Common Pitfalls

  • Global mutable state in objects (shared across threads and tests).

  • Heavy work in an object’s initializer; failures appear as ExceptionInInitializerError and later as NoClassDefFoundError.

  • Expecting a Spark driver object’s state to be visible on executors.

  • Using an object where dependency injection (a parameter) would make testing easier.

Interview Tips

  • Define it as a lazily created singleton that replaces static.

  • List the main uses: utilities, companions, entry points, case objects.

  • Mention per-executor initialization in Spark; it shows practical experience.

8

How do you implement traits in Scala?

A trait is declared with trait Name and can contain abstract members (to be implemented), concrete methods and fields (inherited as-is), and type members; a class implements it with extends (and further traits with with or, in Scala 3, commas), providing every abstract member. Traits are Scala’s main tool for interfaces and reusable behaviour: a class can mix in many traits, traits can extend other traits, Scala 3 traits can take constructor parameters, sealed traits define closed hierarchies for pattern matching, generic traits such as Encoder[A] are the basis of type classes, and objects can implement traits as singletons. Implementing a trait is often as small as one method, with default behaviour inherited from concrete members.

Read full answer

Applies toScala 3 (Scala 2.13 uses `with` instead of commas and has no trait parameters)

What a Trait Can Contain

Member Example Implementing class must
Abstract method def scores(x: Vector[Double]): Vector[Double] Define it
Abstract value def classes: Seq[String] (implementable by a val) Define it
Concrete method def predict(x) = ... Nothing (may override)
Type member type Input Define it
Parameters (Scala 3) trait Named(val name: String) Pass arguments

Code Example (Scala 3)

//> using scala 3.5

// 1. An interface with a default method
trait Classifier:
  def classes: Seq[String]
  def scores(x: Vector[Double]): Vector[Double]
  def predict(x: Vector[Double]): String =
    classes(scores(x).zipWithIndex.maxBy(_._1)._2)   // index of the highest score

// 2. A trait with a parameter (Scala 3)
trait Named(val name: String)

// 3. Implementation: extend the traits and provide the abstract members
final class LinearClassifier(weights: Vector[Vector[Double]], val classes: Seq[String])
    extends Classifier, Named("linear"):
  def scores(x: Vector[Double]): Vector[Double] =
    weights.map(w => w.lazyZip(x).map(_ * _).sum)

// 4. A sealed trait as an ADT
sealed trait Activation
case object Relu extends Activation
final case class Leaky(alpha: Double) extends Activation

def activate(a: Activation, x: Double): Double = a match
  case Relu     => math.max(0.0, x)
  case Leaky(k) => if x >= 0 then x else k * x

// 5. A generic trait used as a type class
trait Describe[A]:
  def describe(a: A): String

given Describe[LinearClassifier] with
  def describe(c: LinearClassifier): String = s"${c.name} model over ${c.classes.mkString("/")}"

@main def traitsDemo(): Unit =
  val clf = LinearClassifier(Vector(Vector(1.0, 0.0), Vector(0.0, 1.0)), Seq("cat", "dog"))
  println(clf.predict(Vector(0.2, 0.9)))     // dog
  println(clf.predict(Vector(2.0, 1.0)))     // cat
  println(clf.name)                          // linear
  println(summon[Describe[LinearClassifier]].describe(clf))   // linear model over cat/dog

  println(activate(Relu, -2.0))              // 0.0
  println(activate(Leaky(0.1), -2.0))        // -0.2

For x = (0.2, 0.9), the scores are 0.2 for “cat” and 0.9 for “dog”, so predict returns “dog”.

Scala 2.13 Syntax

trait Classifier {
  def classes: Seq[String]
  def scores(x: Vector[Double]): Vector[Double]
  def predict(x: Vector[Double]): String = classes(scores(x).zipWithIndex.maxBy(_._1)._2)
}

class LinearClassifier(weights: Vector[Vector[Double]], val classes: Seq[String])
    extends Classifier with Serializable {
  def scores(x: Vector[Double]): Vector[Double] = weights.map(w => w.lazyZip(x).map(_ * _).sum)
}

Trait vs Abstract Class

Trait Abstract class
Multiple per class Yes One
Constructor parameters Scala 3 yes; Scala 2 no Yes
Java interop Interface with default methods Abstract class
Stackable abstract override Yes No

Common Pitfalls

  • Declaring abstract members as val in the trait and using them in the trait’s initializer: they are not yet set during construction. Declare them as def and implement with val.

  • Forgetting override when replacing a concrete trait method.

  • Mixing in two traits with the same concrete method without resolving the conflict (the compiler requires an override).

  • Putting mutable state in widely mixed-in traits.

Interview Tips

  • Show a trait with one abstract and one concrete method, and a class implementing it.

  • Mention Scala 3 trait parameters and sealed traits for ADTs.

  • Connect generic traits to type classes.

9

Why is immutability important, and how does Scala support it?

Immutability means a value cannot change after it is created; to “modify” it you create a new value. It matters because immutable data can be shared between threads without locks, makes code easier to reason about (a value you received cannot be changed behind your back), makes functions easier to test and reuse, works safely as map keys and cache entries, and supports reproducible pipelines: Spark’s whole model (immutable RDDs and DataFrames rebuilt from lineage) depends on it. Scala supports immutability as the default: val for references, immutable collections imported by default (List, Vector, Map, Set), case classes with immutable fields and a copy method, structural sharing so new versions reuse old data, and language features such as final, private constructors, sealed ADTs and opaque types for controlled construction.

Read full answer

Applies toScala 3 and Scala 2.13

Why It Matters

Benefit Example
Thread safety Share a config Map across Futures with no locks
Local reasoning A Vector passed to a function is the same after the call
Safe caching and hashing Case class keys never change their hash
Easier testing Inputs and outputs only; no hidden state
Fault tolerance Spark recomputes lost partitions from immutable inputs
Undo / history Keep old versions (e.g. model configs) without copying everything

Scala Support

Feature Effect
val Reference cannot be reassigned
Immutable collections by default List, Vector, Map, Set cannot be changed in place
Case classes Fields are vals; copy creates modified versions
Structural sharing 0 :: xs reuses xs; Vector.updated copies only a path
final, sealed, private constructors Control who can extend or create values
Opaque types / value classes Type-safe wrappers without mutable state

Code Example (Scala 3)

//> using scala 3.5
import scala.concurrent.{Await, Future}
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration.*

final case class Hyper(lr: Double, epochs: Int, layers: Vector[Int])

@main def immutabilityDemo(): Unit =
  val base  = Hyper(0.01, 10, Vector(64, 32))
  val tuned = base.copy(epochs = 20, layers = base.layers :+ 16)
  println(base)                       // Hyper(0.01,10,Vector(64, 32))
  println(tuned)                      // Hyper(0.01,20,Vector(64, 32, 16))

  val v  = Vector(1, 2, 3)
  val v2 = v.updated(0, 10)
  println(v)                          // Vector(1, 2, 3)
  println(v2)                         // Vector(10, 2, 3)

  // Immutable data shared by many concurrent tasks: no locks needed
  val weights = Map("a" -> 0.5, "b" -> 2.0)
  val tasks = List("a", "b", "a").map(k => Future(weights(k) * 10))
  println(Await.result(Future.sequence(tasks), 2.seconds))   // List(5.0, 20.0, 5.0)

  // Contrast: an Array inside a val is still mutable
  val arr = Array(1, 2, 3)
  arr(0) = 99
  println(arr.mkString(","))          // 99,2,3

Immutability and Performance

Immutability does not mean copying everything: List prepend shares the tail, and Vector/Map updates copy only O(log n) nodes. In measured hot spots, use local mutable builders (List.newBuilder, ArrayBuffer) inside a function and return an immutable result.

Common Pitfalls

  • Thinking val makes the object immutable: val arr = Array(...) and val buf = ArrayBuffer(...) can still change.

  • Case classes holding mutable fields (Array, mutable.Map), which breaks equality and thread safety.

  • Exposing internal mutable collections from a class.

  • Rebuilding large immutable structures in tight loops when a local builder would be cheaper.

Interview Tips

  • Give two concrete benefits (thread safety, reasoning) and one data example (Spark lineage).

  • Explain structural sharing to answer “isn’t it slow?”.

  • Distinguish immutable references (val) from immutable objects.

10

What is the difference between Seq and List in Scala?

Seq is a trait (an interface) for any ordered collection with indexed positions, while List is one concrete implementation of it: an immutable singly linked list. Seq says what you can do (ordered, can be indexed, may contain duplicates); List decides how it is stored, with O(1) access to the head and prepend (::) but O(n) indexing, length and append. Calling Seq(1, 2, 3) gives you a List by default, and other Seq implementations include Vector (fast indexed access, the default IndexedSeq), ArraySeq, Range and LazyList. Since Scala 2.13 the default Seq is scala.collection.immutable.Seq. A common guideline: accept Seq in parameters for flexibility, and choose a concrete type (List for recursion and prepending, Vector for random access) when you build data.

Read full answer

Applies toScala 3 and Scala 2.13 (note on 2.12 below)

Hierarchy

Type Kind Notes
Seq Trait Ordered, indexed positions
LinearSeq Trait Efficient head/tail (e.g. List, LazyList)
IndexedSeq Trait Efficient apply(i) and length (e.g. Vector, ArraySeq, Range)
List Class Immutable linked list (:: and Nil)
Vector Class Immutable tree with effectively constant-time access and update

Performance

Operation List Vector
head, tail, prepend x :: xs O(1) ~O(1)
xs(i), length O(n) ~O(1)
Append xs :+ x O(n) ~O(1)
updated(i, x) O(n) ~O(1)

Code Example (Scala 3)

//> using scala 3.5

def firstTwo(xs: Seq[Int]): String = xs match
  case a +: b +: _ => s"$a,$b"          // works for every Seq
  case _           => "short"

def sumList(xs: List[Int]): Int = xs match
  case Nil    => 0
  case h :: t => h + sumList(t)         // :: patterns only for List

@main def seqVsList(): Unit =
  val s: Seq[Int] = Seq(1, 2, 3)
  println(s)                            // List(1, 2, 3)   the default Seq is a List
  println(s.isInstanceOf[List[?]])      // true
  println(IndexedSeq(1, 2, 3))          // Vector(1, 2, 3)

  println(firstTwo(Vector(7, 8, 9)))    // 7,8
  println(firstTwo(List(1)))            // short

  val xs = List(1, 2, 3)
  println(0 :: xs)                      // List(0, 1, 2, 3)   cheap prepend
  println(xs :+ 4)                      // List(1, 2, 3, 4)   copies the list
  println(sumList(xs))                  // 6
  // sumList(Vector(1, 2))              // does not compile: Vector is a Seq, not a List

Which to Use

Situation Choice
Function parameter that only reads elements Seq[A] (or Iterable[A])
Recursive processing with pattern matching List
Frequent index access, append or update Vector
Wrapping an array without copying ArraySeq
Spark toDF / toDS input Any Seq

Scala 2.12 Note

In 2.12, the default Seq was scala.collection.Seq, which also allowed mutable implementations such as ArrayBuffer. Code moving to 2.13 or Scala 3 may need .toSeq calls where a mutable buffer was passed as a Seq.

Common Pitfalls

  • Indexing a List in a loop (xs(i)), which makes the loop O(n²).

  • Appending to a List repeatedly (xs :+ x); prepend and reverse, or use Vector or a builder.

  • Declaring parameters as List when any Seq would do, forcing callers to convert.

  • Using case h :: t on a parameter typed Seq; use +: patterns instead.

Interview Tips

  • Say “Seq is the interface, List is a linked-list implementation”.

  • Quote the complexity difference for indexing and prepending.

  • Mention that Seq(...) builds a List and IndexedSeq(...) builds a Vector.

11

What are the advantages of using Option in Scala?

Option[A] represents a value that may be absent: it is either Some(value) or None, which replaces null with a type the compiler checks. Its advantages: absence is visible in the signature (def find(id: Long): Option[User] tells the caller to handle “not found”), it prevents NullPointerExceptions, it composes with map, flatMap, filter, getOrElse, fold and for-comprehensions so you write the happy path without nested null checks, it works with pattern matching, and it converts cleanly to and from other types (toRight for Either, toList, Option(javaValue) to wrap nullable Java results). In data work it models missing values honestly, such as nullable Spark columns mapped to Option fields in case classes.

Read full answer

Applies toScala 3 and Scala 2.13

Option vs null

Aspect null Option
Visible in the type No Yes
Compiler reminds you to handle absence No Yes (you must unwrap it)
Failure mode NullPointerException at runtime None flows through the code
Chaining Nested if (x != null) map / flatMap / for
Default value Manual getOrElse, fold, orElse

Code Example (Scala 3)

//> using scala 3.5

val ages = Map("asha" -> 31, "ben" -> 17)

def describe(name: String): String = ages.get(name) match
  case Some(a) if a >= 18 => s"$name is an adult"
  case Some(_)            => s"$name is a minor"
  case None               => s"$name is unknown"

def mean(xs: Seq[Double]): Option[Double] = Option.when(xs.nonEmpty)(xs.sum / xs.size)

@main def optionDemo(): Unit =
  println(ages.get("asha"))                     // Some(31)
  println(ages.get("zoe"))                      // None
  println(ages.get("zoe").getOrElse(0))         // 0
  println(ages.get("asha").map(_ + 1))          // Some(32)
  println(ages.get("ben").filter(_ >= 18))      // None
  println(ages.get("zoe").fold("n/a")(_.toString))   // n/a

  println(describe("asha"))                     // asha is an adult
  println(describe("ben"))                      // ben is a minor
  println(describe("zoe"))                      // zoe is unknown

  // Chaining without null checks
  val total = for
    a <- ages.get("asha")
    b <- ages.get("ben")
  yield a + b
  println(total)                                // Some(48)

  // Safe statistics on empty input
  println(mean(Seq(2.0, 4.0)))                  // Some(3.0)
  println(mean(Seq.empty))                      // None

  // Dropping missing values
  println(List(Some(1), None, Some(3)).flatten) // List(1, 3)
  println(List("4", "x", "6").flatMap(_.toIntOption).sum)   // 10

  // Wrapping a nullable Java value
  val fromJava: String = null
  println(Option(fromJava))                     // None
  println(Option("value"))                      // Some(value)

  // Converting to Either to keep an error message
  println(ages.get("zoe").toRight("user not found"))   // Left(user not found)

Option in Spark (Scala 2.13)

case class Customer(id: Long, email: Option[String], age: Option[Int])

val customers = spark.read.parquet(path).as[Customer]   // null in a column becomes None
val withAge = customers.filter(_.age.exists(_ >= 18))

Declaring a nullable column as Int instead of Option[Int] fails at runtime when a null appears.

Common Pitfalls

  • Calling .get on an Option: it throws on None, bringing back the problem Option solves.

  • Writing Some(x) where x may be null (Some(null) is not None); use Option(x).

  • Returning Option[Option[A]] from map; use flatMap.

  • Using Option for errors that need a reason; use Either or Try.

  • Nesting Option inside Future deeply; consider Future[Option[A]] helpers or a library.

Interview Tips

  • Lead with “absence becomes part of the type, so the compiler forces handling”.

  • Show map, getOrElse and a for-comprehension.

  • Mention Option(...) for Java interop and the .get anti-pattern.

12

What role do implicit parameters play in Scala?

Implicit parameters (called context parameters in Scala 3, declared with using and provided with given) are arguments the compiler fills in automatically from the values in scope whose type matches. They serve two main roles: passing context that many functions need but callers should not have to repeat (an ExecutionContext for futures, a configuration, a Spark SparkSession, a timeout), and type classes, where the compiler finds the right instance for a type (an Ordering[T] for sorted, a Numeric[T] for sum, a Spark Encoder[T] for ds.map, a JSON codec). Callers can still pass the argument explicitly. In Scala 2 the same feature is written with the implicit keyword for both parameters and values.

Read full answer

Applies toScala 3 (`using`/`given`); Scala 2.13 (`implicit`)

Syntax

Purpose Scala 2.13 Scala 3
Declare parameter (implicit ec: ExecutionContext) (using ec: ExecutionContext)
Provide instance implicit val ec: ExecutionContext = ... given ExecutionContext = ...
Fetch in body implicitly[Ordering[T]] summon[Ordering[T]]
Context bound def f[T: Ordering] same
Pass explicitly f(x)(myEc) f(x)(using myEc)

Code Example (Scala 3)

//> using scala 3.5

// 1. Passing context
final case class TrainConfig(seed: Int, verbose: Boolean)

def train(data: Seq[Double])(using cfg: TrainConfig): String =
  s"trained on ${data.size} rows with seed ${cfg.seed}"

// 2. Type class with derived instances
trait Scaler[A]:
  def scale(a: A, factor: Double): A

given Scaler[Double] with
  def scale(a: Double, f: Double): Double = a * f

given [A](using s: Scaler[A]): Scaler[List[A]] with      // instance built from another instance
  def scale(as: List[A], f: Double): List[A] = as.map(s.scale(_, f))

def double[A](a: A)(using s: Scaler[A]): A = s.scale(a, 2.0)

// 3. Context bound: shorthand for (using Ordering[T])
def top2[T: Ordering](xs: Seq[T]): Seq[T] = xs.sorted(using summon[Ordering[T]].reverse).take(2)

@main def usingDemo(): Unit =
  given TrainConfig = TrainConfig(seed = 42, verbose = false)
  println(train(Seq(1.0, 2.0)))                               // trained on 2 rows with seed 42
  println(train(Seq(1.0))(using TrainConfig(7, true)))        // trained on 1 rows with seed 7

  println(double(1.5))                                        // 3.0
  println(double(List(1.0, 2.0)))                             // List(2.0, 4.0)
  println(double(List(List(1.0), List(2.0, 3.0))))            // List(List(2.0), List(4.0, 6.0))

  println(top2(Seq(5, 1, 9, 3)))                              // List(9, 5)
  println(List(3, 1, 2).sorted(using Ordering[Int].reverse))  // List(3, 2, 1)
  // double("text")                                           // does not compile: no given Scaler[String]

The compiler built Scaler[List[List[Double]]] by applying the list instance twice on top of the Double instance. That is type-class derivation through implicit parameters.

Implicit Parameters in Everyday Scala and Spark

Code Implicit parameter supplied
Future { ... } ExecutionContext
xs.sorted, xs.max Ordering[T]
xs.sum Numeric[T]
ds.map(f) in Spark Encoder[U] (from import spark.implicits._)
Await.result, actor ask Timeout / Scheduler
JSON libraries Encoder[A]/Decoder[A] codecs

Scala 2.13 Equivalent

case class TrainConfig(seed: Int, verbose: Boolean)
def train(data: Seq[Double])(implicit cfg: TrainConfig): String =
  s"trained on ${data.size} rows with seed ${cfg.seed}"

implicit val defaultCfg: TrainConfig = TrainConfig(42, verbose = false)
train(Seq(1.0, 2.0))

Resolution Rules (Simplified)

  1. Values in the local scope and explicit imports (import x.given in Scala 3).

  2. The implicit scope of the type: companion objects of the type and its type arguments.

  3. If two candidates are equally specific, the compiler reports an ambiguity error.

Common Pitfalls

  • Using implicit parameters for ordinary arguments, which makes call sites hard to understand.

  • Common types as implicits (given Int, given String): any Int could be picked up. Wrap them in a dedicated type (TrainConfig).

  • Ambiguity from importing two instances of the same type.

  • Missing Spark encoders: “Unable to find encoder for type” means import spark.implicits._ is missing or the type has no encoder.

  • Scala 3: forgetting that import foo.* does not import givens; use import foo.given.

Interview Tips

  • Name the two roles: context passing and type classes.

  • Show using/given and the Scala 2 implicit equivalent.

  • Use real examples: ExecutionContext, Ordering, Spark Encoder.

13

How do for-comprehensions work in Scala?

A for-comprehension is syntax that the compiler rewrites into calls to flatMap, map, withFilter and foreach: each generator x <- xs becomes a flatMap (the last one a map when there is a yield), each guard if cond becomes withFilter, and value definitions y = f(x) become intermediate maps. Because it is only a rewrite, it works with any type that has these methods, not just collections: Option (stop at the first None), Either (stop at the first Left), Try, Future (sequence asynchronous steps), Spark Datasets, and effect types such as Cats Effect IO. Without yield, a for loop is a foreach for side effects.

Read full answer

Applies toScala 3 and Scala 2.13

Desugaring Rules

Comprehension Becomes
for x <- xs yield f(x) xs.map(x => f(x))
for x <- xs; y <- ys yield (x, y) xs.flatMap(x => ys.map(y => (x, y)))
for x <- xs if p(x) yield x xs.withFilter(x => p(x)).map(x => x)
for x <- xs; y = g(x) yield y xs.map { x => val y = g(x); (x, y) }.map { case (x, y) => y }
for x <- xs do println(x) xs.foreach(x => println(x))

Code Example (Scala 3)

//> using scala 3.5

def parsePositive(s: String): Either[String, Int] =
  s.toIntOption.toRight(s"not a number: $s").filterOrElse(_ > 0, s"not positive: $s")

@main def forDemo(): Unit =
  // Collections: nested loops with a guard
  val pairs = for
    x <- List(1, 2, 3)
    if x != 2
    y <- List("a", "b")
  yield s"$x$y"
  println(pairs)                          // List(1a, 1b, 3a, 3b)

  // Hyperparameter grid
  val grid = for
    lr    <- List(0.01, 0.1)
    depth <- List(3, 5)
  yield (lr, depth)
  println(grid)                           // List((0.01,3), (0.01,5), (0.1,3), (0.1,5))

  // Option: stops at the first None
  val config = Map("epochs" -> "20", "batch" -> "64")
  val product = for
    e <- config.get("epochs").flatMap(_.toIntOption)
    b <- config.get("batch").flatMap(_.toIntOption)
    total = e * b                         // value definition
  yield total
  println(product)                        // Some(1280)
  println(for e <- config.get("lr") yield e)   // None

  // Either: stops at the first Left and keeps its error
  val ok = for
    epochs <- parsePositive("20")
    batch  <- parsePositive("64")
  yield (epochs, batch)
  println(ok)                             // Right((20,64))

  val bad = for
    epochs <- parsePositive("20")
    batch  <- parsePositive("-1")
  yield (epochs, batch)
  println(bad)                            // Left(not positive: -1)

  // Pattern in a generator
  val scores = Map("asha" -> 91, "ben" -> 78)
  for (name, score) <- scores.toList.sortBy(_._1) do println(s"$name=$score")
  // asha=91
  // ben=78

With Futures

import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global

def features(id: Int): Future[Vector[Double]] = Future(Vector(id.toDouble))
def predict(v: Vector[Double]): Future[Double] = Future(v.sum * 2)

val result: Future[Double] = for
  f <- features(7)
  p <- predict(f)          // runs after features completes
yield p                    // completes with 14.0

Steps in one comprehension run one after another. To run independent futures in parallel, create them before the for.

Rules to Remember

Rule Consequence
All generators must use the same “container” family Mixing List and Option works only when the inner type converts (e.g. Option to Iterable); mixing Future and Option does not
The first generator decides the result type for x <- List(...); y <- Some(...) returns a List
Guards need withFilter Future supports it (fails with NoSuchElementException); Either does not

Common Pitfalls

  • Expecting parallelism from sequential Future generators.

  • Mixing monad types (Future[Option[A]]), which needs nested comprehensions or a library (OptionT from Cats).

  • Using a guard on Either (it does not compile; use filterOrElse).

  • Long comprehensions with side effects in value definitions.

Interview Tips

  • Write the desugaring of a two-generator comprehension from memory.

  • Show that it works for Option/Either/Future, not just loops.

  • Mention the sequential-futures trap.

14

What is functional programming, and how does Scala support it?

Functional programming (FP) builds programs from pure functions that transform immutable values, instead of from statements that change shared state: functions are first-class values, data is not modified in place, side effects (I/O, randomness, mutation) are pushed to the edges or represented as values, and programs are composed from small functions like map, filter and fold. The payoffs are code that is easier to test, reason about and run in parallel. Scala is a hybrid object-functional language that supports FP fully: first-class and higher-order functions, immutable collections by default and val, case classes and sealed traits / enums (algebraic data types) with pattern matching, Option/Either/Try for errors without exceptions, for-comprehensions, tail-call optimization, type classes via given/using, and libraries such as Cats Effect and ZIO for purely functional effects.

Read full answer

Applies toScala 3 (Scala 2.13 supports the same ideas with older syntax)

FP Principles and Scala Features

Principle Scala feature
Functions as values Function literals x => x + 1, method eta-expansion
Higher-order functions map, filter, foldLeft, user-defined def f(g: A => B)
Immutability val, immutable List/Vector/Map, case class copy
Algebraic data types enum, sealed trait + case class
Pattern matching match with exhaustiveness checks
No nulls / exceptions for control flow Option, Either, Try
Composition andThen, compose, for-comprehensions
Recursion @tailrec
Type classes given / using, context bounds
Pure effects Cats Effect IO, ZIO

Code Example (Scala 3)

//> using scala 3.5

// Algebraic data type + pattern matching
enum Expr:
  case Num(v: Double)
  case Add(a: Expr, b: Expr)
  case Mul(a: Expr, b: Expr)

def eval(e: Expr): Double = e match
  case Expr.Num(v)    => v
  case Expr.Add(a, b) => eval(a) + eval(b)
  case Expr.Mul(a, b) => eval(a) * eval(b)

// Errors as values
def parsePositive(s: String): Either[String, Double] =
  s.toDoubleOption.toRight(s"not a number: $s").filterOrElse(_ > 0, s"not positive: $s")

@main def fpDemo(): Unit =
  import Expr.*
  println(eval(Add(Num(2), Mul(Num(3), Num(4)))))      // 14.0

  // Function composition
  val normalize: Double => Double = _ / 100
  val clip: Double => Double = x => x.max(0.0).min(1.0)
  val pipeline = normalize.andThen(clip)
  println(List(50.0, 150.0, -20.0).map(pipeline))       // List(0.5, 1.0, 0.0)

  // A pipeline of pure transformations on immutable data
  println(List(3, 8, 1, 9).filter(_ > 2).map(_ * 2).sum)   // 40

  // Validation with a for-comprehension over Either
  val lr = for
    a <- parsePositive("0.5")
    b <- parsePositive("0.2")
  yield a * b
  println(lr)                                          // Right(0.1)
  println(parsePositive("-3"))                         // Left(not positive: -3)

Num(2) works because the integer literal widens to the Double parameter. 0.5 * 0.2 prints 0.1 because the double product rounds to the same value as the literal 0.1.

FP vs Imperative Style

// Imperative: mutable accumulator and loop
var total = 0
for x <- List(3, 8, 1, 9) do
  if x > 2 then total += x * 2

// Functional: one expression, no mutation
val total2 = List(3, 8, 1, 9).filter(_ > 2).map(_ * 2).sum   // 40

Why FP Matters for Data and ML Work

Benefit Example
Safe parallelism Pure functions in Spark map can be retried or run anywhere
Reproducibility Same inputs → same features; seeded randomness passed explicitly
Testability Feature functions tested with plain inputs and outputs
Composability Small transforms chained into pipelines

Common Pitfalls

  • Treating FP as “no var ever”: local mutation inside a pure function is fine when performance needs it.

  • Hiding side effects (logging, metrics, random numbers) inside functions that look pure.

  • Deeply nested flatMaps instead of for-comprehensions.

  • Introducing heavy FP libraries (category theory abstractions) into teams not ready for them.

Interview Tips

  • Define FP by pure functions, immutability and composition, then map each to a Scala feature.

  • Show an ADT with pattern matching and an Either-based validation.

  • Mention that Scala is hybrid: you can use OOP for modules and FP for logic.

15

Explain higher-order functions in Scala.

A higher-order function (HOF) is a function that takes other functions as parameters, returns a function, or both. Scala treats functions as first-class values (types like Int => Int or (Double, Double) => Double), so HOFs are everywhere: collection methods (map, filter, foldLeft, sortBy, groupBy) take functions, factories return configured functions (scaler(2.0)), and combinators such as andThen, compose and curried methods build new behaviour from small pieces. HOFs remove repeated loops and boilerplate: the “how” (iteration, retries, timing, resource handling) is written once, and callers pass the “what” as a function.

Read full answer

Applies toScala 3 and Scala 2.13

Kinds of Higher-Order Functions

Kind Example
Takes a function xs.map(f), evaluate(preds, ys)(loss)
Returns a function def scaler(k: Double): Double => Double
Both def andThen[C](g: B => C): A => C
Curried method def scale(k: Double)(x: Double), partially applied as scale(0.5)
By-name control structure def retry[A](n: Int)(op: () => A), Using(resource)(f)

Code Example (Scala 3)

//> using scala 3.5
import scala.util.{Failure, Try}

// Takes a function
def applyTwice(f: Int => Int, x: Int): Int = f(f(x))

// Takes a function as the "strategy"
def evaluate(preds: Seq[Double], ys: Seq[Double])(loss: (Double, Double) => Double): Double =
  preds.lazyZip(ys).map(loss).sum / preds.size

val squared  = (p: Double, y: Double) => (p - y) * (p - y)
val absolute = (p: Double, y: Double) => math.abs(p - y)

// Returns a function
def threshold(t: Double): Double => Int = p => if p >= t then 1 else 0

// Curried method: partially apply to get a function
def scale(factor: Double)(x: Double): Double = x * factor

// Generic retry: the operation is passed in
def retry[A](times: Int)(op: () => A): Try[A] =
  Try(op()) match
    case Failure(_) if times > 1 => retry(times - 1)(op)
    case result                  => result

@main def hofDemo(): Unit =
  println(applyTwice(_ + 3, 10))                                  // 16
  println(evaluate(Seq(2.0, 3.0), Seq(1.0, 5.0))(squared))        // 2.5   (1 + 4) / 2
  println(evaluate(Seq(2.0, 3.0), Seq(1.0, 5.0))(absolute))       // 1.5   (1 + 2) / 2

  val strict = threshold(0.8)
  println(List(0.5, 0.85, 0.9).map(strict))                       // List(0, 1, 1)

  val half = scale(0.5)                                           // Double => Double
  println(List(2.0, 4.0).map(half))                               // List(1.0, 2.0)

  val pipeline = ((x: Double) => x * 10).andThen(x => math.round(x)).andThen(_.toInt)
  println(pipeline(0.46))                                         // 5

  var attempts = 0
  val result = retry(3) { () =>
    attempts += 1
    if attempts < 3 then throw RuntimeException("flaky") else "ok"
  }
  println(s"$result after $attempts attempts")                    // Success(ok) after 3 attempts

  // Standard library HOFs
  val scores = List(("asha", 91), ("ben", 78), ("chen", 85))
  println(scores.sortBy(-_._2).map(_._1))                         // List(asha, chen, ben)
  println(scores.groupBy(_._2 >= 80).view.mapValues(_.size).toList.sortBy(_._1))   // List((false,1), (true,2))

0.46 * 10 is 4.6 (as a double, 4.6000000000000005), and math.round gives 5.

Function Types

Type Meaning
A => B Function1[A, B]
(A, B) => C Function2[A, B, C]
() => A No-argument function
=> A (parameter only) By-name parameter, not a function value
PartialFunction[A, B] Defined for some inputs

Common Pitfalls

  • Overly abstract HOF chains that are harder to read than a simple loop.

  • Passing a method with side effects into map on a lazy view, so it runs later or several times.

  • Creating closures in hot loops that capture large objects.

  • In Spark, passing functions that capture non-serializable objects.

Interview Tips

  • Give one example of taking a function and one of returning a function.

  • Show the “strategy” use (loss functions) and a reusable combinator (retry).

  • Mention currying and eta-expansion (scale(0.5) becomes a function automatically in Scala 3).

16

What are pure functions, and why do they matter in Scala?

A pure function always returns the same output for the same input and has no observable side effects: it does not modify variables outside itself, mutate its arguments, perform I/O, read the clock or a global random generator, or throw for normal control flow. Because a call can be replaced by its result (referential transparency), pure functions are easy to test (no setup or mocks), reason about (only inputs matter), compose, memoize, and run in parallel or retry safely. This matters in Scala data work: Spark may re-run a task on another executor after a failure or for speculative execution, and transformations give consistent results only if the functions inside map and filter are pure.

Read full answer

Applies toScala 3 and Scala 2.13

Pure vs Impure

Function Pure? Why
def add(a: Int, b: Int) = a + b Yes Depends only on inputs
def mean(xs: Seq[Double]) = xs.sum / xs.size Yes Same input → same output
def now() = System.currentTimeMillis() No Reads the clock
def noisy(x: Double) = x + Random.nextDouble() No Global random state
def log(x: Int) = { println(x); x } No I/O side effect
def inc() = { counter += 1; counter } No Mutates external state
def sortInPlace(a: Array[Int]) = java.util.Arrays.sort(a) No Mutates its argument

Code Example (Scala 3)

//> using scala 3.5

// Pure: standardization depends only on its input
def standardize(xs: Vector[Double]): Vector[Double] =
  val mean = xs.sum / xs.size
  val sd = math.sqrt(xs.map(x => (x - mean) * (x - mean)).sum / xs.size)
  xs.map(x => (x - mean) / sd)

// Impure: hidden state changes the result of repeated calls
var calls = 0
def impureScale(x: Double): Double =
  calls += 1
  x * calls

// Pure randomness: the generator state is an explicit input and output
final case class Seed(value: Long):
  def next: (Int, Seed) =
    val n = value * 6364136223846793005L + 1442695040888963407L
    ((n >>> 33).toInt, Seed(n))

// Pure core, impure shell: side effects only at the edge
def summary(xs: Vector[Double]): String = f"n=${xs.size}, max=${xs.max}%.1f"

@main def pureDemo(): Unit =
  println(standardize(Vector(2.0, 4.0, 6.0)).map(v => f"$v%.4f"))   // Vector(-1.2247, 0.0000, 1.2247)

  println(impureScale(10.0))           // 10.0
  println(impureScale(10.0))           // 20.0   same input, different output

  val (a, s1) = Seed(42).next
  val (b, _)  = Seed(42).next
  println(a == b)                      // true: same seed, same number
  println(s1.next._1 == s1.next._1)    // true

  println(summary(Vector(1.0, 3.5, 2.0)))   // n=3, max=3.5   (printing happens here, at the edge)

Benefits in Practice

Benefit How purity helps
Testing assert(standardize(Vector(2, 4, 6)).sum.abs < 1e-9): no mocks or setup
Caching / memoization Results can be stored by input
Parallelism No shared state, so no locks
Spark fault tolerance Re-running a task produces the same partition
Refactoring Reorder or inline calls without changing behaviour
Reproducible ML Seeds passed explicitly give identical features and splits

Handling Effects

Programs need I/O. Two common approaches in Scala:

  1. Functional core, imperative shell: pure functions for logic; a thin outer layer reads input, calls them, and writes output.

  2. Effects as values: Cats Effect IO or ZIO describe effects without running them, keeping even I/O code referentially transparent until the program’s entry point runs it.

Common Pitfalls

  • Functions that look pure but read a global config, a mutable cache or the current time.

  • Using scala.util.Random (shared global state) inside Spark transformations, so retried tasks produce different data; seed per partition instead.

  • Mutating an input collection (e.g. sorting an Array in place) inside a “pure” helper.

  • Throwing exceptions for expected failures instead of returning Option/Either.

Interview Tips

  • Give the two-part definition: same output for same input, and no side effects.

  • Explain referential transparency with “replace the call with its value”.

  • Connect purity to Spark task retries and reproducible ML pipelines.

17

What are partial functions in Scala, and how are they used?

A PartialFunction[A, B] is a function that is defined only for some inputs of type A, and it can report which ones with isDefinedAt. You write one as a block of case clauses, { case "spam" => 1; case "ham" => 0 }; calling it on an input no case matches throws a MatchError, so callers usually check first or use the safe combinators: collect (filter and map in one step), lift (turn it into A => Option[B]), orElse (chain fallbacks), andThen (post-process) and applyOrElse. Partial functions appear across the standard library and frameworks: recover on Future and Try, collect on collections, classic Akka actors’ receive, and route matching in HTTP libraries.

Read full answer

Applies toScala 3 and Scala 2.13

Key Operations

Operation Result
pf.isDefinedAt(x) true if a case matches
pf(x) Value, or MatchError if undefined
pf.lift A => Option[B] (total)
pf.orElse(other) Tries pf, then other
pf.andThen(f) Applies f to the result
pf.applyOrElse(x, default) pf(x) or default(x), evaluating the match once
xs.collect(pf) Keeps elements where pf is defined, mapped by pf

Code Example (Scala 3)

//> using scala 3.5
import scala.util.Try

val parseLabel: PartialFunction[String, Int] = {
  case "spam" => 1
  case "ham"  => 0
}

val fallback: PartialFunction[String, Int] = { case _ => -1 }

val safeDiv: PartialFunction[(Int, Int), Int] = {
  case (a, b) if b != 0 => a / b
}

@main def partialDemo(): Unit =
  println(parseLabel.isDefinedAt("eggs"))          // false
  println(parseLabel("spam"))                      // 1
  println(parseLabel.lift("eggs"))                 // None
  println(parseLabel.lift("ham"))                  // Some(0)

  val labels = List("spam", "eggs", "ham")
  println(labels.collect(parseLabel))              // List(1, 0)          undefined inputs dropped
  println(labels.map(parseLabel.orElse(fallback))) // List(1, -1, 0)      fallback for the rest
  println(parseLabel.andThen((n: Int) => n * 100)("spam"))   // 100

  println(List((10, 2), (1, 0), (9, 3)).collect(safeDiv))   // List(5, 3)

  // collect with type patterns on mixed data
  val mixed: List[Any] = List(1, "a", 2.5, 4)
  println(mixed.collect { case i: Int => i * 2 })  // List(2, 8)

  // recover takes a partial function: only the listed exceptions are handled
  val parsed = Try("x".toInt).recover { case _: NumberFormatException => 0 }
  println(parsed)                                  // Success(0)

  // parseLabel("eggs")                            // throws scala.MatchError: eggs

Partial Function vs Partially Applied Function

Partial function Partially applied function
Meaning Defined for only some inputs Some arguments already supplied
Example { case x if x > 0 => math.sqrt(x) } val add5 = add(5, _)
Type PartialFunction[A, B] Ordinary function type

When to Use Them

  • Filtering and transforming in one pass with collect, instead of filter + map + casts.

  • Handling only some errors with recover/recoverWith.

  • Composable dispatch tables (routing, message handling) with orElse.

  • Validation where invalid inputs are simply not in the domain (lift to get an Option).

Common Pitfalls

  • Calling a partial function directly on untrusted input and getting a MatchError at runtime.

  • Calling isDefinedAt and then apply with expensive guards: the guard runs twice. Use applyOrElse or lift.

  • Writing a case block where a total function is expected (e.g. map { case ... }) without covering all inputs: it compiles but can throw MatchError.

  • Confusing partial functions with partially applied functions.

Interview Tips

  • Define it with isDefinedAt and show the case-block syntax.

  • Show collect, lift and orElse in one example.

  • Mention real uses: recover, actors’ receive, routing.

18

How do closures work in Scala?

A closure is a function value that refers to variables defined outside its own body (free variables) and keeps access to them even after the enclosing scope has finished: def scaler(factor: Double) = (x: Double) => x * factor returns a function that “closes over” factor. Scala closures capture variables, not snapshots of their values: a captured val behaves like a constant, but a captured var is shared, so later changes are visible to the closure and the closure can update it. Closures power higher-order functions (map, filter), callbacks and factories. In Spark, closures are serialized and shipped to executors, so everything they capture must be serializable, and capturing a field silently captures the whole enclosing object.

Read full answer

Applies toScala 3 and Scala 2.13; Spark with Scala 2.13

Code Example (Scala 3)

//> using scala 3.5

// A factory returning a closure over `factor`
def scaler(factor: Double): Double => Double = x => x * factor

// A closure that owns private mutable state
def makeCounter(): () => Int =
  var count = 0
  () =>
    count += 1
    count

@main def closureDemo(): Unit =
  val double = scaler(2.0)
  val half   = scaler(0.5)
  println(double(21))                 // 42.0
  println(half(21))                   // 10.5

  // Captured var: the closure sees the current value, not the value at creation
  var threshold = 0.5
  val isPositive = (p: Double) => p >= threshold
  println(isPositive(0.6))            // true
  threshold = 0.7
  println(isPositive(0.6))            // false

  // Each call to makeCounter creates a new, independent captured variable
  val c1 = makeCounter()
  c1(); c1()
  println(c1())                       // 3
  val c2 = makeCounter()
  println(c2())                       // 1

  // Each loop iteration binds a new `i`, so each closure has its own value
  val fs = (1 to 3).map(i => () => i * 10)
  println(fs.map(_()))                // Vector(10, 20, 30)

  // Closures in higher-order functions
  val minScore = 70
  println(List(65, 80, 92).filter(_ >= minScore))   // List(80, 92)

How Capture Works

Captured Behaviour Implementation
val / parameter Fixed value Copied into the function object
Local var Shared and mutable Boxed in a reference cell (IntRef) shared by the closure and the scope
Field of a class (this.x) Reads the field at call time Captures this, the whole enclosing object

Closures in Spark (Scala 2.13)

import org.apache.spark.sql.{Dataset, SparkSession}

class FeatureJob(spark: SparkSession) {          // not Serializable
  import spark.implicits._
  val factor = 2.0

  // BAD: `factor` means `this.factor`, so the closure captures `this` (FeatureJob)
  def bad(ds: Dataset[Double]): Dataset[Double] = ds.map(_ * factor)
  // -> org.apache.spark.SparkException: Task not serializable

  // GOOD: copy the field into a local val; only a Double is captured
  def good(ds: Dataset[Double]): Dataset[Double] = {
    val f = factor
    ds.map(_ * f)
  }
}

For large read-only data (a lookup table or vocabulary), use spark.sparkContext.broadcast(table) and capture the broadcast handle, so the data is sent once per executor instead of once per task.

Common Pitfalls

  • Capturing a var and being surprised that the closure sees later changes (or changes it).

  • Sharing a captured var between threads (data race); closures do not add synchronization.

  • In Spark: capturing this, a SparkContext, a database connection or a logger, which causes Task not serializable.

  • Updating a driver-side var from a Spark closure: each task updates its own copy, and the driver sees nothing. Use an accumulator.

  • Keeping closures alive that capture large objects, preventing garbage collection.

Interview Tips

  • Define a closure as “function + captured environment” and give the scaler factory example.

  • Explain capture by reference for var, with the threshold example.

  • Bring up Spark closure serialization and the local-val fix; it is the most common practical question.

19

What is tail recursion, and how does Scala optimize it?

A recursive call is a tail call when it is the very last action of the function, with nothing left to compute after it returns; a function whose recursive calls are all tail calls is tail-recursive. The Scala compiler turns direct self tail recursion into a loop (a jump back to the start of the method in bytecode), so it runs in constant stack space and cannot cause a StackOverflowError, even for millions of iterations. The @tailrec annotation asks the compiler to verify this and fails compilation if the method is not tail-recursive. The optimization applies only to self-recursion in methods that cannot be overridden (local, private, final or in an object); mutual recursion needs a trampoline such as scala.util.control.TailCalls, because the JVM itself has no general tail-call elimination.

Read full answer

Applies toScala 3 and Scala 2.13

Tail Call or Not?

Code Tail call? Why
go(n - 1, acc * n) Yes Last action is the call
n * fact(n - 1) No Multiplication happens after the call
1 + count(rest) No Addition after the call
if (x) go(a) else go(b) Yes Each branch ends with the call
try go(n - 1) catch ... No The handler must stay on the stack

Code Example (Scala 3)

//> using scala 3.5
import scala.annotation.tailrec
import scala.util.control.TailCalls.*

// 1. Accumulator style
@tailrec
def sum(xs: List[Long], acc: Long = 0L): Long = xs match
  case Nil    => acc
  case h :: t => sum(t, acc + h)

// 2. Euclid's GCD is naturally tail-recursive
@tailrec
def gcd(a: Int, b: Int): Int = if b == 0 then a else gcd(b, a % b)

// 3. Binary search with a local tail-recursive helper
def search(xs: IndexedSeq[Int], target: Int): Option[Int] =
  @tailrec
  def go(lo: Int, hi: Int): Option[Int] =
    if lo > hi then None
    else
      val mid = lo + (hi - lo) / 2
      if xs(mid) == target then Some(mid)
      else if xs(mid) < target then go(mid + 1, hi)
      else go(lo, mid - 1)
  go(0, xs.size - 1)

// 4. Mutual recursion: not optimized by the compiler, so use a trampoline
def isEven(n: Int): TailRec[Boolean] = if n == 0 then done(true) else tailcall(isOdd(n - 1))
def isOdd(n: Int): TailRec[Boolean] = if n == 0 then done(false) else tailcall(isEven(n - 1))

@main def tailrecDemo(): Unit =
  println(sum(List.range(1L, 100_001L)))      // 5000050000   (100,000 calls, no stack growth)
  println(gcd(48, 18))                        // 6
  println(search(Vector(1, 3, 5, 7, 9, 11), 7))   // Some(3)
  println(search(Vector(1, 3, 5, 7, 9, 11), 4))   // None
  println(isEven(100_001).result)             // false

What the Compiler Produces

For gcd, the compiled code is equivalent to:

def gcdLoop(a0: Int, b0: Int): Int =
  var a = a0
  var b = b0
  while b != 0 do
    val t = a % b
    a = b
    b = t
  a

You write the clear recursive version; the bytecode is a loop.

Rules for the Optimization

Requirement Reason
Self-recursive call in tail position Only direct self-calls are rewritten
Method cannot be overridden (final, private, local, in an object) An override could change which method the call reaches
Use @tailrec Turns a silent missed optimization into a compile error

Converting Non-Tail Recursion

Add an accumulator parameter that carries the partial result, so the work happens before the call:

Non-tail Tail-recursive
def len(xs: List[A]): Int = xs match { case Nil => 0; case _ :: t => 1 + len(t) } def len(xs: List[A], acc: Int = 0): Int = xs match { case Nil => acc; case _ :: t => len(t, acc + 1) }

Tree recursion (e.g. walking both children of a binary tree) cannot be made tail-recursive this simply; use an explicit stack (a List of pending nodes) or a trampoline.

Common Pitfalls

  • Assuming recursion is optimized without @tailrec; a small refactor can silently break it.

  • Putting @tailrec on an overridable public method of a class (compile error).

  • Wrapping the recursive call in try/catch or a lambda, which moves it out of tail position.

  • Expecting the JVM to optimize mutual recursion.

Interview Tips

  • Define a tail call precisely: nothing remains to do after it.

  • Mention @tailrec, the loop rewrite and the restrictions (self-calls, non-overridable).

  • Offer TailCalls (trampolining) for mutual recursion.

20

How do map, flatMap and filter work on Scala collections?

map, flatMap and filter are the three core higher-order functions on Scala collections, and each returns a new collection without changing the original: map(f) applies f to every element and keeps the same number of elements (List(1, 2, 3).map(_ * 2) → List(2, 4, 6)); filter(p) keeps only the elements for which the predicate p is true; and flatMap(f) applies a function that returns a collection (or Option) for each element and flattens the results into one collection, so it can expand, shrink or drop elements. They work the same on List, Vector, Set, Map, Option, Either, Future, Spark Datasets and RDDs, and for-comprehensions are syntax for exactly these calls.

Read full answer

Applies toScala 3 and Scala 2.13 (same methods in Spark's Scala API)

Summary

Method Function type Output size Example
map A => B Same as input List(1, 2, 3).map(_ * 2) → List(2, 4, 6)
filter A => Boolean ≤ input List(1, 2, 3).filter(_ % 2 == 1) → List(1, 3)
flatMap A => IterableOnce[B] Any List("a b", "c").flatMap(_.split(" ")) → List(a, b, c)
collect Partial function ≤ input Filter + map in one step

Code Example (Scala 3)

//> using scala 3.5

final case class User(name: String, age: Int, tags: List[String])

@main def mapFlatMapFilter(): Unit =
  val nums = List(1, 2, 3, 4)
  println(nums.map(_ * 2))                    // List(2, 4, 6, 8)
  println(nums.filter(_ % 2 == 0))            // List(2, 4)
  println(nums.flatMap(n => List(n, -n)))     // List(1, -1, 2, -2, 3, -3, 4, -4)

  // flatMap with Option drops failures
  val raw = List("1", "x", "3")
  println(raw.map(_.toIntOption))             // List(Some(1), None, Some(3))
  println(raw.flatMap(_.toIntOption))         // List(1, 3)

  // Tokenizing text: one line -> many words
  val lines = List("spark makes data", "scala makes spark")
  println(lines.flatMap(_.split(" ")).distinct)   // List(spark, makes, data, scala)

  // A realistic chain
  val users = List(
    User("Asha", 31, List("ml", "scala")),
    User("Ben", 17, List("python")),
    User("Chen", 45, List("scala", "spark"))
  )
  val adultTags = users.filter(_.age >= 18).flatMap(_.tags).distinct.sorted
  println(adultTags)                          // List(ml, scala, spark)
  println(users.map(u => u.name -> u.age).toMap)  // Map(Asha -> 31, Ben -> 17, Chen -> 45)

  // On a Map, map works on (key, value) pairs
  val prices = Map("a" -> 1, "b" -> 2)
  println(prices.map((k, v) => k.toUpperCase -> v * 10))   // Map(A -> 10, B -> 20)

  // collect = filter + map
  println(List(1, 2, 3, 4).collect { case n if n % 2 == 0 => n * 100 })   // List(200, 400)

map vs flatMap with Option

def parse(s: String): Option[Int] = s.toIntOption
def reciprocal(n: Int): Option[Double] = if n == 0 then None else Some(1.0 / n)

Some("4").map(parse)                        // Some(Some(4))  nested
Some("4").flatMap(parse)                    // Some(4)        flattened
Some("4").flatMap(parse).flatMap(reciprocal)   // Some(0.25)
Some("0").flatMap(parse).flatMap(reciprocal)   // None

Use map when the function returns a plain value and flatMap when it returns a wrapped one (Option, List, Future, Either); otherwise you get nested wrappers.

For-Comprehension Equivalence

for
  u <- users if u.age >= 18
  t <- u.tags
yield s"${u.name}:$t"

// is translated to
users.withFilter(_.age >= 18).flatMap(u => u.tags.map(t => s"${u.name}:$t"))

In Spark (Scala 2.13)

val words = spark.read.textFile("docs.txt")        // Dataset[String]
  .flatMap(_.toLowerCase.split("\\W+"))            // one row per word
  .filter(_.nonEmpty)
  .map(w => (w, 1))

Common Pitfalls

  • Using map with a function that returns Option or List and ending up with List[Option[A]].

  • Long chains on a large strict collection building several intermediate collections; use .view, .iterator or a single collect/foldLeft.

  • Side effects inside map (use foreach for side effects).

  • On a Set, map can shrink the result because duplicates collapse: Set(1, 2).map(_ => 0) is Set(0).

Interview Tips

  • Define each by its function type and output size.

  • Show flatMap flattening Options; it shows you understand the monadic pattern.

  • Mention that for-comprehensions desugar to these methods.

21

How does Scala support lazy evaluation?

Scala is strict (eager) by default, but it gives several explicit ways to delay work until a value is needed: lazy val (computed once on first access, then cached), by-name parameters => T (the argument expression is evaluated only when, and each time, the method uses it), lazy collections — views (.view), iterators and LazyList (memoized, possibly infinite) — and short-circuiting operators (&&, ||, getOrElse). Lazy evaluation saves work that may never be needed, allows infinite data structures, avoids building intermediate collections, and lets expensive resources (models, connections) be initialized on demand. Apache Spark applies the same idea at cluster scale: transformations are lazy until an action runs.

Read full answer

Applies toScala 3 and Scala 2.13

Mechanisms

Mechanism Evaluated Cached? Typical use
lazy val x = expr On first access Yes (thread-safe initialization) Expensive setup, config, models
By-name f(x: => T) Each time x is used in f No Logging, getOrElse, custom control structures
.view When a strict result is requested No (recomputed each traversal) Fusing map/filter chains
Iterator On next() No (one pass) Streaming, files
LazyList Element by element on demand Yes (memoized) Infinite or recursive sequences

Code Example (Scala 3)

//> using scala 3.5

// Infinite, memoized sequence
val fibs: LazyList[BigInt] = BigInt(0) #:: BigInt(1) #:: fibs.zip(fibs.tail).map(_ + _)

def logIf(debug: Boolean)(msg: => String): Unit =
  if debug then println(msg)           // msg is evaluated only here

@main def lazyDemo(): Unit =
  // 1. lazy val: computed once, on first use
  lazy val model = { println("loading model..."); "model-v1" }
  println("before")
  println(model)
  println(model)
  // before
  // loading model...
  // model-v1
  // model-v1

  // 2. By-name parameter: the expensive message is never built
  logIf(debug = false) { println("building message"); "details" }   // prints nothing
  logIf(debug = true)("cheap message")                              // cheap message

  // 3. View: only the needed elements are processed
  var calls = 0
  val firstThree = (1 to 1_000_000).view
    .map { x => calls += 1; x * 2 }
    .filter(_ % 3 == 0)
    .take(3)
    .toList
  println(firstThree)                  // List(6, 12, 18)
  println(calls)                       // 9   (not 1,000,000)

  // 4. LazyList: take from an infinite sequence
  println(fibs.take(10).toList)        // List(0, 1, 1, 2, 3, 5, 8, 13, 21, 34)

  // 5. Short-circuit and getOrElse are lazy too
  val cached: Option[Double] = Some(0.9)
  println(cached.getOrElse { println("calling model"); 0.0 })        // 0.9   (no call)

The view stops after x = 9, because 9 × 2 = 18 is the third value divisible by 3; only 9 elements were ever mapped.

Strict vs Lazy in One Table

Expression Work done
(1 to 1_000_000).map(_ * 2).filter(_ % 3 == 0).take(3) 1,000,000 maps + 1,000,000 filters, two intermediate collections
(1 to 1_000_000).view.map(_ * 2).filter(_ % 3 == 0).take(3).toList 9 maps and filters, no intermediate collections

Laziness in Spark

val features = df.filter($"age" > 18).withColumn("x2", $"x" * 2)   // nothing runs yet (transformations)
features.count()                                                    // action: Spark optimizes and runs the whole plan

Spark builds a plan from transformations and optimizes it as a whole before an action triggers execution; cache() then plays the role of memoization.

Common Pitfalls

  • Holding the head of a large LazyList in a val: every computed element stays in memory.

  • Expecting a view to cache results: each traversal recomputes the whole chain, including side effects.

  • Side effects inside lazy code that run later, more than once, or never.

  • A lazy val whose initializer depends on another lazy val that depends back on it (deadlock or StackOverflowError).

  • Overusing lazy val in hot paths; each access checks an initialization flag.

Interview Tips

  • Name the five mechanisms and the “evaluated when / cached?” difference.

  • Contrast lazy val (once) with by-name parameters (every use).

  • Link to Spark’s lazy transformations and actions.

22

What is the difference between mutable and immutable collections in Scala?

An immutable collection never changes after it is created: operations such as :+, + or updated return a new collection and leave the original untouched, while a mutable collection is changed in place by methods like +=, update and remove. Scala imports the immutable versions by default (List, Vector, Map, Set from scala.collection.immutable); mutable ones (ArrayBuffer, ListBuffer, mutable.Map, mutable.Set, ArrayDeque) must be imported from scala.collection.mutable. Immutable collections are thread-safe without locks and easy to reason about, and they avoid full copies through structural sharing (a new list reuses the old one’s nodes). Mutable collections avoid allocation in hot loops and builders. Idiomatic Scala uses immutable collections in APIs and keeps mutation local inside a function.

Read full answer

Applies toScala 3 and Scala 2.13

Comparison

Aspect Immutable Mutable
Default import Yes (List, Vector, Map, Set) No (scala.collection.mutable._)
“Adding” an element Returns a new collection Changes the collection in place
Thread safety Safe to share Needs synchronization or concurrent versions
Memory Structural sharing; some allocation per update In-place; amortized growth
Reasoning No hidden changes; easy to test Aliases can change behind your back
Typical use Domain data, function results, messages Builders, caches, hot loops

Code Example (Scala 3)

//> using scala 3.5
import scala.collection.mutable

@main def collectionsDemo(): Unit =
  // Immutable: new values, originals unchanged
  val xs = List(1, 2, 3)
  val ys = 0 :: xs
  println(xs)              // List(1, 2, 3)
  println(ys)              // List(0, 1, 2, 3)
  println(ys.tail eq xs)   // true: structural sharing, nothing copied

  val m = Map("a" -> 1)
  val m2 = m + ("b" -> 2)
  println(m)               // Map(a -> 1)
  println(m2)              // Map(a -> 1, b -> 2)

  // Mutable: changed in place
  val buf = mutable.ArrayBuffer(1, 2, 3)
  buf += 4
  buf(0) = 10
  println(buf)             // ArrayBuffer(10, 2, 3, 4)

  val counts = mutable.Map.empty[String, Int].withDefaultValue(0)
  for w <- "a b a c a".split(" ") do counts(w) += 1
  println(counts.toList.sorted)   // List((a,3), (b,1), (c,1))

  // Aliasing: two names, one mutable object
  val alias = buf
  alias += 99
  println(buf.last)        // 99

  // Local mutability, immutable result
  def squares(n: Int): List[Int] =
    val b = List.newBuilder[Int]
    for i <- 1 to n do b += i * i
    b.result()
  println(squares(4))      // List(1, 4, 9, 16)

val/var vs Mutable/Immutable

These are independent: val/var controls whether the name can be reassigned; the collection type controls whether the contents can change.

Declaration Reassign name? Change contents? Advice
val xs = List(1, 2) No No Default choice
var xs = List(1, 2) Yes (xs = xs :+ 3) No OK for local accumulation
val buf = ArrayBuffer(1, 2) No Yes OK when kept local
var buf = ArrayBuffer(1, 2) Yes Yes Avoid: two kinds of change

Common Pairs

Immutable Mutable counterpart
List / Vector ListBuffer / ArrayBuffer
Map mutable.Map (HashMap), TrieMap (concurrent)
Set mutable.Set
Queue mutable.Queue, ArrayDeque
Array Mutable, fixed size (JVM array)

Convert with buf.toList, xs.to(mutable.ArrayBuffer), m.toMap.

Common Pitfalls

  • Returning a mutable collection from an API; callers can change your internal state. Return toList/toVector.

  • Mixing scala.collection.Seq (could be mutable) with immutable.Seq in Scala 2.12 code; 2.13+ makes Seq immutable by default.

  • Sharing a mutable collection across threads or Spark closures without synchronization.

  • Using var + mutable collection together.

  • Assuming Array is immutable because it is declared with val.

Interview Tips

  • Explain structural sharing; it answers “isn’t immutability slow?”.

  • Separate val/var from collection mutability with the four-case table.

  • Recommend immutable in APIs and local mutable builders for performance.

23

How do you use tuples in Scala?

A tuple groups a fixed number of values of possibly different types into one value, written (a, b, c) with type (A, B, C); it is useful for returning several values from a function, pairing data (zip, map entries k -> v), and quick intermediate results. You read elements with ._1, ._2, …, or more readably by destructuring (val (lo, hi) = minMax(xs)) and pattern matching (case (name, score) => ...). Tuples are immutable and compare by value. Scala 3 extends them: there is no 22-element limit, tuples support generic operations (++, *:, size, take, toList, indexed t(n)), lambdas can take tuple parameters directly (pairs.map((k, v) => ...)), and Scala 3.7 adds named tuples. For data with meaning that travels far, prefer a case class.

Read full answer

Applies toScala 3 (notes for Scala 2.13)

Code Example (Scala 3)

//> using scala 3.5

def minMax(xs: Seq[Int]): (Int, Int) = (xs.min, xs.max)

@main def tupleDemo(): Unit =
  // Create and access
  val person: (String, Int, Double) = ("Asha", 31, 88.5)
  println(person._1)                       // Asha

  // Destructure
  val (name, age, score) = person
  println(s"$name is $age")                // Asha is 31

  // Return several values
  val (lo, hi) = minMax(Seq(4, 1, 9))
  println(lo -> hi)                        // (1,9)    a -> b builds a pair

  // Pairs from zip, lambdas with tuple parameters
  val pairs = List("a", "b", "c").zip(List(1, 2, 3))
  println(pairs)                           // List((a,1), (b,2), (c,3))
  println(pairs.map((k, v) => s"$k=$v").mkString(","))   // a=1,b=2,c=3
  println(pairs.toMap)                     // Map(a -> 1, b -> 2, c -> 3)

  // Sort by a tuple key: count descending, then name ascending
  val counts = List(("b", 2), ("a", 2), ("c", 1))
  println(counts.sortBy((k, v) => (-v, k)))   // List((a,2), (b,2), (c,1))

  // Pattern matching on tuples
  def label(t: (String, Double)): String = t match
    case (_, s) if s >= 90 => "excellent"
    case (n, s)            => s"$n: ${s.toInt}"
  println(label(("Ben", 72.5)))            // Ben: 72

  // Scala 3 generic tuple operations
  val t = (1, "x") ++ (true, 2.5)
  println(t)                               // (1,x,true,2.5)
  println(t.size)                          // 4
  println(t(2))                            // true
  println(0 *: t.take(2))                  // (0,1,x)
  println(t.toList)                        // List(1, x, true, 2.5)

Tuple vs Case Class

Use a tuple when Use a case class when
Values are short-lived (inside a function or a pipeline step) The data crosses API boundaries or is stored
Position is obvious ((key, value), (min, max)) Fields need names (userId, score)
You need a quick sort or group key You want methods, validation or documentation
You work with Spark Datasets (Dataset[Click] is clearer than Dataset[(Long, String, Double)])

Scala 2.13 vs Scala 3

Feature Scala 2.13 Scala 3
Max size 22 (Tuple22) No limit
pairs.map((k, v) => ...) Not allowed; use { case (k, v) => ... } Allowed (parameter untupling)
Generic ops (++, *:, take) No (needs Shapeless) Built in
Named tuples No Scala 3.7+: (name = "Asha", age = 31)

Common Pitfalls

  • Long chains of ._1._2 that nobody can read; destructure or use a case class.

  • Tuples with many fields in public APIs.

  • In Scala 2, forgetting case in map { case (k, v) => ... }.

  • Expecting (1) to be a tuple; it is just 1. A one-element tuple is Tuple1(1).

  • Spark column names _1, _2 from tuple Datasets; rename with .toDF("id", "score").

Interview Tips

  • Show creation, destructuring and pattern matching in one short example.

  • Mention -> for pairs and zip.

  • Explain when to switch to a case class, and the Scala 3 improvements.

24

What role do iterators play in Scala?

An Iterator[A] is a one-pass, lazy cursor over a sequence of elements, defined by two methods, hasNext and next(). Its role is to process data element by element without holding it all in memory: reading large files line by line (Source.getLines()), streaming database or network results, generating infinite sequences, and chaining transformations (map, filter, grouped, sliding, take) that run lazily only when a result is requested. Every Scala collection can produce one with .iterator, and Spark’s mapPartitions hands each partition to your code as an iterator. The trade-off is that an iterator is consumed as it is used: once traversed (even by size or toList) it is empty, so it must not be reused.

Read full answer

Applies toScala 3 and Scala 2.13

Iterator vs Collection vs View

Iterator Strict collection (List, Vector) View / LazyList
Evaluation Lazy Eager Lazy
Passes One Many Many (LazyList memoizes, views recompute)
Memory One element at a time All elements View: none stored; LazyList: computed part
Typical use Files, streams, batches General data Lazy pipelines over existing data

Code Example (Scala 3)

//> using scala 3.5

// A custom iterator that yields mini-batches
final class Batches[A](data: IndexedSeq[A], size: Int) extends Iterator[IndexedSeq[A]]:
  private var pos = 0
  def hasNext: Boolean = pos < data.size
  def next(): IndexedSeq[A] =
    if !hasNext then throw NoSuchElementException("no more batches")
    val batch = data.slice(pos, pos + size)
    pos += size
    batch

@main def iteratorDemo(): Unit =
  // Lazy chaining; consumed once
  val it = Iterator(1, 2, 3, 4, 5)
  val evensSquared = it.filter(_ % 2 == 0).map(x => x * x)
  println(evensSquared.toList)     // List(4, 16)
  println(evensSquared.toList)     // List()   already consumed

  // Work happens only on demand
  val logged = Iterator.from(1).map { x => println(s"computing $x"); x * 10 }
  println(logged.take(2).toList)
  // computing 1
  // computing 2
  // List(10, 20)

  // Infinite source, finite result
  println(Iterator.iterate(1)(_ * 2).takeWhile(_ < 100).toList)   // List(1, 2, 4, 8, 16, 32, 64)

  // Batching and windows
  println((1 to 10).iterator.grouped(4).map(_.sum).toList)    // List(10, 26, 19)
  println(Iterator(1, 2, 3, 4).sliding(2).map(_.sum).toList)  // List(3, 5, 7)

  // Custom iterator
  println(Batches(Vector(1, 2, 3, 4, 5), 2).toList)           // List(Vector(1, 2), Vector(3, 4), Vector(5))

Reading a Large File

import scala.io.Source
import scala.util.Using

// Count error lines without loading the file into memory
val errors: scala.util.Try[Int] = Using(Source.fromFile("app.log")) { src =>
  src.getLines().count(_.contains("ERROR"))     // consume inside Using, before the file closes
}

Iterators in Spark

// mapPartitions receives an Iterator per partition: open an expensive resource once per partition
val scored = df.as[Features].mapPartitions { rows =>
  val model = loadModel()               // once per partition, not per row
  rows.map(r => model.score(r))         // stays lazy: rows are streamed, not collected
}

Calling rows.toList here would load the whole partition into memory and defeat the purpose.

Useful Methods

Method Effect
grouped(n) Batches of n
sliding(n, step) Moving windows
buffered Peek at head without consuming
duplicate Two independent iterators (buffers the difference)
zipWithIndex Pair elements with their position
distinct, takeWhile, dropWhile Lazy filtering

Common Pitfalls

  • Reusing an iterator after toList, size, foreach or mkString: it is empty now.

  • Calling size “just to check” and losing all elements.

  • Returning getLines() from a Using block: the file is closed before the caller reads it.

  • Calling next() without checking hasNext (throws NoSuchElementException).

  • Mixing side effects into lazy map steps and being surprised by when (or whether) they run.

Interview Tips

  • Define it by hasNext/next(), then say “one-pass and lazy”.

  • Give the memory use case: big files and Spark mapPartitions.

  • Mention the consumption pitfall; interviewers often ask what it.size does to it.

25

What is the difference between fold and reduce in Scala?

Both fold and reduce combine all elements of a collection into one value with a binary operation, but reduce uses the first element as the starting value while fold takes an explicit initial value (the “zero”). That gives three practical differences: reduce throws on an empty collection (UnsupportedOperationException) while fold returns the zero; reduce and fold return the element type (or a supertype), while foldLeft/foldRight can return a different type (for example a String, a tuple or a Map); and the plain fold/reduce may combine in any order (which is how parallel collections and Spark use them), so the operator should be associative and the zero an identity, while foldLeft/reduceLeft process strictly left to right. Use reduceOption when the collection may be empty.

Read full answer

Applies toScala 3 and Scala 2.13 (standard collections; the same ideas apply to Spark RDDs)

Comparison

reduce / reduceLeft fold foldLeft / foldRight
Initial value First element Given (zero) Given
Empty collection Throws Returns zero Returns initial value
Result type Element type (or supertype) Element type (or supertype) Any type B
Order reduce: unspecified; reduceLeft: left to right Unspecified Left to right / right to left
Needs associativity reduce: yes Yes (and zero must be identity) No
Safe variant reduceOption - -

Code Example (Scala 3)

//> using scala 3.5

@main def foldReduceDemo(): Unit =
  val xs = List(1, 2, 3, 4)

  println(xs.reduce(_ + _))                       // 10
  println(xs.fold(0)(_ + _))                      // 10
  println(xs.foldLeft("")((acc, x) => acc + x))   // 1234   (result type String)

  // Empty collections
  println(List.empty[Int].fold(0)(_ + _))         // 0
  println(List.empty[Int].reduceOption(_ + _))    // None
  // List.empty[Int].reduce(_ + _)                // throws UnsupportedOperationException

  // Direction matters for non-associative operations
  val ys = List(1, 2, 3)
  println(ys.foldLeft(0)(_ - _))                  // -6   ((0 - 1) - 2) - 3
  println(ys.foldRight(0)(_ - _))                 // 2    1 - (2 - (3 - 0))
  println(ys.reduceLeft(_ - _))                   // -4   (1 - 2) - 3

  // foldLeft builds other structures
  println(xs.foldLeft(List.empty[Int])((acc, x) => x :: acc))   // List(4, 3, 2, 1)
  println(xs.foldRight(List.empty[Int])(_ :: _))                 // List(1, 2, 3, 4)

  // Mean in one pass: accumulate (sum, count)
  val (sum, count) = List(2.0, 4.0, 9.0).foldLeft((0.0, 0)) { case ((s, c), x) => (s + x, c + 1) }
  println(sum / count)                            // 5.0

  // Word counts into a Map
  val counts = "a b a c a".split(" ").foldLeft(Map.empty[String, Int]) { (m, w) =>
    m.updated(w, m.getOrElse(w, 0) + 1)
  }
  println(counts)                                 // Map(a -> 3, b -> 1, c -> 1)

Why Associativity Matters

In parallel collections and Spark, data is split into partitions, each partition is reduced, and the partial results are reduced again. With + that is fine: (1 + 2) + (3 + 4) = 1 + 2 + 3 + 4. With - it is not: (1 − 2) − (3 − 4) = 0, but ((1 − 2) − 3) − 4 = −8. A fold zero that is not an identity is also wrong in parallel: fold(10)(_ + _) adds 10 once per partition.

In Spark

Spark RDD action Behaves like
rdd.reduce(f) reduce: needs a non-empty RDD and associative, commutative f
rdd.fold(zero)(f) fold: zero applied per partition, so it must be an identity
rdd.aggregate(zero)(seqOp, combOp) foldLeft per partition + merge, can change type

Common Pitfalls

  • reduce on a possibly empty list in production code; use reduceOption or fold.

  • Using foldRight on a huge List and expecting stack safety; in 2.13+ it is stack-safe because it reverses first, but it is still slower than foldLeft.

  • Writing fold with a non-identity zero (fold(1)(_ + _)).

  • Using fold when the result type differs (it does not compile); use foldLeft.

Interview Tips

  • Give the three differences: initial value, empty behaviour, result type.

  • Mention associativity and identity for parallel and Spark use.

  • Show foldLeft building a tuple or a Map; it proves practical fluency.

26

How do Scala parallel collections work, and what are their benefits?

Parallel collections let you run bulk operations such as map, filter, fold and sum on several CPU cores by calling .par on a collection: the data is split into chunks (using splitters), each chunk is processed by a task on a fork/join pool, and the partial results are combined. The benefit is easy data parallelism on one machine with almost no code change, for CPU-heavy work on large collections. Results are deterministic only for pure functions and associative combining operations; side effects and non-associative operations give unpredictable results. Since Scala 2.13 they live in a separate module, scala-parallel-collections, imported with scala.collection.parallel.CollectionConverters._.

Read full answer

Applies toScala 2.13 and Scala 3 with `scala-parallel-collections` 1.x

How It Works

Step Detail
.par Wraps the collection as ParSeq, ParVector, ParRange, ParMap… (cheap for arrays, vectors, ranges; List must be copied)
Split A splitter divides the data recursively into chunks
Execute Tasks run on a ForkJoinPool (one worker per core by default), with work stealing
Combine Partial results are merged with combiners (for map) or the given operator (for fold/reduce)
.seq Back to a sequential collection

Code Example (Scala 3)

//> using scala 3.5
//> using dep org.scala-lang.modules::scala-parallel-collections:1.0.4
import scala.collection.parallel.CollectionConverters.*
import scala.collection.parallel.ForkJoinTaskSupport
import java.util.concurrent.ForkJoinPool

@main def parDemo(): Unit =
  val n = 1_000_000

  // Pure function + associative sum: same answer as sequential
  val seqSum = (1 to n).map(x => x.toLong * x).sum
  val parSum = (1 to n).par.map(x => x.toLong * x).sum
  println(seqSum)                // 333333833333500000
  println(parSum == seqSum)      // true

  // fold needs an identity element and an associative operator
  println((1 to 100).par.fold(0)(_ + _))     // 5050

  // Order is preserved for map/filter results on sequences
  println((1 to 10).par.filter(_ % 3 == 0).map(_ * 10).toList)   // List(30, 60, 90)

  // Control the pool size
  val v = (1 to 1000).toVector.par
  v.tasksupport = ForkJoinTaskSupport(ForkJoinPool(2))
  println(v.map(_ * 2).sum)      // 1001000

  // Non-associative operation: the result depends on how the data was split
  println((1 to 10).par.reduce(_ - _))   // NOT reliable: may differ between runs (sequential gives -53)

Σ k² for k = 1..1,000,000 = n(n+1)(2n+1)/6 = 333,333,833,333,500,000.

Benefits and Limits

Benefits Limits
One-word change (.par) Only one machine (use Spark for clusters)
Uses all cores for CPU-heavy work Overhead dominates for small collections or cheap operations
Work stealing balances uneven tasks Side effects and shared state cause races
Results keep sequence order for map/filter Non-associative reduce/fold give wrong or varying results
foldLeft/foldRight stay sequential

Rules for Correct Use

  1. Use pure functions: no var updates, no printing, no I/O inside the operation.

  2. Use associative operators with a true identity in fold (0 for +, 1 for *).

  3. Use aggregate-style patterns when the result type differs (in 2.13, aggregate is deprecated; map to the result type and reduce, or fold with a combine function).

  4. Start from arrays, vectors or ranges, which split cheaply.

  5. Measure: parallel code is often slower for fewer than ~10,000 cheap elements.

Side-Effect Pitfall

var total = 0
(1 to 10_000).par.foreach(x => total += x)   // race: lost updates, result usually wrong
println(total)                               // varies; never rely on it
// correct: (1 to 10_000).par.sum            // 50005000

Parallel Collections vs Alternatives

Tool Scope Best for
Parallel collections One JVM, data parallel CPU-heavy transforms of in-memory data
Future.traverse One JVM, task parallel Independent, possibly I/O-bound tasks
Akka/Pekko Streams One or more JVMs Streaming with back-pressure
Apache Spark Cluster Data larger than one machine

Common Pitfalls

  • Forgetting the module dependency and import in Scala 2.13+ (.par does not compile).

  • Using .par on a List: it is copied into a ParVector first.

  • Mutating shared variables or collections inside parallel operations.

  • Calling blocking I/O inside parallel tasks, starving the shared fork/join pool.

Interview Tips

  • Explain split → process → combine, and the fork/join pool with work stealing.

  • Stress purity and associativity, with reduce(_ - _) as the classic counterexample.

  • Say when you would use Spark or Futures instead.

27

What are Futures in Scala, and how do you use them asynchronously?

A Future[T] is a placeholder for a value of type T that will be computed asynchronously, usually on a thread pool given by an implicit ExecutionContext; it eventually completes with either a value (Success) or an exception (Failure). You start work with Future { ... }, transform results without blocking using map, flatMap and for-comprehensions, handle errors with recover/recoverWith, combine many futures with Future.sequence, Future.traverse or zip, and react with onComplete. Blocking with Await.result should be limited to the edge of the program (a main method or a test). Futures are eager (they start running as soon as they are created) and memoized (the result is computed once).

Read full answer

Applies toScala 3 and Scala 2.13 (`scala.concurrent`)

Key Operations

Operation Purpose
Future { expr } Run expr asynchronously
map, flatMap, for Chain dependent steps without blocking
zip, Future.sequence, Future.traverse Run independent work in parallel and combine
recover, recoverWith, fallbackTo Handle failures
onComplete, foreach Side effects when done
Promise[T] Complete a future manually (bridge callback APIs)
Await.result(f, timeout) Block (only at program edges)

Code Example (Scala 3)

//> using scala 3.5
import scala.concurrent.{Await, Future, Promise}
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration.*
import scala.util.{Failure, Success}

def fetchFeatures(userId: Int): Future[Vector[Double]] =
  Future { Vector(userId.toDouble, 2.0) }          // e.g. a feature-store call

def score(features: Vector[Double]): Future[Double] =
  Future { features.sum }                          // e.g. a model call

@main def futuresDemo(): Unit =
  // 1. Sequential, dependent steps with a for-comprehension
  val single: Future[Double] =
    for
      f <- fetchFeatures(5)
      s <- score(f)
    yield s
  println(Await.result(single, 2.seconds))         // 7.0

  // 2. Independent work in parallel: start both futures first
  val fa = fetchFeatures(1)
  val fb = fetchFeatures(2)
  val both = for a <- fa; b <- fb yield a.sum + b.sum
  println(Await.result(both, 2.seconds))           // 7.0   (3.0 + 4.0)

  // 3. Many items: traverse runs them concurrently and keeps the input order
  val batch = Future.traverse(List(1, 2, 3))(id => fetchFeatures(id).flatMap(score))
  println(Await.result(batch, 2.seconds))          // List(3.0, 4.0, 5.0)

  // 4. Error handling
  val failing: Future[Double] = Future.failed(RuntimeException("feature store timeout"))
  val safe = failing.recover { case _: RuntimeException => 0.0 }
  println(Await.result(safe, 2.seconds))           // 0.0

  failing.onComplete {
    case Success(v) => println(s"got $v")
    case Failure(e) => println(s"failed: ${e.getMessage}")   // failed: feature store timeout
  }

  // 5. Promise: complete a future from a callback-style API
  val p = Promise[String]()
  Future { p.success("done") }
  println(Await.result(p.future, 2.seconds))       // done

  Thread.sleep(100)   // demo only: give the onComplete callback time to print before exit

Choosing the ExecutionContext

Work ExecutionContext
CPU-bound, short tasks ExecutionContext.global (sized to CPU cores)
Blocking I/O (JDBC, file) Dedicated pool: ExecutionContext.fromExecutor(Executors.newFixedThreadPool(32)), or wrap in blocking { }
JDK 21 blocking calls Virtual-thread executor: Executors.newVirtualThreadPerTaskExecutor()

Futures vs Alternatives

Tool Eager? Cancellable? Notes
Future Yes No In the standard library; simple async
Cats Effect IO / ZIO No (lazy description) Yes Resource safety, retries, structured concurrency
Akka/Pekko Streams Lazy until run Yes Back-pressured pipelines

Common Pitfalls

  • Writing for a <- fetchFeatures(1); b <- fetchFeatures(2) and expecting parallelism: the second call starts only after the first finishes. Create both futures first.

  • Calling Await.result inside other futures or actors (thread starvation, possible deadlock).

  • Running blocking calls on the global pool without blocking.

  • Forgetting that exceptions inside a Future do not propagate unless you handle the Failure.

  • Expecting a Future to be lazy or restartable; it runs once when created.

Interview Tips

  • Explain eager start, memoization and the role of the ExecutionContext.

  • Show the sequential vs parallel for-comprehension difference; it is a common trap.

  • Mention traverse/sequence and recover as everyday tools.

28

What is the difference between an actor and a thread in Scala?

A thread is an execution resource: a JVM (platform) thread maps to an OS thread with its own stack (often about 1 MB), is scheduled preemptively by the OS, and usually shares memory with other threads, so shared state must be protected with locks. An actor is a programming model on top of threads: a lightweight object (hundreds of bytes) with private state and a mailbox that processes one message at a time, scheduled by a dispatcher on a small thread pool. Millions of actors can share a handful of threads, and because no other code touches an actor’s state, it needs no locks. Actors also add supervision (restart on failure) and location transparency (the same ActorRef can point to an actor on another machine). Threads give direct control and the simplest model for a few blocking tasks; actors fit many independent, stateful, message-driven entities.

Read full answer

Applies toScala 3 / 2.13 on the JVM; actors from Akka or Apache Pekko (typed API)

Comparison

Aspect Thread Actor
What it is Unit of execution (OS-scheduled) Unit of state + behaviour (message-driven)
Cost ~1 MB stack, costly to create Hundreds of bytes; millions possible
Scheduling Preemptive, by the OS Runs when it has messages, on a dispatcher thread pool
Communication Shared memory Asynchronous immutable messages
Synchronization Locks, atomics, volatile Not needed for its own state (one message at a time)
Failure handling Uncaught exception kills the thread Supervisor restarts, resumes or stops the actor
Distribution Single JVM Can be remote / clustered
Blocking Acceptable (it is its own thread) Should not block; it holds a shared thread
Ordering No ordering between threads Messages from one sender arrive in order

Code Example: Thread with Shared State (Scala 3)

//> using scala 3.5
import java.util.concurrent.atomic.AtomicInteger

@main def threadDemo(): Unit =
  val processed = AtomicInteger(0)                 // shared, so it must be thread-safe
  val workers = (1 to 4).map { id =>
    Thread(() => for _ <- 1 to 250 do processed.incrementAndGet(), s"worker-$id")
  }
  workers.foreach(_.start())
  workers.foreach(_.join())
  println(processed.get)                           // 1000

Code Example: Actor Owning Its State (Scala 3, Pekko Typed)

//> using scala 3.5
//> using dep org.apache.pekko::pekko-actor-typed:1.1.2
import org.apache.pekko.actor.typed.{ActorRef, ActorSystem, Behavior, Scheduler}
import org.apache.pekko.actor.typed.scaladsl.AskPattern.*
import org.apache.pekko.actor.typed.scaladsl.Behaviors
import org.apache.pekko.util.Timeout
import scala.concurrent.Await
import scala.concurrent.duration.*

object Tracker:
  sealed trait Msg
  case object Processed extends Msg
  final case class Report(replyTo: ActorRef[Int]) extends Msg

  def apply(count: Int = 0): Behavior[Msg] = Behaviors.receiveMessage {
    case Processed => apply(count + 1)            // plain Int: no lock needed
    case Report(replyTo) =>
      replyTo ! count
      Behaviors.same
  }

@main def actorDemo(): Unit =
  val system = ActorSystem(Tracker(), "tracker")
  given Timeout = 3.seconds
  given Scheduler = system.scheduler

  for _ <- 1 to 1000 do system ! Tracker.Processed
  println(Await.result(system.ask(Tracker.Report(_)), 3.seconds))   // 1000
  system.terminate()

The thread version needs an AtomicInteger because four threads write the same counter. The actor version keeps a plain Int because only the actor reads or writes it.

Virtual Threads (JDK 21)

Virtual threads are cheap JVM-managed threads (also thousands to millions), which narrows the cost gap. They make blocking code scale, but they still share memory and need locks; actors still add message-based isolation, supervision and distribution.

When to Use Which

Use threads (or a thread pool / virtual threads) when Use actors when
A few long-running or blocking tasks Many independent stateful entities (users, sessions, devices)
Simple parallel computation Systems that must recover from failures automatically
You need direct control of scheduling State must be distributed across a cluster
Code uses blocking libraries Message-driven workflows and event processing

Common Pitfalls

  • Blocking inside an actor (e.g. Thread.sleep, JDBC), which takes a thread away from other actors.

  • Exposing an actor’s internal mutable state to a Future callback.

  • Creating one platform thread per request: memory and context-switch cost grow quickly.

  • Treating ask as a function call and building long chains of blocking asks.

Interview Tips

  • Say “thread = how code runs; actor = how state and behaviour are organized”.

  • Mention cost, scheduling, communication and failure handling as the four differences.

  • Bring up virtual threads to show current JVM knowledge.

29

How does Scala handle synchronization and concurrency control?

Scala runs on the JVM, so it has all of Java’s concurrency control, plus higher-level tools that avoid shared mutable state: low-level synchronized blocks (intrinsic locks), @volatile fields for visibility, atomic variables (AtomicLong, LongAdder), locks (ReentrantLock, ReadWriteLock), and concurrent collections (ConcurrentHashMap, Scala’s lock-free TrieMap); and at a higher level immutable data (safe to share without locks), Futures for composing asynchronous work, actors (Akka/Pekko) that own state and process one message at a time, and effect systems (Cats Effect Ref, ZIO Ref/STM) with atomic updates. Idiomatic Scala prefers the high-level tools and keeps any locking small and local.

Read full answer

Applies toScala 3 and Scala 2.13 on JDK 17 / 21

Tools by Level

Level Tool Use
Avoid sharing Immutable values, case classes Share freely between threads
Atomic update AtomicLong, AtomicReference, LongAdder Counters, flags, compare-and-set
Mutual exclusion synchronized, ReentrantLock Several fields updated together
Visibility @volatile var One writer, many readers of a flag
Concurrent data TrieMap, ConcurrentHashMap Shared caches and counters
Composition Future, Promise Async pipelines
Ownership Actors Stateful services, many entities
Effects Cats Effect / ZIO Ref, ZIO STM Functional code with atomic state

Code Example (Scala 3)

//> using scala 3.5
import java.util.concurrent.atomic.AtomicLong
import scala.collection.concurrent.TrieMap

def runThreads(n: Int)(body: => Unit): Unit =
  val threads = (1 to n).map(_ => Thread(() => body))
  threads.foreach(_.start())
  threads.foreach(_.join())

final class SyncCounter:
  private var n = 0L
  def inc(): Unit = synchronized { n += 1 }      // lock on this object
  def get: Long = synchronized { n }

@main def syncDemo(): Unit =
  val perThread = 100_000

  // 1. Broken: unsynchronized read-modify-write loses updates
  var unsafe = 0L
  runThreads(4) { for _ <- 1 to perThread do unsafe += 1 }
  println(s"unsafe: $unsafe")       // usually less than 400000, varies run to run

  // 2. synchronized
  val sc = SyncCounter()
  runThreads(4) { for _ <- 1 to perThread do sc.inc() }
  println(s"synchronized: ${sc.get}")   // synchronized: 400000

  // 3. Atomic (lock-free compare-and-set)
  val atomic = AtomicLong(0)
  runThreads(4) { for _ <- 1 to perThread do atomic.incrementAndGet() }
  println(s"atomic: ${atomic.get}")      // atomic: 400000

  // 4. Concurrent map of atomic counters
  val counts = TrieMap.empty[String, AtomicLong]
  val words = Seq("spark", "scala", "spark")
  runThreads(4) {
    for _ <- 1 to 1000; w <- words do counts.getOrElseUpdate(w, AtomicLong(0)).incrementAndGet()
  }
  println(counts.view.mapValues(_.get).toList.sorted)   // List((scala,4000), (spark,8000))

TrieMap.getOrElseUpdate is atomic (since Scala 2.13), so two threads never create two counters for the same word.

@volatile for Flags

final class Worker:
  @volatile private var running = true    // writes become visible to other threads
  def stop(): Unit = running = false
  def loop(): Unit = while running do Thread.onSpinWait()

@volatile guarantees visibility, not atomicity: count += 1 on a volatile field is still a race.

Higher-Level Alternative: Immutable State + Single Writer

import scala.concurrent.{Await, Future}
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration.*

// Each task returns its own partial result; nothing is shared or locked
val partials: Future[Seq[Long]] = Future.traverse(1 to 4)(_ => Future((1 to 100_000).map(_.toLong).sum))
println(Await.result(partials.map(_.sum), 10.seconds))   // 20000200000

Each of the 4 tasks computes 1 + … + 100,000 = 5,000,050,000, so the total is 20,000,200,000.

Common Pitfalls

  • Holding a lock while doing I/O or calling unknown code: slow, and a deadlock risk.

  • Locking on different objects for the same data, so the lock protects nothing.

  • Acquiring two locks in different orders in different threads (deadlock).

  • Using @volatile and assuming += is atomic.

  • Blocking threads of the global ExecutionContext (wrap with blocking {} or use a dedicated pool).

  • Mutating driver-side variables from Spark closures; use accumulators or return values instead.

Interview Tips

  • Start with “avoid shared mutable state”, then list the JVM primitives.

  • Show you know the difference between visibility (@volatile) and atomicity (atomics, locks).

  • Mention actors and effect-system Refs as the idiomatic high-level choices.

30

What is the Akka framework, and how does it relate to Scala?

Akka is a toolkit for building concurrent, distributed and resilient applications on the JVM with the actor model, and it is written in Scala (with Scala and Java APIs). An actor is a lightweight object with private state and a mailbox that processes one message at a time, so you get concurrency without locks; actors form supervision hierarchies that restart failed children (“let it crash”); and the same message-passing code scales from one machine to a cluster (Cluster, Sharding, Persistence). Around the core, Akka Streams gives back-pressured stream processing and Akka HTTP builds REST/gRPC services. Akka replaced Scala’s original scala.actors library (deprecated in Scala 2.10). Since 2022, Akka 2.7+ uses the Business Source License; Apache Pekko is the Apache-licensed fork of Akka 2.6 with the same API under org.apache.pekko.

Read full answer

Applies toAkka 2.6+ / Apache Pekko 1.x, typed actor API, Scala 2.13 and Scala 3

Akka Modules

Module Purpose ML / data use
Actors (Typed) Message-driven concurrency, supervision Model-serving workers, coordinators
Cluster + Sharding Distribute actors by key across nodes Per-user state, online features
Persistence Event-sourced actors with snapshots Audit trails, stateful pipelines
Streams Back-pressured stream processing (Source → Flow → Sink) Feature pipelines, batching predictions
HTTP Server and client Model inference APIs
Alpakka Connectors (Kafka, S3, JDBC…) Ingestion

Code Example: A Typed Actor (Scala 3, Pekko)

//> using scala 3.5
//> using dep org.apache.pekko::pekko-actor-typed:1.1.2
import org.apache.pekko.actor.typed.{ActorRef, ActorSystem, Behavior, Scheduler}
import org.apache.pekko.actor.typed.scaladsl.AskPattern.*
import org.apache.pekko.actor.typed.scaladsl.Behaviors
import org.apache.pekko.util.Timeout
import scala.concurrent.Await
import scala.concurrent.duration.*

object Counter:
  sealed trait Command
  final case class Add(n: Int) extends Command
  final case class Get(replyTo: ActorRef[Int]) extends Command

  // State lives in the behavior; a new behavior is returned for each message
  def apply(total: Int = 0): Behavior[Command] =
    Behaviors.receiveMessage {
      case Add(n) =>
        apply(total + n)
      case Get(replyTo) =>
        replyTo ! total
        Behaviors.same
    }

@main def actorDemo(): Unit =
  val system = ActorSystem(Counter(), "counter")
  given Timeout = 3.seconds
  given Scheduler = system.scheduler

  (1 to 100).foreach(i => system ! Counter.Add(i))   // fire-and-forget messages
  val total = Await.result(system.ask(ref => Counter.Get(ref)), 3.seconds)
  println(total)            // 5050
  system.terminate()

Messages from one sender to one actor arrive in order, so the Get is processed after all 100 Adds. The actor’s total is never shared, so no locks are needed even though the system runs on a thread pool. With Akka, the same code uses the akka.actor.typed packages.

Code Example: Akka/Pekko Streams for Batched Scoring

import org.apache.pekko.stream.scaladsl.{Sink, Source}

// Batch requests into groups of up to 32 (or whatever arrived within 10 ms), score each batch
Source(1 to 1000)
  .groupedWithin(32, 10.millis)
  .mapAsync(parallelism = 4)(batch => scoreBatch(batch))   // scoreBatch: Seq[Int] => Future[Seq[Double]]
  .runWith(Sink.ignore)                                     // needs a given ActorSystem in scope

Back-pressure means a slow model slows the source instead of overflowing memory.

How Akka Relates to Scala

Point Detail
Language Implemented in Scala; idiomatic Scala API with case-class messages and pattern matching
History Replaced scala.actors; part of the Lightbend (Typesafe) stack with Scala and Play
Typed API Uses Scala’s type system: an ActorRef[Command] accepts only Command messages
Spark Spark used Akka for RPC until version 2.0, then replaced it with its own Netty-based RPC
License Akka 2.7+ BSL (commercial license for larger companies); Pekko is Apache 2.0

Common Pitfalls

  • Blocking inside an actor (JDBC calls, Await) and starving the dispatcher; use a separate dispatcher or pipeToSelf with a Future.

  • Sending mutable objects as messages; messages should be immutable case classes.

  • Closing over actor state in a Future callback (it runs on another thread).

  • Using actors for simple request/response work where a Future is enough.

  • Ignoring the license change when choosing between Akka and Pekko.

Interview Tips

  • Describe the actor model in one sentence: private state, mailbox, one message at a time.

  • Mention supervision, clustering and Streams with back-pressure.

  • Show you know the Akka → Pekko licensing situation.

32

How does the Breeze library support scientific computing in Scala?

Breeze is Scala’s main numerical computing library, giving NumPy/SciPy-like functionality on the JVM: dense and sparse vectors and matrices (DenseVector, DenseMatrix, SparseVector, CSCMatrix) with natural operators (+, *, dot, </code> for solving), linear algebra (inverse, determinant, eigen-decomposition, SVD, Cholesky) backed by native BLAS/LAPACK when available, element-wise numeric functions through universal functions (<code>sqrt, exp, log, sigmoid), statistics (mean, variance, stddev, histograms), probability distributions with seeded random sources, and optimization (L-BFGS, OWL-QN, SGD) that works with any differentiable function. Spark MLlib itself uses Breeze internally.

Read full answer

Applies toBreeze 2.x (Scala 2.12 / 2.13 / 3)

Main Modules

Package Provides Example
breeze.linalg Vectors, matrices, slicing, inv, det, eig, svd, “ m b solves m · x = b
breeze.numerics Element-wise functions sqrt(v), exp(v), sigmoid(v)
breeze.stats mean, variance, stddev, median, histograms meanAndVariance(v)
breeze.stats.distributions Gaussian, Poisson, Beta, Dirichlet, sampling Gaussian(0, 1).sample(5)
breeze.optimize L-BFGS, OWL-QN, gradient checks LBFGS.minimize(f, x0)
breeze.signal FFT, filters fourierTr(v)

Code Example (Scala 3)

//> using scala 3.5
//> using dep org.scalanlp::breeze:2.1.0
import breeze.linalg.*
import breeze.numerics.sqrt
import breeze.optimize.{DiffFunction, LBFGS}
import breeze.stats.{mean, variance}
import breeze.stats.distributions.{Gaussian, RandBasis}

@main def breezeDemo(): Unit =
  // Vectors and element-wise operations
  val v = DenseVector(1.0, 2.0, 3.0)
  println(v * 2.0)                    // DenseVector(2.0, 4.0, 6.0)
  println(v dot v)                    // 14.0
  println(sqrt(v))                    // DenseVector(1.0, 1.4142135623730951, 1.7320508075688772)
  println(mean(v))                    // 2.0
  println(variance(v))                // 1.0  (sample variance, n - 1)

  // Matrices and linear algebra
  val m = DenseMatrix((2.0, 0.0), (0.0, 4.0))
  println(m * DenseVector(1.0, 1.0))  // DenseVector(2.0, 4.0)
  println(m \ DenseVector(2.0, 8.0))  // DenseVector(1.0, 2.0)   solves m * x = b
  println(det(m))                     // 8.0
  println(inv(m)(1, 1))               // 0.25

  // Slicing
  val big = DenseMatrix.tabulate(3, 3)((i, j) => (i * 3 + j).toDouble)   // 0..8 row by row
  println(big(::, 1))                 // DenseVector(1.0, 4.0, 7.0)   column 1
  println(sum(big))                   // 36.0

  // Optimization: minimize (x - 3)^2 + (y + 1)^2 with L-BFGS
  val f = new DiffFunction[DenseVector[Double]]:
    def calculate(x: DenseVector[Double]): (Double, DenseVector[Double]) =
      val d = x - DenseVector(3.0, -1.0)
      (d dot d, d * 2.0)              // value and gradient

  val lbfgs = new LBFGS[DenseVector[Double]](maxIter = 100, m = 5)
  val best = lbfgs.minimize(f, DenseVector.zeros[Double](2))
  println(best.map(x => math.round(x * 1000) / 1000.0))   // DenseVector(3.0, -1.0)

  // Reproducible random numbers
  given RandBasis = RandBasis.withSeed(42)
  val samples = Gaussian(0.0, 1.0).sample(1000)
  println(samples.size)               // 1000

The L-BFGS result is rounded because the optimizer stops within a tolerance of the exact minimum (3, −1).

Breeze vs NumPy

Task Breeze NumPy
Create vector DenseVector(1.0, 2.0) np.array([1.0, 2.0])
Matrix product a * b a @ b
Element-wise product a : b a * b
Solve linear system a b np.linalg.solve(a, b)
Column slice m(::, 1) m[:, 1]
Transpose m.t m.T

Note the operator difference: in Breeze * between matrices is matrix multiplication, and : is element-wise.

Performance Notes

  • Breeze 2.x uses the dev.ludovic.netlib BLAS/LAPACK bindings: it calls native libraries (OpenBLAS, MKL) if they are installed and falls back to a pure-JVM implementation otherwise.

  • DenseVector/DenseMatrix are mutable and backed by primitive arrays; in-place operators (+=, :*=) avoid allocation.

  • DenseMatrix is column-major, like Fortran/LAPACK.

Common Pitfalls

  • Confusing * (matrix product) with : (element-wise).

  • Sharing a mutable DenseVector between threads or storing it after an in-place update.

  • Forgetting a seeded RandBasis, so random results change between runs.

  • Mixing Breeze versions with Spark’s bundled Breeze, causing classpath conflicts; align to Spark’s version in Spark jobs.

  • Computing inv(a) * b instead of a \ b (slower and less numerically stable).

Interview Tips

  • Describe Breeze as “NumPy + SciPy for Scala” and name linalg, stats, distributions and optimize.

  • Mention native BLAS support and column-major storage.

  • Point out that Spark MLlib uses Breeze internally, which is a common follow-up.

33

What are Saddle and Algebird, and how do they help ML in Scala?

Saddle and Algebird are two Scala libraries that cover different parts of data work: Saddle is a data-manipulation library (similar to pandas) with indexed, primitive-backed Vec, Mat, Series and Frame types, and Algebird (from Twitter) is an abstract-algebra library that provides Semigroup, Monoid and Group instances plus approximate streaming data structures such as HyperLogLog, Count-Min Sketch, Bloom filters, Q-Tree quantiles and MinHash. For ML operations, Saddle helps with in-memory feature tables and aligned time series on one machine, while Algebird’s main contribution is associative aggregation: because monoids can be combined in any grouping, the same aggregation code runs unchanged in batch (Spark, Scalding), in streaming and in parallel, which makes it well suited for computing features and statistics over huge data with bounded memory.

Read full answer

Applies toAlgebird 0.13.x (Scala 2.12 / 2.13, usable from Spark jobs); Saddle (original project and the maintained `io.github.pityka` fork). Both are niche today: Spark DataFrames and Breeze cover most of Saddle's use cases.

Saddle at a Glance

Type Like Use
Vec[T] NumPy 1-D array Primitive-backed vector with stats (mean, stdev)
Mat[T] NumPy 2-D array Dense matrix
Series[K, V] pandas Series Values with an index; automatic alignment on joins
Frame[R, C, V] pandas DataFrame Rows and columns with indexes

Typical uses: aligning time series by timestamp, quick in-memory analysis, preparing feature matrices for a single-machine model.

Algebird at a Glance

Structure Answers Memory
AveragedValue, Moments Mean, variance, skewness Constant
Max, Min, TopK Extremes and top items Constant / k
HyperLogLog Approximate count of distinct items A few KB
CountMinSketch Approximate frequency of items (heavy hitters) Fixed width × depth
BloomFilter “Have we seen this item?” (no false negatives) Fixed bits
QTree Approximate quantiles Bounded
MinHash Approximate Jaccard similarity Fixed signature
DecayedValue Exponentially decayed averages for recency features Constant

Why Monoids Matter for ML Pipelines

A monoid is a type with an associative plus and an identity zero. Associativity means (a + b) + c == a + (b + c), so:

  • Partitions can be summarized independently and merged (map-side combine in Spark or MapReduce).

  • Streaming jobs can keep a running summary and merge late data.

  • Daily summaries can be added up into weekly or monthly features without re-reading raw data.

Code Example: Algebird (Scala 2.13)

// build.sbt: libraryDependencies += "com.twitter" %% "algebird-core" % "0.13.10"
import com.twitter.algebird._

object AlgebirdDemo {
  def main(args: Array[String]): Unit = {
    // Exact mean via a monoid: partitions can be averaged separately, then merged
    val part1 = Monoid.sum(Seq(2.0, 4.0).map(x => AveragedValue(x)))
    val part2 = Monoid.sum(Seq(9.0).map(x => AveragedValue(x)))
    val merged = Semigroup.plus(part1, part2)
    println(merged.value)    // 5.0   (same as averaging 2, 4, 9 at once)
    println(merged.count)    // 3

    // Approximate distinct users with HyperLogLog (12 bits: ~1.6% standard error)
    val hll = new HyperLogLogMonoid(bits = 12)
    val events = Seq("u1", "u2", "u1", "u3", "u2")
    val sketch = hll.sum(events.map(u => hll.create(u.getBytes("UTF-8"))))
    println(sketch.approximateSize.estimate)   // approximate number of distinct users
  }
}

In Spark, the same sketches are built per partition with mapPartitions or aggregate and merged with the monoid’s plus, so a distinct-count feature over billions of events needs only kilobytes per key.

Where They Fit Today

Need Saddle / Algebird Common modern choice
Tabular manipulation Saddle Frame Spark DataFrame (any size), Smile DataFrame
Linear algebra Saddle Mat Breeze
Mergeable sketches and aggregations Algebird Algebird, Spark’s built-in approx_count_distinct, Apache DataSketches
Streaming aggregations Algebird (Summingbird, Scalding) Flink / Spark Structured Streaming with sketch libraries

Common Pitfalls

  • Treating HyperLogLog or Count-Min results as exact; report them with their error bounds.

  • Merging sketches built with different parameters (bits, width, depth, seeds): the results are invalid.

  • Choosing Saddle for data that does not fit on one machine.

  • Picking a library without checking that it is built for your Scala version (Algebird and Saddle lag behind Scala 3).

Interview Tips

  • Explain monoids in one sentence and why associativity enables parallel and incremental aggregation.

  • Name two Algebird sketches and the question each answers.

  • Be honest that both libraries are niche now, and name the mainstream alternatives.

34

Explain inheritance in Scala.

Inheritance in Scala lets a class reuse and specialize another type’s members with extends: a class can extend one class (abstract or concrete) and mix in any number of traits, overriding inherited members with the override keyword and calling the parent’s version with super. Subclasses can be used wherever the parent type is expected (subtype polymorphism), and method calls are dispatched on the runtime type. Scala adds controls Java lacks or makes optional: override is mandatory when replacing a concrete member, final stops overriding or subclassing, sealed restricts subclasses to one file so pattern matching is exhaustiveness-checked, and linearization gives multiple trait inheritance a defined order. Case classes should not extend other case classes; idiomatic Scala often prefers sealed traits / enums plus composition over deep class hierarchies.

Read full answer

Applies toScala 3 (Scala 2.13 is the same apart from syntax)

Keywords

Keyword Meaning
extends Inherit from a class or trait (Scala 3 also extends A, B for traits)
with Mix in additional traits
override Required when replacing a concrete member
super.m Call the parent’s (or next trait’s) implementation
final No further overriding (member) or subclassing (class)
sealed Subclasses only in the same file; exhaustive matching
open (Scala 3) Marks a class intended for extension in other files
protected Visible to subclasses

Code Example (Scala 3)

//> using scala 3.5

sealed abstract class Activation(val name: String):
  def apply(x: Double): Double
  def describe: String = s"$name(1.0) = ${apply(1.0)}"

object ReLU extends Activation("relu"):
  def apply(x: Double): Double = math.max(0.0, x)

final class LeakyReLU(alpha: Double) extends Activation("leaky_relu"):
  def apply(x: Double): Double = if x >= 0 then x else alpha * x
  override def describe: String = s"${super.describe}, alpha = $alpha"   // extend parent's version

final class Identity extends Activation("identity"):
  def apply(x: Double): Double = x

// Exhaustive match: the compiler knows every subclass because Activation is sealed
def isPiecewiseLinear(a: Activation): Boolean = a match
  case ReLU         => true
  case _: LeakyReLU => true
  case _: Identity  => true

@main def inheritanceDemo(): Unit =
  val acts: List[Activation] = List(ReLU, LeakyReLU(0.1), Identity())

  println(acts.map(_(-2.0)))           // List(0.0, -0.2, -2.0)   dynamic dispatch
  println(acts.map(_.name))            // List(relu, leaky_relu, identity)
  println(LeakyReLU(0.1).describe)     // leaky_relu(1.0) = 1.0, alpha = 0.1
  println(acts.forall(isPiecewiseLinear))   // true

acts.map(_(-2.0)) calls each object’s own apply, even though the list’s type is List[Activation]: that is subtype polymorphism.

Constructors and Initialization

class Base(val id: Int):
  println(s"Base init $id")

class Child(id: Int, val tag: String) extends Base(id):   // pass arguments to the parent
  println(s"Child init $tag")

// Child(1, "x") prints:
// Base init 1
// Child init x

The parent’s constructor body runs first, then the child’s. Do not call overridable methods from a parent constructor: the child’s fields are not initialized yet.

Inheritance vs Composition

Prefer inheritance when Prefer composition when
There is a true “is-a” relation (LeakyReLU is an Activation) It is a “has-a” relation (a Model has an Optimizer)
You need a closed set of variants (sealed hierarchy / enum) You want to swap parts at runtime or in tests
Behaviour is shared and stable Behaviour varies independently along several dimensions

Rules Specific to Scala

  • Case-to-case inheritance is prohibited: a case class cannot extend another case class (equality and copy would break). Use a sealed trait parent.

  • Multiple inheritance of traits is linearized: the rightmost trait wins, and super follows the linearization.

  • val can override a parameterless def, but not the reverse.

Common Pitfalls

  • Forgetting override and getting a compile error, or adding it to a member that does not exist in the parent (also an error, which is a useful check).

  • Deep hierarchies that make behaviour hard to find; Scala code usually keeps hierarchies to one or two levels.

  • Extending a class from a library that was not designed for it (Scala 3 flags this as an “adhoc extension” under -source future unless the class is open).

  • Leaving a hierarchy unsealed and losing exhaustiveness checks in pattern matches.

Interview Tips

  • Mention single class inheritance plus multiple trait mixins and linearization.

  • Explain sealed and why it matters for pattern matching.

  • Say when you would choose composition over inheritance, with a concrete example.

35

How do abstract classes work in Scala?

An abstract class is a class declared with abstract that cannot be instantiated directly; it can contain abstract members (declared without a body), concrete members, fields and constructor parameters, and subclasses must implement every abstract member before they can be created. Scala uses abstract classes for a shared base with state and constructor arguments, such as a model base class that stores a name and provides a concrete evaluate method built on abstract fit/predict. A class can extend only one abstract class but can mix in many traits. Since Scala 3 traits can also take parameters, the main reasons left to choose an abstract class are Java interoperability (Java code can extend it like any Java abstract class), a single clear base type, and constructor-order guarantees.

Read full answer

Applies toScala 3 (Scala 2.13 is the same apart from syntax)

Code Example (Scala 3)

//> using scala 3.5

abstract class Regressor(val name: String):
  // abstract members: no body
  def fit(xs: Seq[Double], ys: Seq[Double]): Regressor
  def predict(x: Double): Double

  // concrete member built on the abstract ones (template method)
  def mse(xs: Seq[Double], ys: Seq[Double]): Double =
    xs.lazyZip(ys).map((x, y) => math.pow(predict(x) - y, 2)).sum / xs.size

final class MeanRegressor(mean: Double = 0.0) extends Regressor("mean"):
  def fit(xs: Seq[Double], ys: Seq[Double]): MeanRegressor = MeanRegressor(ys.sum / ys.size)
  def predict(x: Double): Double = mean

final class SlopeRegressor(slope: Double = 0.0) extends Regressor("slope"):
  // least squares line through the origin: slope = Σxy / Σx²
  def fit(xs: Seq[Double], ys: Seq[Double]): SlopeRegressor =
    SlopeRegressor(xs.lazyZip(ys).map(_ * _).sum / xs.map(x => x * x).sum)
  def predict(x: Double): Double = slope * x

@main def abstractDemo(): Unit =
  val xs = Seq(1.0, 2.0, 3.0)
  val ys = Seq(2.0, 4.0, 6.5)

  val models: List[Regressor] = List(MeanRegressor(), SlopeRegressor())
  for m <- models do
    val fitted = m.fit(xs, ys)
    println(f"${fitted.name}%-5s MSE = ${fitted.mse(xs, ys)}%.3f")
  // mean  MSE = 3.389
  // slope MSE = 0.030

  // new Regressor("x")   // does not compile: Regressor is abstract

The mean model predicts 12.5 / 3 ≈ 4.167 for every x; the slope model fits 29.5 / 14 ≈ 2.107 and is far better on this data. Each subclass overrides fit with a more specific return type, which is allowed (covariant return types).

What an Abstract Class Can Contain

Member Example
Constructor parameters abstract class Regressor(val name: String)
Abstract method def predict(x: Double): Double
Abstract field / type val featureCount: Int, type Input
Concrete methods and fields def mse(...), protected val log = ...
final methods Cannot be overridden by subclasses

Abstract Class vs Trait

Abstract class Trait
Constructor parameters Yes Yes in Scala 3; no in Scala 2
Multiple inheritance One per class Many per class
Java can extend it Yes, directly As an interface (with default methods)
Stackable abstract override No Yes
Typical use Base type with state and a clear constructor Capabilities and mixins

Rule of thumb: start with a trait; use an abstract class when Java code must extend it, when you need a single base with constructor parameters in Scala 2, or when initialization order matters.

Sealed Abstract Classes

sealed abstract class Shape limits subclasses to the same file, so pattern matches over the subclasses are checked for exhaustiveness. In Scala 3, a closed hierarchy is often written as an enum instead.

Common Pitfalls

  • Calling an abstract method in the abstract class’s constructor body; the subclass fields are not initialized yet.

  • Using an abstract class just to share helper methods; a trait or an object is simpler.

  • Forgetting override is optional for implementing abstract members but required for overriding concrete ones.

  • Deep inheritance chains that make behaviour hard to trace; prefer shallow hierarchies plus composition.

Interview Tips

  • Name the two differences from traits: single inheritance and Java-friendly constructors (Scala 3 removed the parameter difference).

  • Show the template-method pattern: concrete mse built on abstract predict.

  • Mention sealed for closed hierarchies and exhaustive matching.

36

What are companion objects in Scala?

A companion object is an object with the same name as a class (or trait) and defined in the same source file; the two can access each other’s private members. Scala has no static keyword, so the companion holds what Java would make static: factory methods (apply), constants, validation logic for smart constructors, extractors (unapply), and type-class instances (given/implicit). Givens placed in a companion are found automatically through the type’s implicit scope, without an import. Every case class gets a compiler-generated companion with apply and unapply, which is why you can write Point(1, 2) and pattern-match on it.

Read full answer

Applies toScala 3 (Scala 2.13 is the same apart from syntax)

What Goes in a Companion

Member Example Why
Factory / smart constructor def apply(...), def fromCelsius(...) Validate input, hide the constructor
Constants val AbsoluteZero One shared instance
Extractor def unapply(...) Pattern matching
Type-class instances given Ordering[Temperature] Found with no import
Helpers Parsing, defaults Grouped with the type

Code Example (Scala 3)

//> using scala 3.5

final class Temperature private (val kelvin: Double):   // constructor is private
  def celsius: Double = kelvin - 273.15
  override def toString: String = f"Temperature($kelvin%.2f K)"

object Temperature:
  val AbsoluteZero: Temperature = new Temperature(0.0)   // companion can call the private constructor

  def apply(kelvin: Double): Temperature =
    require(kelvin >= 0, s"kelvin must be >= 0, got $kelvin")
    new Temperature(kelvin)

  def fromCelsius(c: Double): Either[String, Temperature] =
    if c < -273.15 then Left(s"below absolute zero: $c")
    else Right(new Temperature(c + 273.15))

  // Found automatically when sorting Temperatures: no import needed
  given Ordering[Temperature] = Ordering.by(_.kelvin)

@main def companionDemo(): Unit =
  println(Temperature.fromCelsius(25))          // Right(Temperature(298.15 K))
  println(Temperature.fromCelsius(-300))        // Left(below absolute zero: -300.0)
  println(Temperature.AbsoluteZero.celsius)     // -273.15
  println(List(Temperature(300), Temperature(10)).sorted)
  // List(Temperature(10.00 K), Temperature(300.00 K))
  // new Temperature(5)                         // does not compile: constructor is private

Case Class Companions

final case class Point(x: Int, y: Int)

// The compiler generates roughly:
// object Point:
//   def apply(x: Int, y: Int): Point = new Point(x, y)
//   def unapply(p: Point): Point = p          // Scala 3: used for pattern matching

val p = Point(1, 2)                  // calls Point.apply
val Point(a, b) = p                  // uses the extractor: a = 1, b = 2

You can add your own members to a case class companion (for example a given JSON codec or a parse method) by declaring object Point yourself; the generated methods are still added.

Companion Object vs Java Static

Scala companion object Java static members
Is a real object Yes (a singleton, can extend traits) No
Passed as a value Yes No
Implicit scope Givens/implicits found automatically Not applicable
Access to private class members Yes, both ways Yes
Seen from Java Static forwarders on the class (e.g. Temperature.apply(...)) -

Common Pitfalls

  • Defining the object in a different file: it is then not a companion and cannot see private members.

  • Putting mutable state in a companion; it is global, shared across threads and tests.

  • Forgetting that a hand-written apply in a case class companion does not stop new or copy from bypassing your validation; make the constructor private if needed.

  • Heavy work in the companion’s initializer, which runs on first access and can fail with ExceptionInInitializerError.

Interview Tips

  • Say “same name, same file, mutual private access” in the first sentence.

  • Give the smart-constructor example with a private constructor.

  • Mention implicit scope: givens in the companion need no import.

37

Describe mixins and trait composition in Scala.

A mixin is a trait that adds behaviour to a class when it is “mixed in” with extends ... with ..., and trait composition builds a class from several such traits. Traits can hold abstract and concrete methods and fields, and a class can mix in many of them (unlike single class inheritance). Scala resolves conflicts with linearization: the traits are ordered from right to left, so super in a trait refers to the next trait in that order, not to a fixed parent. This enables stackable modifications with abstract override, where each trait wraps the behaviour of the one below it (logging, clipping, caching, retries). Self-types (self: DataSource =>) let a trait declare that it must be mixed into something else without inheriting from it.

Read full answer

Applies toScala 3 (Scala 2.13 uses `with` instead of commas; the rules are the same)

Core Ideas

Concept Syntax Meaning
Mixin class A extends B with T1 with T2 (Scala 3 also extends B, T1, T2) Add traits’ members to A
Linearization Right to left: T2, T1, B Order used by super and for overriding
Stackable trait abstract override def f = super.f ... Wraps the next implementation
Self-type trait T { self: Dep => } T can be mixed only into something that is also a Dep
Trait parameters (Scala 3) trait Named(val name: String) Traits can take constructor parameters

Code Example 1: Stackable Modifications (Scala 3)

//> using scala 3.5

trait Scorer:
  def score(x: Double): Double

class BaseModel extends Scorer:
  def score(x: Double): Double = 2 * x

trait Clipped extends Scorer:
  abstract override def score(x: Double): Double = super.score(x).max(0.0).min(10.0)

trait Shifted extends Scorer:
  abstract override def score(x: Double): Double = super.score(x) + 1

@main def mixinDemo(): Unit =
  // Linearization: Shifted -> Clipped -> BaseModel, so score = clip(2x) + 1
  val m1 = new BaseModel with Clipped with Shifted
  // Linearization: Clipped -> Shifted -> BaseModel, so score = clip(2x + 1)
  val m2 = new BaseModel with Shifted with Clipped

  println(m1.score(7))     // 11.0   clip(14) = 10, then + 1
  println(m2.score(7))     // 10.0   clip(15) = 10
  println(m1.score(-3))    // 1.0    clip(-6) = 0, then + 1
  println(m2.score(-3))    // 0.0    clip(-5) = 0

The same two traits give different behaviour depending only on the order in which they are mixed in. That is linearization at work.

Code Example 2: Self-Types and Composition

//> using scala 3.5

trait DataSource:
  def rows: Seq[Double]

trait MaxNormalizer:
  self: DataSource =>                     // requires a DataSource, but is not one
  def normalized: Seq[Double] =
    val m = rows.max
    rows.map(_ / m)

trait Summary:
  self: DataSource =>
  def summary: String = s"n=${rows.size}, mean=${rows.sum / rows.size}"

object Job extends DataSource, MaxNormalizer, Summary:
  def rows: Seq[Double] = Seq(1.0, 2.0, 4.0)

@main def selfTypeDemo(): Unit =
  println(Job.normalized)    // List(0.25, 0.5, 1.0)
  println(Job.summary)       // n=3, mean=2.3333333333333335

How Linearization Is Computed

For class C extends A with T1 with T2, the linearization is C, T2, T1, A, AnyRef, Any (each trait appears once, the rightmost first). A call super.score inside T2 goes to T1, inside T1 goes to A. This also solves the diamond problem: shared parents appear once, in a defined position.

Mixins vs Alternatives

Approach Good for Watch out
Trait mixins Reusable cross-cutting behaviour (logging, metrics, validation) Order-dependent; deep stacks are hard to follow
Abstract class One base with constructor and shared state Single inheritance only
Composition (fields) Swapping parts at runtime, testing with fakes More wiring code
Type classes (given) Adding behaviour to types you do not own Less discoverable

Common Pitfalls

  • Forgetting that mixin order changes behaviour; document it or avoid stackable traits for business logic.

  • Initialization order problems: a trait’s val that uses an abstract member defined in a later trait can be null or 0 during construction; use def or lazy val.

  • Using traits only as namespaces of unrelated helpers (“god traits”).

  • Using self-types where a constructor parameter would be simpler.

Interview Tips

  • Define linearization and show how super resolves.

  • Use the stackable abstract override example; it is the classic mixin question.

  • Mention self-types and that Scala 3 traits can take parameters.

38

Write a tail-recursive Scala function to compute a factorial.

A tail-recursive factorial carries the running product in an accumulator so the recursive call is the last thing the function does: loop(n − 1, acc · n). Because nothing is left to do after the call, the Scala compiler rewrites it into a loop, so it uses constant stack space; the @tailrec annotation makes the compiler prove this and fail compilation otherwise. Use BigInt for the result, since Long overflows after 20! (21! > 9.22 × 10¹⁸), and reject negative input with require. The naive n * factorial(n − 1) is not tail-recursive: the multiplication happens after the call returns, so each call needs a stack frame.

Read full answer

Applies toScala 3 (Scala 2.13 is the same apart from syntax)

Code Example (Scala 3)

//> using scala 3.5
import scala.annotation.tailrec

def factorial(n: Int): BigInt =
  require(n >= 0, s"factorial is undefined for negative numbers: $n")

  @tailrec
  def loop(k: Int, acc: BigInt): BigInt =
    if k <= 1 then acc
    else loop(k - 1, acc * k)          // tail call: nothing happens after it

  loop(n, BigInt(1))

// Not tail-recursive: multiplication runs after the recursive call returns
def factorialNaive(n: Int): BigInt =
  if n <= 1 then BigInt(1) else BigInt(n) * factorialNaive(n - 1)

// Same result with a fold (no explicit recursion)
def factorialFold(n: Int): BigInt = (2 to n).foldLeft(BigInt(1))(_ * _)

@main def factorialDemo(): Unit =
  println(factorial(0))                   // 1
  println(factorial(5))                   // 120
  println(factorial(20))                  // 2432902008176640000  (largest that fits in a Long)
  println(factorial(25))                  // 15511210043330985984000000
  println(factorial(1000).toString.length) // 2568  (digits in 1000!)
  println(factorialFold(10))              // 3628800
  println(factorial(20_000).bitLength > 0)    // true: deep "recursion" but no stack growth
  // factorialNaive(100_000)              // StackOverflowError with default stack size
  // factorial(-1)                        // IllegalArgumentException: requirement failed: ...

Trace of factorial(5)

Call k acc
loop(5, 1) 5 1
loop(4, 5) 4 5
loop(3, 20) 3 20
loop(2, 60) 2 60
loop(1, 120) 1 120 → returned

What @tailrec Checks

import scala.annotation.tailrec

@tailrec
def bad(n: Int): BigInt =
  if n <= 1 then BigInt(1) else n * bad(n - 1)
// compile error: cannot rewrite recursive call: it is not in tail position

Without the annotation the code compiles but may overflow the stack at runtime, so add @tailrec whenever you rely on the optimization. Note that Scala only optimizes self-recursion in a final or local method; mutual recursion between two functions is not optimized (use scala.util.control.TailCalls for that).

Comparing Versions

Version Stack use Notes
Naive recursion O(n) frames Clear, but overflows for large n
Tail recursion with accumulator O(1) Compiled to a loop; @tailrec verifies
foldLeft O(1) Most idiomatic for a simple product
Long result - Overflows silently after 20!

Common Pitfalls

  • Returning Int or Long: 13! overflows Int, 21! overflows Long, both without an error.

  • Writing acc * k but forgetting the base case for 0 (0! = 1).

  • Putting @tailrec on a public, non-final class method that can be overridden: the compiler refuses.

  • Assuming try/catch around the recursive call keeps it a tail call; it does not.

Interview Tips

  • Explain why it is tail-recursive: the recursive call is the last operation.

  • Mention @tailrec as a compile-time guarantee and BigInt for overflow.

  • If asked about efficiency for huge n, mention that multiplying big numbers dominates the cost, and split-product algorithms do better.

39

Implement a Scala function that checks whether a list of integers is sorted.

A list is sorted (in non-decreasing order) when every element is less than or equal to the next one, so the check compares each adjacent pair and stops at the first pair that is out of order. In Scala this is a one-liner over adjacent pairs, xs.lazyZip(xs.drop(1)).forall(_ <= _), or a tail-recursive pattern match on a :: b :: rest. Both run in O(n) time and short-circuit. Edge cases matter: an empty list and a single-element list are sorted, and you should decide whether duplicates are allowed (non-decreasing <=) or not (strictly increasing <). A generic version takes an Ordering[T], which also lets you check descending order with Ordering[Int].reverse.

Read full answer

Applies toScala 3 (Scala 2.13 is the same apart from syntax)

Code Example (Scala 3)

//> using scala 3.5
import scala.annotation.tailrec

// 1. Idiomatic: compare adjacent pairs, short-circuits on the first failure
def isSorted(xs: List[Int]): Boolean =
  xs.lazyZip(xs.drop(1)).forall(_ <= _)       // drop(1) is safe on Nil, unlike tail

// 2. Strictly increasing (no duplicates allowed)
def isStrictlySorted(xs: List[Int]): Boolean =
  xs.lazyZip(xs.drop(1)).forall(_ < _)

// 3. Tail-recursive pattern match
@tailrec
def isSortedRec(xs: List[Int]): Boolean = xs match
  case a :: (rest @ b :: _) => if a > b then false else isSortedRec(rest)
  case _                    => true          // Nil or a single element

// 4. Generic: any element type with an Ordering, any order direction
def isSortedBy[T](xs: Seq[T])(using ord: Ordering[T]): Boolean =
  xs.iterator.zip(xs.iterator.drop(1)).forall((a, b) => ord.lteq(a, b))

@main def sortedDemo(): Unit =
  println(isSorted(List(1, 2, 2, 5)))            // true
  println(isStrictlySorted(List(1, 2, 2, 5)))    // false (duplicate 2)
  println(isSorted(List(3, 1, 2)))               // false
  println(isSorted(Nil))                         // true
  println(isSorted(List(7)))                     // true
  println(isSortedRec(List(-3, 0, 4, 4, 9)))     // true
  println(isSortedRec(List(1, 5, 4)))            // false
  println(isSortedBy(Vector("apple", "kiwi", "pear")))           // true
  println(isSortedBy(List(5, 3, 1))(using Ordering[Int].reverse)) // true (descending)

How It Works on List(1, 2, 2, 5)

Pair Check a <= b
(1, 2) true
(2, 2) true
(2, 5) true → sorted

For List(3, 1, 2) the first pair (3, 1) fails, so forall stops and returns false without looking at the rest.

Versions Compared

Version Allocation Notes
lazyZip(...).forall None for pairs Clear and fast; lazyZip does not build tuples
xs.zip(xs.tail) Builds a list of tuples Fails on Nil (tail of empty list throws)
sliding(2).forall One small list per window Readable but slower
Tail recursion None Shows recursion skills; compiled to a loop
xs == xs.sorted Sorts a copy, O(n log n) Simple but wasteful

Common Pitfalls

  • Calling xs.tail on an empty list (throws UnsupportedOperationException); drop(1) is safe.

  • Using xs == xs.sorted: correct but O(n log n) and allocates a copy.

  • Calling xs(i) in an index loop over a List (O(n²) total).

  • Forgetting to agree with the interviewer on duplicates and on ascending vs descending.

Interview Tips

  • State the edge cases (empty, one element, duplicates) before coding.

  • Give the lazyZip one-liner, then the tail-recursive version if asked for recursion.

  • Mention the generic Ordering[T] version and the O(n), short-circuit behaviour.

40

Write a Scala Matrix class with a matrix multiplication method.

A practical Scala Matrix class stores its values in one flat, row-major Array[Double] with the row and column counts, exposes apply(i, j) for reading, and implements multiplication as C(i, j) = Σₖ A(i, k) · B(k, j), which requires A.cols == B.rows and gives a A.rows × B.cols result. Keeping the class immutable (private data, every operation returns a new matrix), validating dimensions with require, and putting factories (fromRows, identity) in the companion object gives a clean API, while the i-k-j loop order over primitive arrays keeps the inner loop sequential in memory and avoids boxing. For real workloads, use Breeze (DenseMatrix, BLAS-backed) or Spark’s distributed BlockMatrix.

Read full answer

Applies toScala 3 (Scala 2.13 is the same apart from syntax)

Code Example (Scala 3)

//> using scala 3.5
import scala.util.Try

final class Matrix private (val rows: Int, val cols: Int, private val data: Array[Double]):

  def apply(i: Int, j: Int): Double = data(i * cols + j)

  def *(that: Matrix): Matrix =
    require(cols == that.rows, s"cannot multiply ${rows}x$cols by ${that.rows}x${that.cols}")
    val out = new Array[Double](rows * that.cols)
    var i = 0
    while i < rows do
      var k = 0
      while k < cols do
        val a = data(i * cols + k)               // reuse A(i, k) across the whole row of B
        var j = 0
        while j < that.cols do
          out(i * that.cols + j) += a * that.data(k * that.cols + j)
          j += 1
        k += 1
      i += 1
    new Matrix(rows, that.cols, out)

  def transpose: Matrix =
    Matrix.tabulate(cols, rows)((i, j) => apply(j, i))

  override def equals(other: Any): Boolean = other match
    case m: Matrix => rows == m.rows && cols == m.cols && java.util.Arrays.equals(data, m.data)
    case _         => false

  override def hashCode: Int = (rows, cols, java.util.Arrays.hashCode(data)).hashCode

  override def toString: String =
    (0 until rows).map(i => (0 until cols).map(j => apply(i, j)).mkString(" ")).mkString("\n")

object Matrix:
  def tabulate(rows: Int, cols: Int)(f: (Int, Int) => Double): Matrix =
    require(rows > 0 && cols > 0, "dimensions must be positive")
    val data = Array.tabulate(rows * cols)(idx => f(idx / cols, idx % cols))
    new Matrix(rows, cols, data)

  def fromRows(rows: Seq[Double]*): Matrix =
    require(rows.nonEmpty && rows.forall(_.size == rows.head.size), "rows must be non-empty and equal length")
    tabulate(rows.size, rows.head.size)((i, j) => rows(i)(j))

  def identity(n: Int): Matrix = tabulate(n, n)((i, j) => if i == j then 1.0 else 0.0)

@main def matrixDemo(): Unit =
  val a = Matrix.fromRows(Seq(1.0, 2.0), Seq(3.0, 4.0))
  val b = Matrix.fromRows(Seq(5.0, 6.0), Seq(7.0, 8.0))
  println(a * b)
  // 19.0 22.0
  // 43.0 50.0

  val c = Matrix.fromRows(Seq(1.0, 2.0, 3.0), Seq(4.0, 5.0, 6.0))       // 2x3
  val d = Matrix.fromRows(Seq(7.0, 8.0), Seq(9.0, 10.0), Seq(11.0, 12.0)) // 3x2
  println(c * d)
  // 58.0 64.0
  // 139.0 154.0

  println(a * Matrix.identity(2) == a)          // true
  println((a * b).transpose == b.transpose * a.transpose)   // true: (AB)^T = B^T A^T
  println(Try(c * c).failed.get.getMessage)     // requirement failed: cannot multiply 2x3 by 2x3

Check of the first entry of c * d: 1·7 + 2·9 + 3·11 = 7 + 18 + 33 = 58.

Why These Design Choices

Choice Reason
Flat Array[Double], row-major Primitive storage (no boxing), one allocation, cache friendly
i-k-j loop order Inner loop walks B and C row by row, so memory access is sequential
private constructor + companion factories Validation in one place; callers cannot pass a wrong-length array
No mutating methods Safe to share between threads
Custom equals Arrays compare by reference; matrices should compare by value

A Purely Functional Version

type Mat = Vector[Vector[Double]]

def multiply(a: Mat, b: Mat): Mat =
  require(a.head.size == b.size, "inner dimensions must match")
  val bt = b.transpose
  a.map(row => bt.map(col => row.lazyZip(col).map(_ * _).sum))

// multiply(Vector(Vector(1.0, 2.0), Vector(3.0, 4.0)), Vector(Vector(5.0, 6.0), Vector(7.0, 8.0)))
// Vector(Vector(19.0, 22.0), Vector(43.0, 50.0))

It is shorter and easy to read but allocates many boxed values; good for interviews, slow for large matrices.

Library Options

Need Use
Fast dense linear algebra on one machine Breeze DenseMatrix (BLAS/LAPACK)
Distributed matrices Spark BlockMatrix, RowMatrix
GPU DJL / ONNX Runtime / TensorFlow Java

Common Pitfalls

  • Forgetting the dimension check, so a mismatch gives an ArrayIndexOutOfBoundsException or a wrong result.

  • Using Vector[Vector[Double]] or List for large matrices (boxing and slow indexing).

  • i-j-k loop order on row-major data: the inner loop jumps across rows of B, hurting cache use.

  • Exposing the internal array, which lets callers mutate an “immutable” matrix.

  • Comparing doubles with == after floating-point computations; use a tolerance in tests.

Interview Tips

  • State the dimension rule and the result shape before writing code.

  • Mention the O(n³) cost and that BLAS libraries use blocking and SIMD to be much faster.

  • Offer the functional version first if asked for idiomatic code, then the array version for performance.

41

Write a Scala program that averages a numeric column from a CSV.

To average a numeric CSV column in Scala, read the header to find the column’s index, stream the remaining lines, parse that field safely with toDoubleOption, and fold the valid values into a running sum and count, skipping (and counting) empty or invalid cells instead of crashing. Use scala.util.Using so the file is always closed, return an Either for errors such as a missing column, and keep memory constant by processing the lines as an iterator. A plain split(",", -1) works for simple files; for quoted fields with embedded commas, use a CSV library (Apache Commons CSV, univocity, scala-csv), or Spark for large files.

Read full answer

Applies toScala 3 (standard library only); Spark 3.5 / 4.x with Scala 2.13 for large files

Code Example (Scala 3)

//> using scala 3.5
import java.nio.file.{Files, Path}
import scala.io.Source
import scala.util.Using

final case class ColumnStats(count: Int, skipped: Int, mean: Option[Double])

def averageOf(path: Path, column: String): Either[String, ColumnStats] =
  Using(Source.fromFile(path.toFile, "UTF-8")) { src =>
    val lines = src.getLines()
    if !lines.hasNext then Left("empty file")
    else
      val header = lines.next().split(",", -1).map(_.trim)
      header.indexOf(column) match
        case -1 => Left(s"column '$column' not found in: ${header.mkString(", ")}")
        case idx =>
          val (sum, n, bad) = lines.filter(_.nonEmpty).foldLeft((0.0, 0, 0)) {
            case ((s, c, b), line) =>
              line.split(",", -1).lift(idx).flatMap(_.trim.toDoubleOption) match
                case Some(v) => (s + v, c + 1, b)
                case None    => (s, c, b + 1)        // empty, "n/a" or missing field
          }
          Right(ColumnStats(n, bad, Option.when(n > 0)(sum / n)))
  }.toEither.left.map(e => s"cannot read file: ${e.getMessage}").flatten

@main def csvAverage(args: String*): Unit =
  // Use the given path, or write a small sample file for the demo
  val path = args.headOption.map(p => Path.of(p)).getOrElse {
    val p = Files.createTempFile("scores", ".csv")
    Files.writeString(p, "id,name,score\n1,Asha,88.5\n2,Ben,\n3,Chen,92\n4,Dina,n/a\n5,Eli,79.5\n")
    p
  }

  averageOf(path, "score") match
    case Right(ColumnStats(n, bad, Some(avg))) => println(f"average score = $avg%.2f over $n rows ($bad skipped)")
    case Right(ColumnStats(_, bad, None))      => println(s"no numeric values ($bad skipped)")
    case Left(err)                             => println(s"error: $err")
  // average score = 86.67 over 3 rows (2 skipped)

  println(averageOf(path, "salary"))
  // Left(column 'salary' not found in: id, name, score)

Run with scala-cli run avg.scala (demo file) or scala-cli run avg.scala -- data.csv. The valid scores are 88.5, 92 and 79.5: (88.5 + 92 + 79.5) / 3 = 260 / 3 = 86.67. Ben’s empty cell and Dina’s “n/a” are skipped.

Design Choices

Choice Why
split(",", -1) Keeps trailing empty fields (“2,Ben,” has 3 fields)
lift(idx) Short rows give None instead of an exception
toDoubleOption Bad numbers become None, not a NumberFormatException
foldLeft over the iterator One pass, constant memory
Using Closes the file even on errors
Either + Option mean Distinguishes “file problem”, “missing column” and “no valid values”

Spark Version for Large Files (Scala 2.13)

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions.avg
import org.apache.spark.sql.types._

val spark = SparkSession.builder.appName("csv-avg").master("local[*]").getOrCreate()

val schema = StructType(Seq(
  StructField("id", IntegerType), StructField("name", StringType), StructField("score", DoubleType)
))

val df = spark.read.option("header", "true").option("mode", "PERMISSIVE").schema(schema).csv("scores.csv")
df.agg(avg("score")).show()
// +-----------------+
// |       avg(score)|
// +-----------------+
// |86.66666666666667|
// +-----------------+

With an explicit schema and permissive mode, the empty cell and “n/a” are read as null, and avg ignores nulls, giving the same 260 / 3.

Common Pitfalls

  • split(",") without -1: trailing empty fields disappear and indices shift for some rows.

  • Hand-splitting files with quoted commas ("Smith, J",90); use a CSV parser.

  • Forgetting to close Source.fromFile (file handle leaks in long-running services).

  • toDouble on dirty data, crashing the whole job on one bad row.

  • Reading the whole file with .toList before processing, which fails on large files.

  • Ignoring the file encoding and locale (1,5 as a decimal in some European files).

Interview Tips

  • Mention header lookup, safe parsing and resource handling (Using) without being asked.

  • Report how many rows were skipped; silent data loss is a common review comment.

  • Offer the Spark version for files that do not fit on one machine.

42

Implement a simple decision tree algorithm in Scala.

A decision tree classifier recursively splits the data on the feature and threshold that make the child nodes purest, then predicts the majority class in each leaf. A compact Scala implementation (the CART approach) needs four pieces: an impurity measure (Gini: 1 − Σ pₖ²), a best-split search over every feature and every midpoint between sorted distinct values, minimizing the size-weighted impurity of the two children, a recursive builder that stops at a maximum depth, a minimum node size, a pure node or no improving split, and a predict function that walks the tree. Scala’s enum (an algebraic data type) models the tree cleanly as Leaf or Node, and pattern matching makes prediction a few lines.

Read full answer

Applies toScala 3 (from scratch). For large data use Spark `DecisionTreeClassifier` (Scala 2.13).

Algorithm

Step Detail
Impurity Gini(S) = 1 − Σ pₖ² (0 when the node holds one class)
Candidate thresholds Midpoints between consecutive sorted distinct values of each feature
Split score (n_L/n)·Gini(L) + (n_R/n)·Gini® — lower is better
Stop depth = maxDepth, fewer than minSize rows, pure node, or no split lowers impurity
Leaf value Majority class (ties go to the smaller label)

Code Example (Scala 3)

//> using scala 3.5
import scala.annotation.tailrec

enum Tree:
  case Leaf(label: Int)
  case Node(feature: Int, threshold: Double, left: Tree, right: Tree)

final case class Row(x: Vector[Double], y: Int)

def gini(rows: Seq[Row]): Double =
  if rows.isEmpty then 0.0
  else
    val n = rows.size.toDouble
    1.0 - rows.groupBy(_.y).values.map(g => math.pow(g.size / n, 2)).sum

def majority(rows: Seq[Row]): Int =
  rows.groupBy(_.y).maxBy((label, g) => (g.size, -label))._1

// Returns (feature, threshold, weighted impurity) of the best split, if any
def bestSplit(rows: Seq[Row]): Option[(Int, Double, Double)] =
  val n = rows.size.toDouble
  val candidates =
    for
      f <- rows.head.x.indices
      values = rows.map(_.x(f)).distinct.sorted
      (lo, hi) <- values.zip(values.tail)
      t = (lo + hi) / 2
      (l, r) = rows.partition(_.x(f) <= t)
    yield (f, t, l.size / n * gini(l) + r.size / n * gini(r))
  candidates.minByOption(_._3)        // the first best split wins ties

def build(rows: Seq[Row], depth: Int = 0, maxDepth: Int = 3, minSize: Int = 2): Tree =
  if depth >= maxDepth || rows.size < minSize || gini(rows) == 0.0 then Tree.Leaf(majority(rows))
  else bestSplit(rows) match
    case Some((f, t, impurity)) if impurity < gini(rows) =>
      val (l, r) = rows.partition(_.x(f) <= t)
      Tree.Node(f, t, build(l, depth + 1, maxDepth, minSize), build(r, depth + 1, maxDepth, minSize))
    case _ => Tree.Leaf(majority(rows))

@tailrec
def predict(tree: Tree, x: Vector[Double]): Int = tree match
  case Tree.Leaf(label)      => label
  case Tree.Node(f, t, l, r) => if x(f) <= t then predict(l, x) else predict(r, x)

def render(t: Tree, names: Vector[String], indent: String = ""): String = t match
  case Tree.Leaf(label) => s"${indent}predict $label"
  case Tree.Node(f, th, l, r) =>
    s"${indent}if ${names(f)} <= $th\n${render(l, names, indent + "  ")}\n${indent}else\n${render(r, names, indent + "  ")}"

@main def treeDemo(): Unit =
  // features: (age, income in thousands) -> bought (1) or not (0)
  val data = Seq(
    Row(Vector(25, 30), 0), Row(Vector(30, 40), 0), Row(Vector(35, 80), 1), Row(Vector(40, 90), 1),
    Row(Vector(45, 35), 0), Row(Vector(50, 85), 1), Row(Vector(55, 95), 0), Row(Vector(60, 100), 0)
  )
  val tree = build(data)

  println(tree)
  // Node(1,60.0,Leaf(0),Node(0,52.5,Leaf(1),Leaf(0)))
  println(render(tree, Vector("age", "income")))
  // if income <= 60.0
  //   predict 0
  // else
  //   if age <= 52.5
  //     predict 1
  //   else
  //     predict 0

  val accuracy = data.count(r => predict(tree, r.x) == r.y).toDouble / data.size
  println(accuracy)                               // 1.0  (training data; use a test set in practice)
  println(predict(tree, Vector(30, 70)))          // 1
  println(predict(tree, Vector(58, 70)))          // 0
  println(predict(tree, Vector(45, 50)))          // 0

Vector(25, 30) is typed as Vector[Double] because Row expects one, so the integers widen to doubles.

How the tree was chosen: the root impurity is 1 − (3/8)² − (5/8)² = 0.469. The best root split is income ≤ 60 (weighted Gini 0.300: three pure non-buyers on the left, 3 buyers and 2 non-buyers on the right). In the right child, both age ≤ 52.5 and income ≤ 92.5 give pure children; age is checked first, so it wins the tie.

Spark Version (Scala 2.13)

import org.apache.spark.ml.classification.DecisionTreeClassifier

val dt = new DecisionTreeClassifier()
  .setLabelCol("label").setFeaturesCol("features")
  .setMaxDepth(3).setImpurity("gini").setSeed(42)
val model = dt.fit(trainingDf)          // trainingDf has "features" (Vector) and "label"
println(model.toDebugString)            // the learned if/else rules

Common Pitfalls

  • Growing the tree without depth or size limits: it memorizes the training data.

  • Evaluating on the training set (the 1.0 above says nothing about generalization).

  • Sorting the data for every candidate threshold: O(n² · d) per node; real implementations sort once per feature and sweep.

  • Not handling ties and empty children deterministically.

  • Treating impurity-based importances as causal.

Interview Tips

  • Write the Gini function and the split score first, then the recursion.

  • State the stopping rules and why they control overfitting.

  • Mention extensions: entropy, regression trees (variance reduction), pruning, and random forests.

43

How do you do NLP text preprocessing in Scala?

Text preprocessing in Scala turns raw strings into clean, consistent tokens with a short pipeline of pure functions: normalize the text (Unicode normalization with java.text.Normalizer to strip accents, consistent apostrophes, lower-casing with a fixed Locale), tokenize with a regular expression or a proper tokenizer, remove stop words, optionally stem or lemmatize (Lucene/Snowball, Stanford CoreNLP, Spark NLP), build n-grams, and convert to features such as bag-of-words counts or TF-IDF. Scala’s immutable collections (map, filterNot, sliding, groupMapReduce) express each step directly. On a cluster, Spark’s RegexTokenizer, StopWordsRemover, NGram and CountVectorizer run the same steps as pipeline stages.

Read full answer

Applies toScala 3 (plain Scala); Spark 3.5 / 4.x with Scala 2.13 for the distributed version

Pipeline Steps

Step Scala tool Example
Unicode normalization Normalizer.normalize(s, NFKD) + remove p{M} “Café” → “Cafe”
Case folding toLowerCase(Locale.ROOT) “LOVE” → “love”
Tokenization Regex.findAllIn “aren’t rising,” → aren't, rising
Stop-word removal filterNot(stopWords) drop “the”, “and”
Stemming / lemmatization Lucene PorterStemFilter, CoreNLP, Spark NLP “loving” → “love”
N-grams sliding(n) cafe_owners
Vectorization groupMapReduce, vocabulary index count vectors

Code Example (Scala 3)

//> using scala 3.5
import java.text.Normalizer
import java.util.Locale

val stopWords: Set[String] = Set("the", "and", "are", "it", "a", "an", "of", "to", "is", "in")
val TokenRe = "[a-z0-9]+(?:'[a-z]+)?".r

def normalize(text: String): String =
  Normalizer.normalize(text, Normalizer.Form.NFKD)
    .replaceAll("\\p{M}", "")       // remove combining accents: é -> e
    .replace('’', '\'')        // curly apostrophe -> straight
    .toLowerCase(Locale.ROOT)       // locale-independent lower case

def tokenize(text: String): List[String] = TokenRe.findAllIn(text).toList

def preprocess(text: String): List[String] = tokenize(normalize(text)).filterNot(stopWords)

def ngrams(tokens: List[String], n: Int): List[String] =
  tokens.sliding(n).filter(_.size == n).map(_.mkString("_")).toList

@main def textDemo(): Unit =
  val docs = List(
    "Café owners LOVE the new menu! Prices aren't rising, and customers are loving it.",
    "The menu is new; the prices are fair."
  )
  val tokenized = docs.map(preprocess)

  println(tokenized.head)
  // List(cafe, owners, love, new, menu, prices, aren't, rising, customers, loving)
  println(tokenized(1))
  // List(menu, new, prices, fair)
  println(ngrams(tokenized.head, 2).take(3))
  // List(cafe_owners, owners_love, love_new)

  // Bag of words over a sorted vocabulary
  val vocab = tokenized.flatten.distinct.sorted.toVector
  println(vocab)
  // Vector(aren't, cafe, customers, fair, love, loving, menu, new, owners, prices, rising)

  val counts = tokenized.map { toks =>
    val c = toks.groupMapReduce(identity)(_ => 1)(_ + _)
    vocab.map(w => c.getOrElse(w, 0))
  }
  counts.foreach(println)
  // Vector(1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1)
  // Vector(0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 0)

Notice that “love” and “loving” stay separate: that is the job of a stemmer or lemmatizer. A hand-written suffix stripper makes mistakes quickly (“rising” → “ris”), so use a library such as Lucene’s EnglishAnalyzer or Spark NLP for that step.

Spark Version (Scala 2.13)

import org.apache.spark.ml.Pipeline
import org.apache.spark.ml.feature.{CountVectorizer, NGram, RegexTokenizer, StopWordsRemover}

val tokenizer = new RegexTokenizer().setInputCol("text").setOutputCol("tokens")
  .setPattern("[a-z0-9]+(?:'[a-z]+)?").setGaps(false).setToLowercase(true)
val stop   = new StopWordsRemover().setInputCol("tokens").setOutputCol("clean")   // built-in English list
val bigram = new NGram().setN(2).setInputCol("clean").setOutputCol("bigrams")
val cv     = new CountVectorizer().setInputCol("clean").setOutputCol("features").setMinDF(2)

val model = new Pipeline().setStages(Array(tokenizer, stop, bigram, cv)).fit(docsDf)

Accent stripping is not built into these stages; apply a Scala UDF with Normalizer first, which runs in the JVM with no Python overhead.

Scala vs Python for NLP Preprocessing

Aspect Scala Python
Libraries Spark NLP, CoreNLP, Lucene, OpenNLP spaCy, NLTK, Hugging Face tokenizers
Scale Spark stages and JVM UDFs at cluster scale pandas / PySpark (Python UDFs are slower)
Transformer tokenizers Via Spark NLP or ONNX/DJL Native
Type safety Compile time Runtime

Common Pitfalls

  • toLowerCase() without a locale: in a Turkish locale “I” becomes “ı”.

  • Splitting on spaces only, leaving punctuation attached (“menu!”).

  • Removing negations (“not”, “aren’t”) as stop words in sentiment tasks.

  • Building the vocabulary on all data instead of the training split.

  • Different preprocessing in training and serving; keep it in one shared function or one Spark pipeline.

Interview Tips

  • Walk the steps in order: normalize, tokenize, filter, stem, n-grams, vectorize.

  • Mention Unicode normalization and locale issues; they show real-world experience.

  • Say you keep preprocessing as pure functions so it is easy to test and reuse in serving.

44

Write a Scala function for element-wise addition of two vectors.

Element-wise addition of two vectors adds the values at the same index: (a + b)ᵢ = aᵢ + bᵢ, which is only defined when both vectors have the same length. In idiomatic Scala the core is one line, a.lazyZip(b).map(_ + _), plus a length check: either require(...) (throws IllegalArgumentException) or a total function that returns Either[String, Vector[Double]]. A Numeric[T] context bound makes it work for Int, Long, Double and BigDecimal; an Array[Double] with a while loop is the fast, unboxed version for hot code; and the Breeze library gives DenseVector with a built-in + operator.

Read full answer

Applies toScala 3 (Scala 2.13 is the same except for syntax); Breeze 2.x

Code Example (Scala 3)

//> using scala 3.5

// 1. Simple version with a precondition
def add(a: Vector[Double], b: Vector[Double]): Vector[Double] =
  require(a.size == b.size, s"length mismatch: ${a.size} vs ${b.size}")
  a.lazyZip(b).map(_ + _)                 // lazyZip avoids building a tuple collection

// 2. Total version: no exceptions
def safeAdd(a: Vector[Double], b: Vector[Double]): Either[String, Vector[Double]] =
  if a.size != b.size then Left(s"length mismatch: ${a.size} vs ${b.size}")
  else Right(a.lazyZip(b).map(_ + _))

// 3. Generic version for any numeric type
def addN[T](a: Seq[T], b: Seq[T])(using num: Numeric[T]): Seq[T] =
  require(a.size == b.size, s"length mismatch: ${a.size} vs ${b.size}")
  a.lazyZip(b).map(num.plus)

// 4. Fast version: primitive arrays, no boxing, no closures
def addArrays(a: Array[Double], b: Array[Double]): Array[Double] =
  require(a.length == b.length, s"length mismatch: ${a.length} vs ${b.length}")
  val out = new Array[Double](a.length)
  var i = 0
  while i < a.length do
    out(i) = a(i) + b(i)
    i += 1
  out

// Operator syntax via an extension method
extension (a: Vector[Double]) def +:+(b: Vector[Double]): Vector[Double] = add(a, b)

@main def vectorAddDemo(): Unit =
  val a = Vector(1.0, 2.0, 3.0)
  val b = Vector(4.0, 5.0, 6.0)

  println(add(a, b))                                    // Vector(5.0, 7.0, 9.0)
  println(a +:+ b)                                      // Vector(5.0, 7.0, 9.0)
  println(safeAdd(a, b))                                // Right(Vector(5.0, 7.0, 9.0))
  println(safeAdd(Vector(1.0), Vector(1.0, 2.0)))       // Left(length mismatch: 1 vs 2)
  println(addN(List(1, 2, 3), List(10, 20, 30)))        // List(11, 22, 33)
  println(addN(Vector(BigDecimal("0.1")), Vector(BigDecimal("0.2"))))   // Vector(0.3)
  println(addArrays(Array(1.0, 2.0, 3.0), Array(4.0, 5.0, 6.0)).mkString(", "))   // 5.0, 7.0, 9.0
  println(add(Vector.empty, Vector.empty))              // Vector()

Breeze Version

//> using scala 3.5
//> using dep org.scalanlp::breeze:2.1.0
import breeze.linalg.DenseVector

@main def breezeAdd(): Unit =
  val a = DenseVector(1.0, 2.0, 3.0)
  val b = DenseVector(4.0, 5.0, 6.0)
  println(a + b)          // DenseVector(5.0, 7.0, 9.0)
  a += b                  // in-place update of a (DenseVector is mutable)
  println(a)              // DenseVector(5.0, 7.0, 9.0)

Breeze checks lengths too and throws on a mismatch.

Which Version to Use

Version Pros Cons
Vector + lazyZip Clear, immutable Boxed doubles, allocation
Either result No exceptions, composes in for-comprehensions Callers must handle Left
Numeric[T] Works for any number type Boxed; slower
Array + while Fastest, unboxed Mutable, more code
Breeze DenseVector Operators, BLAS-backed ops Extra dependency

Common Pitfalls

  • Using zip without a length check: zip silently truncates to the shorter vector, hiding bugs.

  • a.zip(b).map { case (x, y) => x + y } builds an intermediate collection of tuples; lazyZip avoids it.

  • List for numeric vectors: indexed access is O(n).

  • Mutating a shared Array result from several threads.

Interview Tips

  • Mention the length check first, then the one-line lazyZip solution.

  • Offer the Numeric[T] generic version and the Array fast version as follow-ups.

  • Point out the zip truncation trap.

45

How do implicits work in Scala, and how do they enable type conversion?

Implicits let the compiler fill in values or apply conversions that are not written in the code, based on the types involved and what is in scope. There are three main forms: implicit (contextual) parameters, which the compiler supplies (type-class instances like Ordering[T], Spark Encoders, an ExecutionContext); extension methods, which add methods to existing types (Scala 2 implicit class); and implicit conversions, which turn a value of type A into type B when B is expected (Conversion[A, B]). Scala 3 splits these into clearer features: given / using for contextual values, extension for new methods, and given Conversion[A, B] for conversions, which require import scala.language.implicitConversions to avoid a feature warning because hidden conversions make code hard to read.

Read full answer

Applies toScala 3 (with Scala 2.13 equivalents)

Scala 2 vs Scala 3

Purpose Scala 2.13 Scala 3
Type-class instance implicit val ord: Ordering[T] = ... given Ordering[T] = ...
Receive an instance (implicit ord: Ordering[T]) (using ord: Ordering[T])
Get it in the body implicitly[Ordering[T]] summon[Ordering[T]]
Add methods implicit class RichX(x: X) { def f = ... } extension (x: X) def f = ...
Conversion A → B implicit def aToB(a: A): B given Conversion[A, B] = a => ...
Import instances import foo._ import foo.given (givens are imported explicitly)

Code Example (Scala 3)

//> using scala 3.5
import scala.language.implicitConversions

final case class Celsius(value: Double)
final case class Fahrenheit(value: Double)

// 1. Implicit conversion: used where a Fahrenheit is expected but a Celsius is given
given Conversion[Celsius, Fahrenheit] = c => Fahrenheit(c.value * 9 / 5 + 32)

def report(t: Fahrenheit): String = f"${t.value}%.1f F"

// 2. Extension method: an explicit, discoverable alternative to a conversion
extension (c: Celsius) def toF: Fahrenheit = Fahrenheit(c.value * 9 / 5 + 32)

// 3. Type class with given instances and a using parameter
trait FeatureEncoder[A]:
  def encode(a: A): Vector[Double]

given FeatureEncoder[Boolean] with
  def encode(b: Boolean): Vector[Double] = Vector(if b then 1.0 else 0.0)

given FeatureEncoder[Celsius] with
  def encode(c: Celsius): Vector[Double] = Vector(c.value / 100)

def featurize[A](xs: Seq[A])(using enc: FeatureEncoder[A]): Vector[Vector[Double]] =
  xs.map(enc.encode).toVector

@main def implicitsDemo(): Unit =
  println(report(Celsius(100)))                  // 212.0 F   (conversion applied)
  println(report(Celsius(0).toF))                // 32.0 F    (explicit extension)
  println(featurize(Seq(true, false)))           // Vector(Vector(1.0), Vector(0.0))
  println(featurize(Seq(Celsius(50))))           // Vector(Vector(0.5))
  // featurize(Seq("text"))                      // does not compile: no given FeatureEncoder[String]

Scala 2.13 Equivalent

import scala.language.implicitConversions

object Conversions {
  implicit def celsiusToF(c: Celsius): Fahrenheit = Fahrenheit(c.value * 9 / 5 + 32)

  implicit class RichCelsius(private val c: Celsius) extends AnyVal {
    def toF: Fahrenheit = Fahrenheit(c.value * 9 / 5 + 32)
  }
}

Implicits You Already Use in Spark

Code What the implicit does
import spark.implicits._ Brings Encoders for case classes and primitives into scope
Seq(...).toDF("a", "b") Extension method on local collections
$"col" String-interpolator extension producing a Column
ds.map(...) Needs an implicit Encoder for the result type

How the Compiler Finds Them

  1. Local scope and explicit imports (import x.given in Scala 3).

  2. The implicit scope of the type: companion objects of the type, its type arguments and its parents. Putting given instances in the companion object makes them available with no import.

  3. If two candidates are equally specific, the compiler reports an ambiguity error.

Common Pitfalls

  • Broad implicit conversions (e.g. String => Int) that silently change meaning; prefer extension methods.

  • Ambiguous instances from two wildcard imports.

  • Forgetting import spark.implicits._ (Encoder not found) or importing it before the SparkSession value exists.

  • Slow compiles from deep implicit searches (heavy derivation).

  • In Scala 3, expecting import foo._ to bring givens; use import foo.given.

Interview Tips

  • Separate the three uses: contextual parameters, extension methods, conversions.

  • Say conversions are the riskiest and that Scala 3 makes them opt-in with a feature import.

  • Use Spark’s Encoder as a real-world example of implicit parameters.

46

Explain path-dependent types in Scala.

A path-dependent type is a type that belongs to a specific value, written x.T: a type member or inner class declared inside a class or trait is different for each instance, so g1.Node and g2.Node are distinct types even though both come from class Graph { class Node }. The compiler uses this to prevent mixing objects that belong to different owners (nodes from two graphs, keys from two configs, cursors from two databases), and dependent method types let a method’s return type depend on its argument, as in def get(k: Key): k.Value. When you need “a Node of any Graph”, use the type projection Graph#Node (Scala 2) or a shared outer type.

Read full answer

Applies toScala 3 (same feature in Scala 2.13; general type projections `T#X` on abstract types were removed in Scala 3)

Concepts

Term Example Meaning
Inner class per instance g1.Node Nodes created by graph g1
Abstract type member trait Key { type Value } Each key chooses its value type
Dependent method type def get(k: Key): Option[k.Value] Return type depends on the argument
Type projection Graph#Node A Node of any Graph (concrete classes only in Scala 3)
Stable path val g, object o Paths must be stable identifiers (not var, not def)

Code Example 1: Nodes Owned by a Graph (Scala 3)

//> using scala 3.5

class Graph:
  final class Node(val id: Int):
    private var edges: List[Node] = Nil
    def neighbours: List[Node] = edges
    def connectTo(other: Node): Unit = edges = other :: edges   // other must be from this graph

  private var count = 0
  def newNode(): Node =
    count += 1
    Node(count)

@main def graphDemo(): Unit =
  val g1 = Graph()
  val g2 = Graph()
  val a = g1.newNode()
  val b = g1.newNode()
  val c = g2.newNode()

  a.connectTo(b)
  println(a.neighbours.map(_.id))    // List(2)

  // a.connectTo(c)
  // error: Found: (c : g2.Node)  Required: g1.Node

  val anyNodes: List[Graph#Node] = List(a, c)   // projection: a Node of any Graph
  println(anyNodes.map(_.id))        // List(1, 1)

The compiler stops a node of graph g2 from being linked into graph g1, with no runtime check.

Code Example 2: Typed Configuration Keys (Dependent Method Type)

//> using scala 3.5

trait Key:
  type Value
  def name: String

def key[V](n: String): Key { type Value = V } =
  new Key { type Value = V; val name = n }

final class Config(values: Map[String, Any]):
  def get(k: Key): Option[k.Value] =             // return type depends on k
    values.get(k.name).map(_.asInstanceOf[k.Value])

@main def configDemo(): Unit =
  val learningRate = key[Double]("learningRate")
  val epochs       = key[Int]("epochs")
  val cfg = Config(Map("learningRate" -> 0.01, "epochs" -> 20))

  val e: Option[Int] = cfg.get(epochs)            // typed as Int, no cast at call site
  println(e.map(_ + 1))                           // Some(21)
  println(cfg.get(learningRate))                  // Some(0.01)

Each key carries its own value type, so cfg.get(epochs) is an Option[Int] and cfg.get(learningRate) an Option[Double] from the same method. Real libraries (typed settings, typed attributes in HTTP and actor libraries) use this pattern with a safer internal store than Map[String, Any].

Where You See It

Library / pattern Path-dependent use
Scala 3 macros quotes.reflect.Tree belongs to one Quotes instance
Akka typed / Pekko Extensions and attribute keys
Parsers and compilers Symbols and trees tied to one compiler run
Cake pattern Components refer to each other’s type members

Common Pitfalls

  • Using a var or def as the path: paths must be stable (val, object, this).

  • Returning an inner type from a method without the path, which leads to confusing “found g1.Node, required Graph#Node” errors.

  • Overusing the pattern where a simple type parameter (Key[V]) would be clearer.

  • Expecting Scala 3 to allow T#X for abstract T; it only allows projections on concrete classes.

Interview Tips

  • Use the Graph/Node example: same class, different instance, different type.

  • Mention dependent method types (k.Value) as the practical payoff.

  • Contrast path-dependent g.Node with the projection Graph#Node.

47

What is type erasure in Scala, and how can you work around it?

Type erasure means the JVM does not keep generic type arguments at runtime: after compilation, List[Int] and List[String] are both just List, because Scala compiles generics the same way Java does. As a result, a pattern such as case xs: List[String] cannot really check the element type (the compiler warns that it is unchecked), overloads that differ only in type arguments clash, and generic code cannot create an Array[T] or test x.isInstanceOf[T] without help. You work around it by passing type information explicitly: ClassTag[T] (runtime class, enough for arrays and simple type tests), TypeTest in Scala 3, Scala 2 TypeTag (full type, used by Spark encoders), checking element values instead of types, or modelling data as sealed ADTs / case classes so the information is in the value itself.

Read full answer

Applies toScala 3 and Scala 2.13 on the JVM

What Is and Is Not Erased

At runtime you can see Erased
The class: List, Map, Option Type arguments: [Int], [String, Double]
Arrays’ element type: Array[Int] is int[] Abstract type parameters T in generic methods
Case class fields’ actual values Refinements and type aliases

Code Example (Scala 3)

//> using scala 3.5
import scala.reflect.ClassTag

// 1. The problem: the element type is not checked
def isStringList(x: Any): Boolean = x match
  case _: List[String @unchecked] => true
  case _                          => false

// 2. ClassTag: keep the runtime class of T
def onlyOf[T: ClassTag](xs: List[Any]): List[T] =
  xs.collect { case t: T => t }          // uses the ClassTag to test each element

def filled[T: ClassTag](n: Int, v: T): Array[T] = Array.fill(n)(v)   // Array needs the class

// 3. Check values, not types
def allStrings(x: Any): Boolean = x match
  case xs: List[?] => xs.forall(_.isInstanceOf[String])
  case _           => false

// 4. Put the type into the data: a sealed ADT
sealed trait Column
final case class NumericCol(values: Vector[Double]) extends Column
final case class CategoricalCol(values: Vector[String]) extends Column

def describe(c: Column): String = c match
  case NumericCol(vs)     => f"numeric, mean ${vs.sum / vs.size}%.1f"
  case CategoricalCol(vs) => s"categorical, ${vs.distinct.size} levels"

@main def erasureDemo(): Unit =
  println(isStringList(List(1, 2, 3)))              // true  (!) only List was checked
  println(onlyOf[String](List(1, "a", 2.0, "b")))   // List(a, b)
  println(onlyOf[Int](List(1, "a", 2, 3.5)))        // List(1, 2)
  println(filled(3, 0.5).mkString(","))             // 0.5,0.5,0.5
  println(allStrings(List(1, 2, 3)))                // false
  println(allStrings(List("x", "y")))               // true
  println(describe(NumericCol(Vector(1.0, 2.0, 6.0)))) // numeric, mean 3.0
  println(describe(CategoricalCol(Vector("a", "b", "a"))))   // categorical, 2 levels

Overloading Clash

object Stats:
  def mean(xs: List[Int]): Double = xs.sum.toDouble / xs.size
  def mean(xs: List[Double]): Double = xs.sum / xs.size
// error: both methods have the same type after erasure: (xs: List): Double

Fix it with different names, a Numeric[T] type class (def mean[T](xs: List[T])(using n: Numeric[T])), or @targetName in Scala 3 to give one overload a different JVM name.

Options Compared

Tool Gives you Works for Notes
ClassTag[T] Runtime class Arrays, case t: T, classTag[T].runtimeClass Cannot see nested args (List[Int] looks like List)
TypeTest[S, T] (Scala 3) A checked unapply Safe pattern matching on abstract types Compiler warns when the test cannot be checked
TypeTag[T] (Scala 2) Full type, including arguments Spark encoders, reflection Not available in Scala 3
Sealed ADT / case classes Type in the value Domain models Best design option
Value checks Actual element types Small collections Costs a pass over the data

In Spark

Spark’s Encoders for case classes are built from Scala 2 TypeTags (import spark.implicits._), which is why generic helpers need a bound:

import org.apache.spark.sql.{Dataset, Encoder}
def dedupe[T: Encoder](ds: Dataset[T]): Dataset[T] = ds.distinct()

Common Pitfalls

  • Ignoring “unchecked since it is eliminated by erasure” warnings; the match succeeds for the wrong type.

  • Expecting ClassTag to distinguish List[Int] from List[String].

  • Using Any and runtime type tests where a sealed trait or type class would make the code type-safe.

  • Generic Spark helpers without Encoder/TypeTag context bounds: “Unable to find encoder”.

Interview Tips

  • Explain that erasure comes from the JVM and Java compatibility, not from a Scala design choice.

  • Show the List[String] match pitfall and the ClassTag fix.

  • Recommend modelling with ADTs so you rarely need runtime type tests.

48

What are type bounds in Scala? Give an example.

Type bounds restrict which types a type parameter may take: an upper bound T <: Model says T must be Model or a subtype, so the method can call Model‘s members while still returning the precise subtype; a lower bound B >: A says B must be A or a supertype, used to add elements safely to covariant containers (def ::[B >: A](x: B): List[B]); and a context bound T: Ordering (sugar for a using/implicit parameter) requires a type-class instance for T, which is how generic numeric and sorting code is written. Bounds can be combined (T <: Product : Ordering) and can refer to T itself (F-bounded T <: Comparable[T]).

Read full answer

Applies toScala 3 (same concepts in Scala 2.13; context bounds there desugar to `implicit` parameters)

Kinds of Bounds

Bound Syntax Meaning Typical use
Upper T <: Upper T is a subtype of Upper Call methods of Upper, keep the exact subtype
Lower B >: A B is a supertype of A Prepend/append to covariant collections
Both T >: Lower <: Upper T between two types Rare; type-level APIs
Context T: Ordering, T: Numeric A given/implicit Ordering[T] exists Sorting, math, encoders
F-bounded T <: Comparable[T] T compares with itself Self-typed APIs

Code Example (Scala 3)

//> using scala 3.5

trait Model:
  def name: String
  def score(x: Double): Double

final case class Linear(w: Double) extends Model:
  val name = "linear"
  def score(x: Double): Double = w * x

// Upper bound: we can call score, and the result keeps the type M
def best[M <: Model](models: List[M], x: Double): M = models.maxBy(_.score(x))

// Context bound: needs an Ordering[T]
def median[T: Ordering](xs: Seq[T]): T = xs.sorted.apply(xs.size / 2)

// Context bound with Numeric: works for Int, Long, Double, BigDecimal...
def mean[T: Numeric](xs: Seq[T]): Double =
  val num = summon[Numeric[T]]
  xs.map(num.toDouble).sum / xs.size

// Lower bound: append a broader element to an immutable sequence
def withDefault[A, B >: A](xs: List[A], default: B): List[B] = xs :+ default

@main def boundsDemo(): Unit =
  val linear: Linear = best(List(Linear(2.0), Linear(-1.0), Linear(0.5)), 3.0)
  println(linear)                           // Linear(2.0)
  println(linear.w)                         // 2.0  (no cast needed: M = Linear)

  println(median(Seq(5, 1, 3)))             // 3
  println(median(Seq("b", "a", "c")))       // b
  println(mean(Seq(1, 2, 3, 4)))            // 2.5
  println(mean(Seq(1.5, 2.5)))              // 2.0

  val xs: List[Any] = withDefault(List(1, 2), "missing")
  println(xs)                               // List(1, 2, missing)

  // best(List("a", "b"), 1.0)              // does not compile: String is not a subtype of Model

Without the upper bound, best(models: List[Model], ...) would return a Model, and the caller would need a cast to read w.

Upper Bound vs Plain Subtype Parameter

Signature Returns Caller gets
def best(ms: List[Model]): Model Model Must cast to use subtype fields
def best[M <: Model](ms: List[M]): M Exact element type Linear directly

Context Bounds in Spark and ML Code

import org.apache.spark.sql.{Dataset, Encoder}

// Spark needs an Encoder for T to build a Dataset[T]
def sample[T: Encoder](ds: Dataset[T], fraction: Double): Dataset[T] =
  ds.sample(withReplacement = false, fraction, seed = 42)

Common Pitfalls

  • Using an upper bound when a type class (context bound) is the better design, e.g. forcing every model to extend a trait you do not own.

  • Confusing bounds (constraints on T) with variance (+T/-T, how F[T] relates to F[U]).

  • Using Scala 2 view bounds T <% U; they are removed in Scala 3. Use a context bound on T => U or a Conversion.

  • Writing [T <: AnyVal] and expecting fast primitive code; generics are still boxed.

Interview Tips

  • Give one example of each: upper bound (M <: Model), lower bound (B >: A in ::), context bound (T: Ordering).

  • Explain that an upper bound keeps the precise return type.

  • Mention that context bounds are the basis of type classes in Scala.

49

How are covariance and contravariance annotations used in Scala?

Variance annotations say how subtyping of a type argument carries over to the generic type: with covariance +A, if Dog <: Animal then Box[Dog] <: Box[Animal] (good for producers and immutable containers such as List[+A], Option[+A], Vector[+A]); with contravariance -A, the relation flips, so Printer[Animal] <: Printer[Dog] (good for consumers such as Ordering-like handlers and function inputs, Function1[-A, +B]); with no annotation the type is invariant (mutable collections like Array and ArrayBuffer, where both reading and writing happen). The compiler enforces safety: a covariant type may appear only in output positions, so methods that accept it use a lower bound B >: A.

Read full answer

Applies toScala 3 (same rules in Scala 2.13)

Summary

Declaration Dog <: Animal implies Use for Standard library examples
F[+A] covariant F[Dog] <: F[Animal] Producers, read-only/immutable data List, Vector, Option, Either, Future
F[-A] contravariant F[Animal] <: F[Dog] Consumers, handlers, functions’ inputs Function1[-T, +R] input
F[A] invariant No relation Read and write Array, mutable.ArrayBuffer, Set

Code Example (Scala 3)

//> using scala 3.5

sealed trait Animal:
  def name: String
final case class Dog(name: String) extends Animal
final case class Cat(name: String) extends Animal

// Covariant immutable container: produces A values
final case class Stack[+A](items: List[A]):
  def top: Option[A] = items.headOption
  // A cannot appear as a plain parameter type; a lower bound widens instead
  def push[B >: A](b: B): Stack[B] = Stack(b :: items)

// Contravariant consumer: only takes A values
trait Printer[-A]:
  def print(a: A): String

@main def varianceDemo(): Unit =
  val dogs: List[Dog] = List(Dog("Rex"))
  val animals: List[Animal] = dogs                // OK: List is covariant
  println(animals.map(_.name))                    // List(Rex)

  val dogStack = Stack(List(Dog("Rex")))
  val mixed: Stack[Animal] = dogStack.push(Cat("Tom"))
  println(mixed.items)                            // List(Cat(Tom), Dog(Rex))

  val animalPrinter: Printer[Animal] = a => s"Animal: ${a.name}"
  val dogPrinter: Printer[Dog] = animalPrinter    // OK: contravariant
  println(dogPrinter.print(Dog("Rex")))           // Animal: Rex

  // Functions: contravariant in input, covariant in output
  val f: Animal => String = _.name
  val g: Dog => Any = f                           // OK
  println(g(Dog("Fido")))                         // Fido

  // val arr: Array[Animal] = Array[Dog](Dog("Rex"))   // does not compile: Array is invariant

Why the Compiler Rejects Some Definitions

final case class BadBox[+A](value: A):
  def set(a: A): BadBox[A] = BadBox(a)
// error: covariant type A occurs in contravariant position in type A of parameter a

If it compiled, a BadBox[Dog] used as BadBox[Animal] could be given a Cat. Java arrays are covariant and fail this way at runtime with ArrayStoreException; Scala makes Array invariant to catch it at compile time.

Variance in Data/ML Code

Example Variance Reason
Model[-In, +Out] (a predictor) In contravariant, Out covariant A model accepting any Features can serve where a DenseFeatures model is needed
Source[+Row] reading records Covariant Only produces rows
Sink[-Row] writing records Contravariant Only consumes rows
Mutable feature buffer Invariant Reads and writes
Spark Dataset[T] Invariant Dataset[Dog] is not a Dataset[Animal]; use .map or .as[...]

Common Pitfalls

  • Making a type covariant and then trying to accept A as a parameter; use [B >: A].

  • Expecting mutable collections or Dataset[T] to be covariant.

  • Overusing variance on types that are rarely subtyped; invariance is the simpler default.

  • Confusing variance (relation between F[A] and F[B]) with type bounds (constraints on A).

Interview Tips

  • Use the producer/consumer rule: produces → +, consumes → -, both → invariant.

  • Quote Function1[-A, +B] and explain it.

  • Explain the lower-bound trick in List.:: (def ::[B >: A](x: B): List[B]).

50

What are macros in Scala, and what are their implications?

Macros are code that runs inside the compiler to generate or inspect program code at compile time, so libraries can derive boilerplate (JSON codecs, Spark-style encoders, type-class instances), check values before the program runs, or produce better error messages (ScalaTest’s assert shows each sub-expression’s value). Scala 3 replaced Scala 2’s experimental def macros with a principled system: inline methods for guaranteed inlining and compile-time branching, scala.compiletime utilities (error, constValue, summonInline), quotes and splices ('{ ... } and ${ ... }) for typed code generation, and derives with Mirrors for type-class derivation. The implications are real: macros are powerful but increase compile time, complicate tooling and debugging, and Scala 2 macros do not work in Scala 3, which is a major migration cost.

Read full answer

Applies toScala 3 (inline, quotes, derivation); notes on Scala 2.13 macros

Scala 2 vs Scala 3 Metaprogramming

Feature Scala 2.13 Scala 3
Def macros scala.reflect.macros.blackbox/whitebox (experimental) Not supported; rewritten with quotes
Inlining @inline hint (not guaranteed) inline keyword (guaranteed)
Compile-time errors Macro c.abort scala.compiletime.error
Type-class derivation Shapeless / Magnolia macros derives + Mirror (built in)
Typed code generation Quasiquotes q"..." (untyped trees) Quotes '{...} / splices ${...} (typed)
Cross-version use Scala 2 only Scala 3 only

Code Example 1: inline Compile-Time Checks (Scala 3)

//> using scala 3.5
import scala.compiletime.error

inline def percent(inline p: Int): Int =
  inline if p < 0 || p > 100 then error("percent must be between 0 and 100")
  else p

@main def inlineDemo(): Unit =
  println(percent(42))      // 42
  // percent(150)           // does not compile: percent must be between 0 and 100

The check runs while compiling, so an invalid literal never reaches production. With a non-constant argument the inline if cannot be reduced and the compiler reports that instead, so this pattern is for literal configuration values.

Code Example 2: A Quotes Macro (Scala 3)

Macro implementations must be compiled before the code that calls them, so they live in a separate file (or module):

// Debug.scala
import scala.quoted.*

inline def debug[T](inline expr: T): T = ${ debugImpl('expr) }

def debugImpl[T: Type](expr: Expr[T])(using Quotes): Expr[T] =
  val code = Expr(expr.show)            // source of the expression, captured at compile time
  '{
    val value = $expr                    // evaluate once at runtime
    println(${ code } + " = " + value)
    value
  }
// Main.scala
@main def macroDemo(): Unit =
  val lr = 0.1
  val steps = 20
  val total = debug(lr * steps)          // prints the expression's code followed by " = 2.0"
  println(total)                         // 2.0

The generated code is ordinary bytecode: no reflection, no runtime cost beyond the println.

Code Example 3: Derivation Instead of Hand-Written Macros

//> using scala 3.5
//> using dep io.circe::circe-generic:0.14.10

import io.circe.Codec
import io.circe.syntax.*

case class Prediction(id: Long, label: String, score: Double) derives Codec.AsObject

@main def deriveDemo(): Unit =
  println(Prediction(7L, "spam", 0.9).asJson.noSpaces)
  // {"id":7,"label":"spam","score":0.9}

Most application code should use derivation like this rather than write macros.

Implications

Benefit Cost
Removes boilerplate (codecs, encoders, lenses) Longer compile times, especially with heavy derivation
Errors found at compile time Error messages can point at generated code
No runtime reflection; fast generated code Harder debugging and IDE support
Domain-specific checks (SQL in Quill, regexes) Tight coupling to compiler version; Scala 2 macros block migration to Scala 3
Better assertion messages in tests Fewer people on the team can maintain macro code

Where You Meet Macros in Data/ML Work

  • JSON/Avro/Protobuf codecs (circe, jsoniter-scala, zio-json) for feature payloads and model APIs.

  • Type-safe query libraries (Quill, Doobie fragments) that check SQL at compile time.

  • Test frameworks (ScalaTest, MUnit) for rich assertion output.

  • Spark itself uses Scala 2 runtime reflection (TypeTag) for case-class encoders, not macros, which is one reason Spark still targets Scala 2.13.

Common Pitfalls

  • Writing a macro when an inline method, a type class or derives would do.

  • Putting the macro implementation and its call in the same compilation unit (Scala 3 rejects it).

  • Depending on Scala 2 macro libraries in code you plan to move to Scala 3.

  • Generating huge code from derivation of large case classes, slowing builds and hitting method-size limits.

Interview Tips

  • Separate the Scala 3 tools: inline, compiletime, quotes/splices, derives.

  • Give one concrete use (codec derivation or compile-time validation) and one concrete cost (compile time, migration).

  • Say you would prefer libraries that provide derivation over writing custom macros in application code.

51

Demonstrate implementing k-means clustering in Scala.

k-means partitions points into k clusters by repeating two steps until the centroids stop moving: assign every point to its nearest centroid, then move each centroid to the mean of its assigned points (Lloyd’s algorithm). In Scala this maps cleanly onto immutable collections and a tail-recursive loop: points are Vector[Double], assignment is a minBy over centroid indices, the update is a group-and-average, and recursion stops when the largest centroid shift is below a tolerance or the iteration limit is reached. For large data, Spark’s KMeans in spark.ml runs the same algorithm in parallel with k-means|| initialization.

Read full answer

Applies toScala 3 (from scratch); Spark 3.5 / 4.x with Scala 2.13 (distributed)

Algorithm

Step Action
1. Initialize Pick k starting centroids (random points, or k-means++ for better starts)
2. Assign Label each point with the index of its nearest centroid (squared Euclidean distance)
3. Update Each centroid becomes the mean of its points (keep the old centroid if a cluster is empty)
4. Repeat Until the max centroid shift ≤ tolerance or max iterations
Objective Minimize SSE = Σ ‖x − c(x)‖² (within-cluster sum of squares)

Code Example: From Scratch (Scala 3)

//> using scala 3.5
import scala.annotation.tailrec

type Point = Vector[Double]

def dist2(a: Point, b: Point): Double = a.lazyZip(b).map((x, y) => (x - y) * (x - y)).sum

def mean(ps: Seq[Point]): Point = ps.transpose.map(_.sum / ps.size).toVector

def assign(points: Vector[Point], cs: Vector[Point]): Vector[Int] =
  points.map(p => cs.indices.minBy(i => dist2(p, cs(i))))

@tailrec
def kmeans(points: Vector[Point], cs: Vector[Point], maxIter: Int, tol: Double = 1e-9): (Vector[Point], Vector[Int]) =
  val labels = assign(points, cs)
  val newCs = cs.indices.toVector.map { i =>
    val members = points.zip(labels).collect { case (p, `i`) => p }
    if members.isEmpty then cs(i) else mean(members)       // keep an empty cluster's centroid
  }
  val shift = cs.lazyZip(newCs).map(dist2).max
  if shift <= tol || maxIter <= 1 then (newCs, assign(points, newCs))
  else kmeans(points, newCs, maxIter - 1, tol)

def sse(points: Vector[Point], cs: Vector[Point], labels: Vector[Int]): Double =
  points.zip(labels).map((p, l) => dist2(p, cs(l))).sum

@main def runKMeans(): Unit =
  val points: Vector[Point] = Vector(
    Vector(1.0, 1.0), Vector(1.5, 2.0), Vector(1.0, 1.5),   // group A
    Vector(8.0, 8.0), Vector(9.0, 8.5), Vector(8.5, 9.0)    // group B
  )
  // Deterministic start: one point from each end (use k-means++ with a seeded Random in practice)
  val init = Vector(points(0), points(3))

  val (centroids, labels) = kmeans(points, init, maxIter = 100)

  centroids.foreach(c => println(c.map(v => f"$v%.2f").mkString("(", ", ", ")")))
  // (1.17, 1.50)
  // (8.50, 8.50)
  println(labels)                                     // Vector(0, 0, 0, 1, 1, 1)
  println(f"SSE = ${sse(points, centroids, labels)}%.3f")   // SSE = 1.667

Trace: the first assignment already separates A and B, the centroids move to (1.17, 1.50) and (8.50, 8.50), the second pass changes nothing (shift 0), and the recursion stops. SSE is 0.667 for cluster A plus 1.000 for cluster B.

k-means++ Initialization (Seeded)

import scala.util.Random

def kmeansPlusPlus(points: Vector[Point], k: Int, rng: Random): Vector[Point] =
  (1 until k).foldLeft(Vector(points(rng.nextInt(points.size)))) { (cs, _) =>
    val d2 = points.map(p => cs.map(c => dist2(p, c)).min)
    val target = rng.nextDouble() * d2.sum
    val idx = d2.scanLeft(0.0)(_ + _).tail.indexWhere(_ >= target)
    cs :+ points(if idx < 0 then points.size - 1 else idx)
  }
// val init = kmeansPlusPlus(points, 2, Random(42))

Each new centroid is chosen with probability proportional to its squared distance from the nearest existing centroid, which spreads the starts out.

Distributed Version with Spark (Scala 2.13)

import org.apache.spark.ml.clustering.KMeans
import org.apache.spark.ml.evaluation.ClusteringEvaluator
import org.apache.spark.ml.feature.VectorAssembler

val df = Seq((1.0, 1.0), (1.5, 2.0), (1.0, 1.5), (8.0, 8.0), (9.0, 8.5), (8.5, 9.0)).toDF("x", "y")
val features = new VectorAssembler().setInputCols(Array("x", "y")).setOutputCol("features").transform(df)

val model = new KMeans().setK(2).setSeed(42).setMaxIter(20).fit(features)
model.clusterCenters.foreach(println)            // the two group means (order may differ)
val silhouette = new ClusteringEvaluator().evaluate(model.transform(features))

Choosing k

Method How
Elbow Plot SSE (model.summary.trainingCost in Spark) for k = 2..10 and look for the bend
Silhouette ClusteringEvaluator; higher is better
Domain need Number of segments the business can act on

Common Pitfalls

  • Not scaling features: a feature in thousands dominates the distance.

  • A single random start that lands in a poor local minimum; use k-means++ and several restarts.

  • Empty clusters producing NaN centroids (dividing by zero); handle them explicitly as above.

  • Using k-means on non-spherical clusters or categorical data (consider DBSCAN, GMM or k-modes).

  • Comparing SSE across different k as if lower is always better; it always decreases as k grows.

Interview Tips

  • Write the assign and update steps first; they are the whole algorithm.

  • State the complexity: O(n·k·d) per iteration.

  • Mention the convergence guarantee (SSE never increases) and that it reaches a local, not global, minimum.

52

Write a Scala implementation of logistic regression.

Logistic regression models the probability of the positive class as p = σ(w·x + b), where σ(z) = 1 / (1 + e^(−z)), and is trained by minimizing the average log loss with gradient descent: the gradient is simple, ∂L/∂w = (1/n) Σ (pᵢ − yᵢ) xᵢ and ∂L/∂b = (1/n) Σ (pᵢ − yᵢ), so a Scala implementation needs only a sigmoid, a prediction function and an update loop. Idiomatic Scala keeps the model immutable (a case class updated with copy or rebuilt in a foldLeft), standardizes the features first so gradient descent converges, and adds optional L2 regularization. For production-scale data use Spark’s LogisticRegression instead.

Read full answer

Applies toScala 3 (from-scratch version); Spark 3.5 / 4.x with Scala 2.13 for the distributed version

Algorithm

Step Formula
Score z = w·x + b
Probability p = σ(z)
Loss (log loss) L = −(1/n) Σ [y log p + (1 − y) log(1 − p)] + (λ/2)‖w‖²
Gradient ∇w = (1/n) Σ (p − y) x + λw, ∇b = (1/n) Σ (p − y)
Update w ← w − η∇w, b ← b − η∇b
Predict class 1 if p ≥ 0.5

Code Example: From Scratch (Scala 3)

//> using scala 3.5
import scala.math.exp

case class Example(x: Vector[Double], y: Int)

def sigmoid(z: Double): Double = 1.0 / (1.0 + exp(-z))

final case class Model(w: Vector[Double], b: Double):
  def prob(x: Vector[Double]): Double = sigmoid(w.lazyZip(x).map(_ * _).sum + b)
  def predict(x: Vector[Double]): Int = if prob(x) >= 0.5 then 1 else 0

def train(data: Seq[Example], lr: Double, epochs: Int, l2: Double = 0.0): Model =
  val n = data.size.toDouble
  val d = data.head.x.size
  (1 to epochs).foldLeft(Model(Vector.fill(d)(0.0), 0.0)) { (m, _) =>
    val errors = data.map(e => (m.prob(e.x) - e.y, e.x))      // (p - y, x)
    val gradW = Vector.tabulate(d)(j => errors.map((err, x) => err * x(j)).sum / n + l2 * m.w(j))
    val gradB = errors.map(_._1).sum / n
    Model(m.w.lazyZip(gradW).map((wj, gj) => wj - lr * gj), m.b - lr * gradB)
  }

// Standardization parameters learned from the training data only
final case class Scaler(mean: Vector[Double], std: Vector[Double]):
  def apply(x: Vector[Double]): Vector[Double] =
    x.lazyZip(mean).lazyZip(std).map((v, m, s) => if s == 0 then 0.0 else (v - m) / s)

def fitScaler(xs: Seq[Vector[Double]]): Scaler =
  val d = xs.head.size
  val mean = Vector.tabulate(d)(j => xs.map(_(j)).sum / xs.size)
  val std = Vector.tabulate(d)(j => math.sqrt(xs.map(x => math.pow(x(j) - mean(j), 2)).sum / xs.size))
  Scaler(mean, std)

@main def runLogReg(): Unit =
  // One feature (hours studied) -> passed (1) or not (0)
  val raw = Seq(1.0 -> 0, 2.0 -> 0, 3.0 -> 0, 6.0 -> 1, 7.0 -> 1, 8.0 -> 1)
  val scaler = fitScaler(raw.map((h, _) => Vector(h)))
  val data = raw.map((h, y) => Example(scaler(Vector(h)), y))

  val model = train(data, lr = 0.5, epochs = 500, l2 = 0.01)

  val preds = data.map(e => model.predict(e.x))
  println(preds)                                         // List(0, 0, 0, 1, 1, 1)
  val acc = preds.zip(data).count((p, e) => p == e.y).toDouble / data.size
  println(acc)                                           // 1.0
  println(model.w.head > 0)                              // true: more hours -> higher probability
  println(model.predict(scaler(Vector(4.0))))            // 0
  println(model.predict(scaler(Vector(5.0))))            // 1

Run it with scala-cli run logreg.scala. The data is symmetric around 4.5 hours after scaling, so the learned boundary sits at 4.5: 4 hours is predicted 0 and 5 hours is predicted 1. The exact weight values depend on the learning rate and epochs, so only their sign is printed.

Distributed Version with Spark (Scala 2.13)

import org.apache.spark.ml.classification.LogisticRegression
import org.apache.spark.ml.feature.VectorAssembler
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder.appName("lr").master("local[*]").getOrCreate()
import spark.implicits._

val df = Seq((1.0, 0.0), (2.0, 0.0), (3.0, 0.0), (6.0, 1.0), (7.0, 1.0), (8.0, 1.0)).toDF("hours", "label")
val features = new VectorAssembler().setInputCols(Array("hours")).setOutputCol("features").transform(df)

val lrModel = new LogisticRegression()
  .setMaxIter(100).setRegParam(0.01).setStandardization(true)
  .fit(features)

lrModel.transform(features).select("hours", "probability", "prediction").show(truncate = false)
println(lrModel.coefficients)   // positive coefficient for hours

Spark uses L-BFGS / OWL-QN rather than plain gradient descent and standardizes internally.

From Scratch vs Library

Aspect From scratch Spark LogisticRegression
Purpose Learning, interviews, tiny embedded models Real data, many features
Optimizer Batch gradient descent L-BFGS, OWL-QN (L1)
Scale Fits in memory Distributed
Extras Write yourself Elastic net, multinomial, thresholds, summary metrics

Common Pitfalls

  • Not scaling features, so gradient descent needs a tiny learning rate and many epochs.

  • Computing log(p) directly when p rounds to 0 or 1; clip p or use a log-sum-exp form for the loss.

  • Regularizing the bias term.

  • Perfectly separable data without regularization: weights grow without limit.

  • Fitting the scaler on all data instead of the training split.

Interview Tips

  • Derive the gradient (p − y)x out loud; it is the expected follow-up.

  • Explain why the model is kept immutable and trained in a foldLeft.

  • Mention vectorized libraries (Breeze) and Spark for anything beyond a toy.

53

How would you build an ALS recommendation system in Scala?

Alternating Least Squares (ALS) factorizes the sparse user-item rating matrix R into user factors U and item factors V so that R ≈ U·Vᵀ, and Spark’s spark.ml.recommendation.ALS implements it at scale in Scala: it fixes V and solves a regularized least-squares problem for every user, then fixes U and solves for every item, alternating until convergence; each step is embarrassingly parallel across users or items. You give it a DataFrame of (userId, itemId, rating) with integer IDs, set rank, regParam, maxIter, and implicitPrefs for click/view data, evaluate held-out RMSE (explicit) or ranking metrics (implicit), and generate top-N lists with recommendForAllUsers or recommendForUserSubset.

Read full answer

Applies toScala 2.13, Apache Spark 3.5 / 4.x (`spark.ml`)

How ALS Works

Step What happens
Initialize Random user factors U (n_users × rank) and item factors V (n_items × rank)
Fix V, solve U For each user u: minimize Σ (r_ui − u·v_i)² + λ‖u‖² over the items they rated (a small linear system)
Fix U, solve V Same for each item
Repeat maxIter times (10-20 is typical)
Predict r̂_ui = u_u · v_i

For implicit feedback (implicitPrefs = true), ratings become confidence weights c = 1 + α·r on a binary preference, and the loss covers all user-item pairs.

Code Example (Spark, Scala 2.13)

import org.apache.spark.ml.evaluation.RegressionEvaluator
import org.apache.spark.ml.recommendation.ALS
import org.apache.spark.sql.SparkSession

case class Rating(userId: Int, itemId: Int, rating: Float)

object AlsDemo {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder.appName("als").master("local[*]").getOrCreate()
    import spark.implicits._

    // In production: spark.read.parquet(...).as[Rating], with string IDs mapped to Ints first
    val ratings = Seq(
      Rating(1, 10, 5f), Rating(1, 11, 4f), Rating(1, 12, 1f),
      Rating(2, 10, 4f), Rating(2, 12, 2f), Rating(2, 13, 5f),
      Rating(3, 11, 5f), Rating(3, 13, 4f), Rating(3, 14, 1f),
      Rating(4, 10, 1f), Rating(4, 14, 5f), Rating(4, 12, 4f)
    ).toDS()

    val Array(train, test) = ratings.randomSplit(Array(0.8, 0.2), seed = 42)

    val als = new ALS()
      .setUserCol("userId").setItemCol("itemId").setRatingCol("rating")
      .setRank(8)
      .setRegParam(0.1)
      .setMaxIter(15)
      .setNonnegative(true)
      .setColdStartStrategy("drop")   // drop NaN predictions for users/items unseen in training
      .setSeed(42)

    val model = als.fit(train)

    val rmse = new RegressionEvaluator()
      .setMetricName("rmse").setLabelCol("rating").setPredictionCol("prediction")
      .evaluate(model.transform(test))
    println(s"test RMSE = $rmse")

    // Top-3 items per user; returns userId, recommendations: array<struct<itemId, rating>>
    model.recommendForAllUsers(3).show(truncate = false)

    // Recommendations for selected users only
    model.recommendForUserSubset(Seq(1, 2).toDF("userId"), 3).show(truncate = false)

    // Remove items the user already rated before showing them
    // (recommendForAllUsers does not filter them out)

    spark.stop()
  }
}

With twelve ratings this only demonstrates the API; the RMSE and lists depend on the split and are not meaningful at this size.

Production Design

Concern Approach
ID mapping Map string user/item IDs to dense Ints (StringIndexer or a lookup table)
Tuning CrossValidator or TrainValidationSplit over rank, regParam, alpha
Implicit data setImplicitPrefs(true).setAlpha(...); evaluate with precision@k / NDCG, not RMSE
Cold start Fall back to popularity or content-based recommendations for new users/items
Serving Precompute top-N nightly to a key-value store; or export factors to a vector index
Filtering Remove already-seen items and unavailable items after scoring

Scala vs PySpark

Aspect Scala PySpark
ALS implementation Same JVM code Same
Post-processing (filters, business rules) Typed Datasets, JVM UDFs Python UDFs are slower; use built-in functions
Exporting factors to a service JVM services read them directly Usually via Parquet

Common Pitfalls

  • Leaving the default cold-start strategy (nan), which makes the RMSE NaN.

  • Evaluating implicit-feedback models with RMSE.

  • IDs larger than Int.MaxValue or non-integer IDs; ALS requires integer IDs.

  • Random splits that leak future interactions; for real systems split by time.

  • Recommending items the user already bought.

Interview Tips

  • Explain why alternating makes each step a convex least-squares problem that parallelizes per user and per item.

  • Distinguish explicit vs implicit feedback and the matching metrics.

  • Always mention cold start and how you would handle it.

54

How would you use random forests for classification in Scala?

In Scala, random forest classification is usually done with Spark’s RandomForestClassifier from spark.ml, which trains many decision trees in parallel on bootstrap samples of the rows with a random subset of features at each split, then predicts by majority vote (averaging the trees’ class probabilities). You index the string label with StringIndexer, assemble numeric and indexed categorical columns into a feature vector with VectorAssembler, set numTrees, maxDepth, featureSubsetStrategy and seed, fit inside a Pipeline, evaluate with MulticlassClassificationEvaluator, and read featureImportances from the fitted model. Trees need no feature scaling. For single-machine data, the Smile library offers a randomForest function in pure Scala/Java.

Read full answer

Applies toScala 2.13, Apache Spark 3.5 / 4.x (`spark.ml`); Smile 3.x for local training

Main Parameters

Parameter Meaning Typical start
numTrees Number of trees 100-300
maxDepth Maximum depth per tree (Spark max is 30) 5-15
featureSubsetStrategy Features tried per split: auto, sqrt, log2, onethird sqrt for classification
subsamplingRate Fraction of rows per tree 1.0
minInstancesPerNode Minimum rows in a leaf 1-10
maxBins Bins for continuous features; must be ≥ categories in any categorical feature 32-64
impurity gini or entropy gini
seed Reproducibility 42

Code Example (Spark, Scala 2.13)

import org.apache.spark.ml.{Pipeline, PipelineModel}
import org.apache.spark.ml.classification.{RandomForestClassificationModel, RandomForestClassifier}
import org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
import org.apache.spark.ml.feature.{IndexToString, StringIndexer, VectorAssembler}
import org.apache.spark.sql.SparkSession

object RandomForestDemo {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder.appName("rf").master("local[*]").getOrCreate()

    // Columns: petalLength, petalWidth, sepalLength, colour, species
    val data = spark.read.option("header", "true").option("inferSchema", "true").csv(args(0))
    val Array(train, test) = data.randomSplit(Array(0.8, 0.2), seed = 42)

    val labelIdx  = new StringIndexer().setInputCol("species").setOutputCol("label").fit(train)
    val colourIdx = new StringIndexer().setInputCol("colour").setOutputCol("colourIdx").setHandleInvalid("keep")
    val assembler = new VectorAssembler()
      .setInputCols(Array("petalLength", "petalWidth", "sepalLength", "colourIdx"))
      .setOutputCol("features")

    val rf = new RandomForestClassifier()
      .setLabelCol("label").setFeaturesCol("features")
      .setNumTrees(200).setMaxDepth(8)
      .setFeatureSubsetStrategy("sqrt")
      .setSeed(42)

    val toName = new IndexToString().setInputCol("prediction").setOutputCol("predictedSpecies")
      .setLabels(labelIdx.labelsArray(0))

    val model: PipelineModel = new Pipeline()
      .setStages(Array(labelIdx, colourIdx, assembler, rf, toName))
      .fit(train)

    val predictions = model.transform(test)
    val evaluator = new MulticlassClassificationEvaluator().setLabelCol("label")
    println(s"accuracy = ${evaluator.setMetricName("accuracy").evaluate(predictions)}")
    println(s"macro F1 = ${evaluator.setMetricName("f1").evaluate(predictions)}")

    // Feature importances (normalized to sum to 1)
    val forest = model.stages(3).asInstanceOf[RandomForestClassificationModel]
    assembler.getInputCols.zip(forest.featureImportances.toArray)
      .sortBy(-_._2)
      .foreach { case (name, imp) => println(f"$name%-12s $imp%.3f") }

    println(s"trees = ${forest.getNumTrees}")   // trees = 200
    spark.stop()
  }
}

The printed accuracy, F1 and importances depend on the data file, so they are not shown.

Tuning

import org.apache.spark.ml.tuning.{CrossValidator, ParamGridBuilder}

val grid = new ParamGridBuilder()
  .addGrid(rf.numTrees, Array(100, 300))
  .addGrid(rf.maxDepth, Array(5, 10))
  .build()

val cv = new CrossValidator()
  .setEstimator(new Pipeline().setStages(Array(labelIdx, colourIdx, assembler, rf)))
  .setEvaluator(new MulticlassClassificationEvaluator().setMetricName("f1"))
  .setEstimatorParamMaps(grid)
  .setNumFolds(3)
  .setParallelism(2)
  .setSeed(42)

Scala vs PySpark

Aspect Scala PySpark
RandomForestClassifier speed Same JVM implementation Same
Accessing tree internals (trees, toDebugString) Direct Available, fewer helpers
Local alternative Smile scikit-learn
Type safety of the job Compile time Runtime

Common Pitfalls

  • maxBins smaller than the number of categories in an indexed categorical feature: training fails.

  • Fitting the label StringIndexer on all data, so test labels leak into the encoding (minor) or new labels crash (major).

  • Very deep trees with many trees: large model size and slow scoring; maxDepth above ~15 rarely helps.

  • Reading impurity-based importances as causal; they favour high-cardinality features. Check permutation importance on a validation set.

  • Forgetting setSeed, so results differ between runs.

Interview Tips

  • Explain bagging plus feature subsampling and why it lowers variance compared with one tree.

  • Mention that Spark grows the trees in parallel by aggregating split statistics across partitions.

  • Say you would compare with GBTClassifier and pick by validation metric.

55

How does Apache Spark integrate with Scala for machine learning?

Spark is written in Scala, so Scala is its native language: the Spark SQL engine, the spark.ml library and the cluster runtime are Scala/JVM code, and a Scala program calls them directly without any language bridge. For ML this means you can read and clean data as typed Dataset[T]s of case classes, write feature logic as Scala functions or UDFs that run inside the executors’ JVMs with no Python serialization, build spark.ml Pipelines with the same classes PySpark wraps, extend the library with custom Transformers and Estimators written in Scala, and package the job as a jar submitted with spark-submit. Scala jobs are compiled and type-checked, which catches many schema and logic errors before a long cluster run.

Read full answer

Applies toScala 2.13, Apache Spark 3.5 / 4.x

Integration Points

Area How Scala integrates
Data spark.read → DataFrame; .as[CaseClass] → typed Dataset
Features Built-in functions._, Scala UDFs, spark.ml.feature stages
Models spark.ml estimators (logistic regression, trees, GBT, ALS, k-means)
Extension Subclass Transformer / Estimator / UnaryTransformer
Streaming Structured Streaming + PipelineModel.transform for online scoring
Build & deploy sbt or Maven jar, Spark as provided, spark-submit

Build Setup

// build.sbt
ThisBuild / scalaVersion := "2.13.14"

lazy val root = (project in file("."))
  .settings(
    name := "churn-ml",
    libraryDependencies ++= Seq(
      "org.apache.spark" %% "spark-sql"   % "3.5.3" % "provided",
      "org.apache.spark" %% "spark-mllib" % "3.5.3" % "provided"
    )
  )

%% appends the Scala binary version (_2.13), which must match the cluster’s Spark build.

Code Example: Typed Data Prep + Scala UDF + Pipeline

import org.apache.spark.ml.Pipeline
import org.apache.spark.ml.classification.GBTClassifier
import org.apache.spark.ml.feature.{StringIndexer, VectorAssembler}
import org.apache.spark.sql.{Dataset, SparkSession}
import org.apache.spark.sql.functions.{col, udf}

case class Customer(id: Long, plan: String, tenureMonths: Int, monthlySpend: Double, churned: Option[Int])

object ChurnJob {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder.appName("churn").getOrCreate()
    import spark.implicits._

    // Typed read: a missing or misnamed column fails here, before training
    val customers: Dataset[Customer] = spark.read.parquet(args(0)).as[Customer]

    // Typed filtering and cleaning in plain Scala
    val labelled = customers.filter(_.churned.isDefined)
      .map(c => c.copy(plan = c.plan.trim.toLowerCase))

    // Scala UDF: runs in the executor JVM, no Python round trip
    val spendBand = udf((spend: Double) => if (spend >= 100) "high" else if (spend >= 40) "mid" else "low")

    val df = labelled.toDF()
      .withColumn("label", col("churned").cast("double"))
      .withColumn("band", spendBand(col("monthlySpend")))

    val stages = Array(
      new StringIndexer().setInputCols(Array("plan", "band"))
        .setOutputCols(Array("planIdx", "bandIdx")).setHandleInvalid("keep"),
      new VectorAssembler()
        .setInputCols(Array("planIdx", "bandIdx", "tenureMonths", "monthlySpend"))
        .setOutputCol("features"),
      new GBTClassifier().setMaxIter(50).setSeed(42)
    )

    val Array(train, test) = df.randomSplit(Array(0.8, 0.2), seed = 42)
    val model = new Pipeline().setStages(stages).fit(train)
    model.transform(test).select("id", "prediction", "probability").show(5)
    model.write.overwrite().save(args(1))

    spark.stop()
  }
}
// spark-submit --class ChurnJob --master yarn churn-ml_2.13-0.1.jar s3://bucket/customers s3://bucket/models/churn

Scala vs PySpark for Spark ML

Aspect Scala PySpark
UDF performance Native JVM Row serialization to Python (pandas UDFs reduce it)
Type safety Dataset[T], compile time Runtime only
Custom ML stages Full access to internal APIs Wrappers; Python stages are slower
ML/DL ecosystem Smaller Much larger (scikit-learn, PyTorch)
Notebook exploration Possible (Databricks, Almond) Most common
Team skills Data/platform engineers Data scientists

A common split: data scientists prototype in PySpark, and production feature pipelines that need custom UDFs or strict contracts are written in Scala.

Common Pitfalls

  • Scala binary mismatch: a jar built for 2.12 on a 2.13 Spark (or the reverse) fails with NoSuchMethodError.

  • Closures referencing the enclosing object (this) pull non-serializable fields into tasks: Task not serializable.

  • Using Int instead of Option[Int] for nullable columns, which fails at runtime on nulls.

  • Shipping Spark itself in a fat jar instead of marking it provided.

Interview Tips

  • Lead with “Spark is written in Scala, so there is no bridge”, then give concrete gains: UDFs, typed Datasets, custom stages.

  • Be balanced: mention that Python has the bigger ML ecosystem.

  • Mention binary compatibility (%%, _2.13) as a real production concern.

56

What are RDDs and how do they work in Scala?

An RDD (Resilient Distributed Dataset) is Spark’s original core abstraction: an immutable, partitioned collection of objects spread across the cluster, built by coarse-grained transformations and recovered on failure from its lineage. In Scala an RDD[T] looks like a Scala collection: you call map, filter, flatMap, reduceByKey, join with Scala functions, which Spark serializes and ships to executors. Transformations are lazy and only build a lineage graph (DAG); an action (collect, count, reduce, saveAsTextFile) triggers a job, which the scheduler splits into stages at shuffle boundaries. If a partition is lost, Spark recomputes it from the lineage instead of replicating data (“resilient”). Today DataFrames and Datasets are preferred for most work, but RDDs remain the low-level API underneath them.

Read full answer

Applies toScala 2.13, Apache Spark 3.5 / 4.x

Key Properties

Property Meaning
Resilient Lost partitions are recomputed from lineage
Distributed Data split into partitions processed in parallel on executors
Immutable Every transformation returns a new RDD
Lazy Transformations build a DAG; actions run it
Typed RDD[T] checked by the Scala compiler
Persistable cache() / persist(StorageLevel) keeps partitions in memory or disk

Narrow vs Wide Transformations

Type Examples Shuffle?
Narrow map, filter, flatMap, mapPartitions No: each output partition depends on one input partition
Wide reduceByKey, groupByKey, join, distinct, repartition Yes: data moves across the network; new stage

Code Example: Word Count (Spark, Scala 2.13)

import org.apache.spark.sql.SparkSession
import org.apache.spark.storage.StorageLevel

object RddDemo {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder.appName("rdd-demo").master("local[2]").getOrCreate()
    val sc = spark.sparkContext

    val lines = sc.parallelize(Seq(
      "spark makes rdds",
      "rdds are immutable",
      "spark is fast"
    ), numSlices = 2)

    val counts = lines
      .flatMap(_.split(" "))          // narrow
      .map(word => (word, 1))         // narrow
      .reduceByKey(_ + _)             // wide: shuffle, combines on each partition first
      .persist(StorageLevel.MEMORY_ONLY)

    counts.sortBy { case (w, c) => (-c, w) }   // action below triggers the job
      .collect()
      .foreach(println)
    // (rdds,2)
    // (spark,2)
    // (are,1)
    // (fast,1)
    // (immutable,1)
    // (is,1)
    // (makes,1)

    println(counts.count())            // 7  (reuses the cached partitions)
    println(counts.toDebugString)      // prints the lineage: ShuffledRDD <- MapPartitionsRDD <- ...

    spark.stop()
  }
}

RDD Creation and Actions

Create Action
sc.parallelize(seq) collect(), take(n)
sc.textFile(path) count(), reduce(f), fold(z)(f)
df.rdd (from a DataFrame) foreach(f), saveAsTextFile(path)
Transformation of another RDD countByKey(), aggregate

When to Still Use RDDs

  • Unstructured data that does not fit a schema (raw text parsing, binary records).

  • Fine control over partitioning (partitionBy(new HashPartitioner(n))) or mapPartitions with expensive setup per partition.

  • Legacy code and some graph algorithms (GraphX is RDD-based).

For structured data, DataFrames are usually much faster because Catalyst optimizes the query and Tungsten stores rows in compact binary format, while RDD objects are opaque Java objects.

Common Pitfalls

  • groupByKey followed by a sum instead of reduceByKey: it ships all values over the network.

  • Closures capturing a non-serializable object (a DB connection, the enclosing class): Task not serializable. Create the resource inside mapPartitions.

  • Mutating a driver-side var inside foreach and expecting it to change: each executor works on a copy; use an accumulator.

  • Re-running an expensive lineage several times without persist.

  • Calling collect() on a large RDD.

Interview Tips

  • Explain lineage-based fault tolerance and why it avoids replication.

  • Distinguish narrow vs wide transformations and link wide ones to stages and shuffles.

  • Say when you would choose an RDD over a DataFrame, and that the DataFrame is usually the default.

57

Explain Spark's DataFrame API and how to use it with Scala.

A Spark DataFrame is a distributed table with named, typed columns, and in Scala it is literally Dataset[Row]: you create it from files (Parquet, CSV, JSON, Delta), JDBC or local collections, then build a query with lazy transformations (select, filter, withColumn, groupBy/agg, join, window functions) that the Catalyst optimizer rewrites and the Tungsten engine runs as generated JVM code; nothing executes until an action (show, count, collect, write). In Scala you can switch between the untyped DataFrame and a typed Dataset[T] of case classes with .as[T], which gives compile-time checking of field names and types for lambdas such as ds.filter(_.salary > 80).

Read full answer

Applies toScala 2.13, Apache Spark 3.5 / 4.x

DataFrame vs Dataset vs RDD

DataFrame (Dataset[Row]) Dataset[T] RDD
Schema Yes Yes (from case class) No
Compile-time type checks Column names checked at runtime Lambdas checked at compile time Yes
Catalyst optimization Full Full for column expressions; lambdas are opaque None
Typical use SQL-like ETL, features Typed business logic Low-level control

Code Example (Spark, Scala 2.13)

import org.apache.spark.sql.{Dataset, SparkSession}
import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.functions._

case class Employee(name: String, dept: String, salary: Int)

object DataFrameDemo {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder.appName("df-demo").master("local[*]").getOrCreate()
    import spark.implicits._

    val df = Seq(
      ("Asha", "Eng", 120), ("Ben", "Eng", 100),
      ("Chen", "Sales", 90), ("Dina", "Sales", 70), ("Eli", "HR", 60)
    ).toDF("name", "dept", "salary")

    // Transformations (lazy)
    val byDept = df.groupBy($"dept")
      .agg(avg($"salary").as("avg_salary"), count(lit(1)).as("n"))
      .orderBy($"dept")

    // Action
    byDept.show()
    // +-----+----------+---+
    // | dept|avg_salary|  n|
    // +-----+----------+---+
    // |  Eng|     110.0|  2|
    // |   HR|      60.0|  1|
    // |Sales|      80.0|  2|
    // +-----+----------+---+

    // Window function: rank within department
    val w = Window.partitionBy($"dept").orderBy($"salary".desc)
    df.withColumn("rank", rank().over(w))
      .filter($"rank" === 1)
      .orderBy($"dept")
      .select("dept", "name")
      .collect()
      .foreach(println)
    // [Eng,Asha]
    // [HR,Eli]
    // [Sales,Chen]

    // Typed Dataset: field names and types checked by the compiler
    val ds: Dataset[Employee] = df.as[Employee]
    println(ds.filter(_.salary > 80).map(_.name).collect().mkString(", "))
    // Asha, Ben, Chen

    // Spark SQL on the same data
    df.createOrReplaceTempView("employees")
    spark.sql("SELECT COUNT(*) AS n FROM employees WHERE salary >= 90").show()
    // +---+
    // |  n|
    // +---+
    // |  3|
    // +---+

    df.explain()   // prints the physical plan chosen by Catalyst
    spark.stop()
  }
}

Common Operations

Task Scala DataFrame code
Select / rename df.select($"a", $"b".as("c"))
Derived column df.withColumn("x2", $"x" * 2)
Filter df.filter($"age" > 30) or df.where("age > 30")
Missing values df.na.fill(0, Seq("x")), df.na.drop()
Join a.join(b, Seq("id"), "left")
Aggregate df.groupBy("k").agg(sum("v"), max("v"))
Write df.write.mode("overwrite").partitionBy("dt").parquet(path)

Scala vs PySpark DataFrames

Aspect Scala PySpark
Built-in functions Same engine, same speed Same engine, same speed
UDFs JVM, no serialization Python worker; use pandas UDFs or built-ins
Typed Datasets Yes No (DataFrame only)
Errors in column names Runtime (DataFrame), compile time (Dataset lambdas) Runtime

Common Pitfalls

  • Using UDFs for things built-in functions already do; UDFs are opaque to Catalyst and block optimizations such as predicate pushdown.

  • Calling collect() on a large DataFrame and running the driver out of memory.

  • Typed lambdas (ds.filter(_.salary > 80)) deserialize each row into an object; column expressions are faster.

  • Nulls in columns mapped to primitive fields (Int) in a case class: use Option[Int].

  • Reading CSV with inferSchema in production; define an explicit schema instead.

Interview Tips

  • Say clearly that a DataFrame is Dataset[Row] in Scala and explain when you use .as[T].

  • Explain lazy evaluation and that explain() shows the optimized plan.

  • Mention Catalyst and Tungsten as the reasons DataFrames beat hand-written RDD code.

58

How can you use Spark MLlib for machine learning in Scala?

Spark MLlib is Spark’s distributed machine-learning library, and in Scala you use its DataFrame-based API in the org.apache.spark.ml package: you load data into a DataFrame, build feature transformers (StringIndexer, OneHotEncoder, VectorAssembler, StandardScaler, Tokenizer, HashingTF), add an estimator (LogisticRegression, RandomForestClassifier, GBTClassifier, LinearRegression, KMeans, ALS), chain them in a Pipeline, call fit to get a PipelineModel, evaluate with an Evaluator, tune with CrossValidator, and save the model for batch or streaming scoring. Training runs in parallel across the cluster’s executors. The older RDD-based org.apache.spark.mllib API is in maintenance mode and should not be used for new work.

Read full answer

Applies toScala 2.13, Apache Spark 3.5 / 4.x (`spark.ml`). Spark 4 supports Scala 2.13 only; Spark 3.5 supports 2.12 and 2.13.

Building Blocks

Concept Scala type Role
Transformer Tokenizer, VectorAssembler, fitted models transform(df) adds columns
Estimator StringIndexer, LogisticRegression, KMeans fit(df) returns a model (a Transformer)
Pipeline Pipeline().setStages(Array(...)) Chains stages in order; itself an Estimator
Evaluator BinaryClassificationEvaluator, RegressionEvaluator Computes a metric from predictions
Tuning CrossValidator, TrainValidationSplit, ParamGridBuilder Searches hyperparameters
Persistence model.write.overwrite().save(path), PipelineModel.load(path) Reuse fitted pipelines

Code Example: Text Classification Pipeline (Scala 2.13)

import org.apache.spark.ml.{Pipeline, PipelineModel}
import org.apache.spark.ml.classification.LogisticRegression
import org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
import org.apache.spark.ml.feature.{HashingTF, IDF, Tokenizer}
import org.apache.spark.ml.tuning.{CrossValidator, ParamGridBuilder}
import org.apache.spark.sql.SparkSession

object SpamPipeline {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder.appName("spam").master("local[*]").getOrCreate()
    import spark.implicits._

    val data = Seq(
      ("win a free prize now", 1.0), ("cheap pills free offer", 1.0),
      ("claim your free reward", 1.0), ("limited offer win cash", 1.0),
      ("meeting moved to friday", 0.0), ("please review the report", 0.0),
      ("lunch at noon tomorrow", 0.0), ("project update attached", 0.0)
    ).toDF("text", "label")

    val tokenizer = new Tokenizer().setInputCol("text").setOutputCol("words")
    val tf        = new HashingTF().setInputCol("words").setOutputCol("tf").setNumFeatures(1 << 12)
    val idf       = new IDF().setInputCol("tf").setOutputCol("features")
    val lr        = new LogisticRegression().setMaxIter(50)

    val pipeline = new Pipeline().setStages(Array(tokenizer, tf, idf, lr))

    val grid = new ParamGridBuilder()
      .addGrid(lr.regParam, Array(0.01, 0.1))
      .addGrid(lr.elasticNetParam, Array(0.0, 0.5))
      .build()

    val cv = new CrossValidator()
      .setEstimator(pipeline)
      .setEvaluator(new BinaryClassificationEvaluator().setMetricName("areaUnderROC"))
      .setEstimatorParamMaps(grid)
      .setNumFolds(2)
      .setParallelism(2)
      .setSeed(42)

    val model: PipelineModel = cv.fit(data).bestModel.asInstanceOf[PipelineModel]

    val newMail = Seq("free cash prize", "report for friday meeting").toDF("text")
    model.transform(newMail).select("text", "prediction", "probability").show(truncate = false)

    model.write.overwrite().save("/tmp/spam_model")
    val reloaded = PipelineModel.load("/tmp/spam_model")   // same stages, ready for scoring
    println(reloaded.stages.map(_.getClass.getSimpleName).mkString(", "))
    // Tokenizer, HashingTF, IDFModel, LogisticRegressionModel

    spark.stop()
  }
}

A toy dataset of eight rows only shows the API; with real data, keep a held-out test set and report the evaluator’s metric on it.

Running It

sbt package
spark-submit --class SpamPipeline --master local[*] target/scala-2.13/spam_2.13-0.1.jar

Scala vs PySpark for MLlib

Aspect Scala PySpark
MLlib API Native; same classes Thin wrapper over the same JVM classes
Built-in stages speed Same (runs in JVM) Same (runs in JVM)
Custom UDFs / custom Transformers Run in the JVM, no serialization Python UDFs move rows to Python workers (pandas UDFs help)
Type safety Dataset[T], compile-time errors Runtime errors
ML ecosystem Smaller (Smile, Breeze) Larger (scikit-learn, PyTorch, XGBoost)
Team skills Data engineers Data scientists

Common Pitfalls

  • Fitting StringIndexer, StandardScaler or IDF on the full dataset before splitting: put them in the pipeline so they are fit on training folds only.

  • Forgetting setHandleInvalid("keep") on StringIndexer, so unseen categories crash scoring.

  • Mixing the old mllib (RDD, LabeledPoint) and new ml (DataFrame) APIs.

  • Building a jar with Scala 2.12 for a Spark built on 2.13 (NoSuchMethodError at runtime).

  • Marking Spark as a normal dependency instead of % "provided", which bloats the jar and causes version conflicts.

Interview Tips

  • Describe the Transformer / Estimator / Pipeline model and why pipelines prevent leakage.

  • Mention CrossValidator.setParallelism and that each fit is itself distributed.

  • Say that for deep learning you would train in Python and keep Spark/Scala for the data and feature work.

59

What techniques optimize Scala code performance?

Optimizing Scala code means first measuring, then removing the specific costs idiomatic Scala adds on the JVM: choose the right collection (Array/ArrayBuffer/Vector for indexed access, not List), avoid intermediate collections with view, iterator or a single foldLeft, avoid boxing in numeric code by using primitive arrays, use @tailrec or while loops in hot paths, prefer StringBuilder to string concatenation in loops, cache with lazy val or memoization, and use parallelism (Future, parallel collections, Spark) only when the work is large enough. Most of the time the biggest wins are algorithmic (a Map lookup instead of a linear find), and they should be confirmed with JMH and a profiler rather than guessed.

Read full answer

Applies toScala 3 and Scala 2.13 (JDK 17 / 21)

Optimization Checklist

Technique When it helps Example
Better algorithm / data structure Always first Set.contains instead of List.contains
Right collection Indexed or random access Vector/Array instead of List(i)
Avoid intermediate collections Long map/filter chains .view or .iterator, or one foldLeft
Avoid boxing Numeric hot loops Array[Double], while loop
Tail recursion Recursive algorithms @tailrec
StringBuilder Building strings in loops sb.append(...)
Laziness / memoization Expensive values used sometimes lazy val, mutable.Map cache
Local mutability Hot inner loops var inside a method, immutable API outside
Parallelism CPU-heavy, large work Future, .par, Spark

Code Example: Same Result, Different Costs (Scala 3)

//> using scala 3.5

@main def sumSquares(): Unit =
  val n = 1000

  // 1. Readable but allocates a List of 1000 squares, then sums it
  val v1 = (1 to n).toList.map(i => i.toLong * i).sum

  // 2. A view fuses map and sum: no intermediate collection
  val v2 = (1 to n).view.map(i => i.toLong * i).sum

  // 3. Single pass with foldLeft over a Range (no list built)
  val v3 = (1 to n).foldLeft(0L)((acc, i) => acc + i.toLong * i)

  // 4. While loop: no closures, no boxing - fastest in hot code
  var i = 1
  var v4 = 0L
  while i <= n do
    v4 += i.toLong * i
    i += 1

  println(Seq(v1, v2, v3, v4))   // List(333833500, 333833500, 333833500, 333833500)

All four give 333,833,500 (= n(n+1)(2n+1)/6). In most code version 2 or 3 is the right trade-off; version 4 is for measured hot spots.

Code Example: Data Structure Choice

//> using scala 3.5

@main def lookups(): Unit =
  val ids: Vector[Int] = Vector.range(0, 100_000)
  val wanted = Seq(5, 99_999, 123_456)

  // O(n) per lookup on a Vector
  val slow = wanted.map(ids.contains)

  // O(1) average per lookup after building a Set once
  val idSet = ids.toSet
  val fast = wanted.map(idSet.contains)

  println(slow)   // List(true, true, false)
  println(fast)   // List(true, true, false)

  // Building strings: StringBuilder instead of repeated +
  val sb = StringBuilder()
  for k <- 1 to 5 do sb.append(k).append(',')
  println(sb.result())   // 1,2,3,4,5,

Collection Performance Guide

Collection Head / prepend Index (i) Append Notes
List O(1) O(n) O(n) Good for recursion and stack-like use
Vector ~O(1) ~O(1) ~O(1) General-purpose immutable indexed sequence
Array - O(1) copy Mutable, primitive storage, fastest numerics
ArrayBuffer O(n) prepend O(1) amortized O(1) Mutable builder
Map / Set (hash) - O(1) lookup O(1) Use for membership tests

Profiling Workflow

  1. Write a JMH benchmark (sbt-jmh) for the slow function, with realistic input sizes.

  2. Profile the whole application with async-profiler or JDK Flight Recorder to find where time and allocation actually go.

  3. Change one thing, re-run the benchmark, keep the change only if it is faster and still correct.

  4. In Spark jobs, check the Spark UI first: shuffles, skew and serialization usually dominate over Scala code style.

Common Pitfalls

  • Optimizing code that is not on the hot path; the profile usually points somewhere else.

  • Calling list(i) or list.length in a loop over a List (quadratic time).

  • Using parallel collections for tiny workloads; thread coordination costs more than the work.

  • Exposing mutable collections from an API to save copies, then hitting concurrency bugs.

  • Using Option/tuples in tight numeric loops where they cause allocation and boxing.

Interview Tips

  • Start with “measure first” (JMH, profiler), then list techniques by impact: algorithm, data structure, allocation, boxing.

  • Show you know the complexity table for List vs Vector vs Array.

  • Mention keeping mutation local: a var inside a function is fine if the function is pure from the outside.

60

What role does the JVM play in Scala's performance?

Scala compiles to JVM bytecode, so almost all of its runtime performance comes from the Java Virtual Machine: the JIT compiler (HotSpot C1/C2 or GraalVM) turns hot bytecode into optimized machine code at runtime, inlining small methods (including the many tiny closures Scala creates), removing virtual calls and eliminating allocations through escape analysis; the garbage collector (G1 by default, ZGC or Shenandoah for low pause times) handles the many short-lived immutable objects idiomatic Scala produces; and the JVM gives Scala direct access to mature Java libraries and to Spark, Kafka and Flink, which run on the same VM. The costs Scala pays on the JVM are boxing of primitives in generic code, allocation from immutable collections and closures, and warm-up time before the JIT has optimized the hot paths.

Read full answer

Applies toScala 3 and Scala 2.13 on JDK 17 / 21

What the JVM Does for Scala

JVM feature Effect on Scala code
JIT compilation (tiered C1 → C2) Hot loops and methods become native code after warm-up
Inlining Small lambdas passed to map, filter, foldLeft often cost nothing once inlined
Escape analysis Short-lived objects (tuples, Option, iterators) can be scalar-replaced, never allocated
Generational GC Cheap allocation and collection of many small immutable objects
JVM type erasure Generics are erased; List[Int] stores boxed java.lang.Integer
Threads and java.util.concurrent Basis of Future, Akka/Pekko, parallel collections, Spark executors

Where Scala Code Loses Performance on the JVM

  • Boxing: generic code (List[Int], Seq[Double], Option[Int]) stores objects, not primitives. Array[Int] is a real int[] and avoids this.

  • Allocation: map on a List creates a new list; chaining five operations creates five intermediate collections.

  • Megamorphic calls: a call site that sees many different lambda classes is harder for the JIT to inline.

  • Warm-up: short jobs (CLI tools, small Spark tasks) may finish before C2 compiles the hot code.

Code Example: Boxed vs Primitive Storage (Scala 3)

//> using scala 3.5

@main def boxing(): Unit =
  val n = 1_000_000

  // List[Int]: one java.lang.Integer object (plus a cons cell) per element
  val boxed: List[Int] = List.range(1, n + 1)
  println(boxed.foldLeft(0L)(_ + _))        // 500000500000

  // Array[Int]: a primitive int[] - no boxing, contiguous memory, cache friendly
  val prim: Array[Int] = Array.range(1, n + 1)
  var i = 0
  var acc = 0L
  while i < prim.length do
    acc += prim(i)
    i += 1
  println(acc)                               // 500000500000

Both print the same result; the array version uses a fraction of the memory and is the shape the JIT optimizes best (a counted loop over int[]).

Measuring Correctly: JMH

Timing code with System.nanoTime around one call measures interpreter and JIT warm-up, not steady-state speed. Use JMH through the sbt-jmh plugin:

// project/plugins.sbt: addSbtPlugin("pl.project13.scala" % "sbt-jmh" % "0.4.7")
// build.sbt:           enablePlugins(JmhPlugin)
import org.openjdk.jmh.annotations.*

@State(Scope.Thread)
class SumBench:
  val data: Array[Int] = Array.range(0, 100_000)

  @Benchmark
  def foldLeftSum(): Long = data.foldLeft(0L)(_ + _)

  @Benchmark
  def whileSum(): Long =
    var i = 0
    var acc = 0L
    while i < data.length do
      acc += data(i)
      i += 1
    acc
// run: sbt "Jmh/run -wi 5 -i 10 -f 1 SumBench"

JMH handles warm-up iterations, forks a fresh JVM and stops the JIT from removing the work as dead code (returning the value matters).

JVM Settings That Matter

Setting Why
-Xms / -Xmx (equal for services) Avoids heap resizing; size for the working set
-XX:+UseG1GC (default) / -XX:+UseZGC Throughput vs low pause times
JDK 17 or 21 Better JIT, GC and container awareness than JDK 8
-XX:+UseStringDeduplication (G1) Many repeated strings in data jobs
GraalVM JIT / Native Image Better inlining of closures; native image removes warm-up for CLIs

Scala Features That Help the JIT

  • inline (Scala 3) inlines a method at compile time, so the JIT sees straight-line code.

  • Opaque types (Scala 3) and value classes (extends AnyVal) give type safety without a wrapper object at runtime.

  • @specialized (Scala 2) generates primitive versions of generic classes.

  • @tailrec turns self-recursion into a loop, avoiding stack frames.

Common Pitfalls

  • Benchmarking with a single nanoTime measurement and concluding Scala is slow.

  • Using List[Double] for numeric work where Array[Double] (or Breeze) is the right structure.

  • Setting a huge heap for a service and getting long GC pauses instead of using ZGC or a smaller heap.

  • Forgetting that Spark executors are JVMs too: executor memory, GC and serialization tune the same way.

Interview Tips

  • Name the three JVM levers: JIT, GC and the class library/ecosystem.

  • Mention boxing and allocation as the main Scala-specific costs, and the fixes (Array, while, opaque types, inline).

  • Say you measure with JMH and profile with async-profiler or JDK Flight Recorder before optimizing.

61

How can you manage memory effectively in Scala applications?

Memory management in Scala is handled by the JVM garbage collector, so the goal is to allocate less, hold references shorter, and size and tune the JVM. Practical techniques: prefer streaming and iterators (Iterator, LazyList, fs2/Akka Streams) to loading everything into memory; use primitive arrays (Array[Double]) for numeric data instead of boxed collections like List[Double]; avoid accidental retention through closures, caches and long-lived collections; use @specialized-aware libraries such as Breeze for numerics; choose and tune the GC (G1 by default, ZGC for low latency) and heap size (-Xmx); and measure with heap dumps, JFR and GC logs. In Spark, also manage executor memory, caching (persist with the right storage level, unpersist) and partition sizes.

Read full answer

Applies toScala 3 on JDK 21; Spark 3.5 / 4.x tips

Techniques

Technique Why
Iterator / streaming over files Constant memory for large inputs
Array[Double] instead of List[Double] No boxing: about 8 bytes per value instead of an object plus a list node
Vector / ArraySeq for indexed data Compact, efficient random access
Avoid toList on huge streams Materializes everything
Bounded caches (Caffeine) Prevent unbounded growth
Release references Clear large collections no longer needed
JVM flags -Xmx, -XX:+UseZGC, GC logging -Xlog:gc
Spark persist(StorageLevel.MEMORY_AND_DISK), unpersist(), avoid collect()

Code Example: Constant-Memory Statistics over a Large Stream

final case class Running(count: Long, mean: Double, m2: Double):
  def add(x: Double): Running =                      // Welford's online algorithm
    val n = count + 1
    val delta = x - mean
    val newMean = mean + delta / n
    Running(n, newMean, m2 + delta * (x - newMean))
  def variance: Double = if count < 2 then 0.0 else m2 / (count - 1)

@main def stats(): Unit =
  // A lazy iterator: values are produced one at a time and never stored together
  val values: Iterator[Double] = Iterator.range(1, 10_000_001).map(_.toDouble)
  val result = values.foldLeft(Running(0, 0.0, 0.0))(_ add _)
  println(f"count=${result.count} mean=${result.mean}%.1f")
// count=10000000 mean=5000000.5

  // Primitive array: compact storage when data must be held in memory
  val arr = Array.tabulate(1_000_000)(_.toDouble)
  println(arr.sum)
// 4.999995E11

Ten million values are summarized with a single small object, whereas (1 to 10_000_000).map(_.toDouble).toList would allocate tens of millions of objects. The array of one million doubles uses about 8 MB.

Common Pitfalls

  • Holding the head of a LazyList in a val, which keeps every evaluated element alive.

  • Boxed numeric collections in hot paths.

  • Caching Spark DataFrames that are used only once.

62

Which frameworks are commonly used for testing Scala applications?

The most common Scala testing frameworks are ScalaTest, munit, specs2 and ScalaCheck. ScalaTest is the most widely used, offering many styles (AnyFunSuite, AnyFlatSpec, AnyWordSpec) and rich matchers. munit is a lightweight, fast framework with clear diffs, popular in newer Scala 3 projects. specs2 focuses on behaviour-driven specifications. ScalaCheck provides property-based testing and integrates with the others. Around them: Mockito-Scala or ScalaMock for mocks, Testcontainers-scala for real databases and Kafka in Docker, weaver-test and ZIO Test for effectful code, Akka/Pekko TestKit for actors, and spark-fast-tests or a local SparkSession for Spark jobs.

Read full answer

Applies toScala 3 (frameworks also support Scala 2.13)

Frameworks

Framework Strength Typical use
ScalaTest Many styles, matchers, mature General unit and integration tests
munit Minimal, fast, good diffs Scala 3 libraries and apps
specs2 BDD-style specifications Acceptance-style tests
ScalaCheck Property-based testing Invariants, edge cases
ScalaMock / Mockito-Scala Mocks and stubs Isolating dependencies
Testcontainers-scala Real services in Docker Integration tests
ZIO Test / weaver Effect-aware tests ZIO and Cats Effect code
Akka/Pekko TestKit Actor testing Message-passing systems
spark-fast-tests DataFrame comparisons Spark jobs

Code Example: The Same Test in ScalaTest and munit

//> using scala 3.5
//> using dep org.scalatest::scalatest:3.2.19
//> using dep org.scalameta::munit:1.0.2

def tokenize(s: String): List[String] =
  s.toLowerCase.split("\\W+").filter(_.nonEmpty).toList

// ScalaTest, FunSuite style with matchers
import org.scalatest.funsuite.AnyFunSuite
import org.scalatest.matchers.should.Matchers

class TokenizeScalaTest extends AnyFunSuite with Matchers:
  test("splits on punctuation and lowercases") {
    tokenize("Hello, World!") shouldBe List("hello", "world")
  }

// munit
class TokenizeMunit extends munit.FunSuite:
  test("ignores empty tokens") {
    assertEquals(tokenize("  a--b  "), List("a", "b"))
  }

Save this as tokenize.test.scala and run scala-cli test tokenize.test.scala; in sbt, add the libraries with % Test and run sbt test. Most projects pick one main framework and add ScalaCheck for property tests.

Interview Tips

  • Mention ScalaTest and munit first, then ScalaCheck for properties.

  • For data work, mention testing Spark transformations with a small local SparkSession.

63

How do you unit test code in Scala?

Unit testing in Scala means writing small, fast tests for individual functions or classes with a test framework, usually ScalaTest or munit, running them with sbt (sbt test) or scala-cli (scala-cli test .). Tests live under src/test/scala, each suite groups related cases, and assertions compare actual and expected values. Scala’s style helps testing: pure functions are tested by input and output alone, immutable case classes compare by value, and dependencies are injected through constructor parameters or traits so they can be replaced with fakes. Property-based testing with ScalaCheck checks laws over many generated inputs.

Read full answer

Applies toScala 3, munit 1.x, ScalaCheck 1.18 (via munit-scalacheck)

Process

Step Practice
Structure Code in src/main/scala, tests in src/test/scala
Dependencies libraryDependencies += "org.scalameta" %% "munit" % "1.0.2" % Test
Write tests One behaviour per test, clear names
Isolate Inject dependencies via traits; use fakes instead of real databases
Properties ScalaCheck generators for invariants
Run sbt test, sbt testOnly *ScalerSuite, CI on every commit
Coverage sbt-scoverage

Code Example

// src/main/scala/Scaler.scala
final case class Scaler(mean: Double, std: Double):
  def transform(x: Double): Double = if std == 0 then 0.0 else (x - mean) / std

object Scaler:
  def fit(xs: Seq[Double]): Scaler =
    require(xs.nonEmpty, "cannot fit on empty data")
    val m = xs.sum / xs.size
    val variance = xs.map(x => math.pow(x - m, 2)).sum / xs.size
    Scaler(m, math.sqrt(variance))

trait PriceSource:
  def prices(symbol: String): Seq[Double]

final class Normalizer(source: PriceSource):
  def normalized(symbol: String): Seq[Double] =
    val ps = source.prices(symbol)
    val s = Scaler.fit(ps)
    ps.map(s.transform)
// src/test/scala/ScalerSuite.scala
import munit.ScalaCheckSuite
import org.scalacheck.{Gen, Prop}

class ScalerSuite extends ScalaCheckSuite:
  test("fit computes mean and population std") {
    val s = Scaler.fit(Seq(2.0, 4.0, 6.0))
    assertEquals(s.mean, 4.0)
    assertEqualsDouble(s.std, math.sqrt(8.0 / 3), 1e-9)
  }

  test("constant data maps to zero") {
    assertEquals(Scaler(5.0, 0.0).transform(7.0), 0.0)
  }

  test("empty input is rejected") {
    intercept[IllegalArgumentException](Scaler.fit(Seq.empty))
  }

  test("Normalizer works with a fake data source") {
    val fake = new PriceSource { def prices(symbol: String) = Seq(1.0, 2.0, 3.0) }
    val out = Normalizer(fake).normalized("ANY")
    assertEqualsDouble(out.sum, 0.0, 1e-9)
  }

  property("transformed data has mean close to zero") {
    Prop.forAll(Gen.nonEmptyListOf(Gen.choose(-1000.0, 1000.0))) { xs =>
      val s = Scaler.fit(xs)
      math.abs(xs.map(s.transform).sum / xs.size) < 1e-6
    }
  }

The fake PriceSource replaces a real data feed, so the test is fast and deterministic. Add "org.scalameta" %% "munit-scalacheck" % "1.0.0" % Test for the property test.

Common Pitfalls

  • Testing implementation details instead of behaviour.

  • Exact equality on floating-point results (use tolerances).

  • Tests that depend on time, randomness or network without control (inject clocks and seeds).

64

How would you debug a Scala application that is not performing as expected?

Debugging a Scala application starts by deciding whether the problem is wrong results or poor performance. For wrong results: reproduce with a minimal input and a failing unit test, add structured logging (SLF4J/Logback) at boundaries, inspect values in the REPL or scala-cli, step through with a debugger (IntelliJ IDEA or Metals in VS Code with breakpoints), and check common culprits: nulls from Java APIs, unhandled Option/Either cases, integer division, and mutable state shared between threads. For performance: measure first with JMH microbenchmarks or simple timing, profile with async-profiler, VisualVM or Java Flight Recorder, check GC logs, and for Spark jobs use the Spark UI (stages, skew, spills, shuffle sizes).

Read full answer

Applies toScala 3; Spark 3.5 / 4.x for Spark-specific tips

Debugging Toolkit

Problem Tools
Logic errors Unit tests (munit, ScalaTest), REPL, debugger breakpoints
Unexpected values Logging with context, assert/require checks
Exceptions Full stack traces, Try/Either with error messages
Slow code async-profiler, JFR, VisualVM, JMH
Memory problems Heap dumps (jcmd GC.heap_dump), Eclipse MAT, GC logs
Concurrency bugs Thread dumps (jstack), reduce shared mutability
Spark jobs Spark UI, explain(), executor logs, skew detection

Code Example: Making a Bug Visible with Types, Tests and Timing

//> using scala 3.5
//> using dep org.scalameta::munit:1.0.2

import scala.util.Try

// Bug: integer division silently truncates the average
def averageBuggy(xs: List[Int]): Int = xs.sum / xs.size

// Fixed: explicit Double result and empty-input handling
def average(xs: List[Int]): Option[Double] =
  if xs.isEmpty then None else Some(xs.sum.toDouble / xs.size)

def timed[A](label: String)(body: => A): A =
  val start = System.nanoTime()
  val result = body
  println(f"$label took ${(System.nanoTime() - start) / 1e6}%.2f ms")
  result

class AverageSuite extends munit.FunSuite:
  test("average keeps the fraction") {
    assertEquals(average(List(1, 2)), Some(1.5))
  }
  test("buggy version truncates (documents the old bug)") {
    assertEquals(averageBuggy(List(1, 2)), 1)
  }
  test("empty input is handled") {
    assertEquals(average(Nil), None)
    assert(Try(averageBuggy(Nil)).isFailure)          // ArithmeticException: / by zero
  }

@main def demo(): Unit =
  val data = List.tabulate(1_000_000)(_ % 100)
  println(timed("average")(average(data)))
// average took ... ms (varies)
// Some(49.5)

Save the file as average.test.scala (scala-cli treats *.test.scala files as test sources) and run scala-cli test average.test.scala; the failing case pinpoints the bug before any debugger is needed. Timing like this is only a first look; use JMH for reliable benchmarks because of JIT warm-up.

Common Pitfalls

  • Optimizing before profiling.

  • Benchmarking without JVM warm-up.

  • In Spark, debugging with collect() on large data instead of show, limit and the UI.

65

Can you describe a case study of Scala in an ML pipeline?

A typical case study is a churn prediction pipeline for a subscription business running on Spark with Scala. Event logs (logins, plays, payments) land in a data lake; a nightly Scala Spark job builds point-in-time features per customer with window functions and typed Dataset[CustomerFeatures]; a spark.ml Pipeline (string indexing, vector assembly, gradient-boosted trees) is retrained weekly with CrossValidator; the model is saved and versioned; a daily scoring job writes churn probabilities to a table used by the CRM; and monitoring jobs compare score distributions and realized churn over time. Scala’s case classes give a compile-time-checked schema across all these jobs, and the same JAR runs in development and on the cluster.

Read full answer

Applies toScala 2.13, Spark 3.5 / 4.x

Pipeline Stages

Stage Implementation
Ingest Spark reads Parquet/Delta event tables
Features Windowed aggregates per customer as of a snapshot date
Labels Churn within 30 days after the snapshot
Train GBTClassifier in a Pipeline, tuned with CrossValidator
Evaluate AUC on a later snapshot (out-of-time)
Score Daily batch PipelineModel.transform → predictions table
Monitor Score drift and realized churn per cohort

Code Example: Typed Feature Building

import org.apache.spark.sql.{Dataset, SparkSession}
import org.apache.spark.sql.functions._

final case class Event(customerId: String, eventDate: java.sql.Date, kind: String, amount: Double)
final case class CustomerFeatures(customerId: String, logins30d: Long, spend30d: Double, daysSinceLast: Int)

object BuildFeatures {
  def build(events: Dataset[Event], snapshot: String)(implicit spark: SparkSession): Dataset[CustomerFeatures] = {
    import spark.implicits._
    val snap = to_date(lit(snapshot))
    events
      .filter($"eventDate" < snap && $"eventDate" >= date_sub(snap, 30))      // point-in-time window
      .groupBy($"customerId")
      .agg(
        count(when($"kind" === "login", 1)).as("logins30d"),
        coalesce(sum(when($"kind" === "payment", $"amount")), lit(0.0)).as("spend30d"),
        datediff(snap, max($"eventDate")).as("daysSinceLast"))
      .as[CustomerFeatures]
  }

  def main(args: Array[String]): Unit = {
    implicit val spark: SparkSession = SparkSession.builder.master("local[*]").appName("features").getOrCreate()
    import spark.implicits._
    val events = Seq(
      Event("c1", java.sql.Date.valueOf("2026-10-20"), "login", 0.0),
      Event("c1", java.sql.Date.valueOf("2026-10-25"), "payment", 9.99),
      Event("c2", java.sql.Date.valueOf("2026-10-05"), "login", 0.0)
    ).toDS()
    build(events, "2026-11-01").orderBy("customerId").show()
    // c1 | logins30d 1 | spend30d 9.99 | daysSinceLast 7
    // c2 | logins30d 1 | spend30d 0.0  | daysSinceLast 27
    spark.stop()
  }
}

The .as[CustomerFeatures] call fails at analysis time if a column is missing or mistyped, which catches schema drift early.

Lessons from Such Projects

  • Keep feature logic in one shared Scala module used by training and scoring jobs.

  • Validate on a later time period; random splits overstate churn models.

  • Pin Spark and Scala versions in sbt to match the cluster.

66

How can Scala process large-scale graph data for machine learning?

Scala’s main tool for large-scale graphs is Spark: GraphX (Scala-only API in Spark core) represents a graph as distributed vertex and edge RDDs and provides algorithms such as PageRank, connected components, triangle counting, label propagation and the Pregel API for custom iterative algorithms. GraphFrames (a Spark package) offers a DataFrame-based API with motif finding and the same algorithms. For ML, these compute graph features (degree, PageRank, component size, triangle counts, neighbour aggregates) that are joined to entity tables and fed into spark.ml models, for example in fraud detection, recommendations or influence scoring. Graph neural networks are usually trained in Python (PyTorch Geometric, DGL) on data prepared with Spark.

Read full answer

Applies toScala 2.13, Spark 3.5 / 4.x (GraphX)

Graph Features for ML

Feature GraphX
Degree graph.degrees, inDegrees, outDegrees
Importance graph.pageRank(tol)
Communities connectedComponents(), LabelPropagation.run
Clustering triangleCount()
Custom propagation graph.pregel(...), aggregateMessages

Code Example: PageRank and Components as Features

import org.apache.spark.graphx.{Edge, Graph}
import org.apache.spark.sql.SparkSession

object GraphFeatures {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder.master("local[*]").appName("graph-features").getOrCreate()
    val sc = spark.sparkContext

    // Accounts (vertices) and money transfers (edges)
    val vertices = sc.parallelize(Seq((1L, "a1"), (2L, "a2"), (3L, "a3"), (4L, "a4"), (5L, "a5")))
    val edges = sc.parallelize(Seq(Edge(1L, 2L, 100.0), Edge(2L, 3L, 50.0), Edge(3L, 1L, 75.0), Edge(4L, 5L, 20.0)))
    val graph = Graph(vertices, edges)

    val ranks      = graph.pageRank(tol = 0.0001).vertices
    val components = graph.connectedComponents().vertices
    val degrees    = graph.degrees

    val features = vertices
      .join(ranks).join(components).leftOuterJoin(degrees)
      .map { case (id, (((name, rank), comp), deg)) => (id, name, rank, comp, deg.getOrElse(0)) }
      .sortBy(_._1)
      .collect()

    features.foreach { case (id, name, rank, comp, deg) =>
      println(f"$name%s rank=$rank%.3f component=$comp%d degree=$deg%d")
    }
    // a1, a2, a3 share component 1 with degree 2 each and equal PageRank (a 3-cycle);
    // a4 and a5 form component 4, each with degree 1, and a5 ranks above a4 (it receives the transfer).
    spark.stop()
  }
}

collect() is safe here only because the result is tiny; on real graphs write the feature table out with Spark instead.

Common Pitfalls

  • Using collect() on large vertex sets.

  • Hub vertices (very high degree) creating skewed partitions; partition edges with PartitionStrategy.

  • Computing graph features with edges that occurred after the prediction time (leakage).

67

How can Scala be applied to natural language processing?

A common Scala NLP application is large-scale text processing on Spark, for example classifying millions of support tickets or extracting entities from documents. Spark NLP (John Snow Labs), written in Scala, provides annotators for tokenization, lemmatization, named-entity recognition, sentiment and transformer-based embeddings (BERT and others) that run distributed across a cluster. For simpler pipelines, spark.ml.feature offers Tokenizer, RegexTokenizer, StopWordsRemover, NGram, HashingTF, IDF, CountVectorizer and Word2Vec, which combine with classifiers in one Pipeline. Stanford CoreNLP and OpenNLP (Java libraries) are also callable directly from Scala.

Read full answer

Applies toScala 2.13, Spark 3.5 / 4.x (spark.ml)

Tools

Library Use
spark.ml.feature Tokenizers, stop words, n-grams, TF-IDF, Word2Vec
Spark NLP Production NLP: NER, embeddings, transformers, multilingual pipelines
Stanford CoreNLP / OpenNLP Parsing, POS tagging, NER (JVM libraries)
Plain Scala collections Small-scale preprocessing and prototyping

Code Example: Ticket Classification with spark.ml

import org.apache.spark.ml.Pipeline
import org.apache.spark.ml.classification.LogisticRegression
import org.apache.spark.ml.feature.{HashingTF, IDF, RegexTokenizer, StopWordsRemover}
import org.apache.spark.sql.SparkSession

object TicketClassifier {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder.master("local[*]").appName("tickets").getOrCreate()
    import spark.implicits._

    val train = Seq(
      ("refund not received for my order", 1.0),
      ("I want my money back", 1.0),
      ("charged twice please refund", 1.0),
      ("how do I change my password", 0.0),
      ("cannot log in to my account", 0.0),
      ("reset password link expired", 0.0)
    ).toDF("text", "label")

    val tokenizer = new RegexTokenizer().setInputCol("text").setOutputCol("tokens").setPattern("\\W+")
    val stopWords = new StopWordsRemover().setInputCol("tokens").setOutputCol("filtered")
    val tf        = new HashingTF().setInputCol("filtered").setOutputCol("tf").setNumFeatures(1 << 12)
    val idf       = new IDF().setInputCol("tf").setOutputCol("features")
    val lr        = new LogisticRegression().setMaxIter(50)

    val model = new Pipeline().setStages(Array(tokenizer, stopWords, tf, idf, lr)).fit(train)

    val test = Seq("please refund my order", "forgot my password").toDF("text")
    model.transform(test).select("text", "prediction").show(truncate = false)
    // Expected: "please refund my order" -> 1.0 (billing), "forgot my password" -> 0.0 (account)
    spark.stop()
  }
}

The same pipeline scales from this toy example to millions of tickets on a cluster without code changes. For higher accuracy, replace TF-IDF with Spark NLP’s transformer embeddings.

Common Pitfalls

  • Default stop-word lists removing negations (“not”) that matter for sentiment.

  • Tokenizers that do not suit the language (e.g. whitespace tokenizers for Chinese or Japanese).

  • Running Python NLP libraries via UDFs inside Spark, which is slow; prefer JVM annotators.

68

How is Scala used in finance for algorithmic trading systems?

Scala is used in finance for trading and risk systems because it combines JVM performance with a strong type system and functional style. Banks and trading firms use it for market data pipelines (Kafka, Akka Streams, Flink), strategy and signal engines where immutable data and pure functions make logic testable and reproducible, low-latency services on the JVM, risk and pricing analytics on Spark for large portfolios, and backtesting frameworks. Case classes and sealed traits model orders, events and states precisely, so the compiler catches many errors (missing cases in pattern matches, wrong currency types) before they reach production.

Read full answer

Applies toScala 3

Uses in Trading

Area Scala role
Market data ingestion Kafka consumers, Akka Streams, back-pressure
Strategy logic Pure functions over immutable price series
Order management Sealed trait ADTs for order states, actor-based state handling
Risk / pricing Spark jobs, Monte Carlo simulations
Backtesting Deterministic replay of historical events
Typing for safety Opaque types for prices, quantities, currencies

Code Example: Moving-Average Crossover Backtest (Scala 3)

enum Signal:
  case Buy, Sell, Hold

opaque type Price = Double
object Price:
  def apply(d: Double): Price = d
extension (p: Price) def value: Double = p

def sma(prices: Vector[Price], window: Int): Vector[Option[Double]] =
  prices.indices.toVector.map { i =>
    if i + 1 < window then None
    else Some(prices.slice(i + 1 - window, i + 1).map(_.value).sum / window)
  }

def signals(prices: Vector[Price], fast: Int, slow: Int): Vector[Signal] =
  sma(prices, fast).zip(sma(prices, slow)).map {
    case (Some(f), Some(s)) if f > s => Signal.Buy
    case (Some(f), Some(s)) if f < s => Signal.Sell
    case _                           => Signal.Hold
  }

@main def backtest(): Unit =
  val prices = Vector(10.0, 10.5, 11.0, 11.5, 11.0, 10.0, 9.5, 9.0).map(Price(_))
  val sigs = signals(prices, fast = 2, slow = 3)
  println(sigs.mkString(", "))
// Hold, Hold, Buy, Buy, Buy, Sell, Sell, Sell

At index 4 the fast average (11.5 + 11.0) / 2 = 11.25 is still above the slow average (11.0 + 11.5 + 11.0) / 3 ≈ 11.17, so the signal stays Buy; at index 5 the fast average 10.5 falls below 10.83, giving Sell. The opaque Price type prevents mixing prices with quantities at compile time with no runtime cost.

Common Pitfalls

  • Using Double for money in accounting logic; use BigDecimal or integer minor units.

  • Garbage-collection pauses in latency-critical paths; tune the GC (ZGC, Shenandoah) and reduce allocation.

  • Backtests with look-ahead bias (signals using the same bar’s close they trade on).

69

How does Scala address scalability in large ML systems?

Scala supports scalability in two ways: the language and the ecosystem built on it. As a language it runs on the JVM (mature JIT compilation, garbage collection, multithreading), encourages immutability and pure functions that are safe to run in parallel, and has a strong static type system that keeps large codebases maintainable. Its ecosystem provides the main tools for scaling out: Apache Spark (distributed data processing and spark.ml, written in Scala), Akka / Apache Pekko (actor-based concurrency and clustering), Kafka clients and Flink for streaming, plus Futures and effect libraries (Cats Effect, ZIO) for asynchronous, non-blocking services.

Read full answer

Applies toScala 3 (language examples); Spark 3.5 / 4.x with Scala 2.13 for distributed work

Scalability Mechanisms

Need Scala mechanism
Use many cores Futures, parallel collections (separate module), Cats Effect / ZIO fibres
Use many machines Spark (RDD/Dataset), Akka Cluster, Flink
Safe parallelism Immutable data, pure functions, no shared mutable state
Streaming Spark Structured Streaming, Akka Streams, Flink, Kafka Streams
Large codebases Static types, case classes, traits, compiler-checked refactoring
Performance JVM JIT, primitive arrays, specialized libraries (Breeze)

Code Example: Parallel Feature Computation with Futures (Scala 3)

import scala.concurrent.{Await, Future}
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration.*

final case class UserStats(userId: Int, total: Double, count: Int)

def computeStats(userId: Int, events: Vector[Double]): UserStats =
  UserStats(userId, events.sum, events.size)        // pure: safe to run in parallel

@main def runParallel(): Unit =
  val data: Map[Int, Vector[Double]] =
    (1 to 4).map(id => id -> Vector.tabulate(5)(i => (id * 10 + i).toDouble)).toMap

  val futures = data.toSeq.sortBy(_._1).map { case (id, events) => Future(computeStats(id, events)) }
  val results = Await.result(Future.sequence(futures), 5.seconds)
  results.foreach(println)
// UserStats(1,60.0,5)
// UserStats(2,110.0,5)
// UserStats(3,160.0,5)
// UserStats(4,210.0,5)

User 1’s events are 10, 11, 12, 13, 14, which sum to 60. Because computeStats is pure and the data is immutable, the work can be spread across threads, or across a Spark cluster with the same function, without locks.

Common Pitfalls

  • Shared mutable state accessed from many threads or Spark tasks.

  • Blocking calls (Await) inside request handlers; keep them at the program edge.

  • Assuming the JVM scales automatically; tune executor memory, partitions and GC for big jobs.

70

How would you implement a distributed ML system using Scala?

A distributed ML system in Scala is usually built on Apache Spark, whose core is written in Scala: data is loaded and transformed as distributed Datasets/DataFrames, features and models are defined with spark.ml Pipelines (estimators such as LogisticRegression, RandomForestClassifier, ALS, KMeans train in parallel across executors), hyperparameters are tuned with CrossValidator or TrainValidationSplit, and fitted PipelineModels are saved and reused for batch or streaming scoring (Structured Streaming). Around Spark, Scala is also used for Akka-based services (actor systems for online serving and coordination), Kafka consumers for event streams, and type-safe data contracts with case classes. Training of deep models is usually delegated to Python frameworks, with Scala handling the data plane.

Read full answer

Applies toScala 2.13, Apache Spark 3.5 / 4.x (spark.ml)

Architecture

Layer Scala / JVM tool
Ingestion Spark readers (Parquet, Delta, JDBC), Kafka + Structured Streaming
Data contracts Case classes → Dataset[T]
Feature engineering DataFrame API, Spark SQL, window functions
Training spark.ml Pipeline on a cluster (YARN, Kubernetes)
Tuning CrossValidator, ParamGridBuilder with setParallelism
Model storage PipelineModel.write.save(...), MLflow
Serving Batch scoring jobs; streaming scoring; Akka HTTP service for low latency
Monitoring Metrics tables, Spark UI, drift jobs

Code Example: Distributed Training Job (Spark, Scala 2.13)

import org.apache.spark.ml.Pipeline
import org.apache.spark.ml.classification.LogisticRegression
import org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
import org.apache.spark.ml.feature.{StringIndexer, VectorAssembler, StandardScaler}
import org.apache.spark.ml.tuning.{CrossValidator, ParamGridBuilder}
import org.apache.spark.sql.SparkSession

object TrainChurn {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder.appName("train-churn").getOrCreate()

    val data = spark.read.parquet(args(0))            // columns: plan, tenure, usage, label
    val Array(train, test) = data.randomSplit(Array(0.8, 0.2), seed = 42)

    val planIdx   = new StringIndexer().setInputCol("plan").setOutputCol("planIdx").setHandleInvalid("keep")
    val assembler = new VectorAssembler().setInputCols(Array("planIdx", "tenure", "usage")).setOutputCol("raw")
    val scaler    = new StandardScaler().setInputCol("raw").setOutputCol("features")
    val lr        = new LogisticRegression().setLabelCol("label").setFeaturesCol("features")
    val pipeline  = new Pipeline().setStages(Array(planIdx, assembler, scaler, lr))

    val grid = new ParamGridBuilder()
      .addGrid(lr.regParam, Array(0.01, 0.1))
      .addGrid(lr.elasticNetParam, Array(0.0, 0.5))
      .build()
    val evaluator = new BinaryClassificationEvaluator().setMetricName("areaUnderROC")
    val cv = new CrossValidator()
      .setEstimator(pipeline).setEstimatorParamMaps(grid).setEvaluator(evaluator)
      .setNumFolds(3).setParallelism(4).setSeed(42)

    val model = cv.fit(train)                          // fits run in parallel on the cluster
    println(s"test AUC = ${evaluator.evaluate(model.transform(test))}")
    model.bestModel.asInstanceOf[org.apache.spark.ml.PipelineModel]
      .write.overwrite().save(args(1))
    spark.stop()
  }
}

Submit with spark-submit --class TrainChurn train-churn.jar s3://bucket/features s3://bucket/models/churn-v3. The same saved PipelineModel is loaded by scoring jobs with PipelineModel.load(path).

Common Pitfalls

  • Building the JAR against a Scala version different from the cluster’s Spark build (2.12 vs 2.13).

  • Closures capturing non-serializable objects (database clients), causing Task not serializable.

  • Collecting large DataFrames to the driver with collect().

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 70 Scala answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Machine Learning & Data Science subjects