Sunday, 8 January 2017

Difference between filter and where in scala spark sql ?

Filters rows using the given condition. This is an alias for filterwhere Docmentation

There is no confusion in Filter and where in spark sql since both gives same result.

// The following are equivalent:
employee.filter($"age" > 15)
employee.where($"age" > 15)
employees.filter($"emp_id".isin(items:_*)).show
employees.where($"emp_id".isin(items:_*)).show

Result is same for the both
 Filter is simply the standard Scala name for such a function, and where is for people who prefer SQL.