Catch-It-All Page

Even more commands, procedures, links and resources

• Setting variables with multiple types in a function

• To get float division (complete division with decimals) of integer numbers, or other types, into float format, use .toFloat on one of them. e.g. 1/(df.count -1).toFloat source

• To raise a number to power, use pow function like so, pow('colName, integer_value_of_power) e.g. pow('salary, 3) to take third exponent of salary column. NOTE: exp does NOT raise to power. It takes the exponential with base e the Euler number.

Reading the Newly Vamped Docs 3.1

I admit it's irritating at first to navigat the new docs for either APIs. So here is how it works,

• You still need to know where to look beforehand in terms of API, and package (module). Python API means PySpark, and Scala API means Spark Scala.

• Googling "spark docs latest", or "spark 3.1 official docs" will lead you to the familiar page https://spark.apache.org/docs/latest/ with main tabs on top. Hovering over "API Docs" tab shows a dropdown list of the APIs. Choose Python for Pyspark, and Scala for Spark Scala.

• Let's choose Scala. That will lead you to another confusing page https://spark.apache.org/docs/latest/api/scala/org/apache/spark/index.html though if you look closely, you'll see that the "org.apache.spark" is highlighted in the menu to the right, all the other modules are under it. Say you want to see all column function. To do that, select "SQL", which leads to this page https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/index.html It's hard to see, but "functions" are towards the end of the righthand menu, after all the green and blue dots. https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/functions$.html

• Similarly, you can find other modules like "org.apache.spark.sql.types"; or "org.apache.spark.ml" and all its submodules, https://spark.apache.org/docs/latest/api/scala/org/apache/spark/ml/index.html