Update of Apache Spark and Python 3.6 compatibility

For the last time I wrote an article about the compatibility of Apache Spark and Python3.6 on macOS.


Image result for apache spark



Now the Apache-Spark has released two stable version, 2.1.1 and 2.2.0. The compatibility problem has been solved, you don’t need to try pyenv anymore. Just install apache-spark from home-brew then you may use it.



But if you are going to use python3.6 and your default python in terminal is python2.7, you would find that the REPL would still be the python2.7 version.




In order to fix that, you should input


sudo PYSPARK_PYTHON=ptipython3 pyspark



As you can see, I am using ptipython3 as my Python 3 REPL. You can use the default interpreter by changing it to python3 and that will be good for you to use.


Ptpython is a better python interpreter, here is the link:



Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.