The error ModuleNotFoundError: No module named ‘py4j‘ is as shown below and getting it while importing some modules.
/opt/spark/python/pyspark/context.py in <module> 27 from tempfile import NamedTemporaryFile 28 ---> 29 from py4j.protocol import Py4JError 30 31 from pyspark import accumulators ModuleNotFoundError: No module named 'py4j'
If you can run spark directly, you may need to modify the PYTHONPATH
environment setting. Check the filename in the $SPARK HOME/python/lib/ directory. If you’re using Spark 2.4.3, the file is py4j-0.10.7-src.zip
then run:
export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.7-src.zip:$PYTHONPATH
Hope it solves your issue.
Similar Errors:
Failed to save file, EACCESS permision denied