Python dependencies in kubeflow spark operator

I wanted to ask if there is a way to use python as a .wheel or .egg or just .py dependency in kubeflow spark operator.

The resulting file i have in mind would look something like this, the dependecy would be either under jars or files, i presume files would make more sense:

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi-python
  namespace: default
spec:
  type: Python
  pythonVersion: "3"
  mode: cluster
  image: spark:3.5.3
  imagePullPolicy: IfNotPresent
  mainApplicationFile: local:///path/to/my/python/script.py
  deps:
    jars:
      - local:///path/to/python/functions.py
    files:
      - gs://path/to/python/functions.py
  sparkVersion: 3.5.3
  driver:
    cores: 1
    memory: 512m
    serviceAccount: spark-operator-spark
  executor:
    instances: 1
    cores: 1
    memory: 512m

Thank you in advance

You need to sign in to view this answers

Related Post