• Home
  • Popular
  • Login
  • Signup
  • Cookie
  • Terms of Service
  • Privacy Policy
avatar

Posted by G Bot


21 Feb, 2025

Updated at 24 Mar, 2025

Error 500 when trying to create a Dataproc cluster with a history serve

Hello,

 
I am trying to recreate a dataproc cluster that was previously created with the following configuration ( retrieved from the command 'gcloud dataproc clusters export'):
 
virtualClusterConfig:
  auxiliaryServicesConfig:
    sparkHistoryServerConfig:
      dataprocCluster: projects/***/regions/europe-west2/clusters/dpgke-hs-stg
  kubernetesClusterConfig:
    gkeClusterConfig:
      gkeClusterTarget: projects/***/locations/europe-west2/clusters/dpgke-stg
      nodePoolTarget:
      - nodePool: projects/***/locations/europe-west2/clusters/dpgke-stg/nodePools/dp-default
        roles:
        - DEFAULT
      - nodePool: projects/***/locations/europe-west2/clusters/dpgke-stg/nodePools/dp-driver
        roles:
        - SPARK_DRIVER
      - nodePool: projects/***/locations/europe-west2/clusters/dpgke-stg/nodePools/dp-executor
        roles:
        - SPARK_EXECUTOR
    kubernetesNamespace: dpgke-staging
    kubernetesSoftwareConfig:
      componentVersion:
        SPARK: 3.5-dataproc-17
      properties:
        ( ... )
  stagingBucket: staging-bucket-dataproc-stg
 
I have tried creating the cluster using the UI Console, using the terraform resource "google_dataproc_cluster" ( terraform Google provider version 6.7) and by CLI ( gcloud SDK 511.0.0). In every case, i get an error 500 from dataproc.googleapis.com with a "Unknown Error" message ( see screenshots). 
 
When removing the Spark history server config, the cluster is created without any problems though.
 
I have tried the suggested actions in the "Troubleshoot cluster creation issues" doc without any success.
 
Any suggestions or help would be much appreciated.
 
Capture d’écran 2025-02-21 à 13.29.19.pngCapture d’écran 2025-02-21 à 13.42.28.png
 
 
Thank you