databricks not working with Geoparquet for write data

Question

databricks not working with Geoparquet for write data

Anil Kasar 5

Py4JJavaError: An error occurred while calling o535.save. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 17.0 failed 4 times, most recent failure: Lost task 0.3 in stage 17.0 (TID 45) (10.137.237.68 executor 0): java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(Ljava/lang/String;Lorg/apache/hadoop/mapreduce/TaskAttemptContext;)V at org.apache.spark.sql.execution.datasources.parquet.GeoParquetFileFormat$$anon$1.newInstance(GeoParquetFileFormat.scala:162) at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:186) at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:168) at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:544) at org.apache.spark.sql.execution.datasources.WriteFilesExec.$anonfun$doExecuteWrite$1(WriteFiles.scala:117)

Venkat Reddy Navari 5,255 Reputation points Microsoft External Staff Moderator

2025-08-07T11:33:38.0333333+00:00

Hi Anil Kasar That error looks like a compatibility issue between the GeoParquet library you're using and the version of Spark that comes with your Databricks Runtime. Specifically, the NoSuchMethodError around ParquetOutputWriter usually means the library is expecting a method or class that doesn’t exist in your current Spark version.

A few things you might want to check:

Databricks Runtime version – Not all versions are compatible with every library, especially ones that hook into low-level file I/O like GeoParquet. You can check the Spark version tied to your DBR here: https://docs.databricks.com/release-notes/runtime/index.html

GeoParquet version – If you're pulling the library from GitHub or Maven, make sure it's tested with your Spark/DBR version. Sometimes newer releases drop support for older Spark APIs.

Parquet format support in Databricks – This doc might help clarify how Databricks handles Parquet under the hood: https://learn.microsoft.com/en-us/azure/databricks/query/formats/parquet

Dependency conflicts – If you’re including additional JARs (e.g., for GeoParquet or Parquet libraries), it’s possible there’s a version clash with what Databricks already includes. Worth double-checking what's loaded on your cluster.

As a workaround, you can write geospatial data to standard Parquet files by converting geometries to WKT or WKB format. This avoids the need for GeoParquet and tends to be more reliable across Spark versions.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

1 answer

Your answer

Venkat Reddy Navari 5,255 Reputation points Microsoft External Staff Moderator

2025-08-07T11:33:38.0333333+00:00

Hi Anil Kasar That error looks like a compatibility issue between the GeoParquet library you're using and the version of Spark that comes with your Databricks Runtime. Specifically, the NoSuchMethodError around ParquetOutputWriter usually means the library is expecting a method or class that doesn’t exist in your current Spark version.

A few things you might want to check:

Databricks Runtime version – Not all versions are compatible with every library, especially ones that hook into low-level file I/O like GeoParquet. You can check the Spark version tied to your DBR here: https://docs.databricks.com/release-notes/runtime/index.html

GeoParquet version – If you're pulling the library from GitHub or Maven, make sure it's tested with your Spark/DBR version. Sometimes newer releases drop support for older Spark APIs.

Parquet format support in Databricks – This doc might help clarify how Databricks handles Parquet under the hood: https://learn.microsoft.com/en-us/azure/databricks/query/formats/parquet

Dependency conflicts – If you’re including additional JARs (e.g., for GeoParquet or Parquet libraries), it’s possible there’s a version clash with what Databricks already includes. Worth double-checking what's loaded on your cluster.

As a workaround, you can write geospatial data to standard Parquet files by converting geometries to WKT or WKB format. This avoids the need for GeoParquet and tends to be more reliable across Spark versions.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Answer 1

Anil Kasar 5

Thanks @Venkat Reddy Navari ,

Till day before it was working fine , and from yesterday its started giving problem. I have not made any changes in Library and the runtime version.

Venkat Reddy Navari 5,255 Reputation points Microsoft External Staff Moderator

2025-08-07T13:41:36.81+00:00
Anil Kasar Thanks for the update. Since you mentioned the code was working fine until recently and you haven’t made any changes to the library or Databricks runtime, it’s possible that:

A system-level change (e.g., library or dependency update at cluster level) may have occurred.

Cluster restart or auto-upgrade might have refreshed dependency resolution.

A transient issue or corruption in cached dependencies could be affecting execution.

Suggested Next Steps:

Restart the cluster if you haven’t already this often clears transient or cache-related issues.

Check the cluster libraries tab and confirm if any new libraries (JARs or wheels) were auto installed or updated.

Try running the job on a new cluster with the same runtime and settings to rule out cluster-specific problems.

If the issue persists, consider switching to a standard Parquet write by converting geometries to WKT/WKB format temporarily as a workaround.

Let me know if nothing is working out.

Hope this helps. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Anil Kasar 5 Reputation points

2025-08-07T14:01:27.2066667+00:00

Hi @Venkat Reddy Navari ,

I restarted compute cupola of time , removed all libraries and added them back. Problem is only for write the geoparquet. This issue is not only one compute and workspace but the other compute and workspace also similar issues where earlier code was working fine but now not able to write data in geoparquet.

I spent day to check all this think and later I added comment for help.
Venkat Reddy Navari 5,255 Reputation points Microsoft External Staff Moderator

2025-08-08T17:28:33.34+00:00

Anil Kasar Could you please share the details I requested in the private message?
Venkat Reddy Navari 5,255 Reputation points Microsoft External Staff Moderator

2025-08-11T12:26:02.7266667+00:00

Anil Kasar Thanks for sharing the details SR# 2508080050000959. We’ll coordinate with our team and get back to you with an update.

Share via

databricks not working with Geoparquet for write data

1 answer

Your answer