HDInsight Hive error: Unable to rename

2024-01-17

When you use Azure PowerShell to run HiveQL statements against HDInsight or HDInsight emulator, you might get the following error:

Failed with exception Unable to rename: 
wasb://xxxxxxx@xxxxxxxx.blob.core.windows.net/hive/scratch/hive_2014-02-12_14-58-27_594_640725686036024640-1/-ext-10001 
to: wasb://xxxxxxxx@xxxxxxxx.blob.core.windows.net/apps/hive/warehouse/delays
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask

Creating a /apps/hive/warehouse folder on HDInsight can bypass this error message, but could run into another problem. You might notice a newly created Hive table doesn't contain any records. In the error log, you could find the data file was moved to the /apps/hive/warehouse folder instead of the /hive/warehouse folder.

This problem was fixed in HDInsight clusters version 3.1 and higher, with the exception of HBase deployments (fix in the works).
**
To automate this fix where needed:
**You can use PowerShell to deploy HDInsight following this blog article and this documentation with a custom configuration for the Hadoop Core settings, to include this workaround each time you create a cluster from PowerShell scripts.

These are the two key syntax lines to use with a code sample.

# Customize Core configuration options
$CoreConfigValues = @{ "templeton.hive.properties"="hive.metastore.local=false,hive.metastore.uris=thrift://headnodehost:9083,hive.metastore.warehouse.dir=/hive/warehouse/" }

...
| Add-AzureHDInsightConfigValues -Core $CoreConfigValues
...

This ends up putting the value in this core-site.xml file on the nodes:
"C:\apps\dist\hadoop-2.4.0.2.1.6.0-2103\etc\hadoop\core-site.xml"

<property>
<name>templeton.hive.properties</name>
<value>hive.metastore.local=false,hive.metastore.uris=thrift://headnodehost:9083,hive.metastore.warehouse.dir=/hive/warehouse/</value>
</property>

To fix the problem on HDInsight cluster version 3.0:

Connect to the cluster head node using RDP. For instructions, see Administer HDInsight clusters using Management portal.
Use a text editor to open C:\apps\dist\hcatalog-0.12.0.2.0.7.0-1528\etc\webhcat\webhcat-site.xml.

Find the following tags:

<property>
    <name>templeton.hive.properties</name>
    <value>hive.metastore.local=false,hive.metastore.uris=thrift://headnodehost:9083,hive.metastore.warehouse.dir=/apps/hive/warehouse</value>
    <description>Properties to set when running hive.</description>
</property>

Replace /apps/hive/warehouse to /hive/warehouse.
Save the file.
Open Service from Administrative Tools, and restart Apache Hadoop templeton.

To fix the problem on HDInsight cluster version 2.1:

Connect to the cluster head node using RDP. For instructions, see Administer HDInsight clusters using Management portal.
Use a text editor to open C:\apps\dist\hcatalog-0.11.0.1.3.2.0-05\conf\webhcat-site.xml.

Find the following tags:

<property>
    <name>templeton.hive.properties</name>
    <value>hive.metastore.local=false,hive.metastore.uris=thrift://headnodehost:9083,hive.metastore.warehouse.dir=/apps/hive/warehouse</value>
    <description>Properties to set when running hive.</description>
</property>

Replace /apps/hive/warehouse to /hive/warehouse.
Save the file.
Open Service from Administrative Tools, and restart Apache Hadoop templeton.

To fix the problem on HDInsight emulator:

Use a text editor to open C:\Hadoop\hive-0.9.0\conf\hive-site.xml.

Find the following tags:

<property>
  <name>hive.metastore.warehouse.dir</name>
  <value>/apps/hive/warehouse</value>
  <description>location of default database for the warehouse</description>
</property>

Replace /apps/hive/warehouse to /hive/warehouse.
Save the file.
Open Service from Administrative Tools, and restart Apache Hadoop templeton.

Share via

HDInsight Hive error: Unable to rename

Additional resources