日期:2014-05-16  浏览次数:20609 次

使用JDBC执行impala SQL出现的问题
   impala版本:1.1.1
   hive版本:0.10

   最近在使用JDBC执行impala sql的时候遇到一个问题,使用JDBC执行insert overwrite/into table...select...语句的时候,执行的结果显示是成功,但是查看表中的数据的时候,发现数据并没有插入到目标表中。通过查看http://impala-node-hostname:25000/queries 发现刚才执行的SQL的状态是Exception.说明确实执行失败。

   出现这种情况的原因是由于hive驱动的bug造成的,因为sessions在执行的时候,impala将取消正在运行的查询。现在hive 0.10以及 0.11的驱动都存在这样的问题,估计能在0.12的版本中解决这个问题。

   解决方案: 我们到/hive/src/jdbc/src/java/org/apache/hive/jdbc这个目录下找到HiveStatement.java这个文件,在这个类中有个execute方法,我们提交的SQL就是通过这个方法来执行的,它的代码如下:
  public boolean execute(String sql) throws SQLException {
    if (isClosed) {
      throw new SQLException("Can't execute after statement has been closed");
    }

    try {
      closeClientOperation();
      TExecuteStatementReq execReq = new TExecuteStatementReq(sessHandle, sql);
      execReq.setConfOverlay(sessConf);
      TExecuteStatementResp execResp = client.ExecuteStatement(execReq);
      if (execResp.getStatus().getStatusCode().equals(TStatusCode.STILL_EXECUTING_STATUS)) {
        warningChain = Utils.addWarning(warningChain, new SQLWarning("Query execuing asynchronously"));
      } else {
        Utils.verifySuccessWithInfo(execResp.getStatus());
      }
      stmtHandle = execResp.getOperationHandle();
    } catch (SQLException eS) {
      throw eS;
    } catch (Exception ex) {
      throw new SQLException(ex.toString(), "08S01", ex);
    }

    if (!stmtHandle.isHasResultSet()) {
      return false;
    }
    resultSet =  new HiveQueryResultSet.Builder().setClient(client).setSessionHandle(sessHandle)
        .setStmtHandle(stmtHandle).setMaxRows(maxRows).setFetchSize(fetchSize)
        .setScrollable(isScrollableResultset)
        .build();
    return true;
  }


修改上述代码的
    if (!stmtHandle.isHasResultSet()) {
      return false;
    }

部分,修改后的代码如下:
  public boolean execute(String sql) throws SQLException {
    if (isClosed) {
      throw new SQLException("Can't execute after statement has been closed");
    }

    try {
      closeClientOperation();
      TExecuteStatementReq execReq = new TExecuteStatementReq(sessHandle, sql);
      execReq.setConfOverlay(sessConf);
      TExecuteStatementResp execResp = client.ExecuteStatement(execReq);
      if (execResp.getStatus().getStatusCode().equals(TStatusCode.STILL_EXECUTING_STATUS)) {
        warningChain = Utils.addWarning(warningChain, new SQLWarning("Query execuing asynchronously"));
      } else {
        Utils.verifySuccessWithInfo(execResp.getStatus());
      }
      stmtHandle = execResp.getOperationHandle();
    } catch (SQLException eS) {
      throw eS;
    } catch (Exception ex) {
      throw new SQLException(ex.toString(), "08S01", ex);
    }

    if (!stmtHandle.isHasResultSet()) {
       // Poll until the query has completed one way or another. DML queries will not return a result
       // set, but we should not return from this method until the query has completed to avoid
       // racing with possible subsequent session shutdown, or queries that depend on the results
       // materialised here.
       TGetOperationStatusReq statusReq = new TGetOperationStatusReq(stmtHandle);
       boolean requestComplete = false;
       while (!requestComplete) {
       try {
       TGetOperationStatusResp statusResp = client.GetOperationStatus(statusReq);
       Utils.verifySuccessWithInfo(statusResp.getStatus());
       if (statusResp.isSetOperationState()) {
       switch (statusResp.getOperationState()) {
       case CLOSED_STATE:
       case FINISHED_STATE:
         return false;
       case CANCELED_STATE:
       // 01000 -> warning
       throw new SQLException("Query was cancelled", "01000");
       case ERROR_STATE:
       // HY000 -> general error
       throw new SQLException("Query failed", "HY000");
       case UKNOW