Win7平台Python3使用impyla连接Hive遇到的坑

mengpegnfei

环境硬件配置及Hadoop，Hive版本此博客置顶文章中有
安装步骤pip install pure-sasl

• 1
  

  Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
  Collecting pure-sasl
    Downloading https://pypi.tuna.tsinghua.edu.cn/packages/16/83/30eaf3765de898083
  75a8358f9c15d45a3dd44ed26be991471abc0b4480b/pure_sasl-0.5.1-py2.py3-none-any.whl
    Installing collected packages: pure-sasl
  Successfully installed pure-sasl-0.5.1

pip install thrift_sasl==0.2.1 --no-deps

• 1
  

  Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
  Collecting thrift_sasl==0.2.1
    Downloading https://pypi.tuna.tsinghua.edu.cn/packages/80/36/16dfe92d32df63cc2
  b7b7be8d0e4a736617b7e52daaa7f83ae386a89d179/thrift_sasl-0.2.1.tar.gz
  Installing collected packages: thrift-sasl
    Running setup.py install for thrift-sasl … done
  Successfully installed thrift-sasl-0.2.1

pip install thrift==0.9.3

• 1
  

  Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
  Collecting thrift==0.9.3
    Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ae/58/35e3f0cd290039ff8
  62c2c9d8ae8a76896665d70343d833bdc2f748b8e55/thrift-0.9.3.tar.gz
  Installing collected packages: thrift
    Running setup.py install for thrift … done
  Successfully installed thrift-0.9.3

pip install impyla

• 1
  

  Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
  Collecting impyla
    Downloading https://pypi.tuna.tsinghua.edu.cn/packages/80/93/f0d226061ee4679d5
  b593c88c7b2e9e077a271c799d29facf31bf03666c1/impyla-0.14.1.tar.gz (151kB)
      40% |█████████████                   | 61kB 1.3MB/s eta 0:00:01
      47% |███████████████                 | 71kB 1.1MB/s eta 0:00:
      53% |█████████████████               | 81kB 1.1MB/s eta 0:0
      60% |███████████████████▌            | 92kB 1.2MB/s eta
      67% |█████████████████████▌          | 102kB 1.1MB/s e
      74% |████████████████████████        | 112kB 1.1MB/s
      80% |██████████████████████████      | 122kB 759kB
      87% |████████████████████████████    | 133kB 786
      94% |██████████████████████████████  | 143kB 8
      100% |████████████████████████████████| 153k
  B 1.0MB/s
  Requirement already satisfied: six in c:\program files\python36\lib\site-package
  s (from impyla) (1.11.0)
  Collecting bitarray (from impyla)
    Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e2/1e/b93636ae36d08d0ee
  3aec40b08731cc97217c69db9422c0afef6ee32ebd2/bitarray-0.8.3.tar.gz
  Collecting thriftpy>=0.3.5 (from impyla)
    Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f4/19/cca118cf7d2087310
  dbc8bd70dc7df0c1320f2652873a93d06d7ba356d4a/thriftpy-0.3.9.tar.gz (208kB)
      39% |████████████▌                   | 81kB 875kB/s eta 0:00:01
      44% |██████████████                  | 92kB 984kB/s eta 0:00:0
      49% |████████████████                | 102kB 820kB/s eta 0:0
      54% |█████████████████               | 112kB 820kB/s eta 0:
      59% |███████████████████             | 122kB 820kB/s eta
      63% |████████████████████▌           | 133kB 729kB/s et
      68% |██████████████████████          | 143kB 937kB/s e
      73% |███████████████████████▌        | 153kB 937kB/s
      78% |█████████████████████████       | 163kB 820kB/
      83% |███████████████████████████     | 174kB 937k
      88% |████████████████████████████▌   | 184kB 93
      93% |██████████████████████████████  | 194kB 9
      98% |███████████████████████████████▌| 204kB
      100% |████████████████████████████████| 215k
  B 1.1MB/s
  Requirement already satisfied: ply<4.0,>=3.4 in c:\program files\python36\lib\si
  te-packages (from thriftpy>=0.3.5->impyla) (3.11)
  thrift-sasl 0.2.1 requires sasl>=0.2.1, which is not installed.
  Installing collected packages: bitarray, thriftpy, impyla
    Running setup.py install for bitarray … done
    Running setup.py install for thriftpy … done
    Running setup.py install for impyla … done
  Successfully installed bitarray-0.8.3 impyla-0.14.1 thriftpy-0.3.9

执行数据库连接后，出现问题

  ThriftParserError: ThriftPy does not support generating module with path in protocol ‘c’

定位到 \Lib\site-packages\thriftpy\parser\parser.py的
if url_scheme == '':    with open(path) as fh:        data = fh.read()elif url_scheme in ('http', 'https'):    data = urlopen(path).read()else:    raise ThriftParserError('ThriftPy does not support generating module '                            'with path in protocol \'{}\''.format(                                url_scheme))

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
  

更改为
if url_scheme == '':    with open(path) as fh:        data = fh.read()elif url_scheme in ('c', 'd','e','f''):    with open(path) as fh:        data = fh.read()elif url_scheme in ('http', 'https'):    data = urlopen(path).read()else:    raise ThriftParserError('ThriftPy does not support generating module '                            'with path in protocol \'{}\''.format(                                url_scheme))

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
  

执行数据库连接后，再次出现问题

  TypeError: can’t concat str to bytes

定位到错误的最后一条，在init.py第94行
...header = struct.pack(">BI", status, len(body))self._trans.write(header + body)...

• 1
• 2
• 3
• 4
  

修改为
...header = struct.pack(">BI", status, len(body))if(type(body) is str):    body = body.encode() self._trans.write(header + body)...

• 1
• 2
• 3
• 4
• 5
• 6
  

执行连接 成功
最终模块  模块  版本
  pure_sasl  0.5.1
  thrift_sasl  0.2.1
  thrift  0.9.3
  bitarray  0.8.3
  thriftpy  0.3.9
  impyla  0.14.1连接代码from impala.dbapi import connectfrom impala.util import as_pandasconn = connect(host='***', port=10000, auth_mechanism='PLAIN', user='***', password='***', database='***')cursor = conn.cursor()cursor.execute('show databases')print(as_pandas(cursor))

• 1
• 2
• 3
• 4
• 5
• 6
• 7
  

遇到的问题除去以上两个问题还有

‘TSocket’ object has no attribute ‘isOpen’

Can’t connect to unsecured hive. SASL error: TTransportException: Could not start SASL: Error in sasl_client_start (-4) SASL(-4): no mechanism available: Unable to find a callback: 2

TTransportException: Could not start SASL: Error in sasl_client_start (-4) SASL(-4): no mechanism available: Unable to find a callback: 2

‘No protocol version header’)thriftpy.protocol.exc.TProtocolException: TProtocolException(type=4)

以上问题均可以
pip uninstall

• 1
  

卸载所有相关模块之后， 使用本文的顺序和版本号安装来保证运行正确
---------------------作者：岚匀 来源：CSDN 原文：https://blog.csdn.net/Xiblade/article/details/82318294?utm_source=copy 版权声明：本文为博主原创文章，转载请附上博文链接！