PIP で NUMBA をインストールする時にエラーになる

GitHub にある誰かが作った Python のソースを動かしたくて、実行してみたのですが、Numba を使っているようで、Numba をインストールしようとしました。

Numba?

Numba gives you the power to speed up your applications with high performance functions written directly in Python. Numba works by generating optimized machine code using the LLVM compiler infrastructure at import time, runtime, or statically (using the included pycc tool).

Numba — Numba

Numba はスピードアップのために使うものだと解釈しました。

Numba のインストール方法

Installing The easiest way to install numba and get updates is by using the Anaconda Distribution: http://continuum.io/downloads.html

If you have Anaconda installed already:
$ conda install numba
or
$ conda update numba
For custom python environments see the README on Github.

Numba — Numba

Numba の公式のサイトのインストール方法を見ると、Anaconda Distribution を使うことを推奨しているようです。

Anaconda is the leading open data science platform powered by Python. The open source version of Anaconda is a high performance distribution of Python and R and includes over 100 of the most popular Python, R and Scala packages for data science.

Download Anaconda Now! | Continuum

Anaconda はデータサイエンスのためのパッケージを 100 以上含めた環境のようです。

Custom Python Environments

or simply
$ pip install numba
GitHub - numba/numba: NumPy aware dynamic Python compiler using LLVM

Anaconda 以外で Numba を使うには pip を使うようです。今回は pip を使う方法で進めました。

Numba のインストール

$ pip install numba
Collecting numba
  Downloading numba-0.31.0.tar.gz (1.2MB)
    100% |████████████████████████████████| 1.2MB 398kB/s
Collecting llvmlite (from numba)
  Downloading llvmlite-0.16.0.tar.gz (95kB)
    100% |████████████████████████████████| 102kB 527kB/s
Requirement already satisfied: numpy in ./lib/python3.6/site-packages (from numba)
Building wheels for collected packages: numba, llvmlite
  Running setup.py bdist_wheel for numba ... done
  Stored in directory: /Users/user/Library/Caches/pip/wheels/1a/93/9e/25e3fef05380f6db4273de73ea0fd6645bcdba14659f1e4e2d
  Running setup.py bdist_wheel for llvmlite ... error
  Complete output from command /Users/user/ENV/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-o1x6nnt4/llvmlite/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/tmpg1o_xzcopip-wheel- --python-tag cp36:
  running bdist_wheel
  running build
  got version from file /private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-o1x6nnt4/llvmlite/llvmlite/_version.py {'version': '0.16.0', 'full': '964cf1d6d725b000c769379a5efe11932fcfc551'}
  running build_ext
  /Users/user/ENV/bin/python3.6 /private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-o1x6nnt4/llvmlite/ffi/build.py
  LLVM version... Traceback (most recent call last):
    File "/private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-o1x6nnt4/llvmlite/ffi/build.py", line 105, in main_posix
      out = subprocess.check_output([llvm_config, '--version'])
    File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 336, in check_output
      **kwargs).stdout
    File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 403, in run
      with Popen(*popenargs, **kwargs) as process:
    File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 707, in __init__
      restore_signals, start_new_session)
    File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 1326, in _execute_child
      raise child_exception_type(errno_num, err_msg)
  FileNotFoundError: [Errno 2] No such file or directory: 'llvm-config'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-o1x6nnt4/llvmlite/ffi/build.py", line 152, in <module>
      main()
    File "/private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-o1x6nnt4/llvmlite/ffi/build.py", line 146, in main
      main_posix('osx', '.dylib')
    File "/private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-o1x6nnt4/llvmlite/ffi/build.py", line 108, in main_posix
      "to the path for llvm-config" % (llvm_config,))
  RuntimeError: llvm-config failed executing, please point LLVM_CONFIG to the path for llvm-config
  error: command '/Users/user/ENV/bin/python3.6' failed with exit status 1

  ----------------------------------------
  Failed building wheel for llvmlite
  Running setup.py clean for llvmlite
Successfully built numba
Failed to build llvmlite
Installing collected packages: llvmlite, numba
  Running setup.py install for llvmlite ... error
    Complete output from command /Users/user/ENV/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-o1x6nnt4/llvmlite/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-4lv7jqpk-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/user/ENV/bin/../include/site/python3.6/llvmlite:
    running install
    running build
    got version from file /private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-o1x6nnt4/llvmlite/llvmlite/_version.py {'version': '0.16.0', 'full': '964cf1d6d725b000c769379a5efe11932fcfc551'}
    running build_ext
    /Users/user/ENV/bin/python3.6 /private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-o1x6nnt4/llvmlite/ffi/build.py
    LLVM version... Traceback (most recent call last):
      File "/private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-o1x6nnt4/llvmlite/ffi/build.py", line 105, in main_posix
        out = subprocess.check_output([llvm_config, '--version'])
      File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 336, in check_output
        **kwargs).stdout
      File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 403, in run
        with Popen(*popenargs, **kwargs) as process:
      File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 707, in __init__
        restore_signals, start_new_session)
      File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 1326, in _execute_child
        raise child_exception_type(errno_num, err_msg)
    FileNotFoundError: [Errno 2] No such file or directory: 'llvm-config'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-o1x6nnt4/llvmlite/ffi/build.py", line 152, in <module>
        main()
      File "/private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-o1x6nnt4/llvmlite/ffi/build.py", line 146, in main
        main_posix('osx', '.dylib')
      File "/private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-o1x6nnt4/llvmlite/ffi/build.py", line 108, in main_posix
        "to the path for llvm-config" % (llvm_config,))
    RuntimeError: llvm-config failed executing, please point LLVM_CONFIG to the path for llvm-config
    error: command '/Users/user/ENV/bin/python3.6' failed with exit status 1

    ----------------------------------------
Command "/Users/user/ENV/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-o1x6nnt4/llvmlite/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-4lv7jqpk-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/user/ENV/bin/../include/site/python3.6/llvmlite" failed with error code 1 in /private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-o1x6nnt4/llvmlite/

次のようなエラーになっているようです。

FileNotFoundError: [Errno 2] No such file or directory: 'llvm-config'
RuntimeError: llvm-config failed executing, please point LLVM_CONFIG to the path for llvm-config

llvm-config コマンドが必要のようですが、ないのでエラーになっているようです。

LLVM 自体は Xcode コマンドラインツールをインストールした時のものがあるようですが、llvm-config は確かにないみたいです。

llvm-config?

DESCRIPTION

llvm-config makes it easier to build applications that use LLVM. It can print the compiler flags, linker flags and object libraries needed to link against LLVM.

llvm-config - Print LLVM compilation options — LLVM 5 documentation

LLVM のコンパイルオプションを出力するものだと解釈しました。

LLVM のインストール方法

llvm-config を含む LLVM をインストールする必要がありそうでしたので、インストール方法を調べました。

Getting Started Quickly (A Summary)

The LLVM Getting Started documentation may be out of date. So, the Clang Getting Started page might also be a good place to start.

Here’s the short story for getting up and running quickly with LLVM:

Getting Started with the LLVM System — LLVM 5 documentation

LLVM の公式のホームページにはソースからビルドするような記載がありました。面倒なので、今回は Homebrew からインストールすることにしました。

LLVM のインストール

$ brew install llvm
==> Installing dependencies for llvm: libffi
==> Installing llvm dependency: libffi
==> Downloading https://homebrew.bintray.com/bottles/libffi-3.2.1.sierra.bottle.
######################################################################## 100.0%
==> Pouring libffi-3.2.1.sierra.bottle.tar.gz
==> Caveats
This formula is keg-only, which means it was not symlinked into /usr/local.

Some formulae require a newer version of libffi.

For compilers to find this software you may need to set:
    LDFLAGS:  -L/usr/local/opt/libffi/lib

==> Summary
🍺  /usr/local/Cellar/libffi/3.2.1: 16 files, 297.0KB
==> Installing llvm
==> Downloading https://homebrew.bintray.com/bottles/llvm-4.0.0.sierra.bottle.ta
######################################################################## 100.0%
==> Pouring llvm-4.0.0.sierra.bottle.tar.gz
==> Caveats
LLVM executables are installed in /usr/local/opt/llvm/bin.
Extra tools are installed in /usr/local/opt/llvm/share/llvm.
To use the bundled libc++ please add the following LDFLAGS:
  LDFLAGS="-L/usr/local/opt/llvm/lib -Wl,-rpath,/usr/local/opt/llvm/lib"

This formula is keg-only, which means it was not symlinked into /usr/local.

macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have this software first in your PATH run:
  echo 'export PATH="/usr/local/opt/llvm/bin:$PATH"' >> ~/.bash_profile

For compilers to find this software you may need to set:
    LDFLAGS:  -L/usr/local/opt/llvm/lib
    CPPFLAGS: -I/usr/local/opt/llvm/include


If you need Python to find bindings for this keg-only formula, run:
  echo /usr/local/opt/llvm/lib/python2.7/site-packages >> /usr/local/lib/python2.7/site-packages/llvm.pth
  mkdir -p /Users/user/Library/Python/2.7/lib/python/site-packages
  echo 'import site; site.addsitedir("/usr/local/lib/python2.7/site-packages")' >> /Users/user/Library/Python/2.7/lib/python/site-packages/homebrew.pth
==> Summary
🍺  /usr/local/Cellar/llvm/4.0.0: 2,245 files, 1GB

1GB もありました。

必要であれば環境変数の PATH を設定するように書いてあります。

$ echo 'export PATH="/usr/local/opt/llvm/bin:$PATH"' >> ~/.bash_profile

Python to find bindings が必要であれば次のように設定するようです。

$ echo /usr/local/opt/llvm/lib/python2.7/site-packages >> /usr/local/lib/python2.7/site-packages/llvm.pth
$ mkdir -p /Users/user/Library/Python/2.7/lib/python/site-packages
$ echo 'import site; site.addsitedir("/usr/local/lib/python2.7/site-packages")' >> /Users/user/Library/Python/2.7/lib/python/site-packages/homebrew.pth

Python to find bindings って何かな？と思って調べてみたところ、

clang ASTの情報が欲しかったので，python bindingsを使おうとしたらまたつまりました．教えてもらったのをメモ！

…略…

動かしたサンプルはこれ．

[main.py]
import sys

import clang.cindex
from clang.cindex import Index
from clang.cindex import Config

def print_node_tree(node):
    print "%s : %s" % (node.kind.name, node.displayname)
    for child in node.get_children():
        print_node_tree(child)

index = Index.create()
tu = index.parse("test.cpp")
print_node_tree(tu.cursor)
Clangのpython bindingsを使う - 脱初心者を目指す

Python のソースコードの中から C++ 等のソースを呼び出すようなものらしいと解釈しました。今回は Python bindings は必要なさそうでしたので設定しませんでした。

Numba のインストール(2回目)

$ pip install numba
Collecting numba
Requirement already satisfied: numpy in ./lib/python3.6/site-packages (from numba)
Collecting llvmlite (from numba)
  Using cached llvmlite-0.16.0.tar.gz
Building wheels for collected packages: llvmlite
  Running setup.py bdist_wheel for llvmlite ... error
  Complete output from command /Users/user/ENV/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-d41q9kgl/llvmlite/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/tmpoogsp4e5pip-wheel- --python-tag cp36:
  running bdist_wheel
  running build
  got version from file /private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-d41q9kgl/llvmlite/llvmlite/_version.py {'version': '0.16.0', 'full': '964cf1d6d725b000c769379a5efe11932fcfc551'}
  running build_ext
  /Users/user/ENV/bin/python3.6 /private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-d41q9kgl/llvmlite/ffi/build.py
  LLVM version... 4.0.0

  Traceback (most recent call last):
    File "/private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-d41q9kgl/llvmlite/ffi/build.py", line 152, in <module>
      main()
    File "/private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-d41q9kgl/llvmlite/ffi/build.py", line 146, in main
      main_posix('osx', '.dylib')
    File "/private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-d41q9kgl/llvmlite/ffi/build.py", line 119, in main_posix
      raise RuntimeError(msg)
  RuntimeError: Building llvmlite requires LLVM 3.9.x. Be sure to set LLVM_CONFIG to the right executable path.
  Read the documentation at http://llvmlite.pydata.org/ for more information about building llvmlite.

  error: command '/Users/user/ENV/bin/python3.6' failed with exit status 1

  ----------------------------------------
  Failed building wheel for llvmlite
  Running setup.py clean for llvmlite
Failed to build llvmlite
Installing collected packages: llvmlite, numba
  Running setup.py install for llvmlite ... error
    Complete output from command /Users/user/ENV/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-d41q9kgl/llvmlite/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-o46hy98_-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/user/ENV/bin/../include/site/python3.6/llvmlite:
    running install
    running build
    got version from file /private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-d41q9kgl/llvmlite/llvmlite/_version.py {'version': '0.16.0', 'full': '964cf1d6d725b000c769379a5efe11932fcfc551'}
    running build_ext
    /Users/user/ENV/bin/python3.6 /private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-d41q9kgl/llvmlite/ffi/build.py
    LLVM version... 4.0.0

    Traceback (most recent call last):
      File "/private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-d41q9kgl/llvmlite/ffi/build.py", line 152, in <module>
        main()
      File "/private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-d41q9kgl/llvmlite/ffi/build.py", line 146, in main
        main_posix('osx', '.dylib')
      File "/private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-d41q9kgl/llvmlite/ffi/build.py", line 119, in main_posix
        raise RuntimeError(msg)
    RuntimeError: Building llvmlite requires LLVM 3.9.x. Be sure to set LLVM_CONFIG to the right executable path.
    Read the documentation at http://llvmlite.pydata.org/ for more information about building llvmlite.

    error: command '/Users/user/ENV/bin/python3.6' failed with exit status 1

    ----------------------------------------
Command "/Users/user/ENV/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-d41q9kgl/llvmlite/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-o46hy98_-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/user/ENV/bin/../include/site/python3.6/llvmlite" failed with error code 1 in /private/var/folders/3m/p627w85d35b_tt0lrxzndm4c0000gn/T/pip-build-d41q9kgl/llvmlite/

次のようなエラーになっているようです。

RuntimeError: Building llvmlite requires LLVM 3.9.x. Be sure to set LLVM_CONFIG to the right executable path.

Numba が依存している Python のパッケージである llvmlite が必要としているのが LLVM 3.9.x で、Homebrew でインストールした LLVM が LLVM version... 4.0.0 だからのようです。

対処

インストールしたLLVMのヴァージョンは3.9.1で、llvmliteが必要とするLLVMのヴァージョンは3.8.xらしいので、ヴァージョンを落としてLLVMをインストールする必要があるみたいです。

上手くいった手順

s-1. llvmのインストール brew install homebrew/versions/llvm38
s-2. export LLVM_CONFIG=/usr/local/Cellar/llvm38/3.8.1/bin/llvm-config-3.8
s-3. llvmlite のインストール pip3 install llvmlite
s-4. numbaのインストール pip3 install numba

Python Numba のインストール - サブロウ丸

LLVM のバージョンを落としてインストールすれば良さそうです。

続く

Python の環境設定についての記事で言われていることが少し理解できてきました。

初めてで知らないことが多いため調べながら進めることになるわけですが、本来の目的と違うことに時間がかかりすぎてしまって、意味があるのか考えてしまいました。

対処にあるようにバージョンを合わせてインストールするのは面倒なので、もう素直に Anaconda をインストールしようと思います。

最初から Anaconda を使えばよかった。

8 April 2017 追記

次の記事を書きました。

Docker のインストールから Docker の Anaconda(Jupyter, Numba) を使うまで

pip で Numba をインストールする時にエラーになる