From Mageia wiki
Jump to: navigation, search
Warning.png
Warning!
pyc/pyo Policy is for Mageia 4 and later.



Rules

Naming Conventions

These naming guidelines only apply to Python packages whose main purpose is providing a Python library; packages that mainly provide user-level tools that happen to be written in Python do not need to follow these naming guidelines.

  • The name of a Python extension/library package must be of the form python-UPSTREAM. If the upstream name UPSTREAM contains “python”, that should be dropped from the name.
  • Packages for Python 3 must be of the form python3-UPSTREAM.
  • Please use lower case.

Rationale:

  • Using a prefix is serving as a name space, and clearly shows how to find python modules.
  • Having everything in lower case is easier to read, and offers a consistent sorting.

pyc/pyo Policy

Files should be compiled before rpm creation and includes. If you do not include them in your packages, python will try to create them when the user runs the program. If the system administrator uses them, then the files will be successfully written. Later, when the package is removed, the .pyc and .pyo files will be left behind on the filesystem. To prevent that you need to byte compile the files when building your package and include the files in the %files section.

As often .pyo files are exactly the same as .pyc, the Mageia build system has brp-python-hardlink enabled. This will hardlink these files together (like fdupes does).

Files to Include

When installing python modules we include several different types of files.

  • *.py source files because they are used when generating tracebacks
  • *.pyc and *.pyo byte compiled files
  • *.egg-info files or directories. If these are generated by the module's build scripts they must be included in the package because they might be needed by other applications and modules at run-time.

Building for Several Versions of Python

As of Mageia 4, Python 2.7 still is the standard Python, while Python 3.3 is delivered, too. To support developers aiming to move forward to Python 3, we ask you to provide packages for both version whenever possible. If the same version of the package can be used for both Python-versions, please put building for both into one spec-file. The examples below are showing this.

Bytecompiling with the correct python version

When byte compiling a .py file, python embeds a magic number in the byte compiled files that correspond to the runtime. The Python interpreter checks this magic number when loading the byte compiled file. If it does not match the current runtime, the source is compiled again, which basically renders the cached byte code useless. So the make the caching work as expected, files in %{python_sitelib} and %{python_sitearch} must correspond to respective runtime. For instance files below %{_usr}/lib/python3.1/site-packages are expected to be compiled for the 3.1 runtime.

The brp-python-bytecompile script tries to figure this out for you. The script determines which interpreter to use when byte compiling the module by following these steps:

  1. what directory is the module installed in? If it's /usr/lib{,64}/pythonX.Y, then pythonX.Y is used to byte compile the module. If pythonX.Y is not installed, then an error is returned and the rpm build process will exit on an error so remember to BuildRequire the proper python package.
  2. the script interpreter defined in %{__python} is used to compile the modules. This defaults to the latest python2 version on Mageia 4. If you need to compile this module for python3, set it to /usr/bin/python3 instead:
    %define __python %{__python3}
    

    Doing this is useful when you have a python3 application that's installing a private module into its own directory. For instance, if the foobar application installs a module for use only by the command line application in %{_datadir}/foobar. Since these files are not in one of the python3 library paths (ie. /usr/lib/python3.1) you have to override %{__python} to tell brp-python-bytecompile to use the python3 interpreter for byte compiling.

These settings are enough to properly byte compile any package that builds python modules in %{python_sitelib} or %{python_sitearch} or builds for only a single python interpreter. However, if the application you're packaging needs to build with both python2 and python3 and install into a private module directory (perhaps because it provides one utility written in python2 and a second utility written in python3) then you need to do this manually. Here's a sample spec file snippet that shows what to do:

# Turn off the brp-python-bytecompile script 
%define __brp_python_bytecompile %{nil}

# Buildrequire both python2 and python3
BuildRequires: python2-devel python3-devel
[...]

%install
# Installs a python2 private module into %{buildroot}%{_datadir}/mypackage/foo
# and installs a python3 private module into %{buildroot}%{_datadir}/mypackage/bar
make install DESTDIR=%{buildroot}

# Manually invoke the python byte compile macro for each path that needs byte
# compilation.
%py_byte_compile %{__python2} %{buildroot}%{_datadir}/mypackage/foo
%py_byte_compile %{__python3} %{buildroot}%{_datadir}/mypackage/bar

The %py_byte_compile macro takes two arguments. The first is the python interpreter to use for byte compiling. The second is a file or directory to byte compile. If the second argument is a directory, the macro will recursively byte compile any *.py file in the directory.

No %{} for py_byte_compile: RPM macros can only take arguments when they do not have curly braces around them. Therefore, py_byte_compile won't work correctly if you write: %{py_byte_compile} %{__python}}

Writing the spec-File

Tags

Group should be Development/Python:

 Group: Development/Python

For pure Python packages, BuildArch must be noarch:

 BuildArch: noarch

BuildRequires

In the typical case, just use:

 BuildRequires: python

If you are building a C extension-modules, use python-devel, but leave away python (which is requires by python-devel anyway):

 BuildRequires: pkgconfig(python)

As needed add one of:

 BuildRequires: python-setuptools
 BuildRequires: python-pip

Requires

  • Please do not specify `python` as a requirement for your package. It will be added automatically in the correct way.
  • The %py_requires macro has been deprecated since long and should not be used. It will be removed soon, latest for Mageia 5.

Some times we need to exclude modules from autorequires  :

%global __requires_exclude pythonegg\\(3\\)\\(enum34\\)

Provides

Packages shouldn’t have explicit provides, as it is automatically added in rpmbuild itself.

But usually we don't want to provide private python extension libs so please consider using the following:

 # we don't want to provide private python extension libs
 %define _exclude_files_from_autoprov %{python2_sitearch}/.*\\.so\\|%{python3_sitearch}/.*\\.so

%prep

Typically:

 %setup -q -n %{fname}-%{version}
 %apply_patches

If you are building for Python 3 in addition to 2.x, add this line:

 cp -a . %{py3dir}

%build

Typically:

%{__python2} setup.py build

If you are building for Python 3 in addition to 2.x, add this line:

pushd %{py3dir} 
%{__python3} setup.py build
popd

Do not use the following options for setup.py build:

  • '--executable'
  • '--plat-name'

%install

Typically:

 %{__python} setup.py install --root=%{buildroot} --skip-build
 
 pushd %{py3dir} 
 %{__python3} setup.py install --root=%{buildroot} --skip-build
 popd

If you are building for Python 2.x, only, leave off the Python3-lines, of course.

Do not use the following options for setup.py install:

  • '--optimize' or -O': see above
  • '--record': this is done automatically
  • '--compile' (is default anyway), '--no-compile' (brb-python-bytecompile would compile later anyway)

%check

For example:

 %check
 pushd %{py3dir}/build/lib/astroid
 PYTHONPATH=%{buildroot}%{python3_sitelib} %{__python3} -m unittest discover -s test -p "unittest*.py"
 popd
 pushd build/lib/astroid
 PYTHONPATH=%{buildroot}%{python2_sitelib} %{__python} -m unittest discover -s test -p "unittest*.py"
 popd

If a module don't have a test suite, a minimal check for a module can be:

 %check
 export PYTHONPATH=$RPM_BUILD_ROOT%{python_sitearch}
 %{__python2} -c 'import %{module}; print(%{module}.version)'
 pushd %{py3dir}
 export PYTHONPATH=$RPM_BUILD_ROOT%{python3_sitearch}
 %{__python3} -c 'import %{module}; print(%{module}.version)'
 popd

%files

Typically:

 %doc README
 %{python2_sitelib}/*egg-info
 %{python2_sitelib}/%{fname}

Macros

The following macros are defined in /etc/rpm/macros.d/:

 %python_sitearch %(%{__python} -c "from distutils.sysconfig import get_python_lib; print get_python_lib(1)")
 %python_sitelib	%(%{__python} -c "from distutils.sysconfig import get_python_lib; print get_python_lib()")
 %python_version	%(%{__python} -c "import sys; print(sys.version[0:3])")
 
 %pyver	%(python -V 2>&1 | cut -f2 -d" " | cut -f1,2 -d".")
 
 %py_prefix              %(python -c "import sys; print sys.prefix" 2>/dev/null || echo PYTHON-NOT-FOUND)
 # backward compatibility
 %py_libdir              %py_purelibdir
 %py_platlibdir          %(python -c 'import distutils.sysconfig; print distutils.sysconfig.get_python_lib(standard_lib=1,plat_specific=1)' 2>/dev/null || echo PYTHON-LIBDIR-NOT-FOUND)
 %py_purelibdir          %(python -c 'import distutils.sysconfig; print distutils.sysconfig.get_python_lib(standard_lib=1,plat_specific=0)' 2>/dev/null || echo PYTHON-LIBDIR-NOT-FOUND)
 %py_incdir              %(python -c 'import distutils.sysconfig; print distutils.sysconfig.get_python_inc()' 2>/dev/null || echo PYTHON-INCLUDEDIR-NOT-FOUND)
 %py_sitedir             %py_puresitedir
 %py_platsitedir         %(python -c 'import distutils.sysconfig; print distutils.sysconfig.get_python_lib(plat_specific=1)' 2>/dev/null || echo PYTHON-LIBDIR-NOT-FOUND)
 %py_puresitedir         %(python -c 'import distutils.sysconfig; print distutils.sysconfig.get_python_lib()' 2>/dev/null || echo PYTHON-LIBDIR-NOT-FOUND)
 
 %py_compile(O)  \
 find %1 -name '*.pyc' -exec rm -f {} \\; \
 python -c "import sys, os, compileall; br='%{buildroot}'; compileall.compile_dir(sys.argv[1], ddir=br and (sys.argv[1][len(os.path.abspath(br)):]+'/') or None)" %1 \
 %{-O: \
 find %1 -name '*.pyo' -exec rm -f {} \\; \
 python -O -c "import sys, os, compileall; br='%{buildroot}'; compileall.compile_dir(sys.argv[1], ddir=br and (sys.argv[1][len(os.path.abspath(br)):]+'/') or None)" %1 \
 }
 
 %__python3 /usr/bin/python3
 %python3_sitelib %(%{__python3} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")
 %python3_sitearch %(%{__python3} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib(1))")
 %python3_version %(%{__python3} -c "import sys; print(sys.version[0:3])")
 %py3_prefix %(%{__python3} -c "import sys; print(sys.prefix)" 2>/dev/null || echo PYTHON-NOT-FOUND)
 %py3_platsitedir %python3_sitearch
 %py3_puresitedir %python3_sitelib
 %py3_incdir %(%{__python3} -c 'from distutils.sysconfig import get_python_inc; print(get_python_inc())' 2>/dev/null || echo PYTHON-INCLUDEDIR-NOT-FOUND)
 %py3dir %{_builddir}/python3-%{name}-%{version}-%{release}

New in Mageia6

 %__python2 /usr/bin/python2
 %python2_sitelib %(%{__python2} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")
 %python2_sitearch %(%{__python2} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib(1))")
 %py2ver %(%{__python2} -V 2>&1 | cut -f2 -d" " | cut -f1,2 -d".")
 %python2_version %(%{__python2} -c "import sys; print(sys.version[0:2])")
 %py2_ver %(%{__python2} -c "import sys; v=sys.version_info[:2]; print('%%d.%%d'%%v)" 2>/dev/null || echo PYTHON-NOT-FOUND)
 %py2_prefix %(%{__python2} -c "import sys; print(sys.prefix)" 2>/dev/null || echo PYTHON-NOT-FOUND)
 %py2_platsitedir %python2_sitearch
 %py2_puresitedir %python2_sitelib
 %py2_incdir %(%{__python2} -c 'from distutils.sysconfig import get_python_inc; print(get_python_inc())' 2>/dev/null || echo PYTHON-INCLUDEDIR-NOT-FOUND)
 %py2dir %{_builddir}/python2-%{name}-%{version}-%{release}
 
 %py2_setup setup.py
 
 %py2_shbang_opts -s
 
 %py2_build() %{expand:\
 CFLAGS="%{optflags}" %{__python2} %{py2_setup} %{?py2_setup_args} build --executable="%{__python2} %{py2_shbang_opts}" %{?1}\
 }
 
 %py2_install() %{expand:\
 CFLAGS="%{optflags}" %{__python2} %{py2_setup} %{?py2_setup_args} install -O1 --skip-build --root %{buildroot} %{?1}\
 }
 
 %py3_setup setup.py
 
 %py3_shbang_opts -s
 
 %py3_build() %{expand:\
 CFLAGS="%{optflags}" %{__python3} %{py3_setup} %{?py3_setup_args} build --executable="%{__python3} %{py3_shbang_opts}" %{?1}\
 }
 
 %py3_install() %{expand:\
 CFLAGS="%{optflags}" %{__python3} %{py3_setup} %{?py3_setup_args} install -O1 --skip-build --root %{buildroot} %{?1}\
 }


Usage sample of %py2_build :

If you need something like :

 python2 setup.py build --use-system-libraries --offline

you can write :

%py2_build -- --use-system-libraries --offline

Tools

There is a tool that can help you to package Python modules from PyPI : https://bitbucket.org/bkabrda/pyp2rpm

It is packaged for Mageia, so you just have to install python-pyp2rpm.


Samples

First Sample

Here is a sample of a package build for both Python 2.x and Python 3 and even including a -doc-subpackage. It builds both versions in the same directory; this is possible because the build products for different versions of Python usually do not conflict.

There are cases where it is not possible to build in a single directory. Most commonly this happens when the sources are modified during the build process to convert them from python2 to python3 using the the 2to3 tool. In that case, please see the second sample.


%define upstream_name	fdb

Name:		python-fdb
Summary:	Firebird RDBMS bindings for Python
Version:	1.7
Release:	%mkrel 3
Source0:	https://pypi.io/packages/source/f/%{upstream_name}/%{upstream_name}-%{version}.tar.gz
URL:		www.firebirdsql.org/
Group:		Development/Python
License:	BSD
BuildArch:	noarch
BuildRequires:	firebird-devel
BuildRequires:	python2dist(setuptools)
BuildRequires:  python3dist(setuptools)

%description
Set of Firebird RDBMS bindings for Python. 

%package -n python2-%{upstream_name}
Summary:        Firebird RDBMS bindings for Python 2
%{?python_provide:%python_provide python2-%{upstream_name}}


%description -n python2-%{upstream_name}
Set of Firebird RDBMS bindings for Python.


%package -n python3-%{upstream_name}
Summary:        Firebird RDBMS bindings for Python 3
%{?python_provide:%python_provide python3-%{upstream_name}}


%description -n python3-%{upstream_name}
Set of Firebird RDBMS bindings for Python.


%package doc
Summary:        Documentation for %{name}
Group:          Documentation
License:        BSD


%description doc
Documentation files for %{name}. It works on Python2.6+ and Python3.


%prep
%autosetup -n %{upstream_name}-%{version}
rm -rf %{upstream_name}.egg-info


%build
%py2_build
%py3_build


%install
%py2_install
%py3_install


%files -n python2-%{upstream_name}
%doc README
%license LICENSE.TXT
%{python2_sitelib}/*


%files -n python3-%{upstream_name}
%doc README
%license LICENSE.TXT
%{python3_sitelib}/*


%files doc
%doc docs/* 
%license LICENSE.TXT


Second Sample

Sometimes it is impossible to build both versions from the same source directory. Most often this happens when sources are "translated" to python3 in the source directory and made incompatible with python2 in the process. This used to be fairly common, but is fortunately much rarer now. Some things to look for are:

  • Sources are not Python 3 compatible (print without parentheses is used, old module names like ConfigParser are imported),
  • six module is not used,
  • 2to3 is run in setup.py without creating a separate build directory.

Our method in building from the same code to make the two separate modules is to keep each build as independent as possible. To do that, we copy the source tree to python3 so that the python 2 sources are entirely independent from the python 3 sources.

%prep
%setup -q -n %{fname}-%{version} 
%apply_patches
cp -a . %{py3dir} 

%build
%py2_build

pushd %{py3dir} 
%py3_build
popd

%install
%py2_install

pushd %{py3dir} 
%py3_install
popd

Avoiding collisions between the python 2 and python 3 stacks

The python 2 and python 3 stacks are intended to be fully-installable in parallel. When generalizing the package for both python 2 and python 3, it is important to ensure that two different built packages do not attempt to place different payloads into the same path.

Executables in /usr/bin

The problem

Many existing python packages install executables into /usr/bin.

For example if we have a console_scripts in a setup.py shared between python 2 and python 3 builds: these will spit out files in /usr/bin/, and these will collide.

For example python-coverage has a setup.py that contains:

    entry_points = {
        'console_scripts': [
            'coverage = coverage:main',
            ]
        },

which thus generates a /usr/bin/coverage executable (this is a python script that runs another python script whilst generating code-coverage information on the latter).

Similarly for the 'scripts' clause; see e.g. python-pygments: Pygments-1.1.1/setup.py has:

    scripts = ['pygmentize'],

which generates a /usr/bin/pygmentize (this is a python script that leverages the pygments syntax-highlighting module, giving a simple command-line interface for generating syntax-highlighted files)

Guidelines

If the executables provide the same functionality independent of whether they are run on top of Python 2 or Python 3, then only one version of the executable should be packaged. Currently it will be the python 2 implementation, but once the Python 3 implementation is proven to work, the executable can be retired from the python 2 build and enabled in the python 3 package. Be sure to test the new implementation. Transitioning from python2 to python3 is left to individual package maintainers.

Examples of this:

  • /usr/bin/pygmentize ought to generate the same output regardless of whether it's implemented via Python 2 or Python 3, so only one version needs to be shipped.

If the executables provide different functionality for Python 2 and Python 3, then both versions should be packaged.

Examples of this:

  • /usr/bin/coverage runs a python script, augmenting the interpreter with code-coverage information. Given that the interpreter itself is the thing being worked with, it's reasonable to package both versions of the executable.
  • /usr/bin/bpython augments the interpreter with a "curses" interface. Again, it's reasonable to package both versions of this.
  • /usr/bin/easy_install installs a module into one of the Python runtimes: we need a version for each runtime.

As an exception, for the rpms that are part of a python runtime itself, we plan to package both versions of the executables, so that e.g. both the python 2 and python 3 versions of 2to3 are packaged.

Naming

Many executables already contain a "-MAJOR.MINOR" suffix, for example /usr/bin/easy_install-3.1. These obviously can be used as-is, as they won't conflict.

For other executables, the general rule is:

  • if only one executable is to be shipped, then it owns its own slot
  • if executables are to be shipped for both python 2 and python 3 :
    • Both python 2 and python 3 variants must provide symlinks with a '-X' and '-X.Y' suffix (python runtime major version, or python runtime major.minor version), unless upstream already provides appropriately versioned executables without the dash.
    • The unversioned executable must be the python2 version.

References

Policy in other distributions