[Rpmlint-discuss] PATCH: verify timestamps of packaged python bytecode files
dmalcolm at redhat.com
Fri Oct 23 22:43:24 CEST 2009
Python's runtime compiles .py source files to bytecode "code objects",
and saves a cache of this data as .pyc and .pyo files on the filesystem.
It looks for .pyc files before parsing a .py file, using the .pyc file
if it can, to avoid having to parse the file.
Upstream rpm.org has an rpmbuild script brp-python-bytecompile, which we
use in Fedora and derived distributions to ship pre-parsed bytecode
files for .py files in package payloads. These are typically
non-writable by non-root.
The python runtime does some checking of a .pyc file before it will use
it as a cached .py file: it requires that a 4-byte value stored in the
header of the pyc file equals the mtime of the .py file; similarly it
checks a 4-byte "magic" ABI value which has to match that of the python
(this is implemented in Python/import.c:check_compiled_module; see also
the Lib/importlib/_bootstrap.py implementation within Python 3)
If the values in the header aren't correct, the .pyc file is treated as
stale, and the .py file is parsed, and an updated .pyc file will be
written out, if Python can.
If this happens for a packaged python module, the .pyc files aren't
writable by non-root, and so python will typically fail to write out an
updated cache (if running as non-root), and will constantly have to
reparse the code, without any visible indication apart from a big loss
It _is_ possible for this to go wrong; for example I managed to mess up
in my attempts to create a python 3 rpm for Fedora (see
Attached is a patch against svn trunk that adds testing for .pyo/.pyc
files to rpmlint. For any .pyo/.pyc file it verifies that the mtime
stored in the header corresponds to the mtime of the .py file.
For example, on my broken rpms it now emits messages like this:
1256322858 /usr/lib/python3.1/html/__init__.py 1256322906
python-bytecode-inconsistent-mtime /usr/lib/python3.1/distutils/version.pyo 1256322866 /usr/lib/python3.1/distutils/version.py 1256322905
The patch also makes rpmlint verify that the .py file is present. I
belive this is an implicit policy assumption for Fedora/RHEL rpms; is it
the case for all other mainstream users of rpm?
(I also extract the magic ABI value for each .pyc file as ultimately I
want to also add checking for this; I hope to support multiple parallel
installs of python with differing ABI via RPM, and I want to make sure
that we get this right in our RPMs, so it will help if rpmlint can check
for this; see https://fedoraproject.org/wiki/Features/Python3F13)
(it's still unclear to me how to login to the Trac instance's ticketing
system, hence sending this via email; sorry!)
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 3122 bytes
Desc: not available
More information about the Rpmlint-discuss