tests: detect stale Cython .so files at test startup by mykaul · Pull Request #756 · scylladb/python-driver

mykaul · 2026-03-19T12:57:43Z

Add a pytest_configure hook in tests/conftest.py that compares mtime of each compiled .so against its .py source and warns when the source is newer. This prevents silently testing stale compiled code after editing a Cython-compiled module without rebuilding.
(Bit me here - #749 )

Also document the rebuild requirement in CONTRIBUTING.rst.

Pre-review checklist

I have split my patch into logically separate commits.
All commit messages clearly explain what they change and why.
I added relevant tests for new features and bug fixes.
All commits compile, pass static checks and pass test.
PR description sums up the changes and reasons why they should be introduced.
I have provided docstrings for the public items that I want to introduce.
I have adjusted the documentation in ./docs/source/.
I added appropriate Fixes: annotations to PR description.

Copilot

Pull request overview

Adds early test-session detection of stale in-place Cython extension builds (to avoid running tests against outdated compiled code) and documents the rebuild requirement for contributors.

Changes:

Add pytest_configure hook in tests/conftest.py to warn when a compiled extension is older than its corresponding cassandra/*.py source.
Document the required python setup.py build_ext --inplace rebuild step in CONTRIBUTING.rst and note that tests will warn when staleness is detected.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
`tests/conftest.py`	New pytest startup hook that scans for stale compiled extensions vs `.py` sources and emits a warning.
`CONTRIBUTING.rst`	Adds contributor guidance about rebuilding extensions to avoid testing stale compiled code.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

CONTRIBUTING.rst

tests/conftest.py

Add a pytest_configure hook in tests/conftest.py that compares mtime of each compiled .so against its .py source and warns when the source is newer. This prevents silently testing stale compiled code after editing a Cython-compiled module without rebuilding. Also document the rebuild requirement in CONTRIBUTING.rst.

Copilot

Pull request overview

Adds a pytest startup check to help developers avoid accidentally running tests against stale Cython-compiled extensions when the corresponding cassandra/*.py sources have been edited without rebuilding, and documents the rebuild requirement for contributors.

Changes:

Introduces a pytest_configure hook that scans for Cython extension modules and warns when a compiled extension appears older than its .py source.
Documents the need to rebuild extensions after editing Cython-compiled cassandra/ modules, noting that the test suite will warn on staleness.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
`tests/conftest.py`	Adds a pytest session-start hook that attempts to detect stale Cython extension artifacts and emits a warning.
`CONTRIBUTING.rst`	Adds contributor guidance about rebuilding extensions after modifying Cython-compiled driver modules.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-20T12:43:05Z

tests/conftest.py

+    seen = set()
+    stale = []
+    # Use the current interpreter's extension suffixes so we only check
+    # extensions that would actually be loaded (correct ABI tag), and
+    # handle both .so (POSIX) and .pyd (Windows) automatically.
+    for suffix in importlib.machinery.EXTENSION_SUFFIXES:
+        for ext_path in glob.glob(os.path.join(_CASSANDRA_DIR, f"*{suffix}")):
+            module_name = os.path.basename(ext_path).split(".")[0]
+            if module_name in seen:
+                continue
+            py_path = os.path.join(_CASSANDRA_DIR, module_name + ".py")
+            if os.path.exists(py_path) and os.path.getmtime(py_path) > os.path.getmtime(
+                ext_path
+            ):
+                seen.add(module_name)
+                stale.append((module_name, ext_path, py_path))


Using glob('*{suffix}') while iterating importlib.machinery.EXTENSION_SUFFIXES can yield false positives because the generic suffixes (e.g. .so/.pyd) will match ABI-tagged extension files that the import system would not load for that suffix (and can also match extensions built for other Python versions, like query.cpython-310-...so). This may warn that tests will run stale compiled code even when the current interpreter would import the .py (or a different, newer extension).

Consider iterating over .py sources and, per module, selecting the first exact existing module_name + suffix in EXTENSION_SUFFIXES order (i.e., the one Python would actually import) and only comparing that file’s mtime to the .py mtime; also mark the module as processed regardless of staleness to avoid redundant checks.

Suggested change

seen = set()

stale = []

# Use the current interpreter's extension suffixes so we only check

# extensions that would actually be loaded (correct ABI tag), and

# handle both .so (POSIX) and .pyd (Windows) automatically.

for suffix in importlib.machinery.EXTENSION_SUFFIXES:

for ext_path in glob.glob(os.path.join(_CASSANDRA_DIR, f"*{suffix}")):

module_name = os.path.basename(ext_path).split(".")[0]

if module_name in seen:

continue

py_path = os.path.join(_CASSANDRA_DIR, module_name + ".py")

if os.path.exists(py_path) and os.path.getmtime(py_path) > os.path.getmtime(

ext_path

):

seen.add(module_name)

stale.append((module_name, ext_path, py_path))

stale = []

# Iterate over .py sources and, for each module, look for the first

# existing compiled extension in EXTENSION_SUFFIXES order. This mirrors

# how Python's import machinery selects an extension module, and avoids

# globbing patterns like "*{suffix}" that can pick up ABI-tagged

# extensions built for other Python versions.

if os.path.isdir(_CASSANDRA_DIR):

for entry in os.listdir(_CASSANDRA_DIR):

if not entry.endswith(".py"):

continue

module_name, _ = os.path.splitext(entry)

py_path = os.path.join(_CASSANDRA_DIR, entry)

# For this module, find the first extension file Python would load.

for suffix in importlib.machinery.EXTENSION_SUFFIXES:

ext_path = os.path.join(_CASSANDRA_DIR, module_name + suffix)

if not os.path.exists(ext_path):

continue

if os.path.getmtime(py_path) > os.path.getmtime(ext_path):

stale.append((module_name, ext_path, py_path))

# Only consider the first matching suffix; this is the one

# the import system would actually use.

break

@mykaul , this one is valid, could you please address it.

…sions Rewrite the stale-extension scan to iterate over .py source files and check for the first matching compiled extension per EXTENSION_SUFFIXES order. This mirrors Python's import machinery and avoids false positives from ABI-tagged extensions built for other Python versions that the glob approach could pick up. Also removes the now-unused glob import.

Lorak-mmk · 2026-03-31T18:38:45Z

CONTRIBUTING.rst

+**Important:** After modifying any ``.py`` file under ``cassandra/`` that is
+Cython-compiled (such as ``query.py``, ``protocol.py``, ``cluster.py``, etc.),
+you **must** rebuild extensions before running tests::
+
+    python setup.py build_ext --inplace
+
+Without rebuilding, Python will load the stale compiled extension (``.so`` / ``.pyd``)
+instead of your modified ``.py`` source, and your changes will not actually be tested.
+The test suite will emit a warning if it detects this situation.
+


No no no, calling setup.py directly is deprecated for many years now and shoud never be done.
The tool to work with the driver is uv, and it is the one that should be used in such guides.

How did you even encounter this problem? In pyproject.toml section we have this section:

[tool.uv] cache-keys = [ { file = "pyproject.toml" }, { file = "setup.py" }, # Cythonized modules { file = "cassandra/io/libevwrapper.c" }, { file = "cassandra/cmurmur3.c" }, { file = "cassandra/cluster.py" }, { file = "cassandra/concurrent.py" }, { file = "cassandra/connection.py" }, { file = "cassandra/cqltypes.py" }, { file = "cassandra/metadata.py" }, { file = "cassandra/pool.py" }, { file = "cassandra/protocol.py" }, { file = "cassandra/query.py" }, { file = "cassandra/util.py" }, { file = "cassandra/shard_info.py" }, { file = "cassandra/*.pyx" }, { file = "cassandra/*.pxd" }, # Env variables used in setup.py { env = "CASS_DRIVER_LIBEV_INCLUDES" }, { env = "CASS_DRIVER_LIBEV_LIBS" }, { env = "CASS_DRIVER_NO_EXTENSIONS" }, { env = "CASS_DRIVER_NO_LIBEV" }, { env = "CASS_DRIVER_NO_CYTHON" }, { env = "CASS_DRIVER_BUILD_CONCURRENCY" }, { env = "CASS_DRIVER_BUILD_EXTENSIONS_ARE_MUST" }, # used by setuptools_scm { git = { commit = true, tags = true } }, ]

It already achieves what this PR is trying to achieve, assuming you use uv (which you should).

https://python-driver.docs.scylladb.com/stable/installation.html#speeding-up-installation for example...

We should remove those, not add new ones.

Docs fixed in 2 additional commits. I hope it's OK.

….rst Address review feedback: replace deprecated 'python setup.py build_ext --inplace' instructions with 'uv sync --reinstall-package scylla-driver' and note that 'uv run pytest' handles rebuilds automatically via cache-keys in pyproject.toml.

Replace all 'python setup.py install' instructions with 'pip install .' or 'pip install scylla-driver' equivalents. Replace setup.py-specific command-line flags (--no-cython, --no-extensions, etc.) with their environment variable equivalents (CASS_DRIVER_NO_CYTHON, CASS_DRIVER_NO_EXTENSIONS, CASS_DRIVER_NO_LIBEV). Remove deprecated pip --install-option usage.

mykaul requested a review from Copilot March 19, 2026 17:13

Copilot started reviewing on behalf of mykaul March 19, 2026 17:13 View session

Copilot AI reviewed Mar 19, 2026

View reviewed changes

CONTRIBUTING.rst Outdated Show resolved Hide resolved

tests/conftest.py Outdated Show resolved Hide resolved

tests/conftest.py Outdated Show resolved Hide resolved

Lorak-mmk self-requested a review March 19, 2026 17:27

mykaul force-pushed the fix/detect-stale-cython-extensions branch from 91aeef3 to 668275b Compare March 20, 2026 10:46

mykaul requested a review from Copilot March 20, 2026 12:38

Copilot started reviewing on behalf of mykaul March 20, 2026 12:38 View session

Copilot AI reviewed Mar 20, 2026

View reviewed changes

dkropachev approved these changes Mar 31, 2026

View reviewed changes

Lorak-mmk requested changes Mar 31, 2026

View reviewed changes

mykaul added 2 commits March 31, 2026 23:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests: detect stale Cython .so files at test startup#756

tests: detect stale Cython .so files at test startup#756
mykaul wants to merge 4 commits intoscylladb:masterfrom
mykaul:fix/detect-stale-cython-extensions

mykaul commented Mar 19, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 20, 2026

Uh oh!

dkropachev Mar 31, 2026

Uh oh!

Lorak-mmk Mar 31, 2026

Uh oh!

mykaul Mar 31, 2026

Uh oh!

Lorak-mmk Mar 31, 2026

Uh oh!

mykaul Apr 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

mykaul commented Mar 19, 2026

Pre-review checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 20, 2026

Choose a reason for hiding this comment

Uh oh!

dkropachev Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Lorak-mmk Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

mykaul Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Lorak-mmk Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

mykaul Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants