Skip to content

Resolve AttributeError: 'ScalaFunction1' object has no attribute 'hashCode'. #184

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
poolis wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Resolve AttributeError: 'ScalaFunction1' object has no attribute 'hashCode'. #184

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c79da8c
Resolve AttributeError: 'ScalaFunction1' object has no attribute 'has…
poolis Jan 3, 2024
c01b232
Address CI issue with return types.
poolis Jan 23, 2024
d417a24
Use Integer.hashCode.
poolis Jan 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions pydeequ/scala_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ def apply(self, arg):
"""Implements the apply function"""
return self.lambda_function(arg)

Copy link
Copy Markdown

@github-actions github-actions Bot May 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EDGE_CASE: Using hash() on a lambda function returns a value based on the lambda's identity (memory address), not its logical content. Two identical lambda expressions (e.g., lambda x: x > 10 defined twice) will produce different hash values, making hashCode() non-deterministic across equivalent functions. More critically, hash() in Python can return negative values or values outside Java's int range, and Integer.hashCode(int) on the JVM expects a Java int. If hash(self.lambda_function) returns a Python int larger than Integer.MAX_VALUE or smaller than Integer.MIN_VALUE, this could cause unexpected behavior or overflow when passed through py4j.

Line 39: return self.gateway.jvm.java.lang.Integer.hashCode(hash(self.lambda_function)). Python's hash() returns a 64-bit integer on 64-bit platforms, but java.lang.Integer.hashCode(int) expects a 32-bit int. The test at test_scala_utils.py line 28 (self.assertNotEqual(greaterThan10.hashCode(), notNoneTest.hashCode())) passes only because the two lambdas happen to have different object identities, not because of logical equivalence checking.

def hashCode(self):
return self.gateway.jvm.java.lang.Integer.hashCode(hash(self.lambda_function))

class Java:
"""scala.Function1: a function that takes one argument"""

Expand All @@ -55,6 +58,9 @@ def apply(self, t1, t2):
"""Implements the apply function"""
return self.lambda_function(t1, t2)

def hashCode(self):
return self.gateway.jvm.java.lang.Integer.hashCode(hash(self.lambda_function))

class Java:
"""scala.Function2: a function that takes two arguments"""

Expand Down
23 changes: 22 additions & 1 deletion tests/test_checks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1366,4 +1366,25 @@ def test_fail_list_of_constraints(self):
.run()

df = VerificationResult.checkResultsAsDataFrame(self.spark, result)
self.assertEqual(df.select("constraint_status").collect(), [Row(constraint_status="Success"), Row(constraint_status="Success")])
self.assertEqual(df.select("constraint_status").collect(), [Row(constraint_status="Success"), Row(constraint_status="Success")])

def test_hash_code(self):
Copy link
Copy Markdown

@github-actions github-actions Bot May 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DESIGN: The test repeatedly calls vrb.addCheck(check) with the same check object after mutating it in-place. This adds the same Check instance multiple times to the VerificationRunBuilder, which means the JVM receives the same underlying _Check Java object multiple times. This doesn't properly test the scenario described in issue #91 where multiple different checks with different ScalaFunction1 instances trigger the hashCode call. The test is fragile and may not reliably reproduce the original bug.

Lines 1378-1389: check.isComplete('b') mutates check._Check in place, then vrb.addCheck(check) adds it. Then check.containsEmail('email') mutates the same object again, and vrb.addCheck(check) adds the same Python object (now pointing to a different _Check Java object). The existing feedback confirms CI is failing on this test.

"""
Lack of Exception is passing. Previously this test would fail with:
AttributeError: 'ScalaFunction1' object has no attribute 'hashCode'
"""
vrb = VerificationSuite(self.spark) \
.onData(self.df)
check = Check(self.spark, CheckLevel.Error, "Enough checks to trigger a hashCode not an attribute of ScalaFunction1")
check.isComplete('b')
vrb.addCheck(check)
check.containsEmail('email')
vrb.addCheck(check)
check.isGreaterThanOrEqualTo("d", "b")
vrb.addCheck(check)
check.isLessThanOrEqualTo("b", "d")
vrb.addCheck(check)
check.hasDataType("d", ConstrainableDataTypes.String, lambda x: x >= 1)
vrb.addCheck(check)
Copy link
Copy Markdown
Contributor

@chenliu0831 chenliu0831 Jan 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to add and verify one by one?

Copy link
Copy Markdown
Contributor Author

@poolis poolis Jan 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because that is the use case for triggering the exception. If I change the test like so, it does not use hashCode.

        vrb = VerificationSuite(self.spark) \
            .onData(self.df)
        check = Check(self.spark, CheckLevel.Error, "Enough checks to trigger a hashCode not an attribute of ScalaFunction1")
        check.addConstraints([
            check.isComplete('b'),
            check.containsEmail('email'),
            check.isGreaterThanOrEqualTo("d", "b"),
            check.isLessThanOrEqualTo("b", "d"),
            check.hasDataType("d", ConstrainableDataTypes.String, lambda x: x >= 1)])

         result = vrb.addCheck(check).run()

Copy link
Copy Markdown
Contributor

@chenliu0831 chenliu0831 Jan 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it only fail at the magic 5th one? It's a big strange if so.. btw CI is failing on this test

Copy link
Copy Markdown
Contributor Author

@poolis poolis Jan 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue write up has more info: #91

Copy link
Copy Markdown
Contributor Author

@poolis poolis Jan 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't replicate the error in CI but submitted an attempt to fix it.


result = vrb.run()
Copy link
Copy Markdown

@github-actions github-actions Bot May 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MISSING_TEST: The test never asserts anything about the result of vrb.run(). It only relies on the absence of an exception as the pass condition, but doesn't verify that the verification actually completed successfully (e.g., checking result.status or the check results). If run() silently fails or returns an error status, this test would still pass.

Line 1390: result = vrb.run() — the variable result is assigned but never used in any assertion. The docstring says 'Lack of Exception is passing' but a proper test should also assert the result is valid.

13 changes: 12 additions & 1 deletion tests/test_scala_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,24 @@ def test_scala_function1(self):
self.assertFalse(notNoneTest.apply(None))
self.assertTrue(notNoneTest.apply("foo"))

# Test hashCode()
self.assertNotEqual(greaterThan10.hashCode(), notNoneTest.hashCode())
self.assertTrue(isinstance(greaterThan10.hashCode(), int))

appendTest = ScalaFunction1(self.sc._gateway, "{}test".format)
self.assertEqual("xtest", appendTest.apply("x"))

def test_scala_function2(self):
concatFunction = ScalaFunction2(self.sc._gateway, lambda x, y: x + y)
lambda_func = lambda x, y: x + y
concatFunction = ScalaFunction2(self.sc._gateway, lambda_func)
self.assertEqual("ab", concatFunction.apply("a", "b"))

anotherConcatFunction = ScalaFunction2(self.sc._gateway, lambda_func)

# Test hashCode()
self.assertEqual(concatFunction.hashCode(), anotherConcatFunction.hashCode())
self.assertTrue(isinstance(concatFunction.hashCode(), int))


if __name__ == "__main__":
unittest.main()