INTPYTHON-527 Add Queryable Encryption support #329

aclark4life · 2025-06-27T02:16:28Z

Previous attempts and additional context here:

INTPYTHON-527 Add Queryable Encryption config #318
- Branched from transaction support and includes only helpers
INTPYTHON-527 Add queryable encryption support #319
- In which this bug in libmongocrypt was discovered
- A fix for that bug in PyMongo is being discussed here.
INTPYTHON-527 Add Queryable Encryption support #323
- Before I understood that the shared lib for mongocrypt was included in the pymongocrypt wheel.

tests/encryption_/tests.py

django_mongodb_backend/models.py

django_mongodb_backend/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/base.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/features.py

django_mongodb_backend/routers.py

django_mongodb_backend/schema.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

tests/encryption_/tests.py

tests/encryption_/routers.py

tests/encryption_/models.py

tests/backend_/test_features.py

django_mongodb_backend/schema.py

tests/encryption_/models.py

django_mongodb_backend/schema.py

django_mongodb_backend/encryption.py

tests/encryption_/routers.py

django_mongodb_backend/schema.py

django_mongodb_backend/models.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

docs/source/ref/encryption.rst

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

django_mongodb_backend/schema.py

tests/encryption_/tests.py

timgraham · 2025-09-04T01:05:24Z

django_mongodb_backend/schema.py

+        field_list = []
+        for field in fields:
+            if getattr(field, "encrypted", False):
+                key_alt_name = f"{db_table}_{field.column}"


Is this sufficient to avoid collisions? Unlikely as it may be:

db_table = "foo" column = "bar_id" db_table = "foo_bar" column = "id"

Well now that you mention it, how about a double separator ? {db_table}__{field.column} would pass that test with these key alt names:

foo__bar_id

foo_bar__id

A period may be a more typical separator, including to indicate nested fields.

The spec says: "encryptedFieldsMap maps a collection namespace to an encryptedFields." In this commit, we clarify the distinction between encrypted fields map and encrypted fields.

- Test Python tutorial models (still unencrypted in this commit) - Test all supported encrypted fields. - Test equality and range query type queries - Test management command - Test schema - Removed test router - Removed test_base

- EncryptedEmbeddedModel for encrypted objects - EmbeddedModel for models with encrypted fields

timgraham · 2025-09-19T20:44:25Z

django_mongodb_backend/schema.py

        db_table = model._meta.db_table
        field_list = []
        for field in fields:
+            if isinstance(field, EmbeddedModelField):


How about PolymorphicEmbeddedFields or EmbeddedModelArrayFIeld? The design doc still needs a list of all encrypted fields that will be implemented, I think.

Those could be added because I think the existing embedded models are doing all the work.

timgraham · 2025-09-19T20:45:42Z

django_mongodb_backend/models.py

+class EncryptedEmbeddedModel(EmbeddedModel):
+    encrypted = True
+
+    class Meta:
+        abstract = True
+        required_db_features = {"supports_queryable_encryption"}


EncryptedEmbeddedModel isn't present in the design doc. Did you thinking change? I don't see what purpose it serves.

Without it, I get this:

django.core.exceptions.ImproperlyConfigured: No kms_provider found in database router.

EncryptedEmbeddedModel isn't present in the design doc. Did you thinking change? I don't see what purpose it serves.

And yes the design doc needs to be updated.

Without it, I get this:

django.core.exceptions.ImproperlyConfigured: No kms_provider found in database router.

The encryption_ tests pass for me without it. We shouldn't be trying to resolve a kms_provider for an embedded model anyway.

timgraham · 2025-09-19T20:46:30Z

tests/encryption_/models.py

    patient_name = models.CharField(max_length=255)
    patient_id = models.BigIntegerField()
-    patient_record = EmbeddedModelField(PatientRecord)
+    patient_record = EncryptedEmbeddedModelField(PatientRecord)


Per design doc, this is supposed to be EmbeddedModelField.

Good catch!

timgraham · 2025-09-19T20:47:40Z

django_mongodb_backend/models.py

+class EncryptedEmbeddedModel(EmbeddedModel):
+    encrypted = True
+
+    class Meta:
+        abstract = True
+        required_db_features = {"supports_queryable_encryption"}


Embedded models aren't created anyway, so required_db_features = {"supports_queryable_encryption"} would serve no purpose.

This seems to facilitate use of the skip decorator.

The purpose of required_db_features is to prevent the model from being created if the database doesn't support it. Since SchemaEditor already ignores embedded models, required_db_features serves no purpose here.

timgraham · 2025-09-19T20:49:49Z

django_mongodb_backend/utils.py

+        if getattr(field, "encrypted", False):
+            return True
+
+    return bool(issubclass(model, EncryptedEmbeddedModel))


If the model has encrypted fields, the loop above would already have returned True. This would only be executed if the model doesn't have any encrypted fields. :-D

This is for the case when an EncryptedEmbeddedModel has no encrypted fields but itself is encrypted e.g. Billing.

The code that calls model_has_encrypted_fields() shouldn't receive embedded models. Embedded models don't have their own collection, so they aren't going to be passed to SchemaEditor._create_collection() and neither are they returned by router.get_migratable_models() in showencryptedfieldsmap.py.

I don't see EncryptedEmbeddedModel serving any purpose. If I missed something, revert my commit.

aclark4life commented Jun 27, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/models.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/encryption.py Outdated Show resolved Hide resolved

timgraham reviewed Jun 27, 2025

View reviewed changes

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jun 30, 2025

View reviewed changes

django_mongodb_backend/features.py Outdated Show resolved Hide resolved

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/routers.py Outdated Show resolved Hide resolved

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jul 4, 2025

View reviewed changes

tests/encryption_/models.py Show resolved Hide resolved

timgraham reviewed Jul 7, 2025

View reviewed changes

tests/encryption_/models.py Outdated Show resolved Hide resolved

aclark4life commented Jul 7, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jul 9, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

aclark4life commented Jul 11, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

This comment was marked as outdated.

Sign in to view

aclark4life requested a review from Copilot August 19, 2025 23:18

This comment was marked as resolved.

Sign in to view

aclark4life force-pushed the INTPYTHON-527 branch 5 times, most recently from 26874aa to f32d62b Compare August 25, 2025 17:55

timgraham reviewed Sep 4, 2025

View reviewed changes

aclark4life force-pushed the INTPYTHON-527 branch 3 times, most recently from 1169c04 to d04bdbb Compare September 10, 2025 11:55

aclark4life force-pushed the INTPYTHON-527 branch 2 times, most recently from ccd2f36 to 94c1dc2 Compare September 17, 2025 21:25

aclark4life requested review from timgraham, a team and addaleax and removed request for addaleax September 19, 2025 19:58

aclark4life added 5 commits September 19, 2025 16:01

INTPYTHON-527 Add Queryable Encryption support

67542b8

encrypted fields map != encrypted fields

fd6bcab

The spec says: "encryptedFieldsMap maps a collection namespace to an encryptedFields." In this commit, we clarify the distinction between encrypted fields map and encrypted fields.

Use dot separator

8f6c988

Refactor tests

100a4d0

- Test Python tutorial models (still unencrypted in this commit) - Test all supported encrypted fields. - Test equality and range query type queries - Test management command - Test schema - Removed test router - Removed test_base

Add embedding

d48ebc9

- EncryptedEmbeddedModel for encrypted objects - EmbeddedModel for models with encrypted fields

aclark4life force-pushed the INTPYTHON-527 branch from e1077a1 to d48ebc9 Compare September 19, 2025 20:01

timgraham reviewed Sep 19, 2025

View reviewed changes

Review feedback

4992d1f

aclark4life requested a review from timgraham September 19, 2025 21:14

timgraham added 3 commits September 22, 2025 21:20

remove EncryptedEmbeddedModel

d6813bb

PatientRecord shouldn't be encrypted

4c30590

fix linting of kms_provider() docstring line length

16922e6

INTPYTHON-527 Add Queryable Encryption support #329

Are you sure you want to change the base?

INTPYTHON-527 Add Queryable Encryption support #329

Uh oh!

Conversation

aclark4life commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as outdated.

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

aclark4life commented Jun 27, 2025 •

edited

Loading