INTPYTHON-527 Add queryable encryption support by aclark4life · Pull Request #319 · mongodb/django-mongodb-backend

aclark4life · 2025-06-19T13:43:31Z

Completed most setup in qe.py
Completed some setup in Django
- Refactored get_auto_encryption_options to support ephemeral local KMS and allow crypt_shared_lib_path to be passed in
- Add check for presence of libmongocrypt to supports_queryable_encryption feature check
Started thinking about models, fields, migrations and planning an issubclassof check for EncryptedModel to call create_encrypted_collection instead of create_collection but need to pass encrypted_client to ClientEncryption somewhere in Django.

TODO

Decide on a configurable way to support local KMS so we can add other KMS later using the same configuration framework.
Decide on how to configure crypt_shared_lib_path. I don't like setting an environment variable for this configuration task so I settled on passing to get_auto_encryption_options in the short term but maybe a Django setting or something else in long term.
Decide on EncryptedModel and EncryptedField (not EncryptedModelField or EncryptedEmbeddedModelField yet!) implementation to support this:

from django_mongodb_backend.models import EncryptedModel
from django_mongodb_backend.models import EncryptedField
from django import models

class Person(EncryptedModel):
  ssn = EncryptedField(models.CharField)

Question

Lastly, I'm not sure why PyMongo is trying to do explicit here or at least that is what it appears to be doing despite the auto config:

% j q
python qe.py
Traceback (most recent call last):
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 124, in _wrap_encryption_errors
    yield
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 861, in create_data_key
    self._encryption.create_data_key(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        kms_provider,
        ^^^^^^^^^^^^^
    ...<2 lines>...
        key_material=key_material,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    ),
    ^
  File "/Users/alexclark/Developer/django-mongodb-cli/.venv/lib/python3.13/site-packages/pymongocrypt/synchronous/explicit_encrypter.py", line 66, in create_data_key
    with self.mongocrypt.data_key_context(kms_provider, opts) as ctx:
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexclark/Developer/django-mongodb-cli/.venv/lib/python3.13/site-packages/pymongocrypt/mongocrypt.py", line 284, in data_key_context
    return DataKeyContext(
        self._create_context(),
    ...<3 lines>...
        self.__callback,
    )
  File "/Users/alexclark/Developer/django-mongodb-cli/.venv/lib/python3.13/site-packages/pymongocrypt/mongocrypt.py", line 560, in __init__
    raise ValueError(f"unknown kms_provider: {kms_provider}")
ValueError: unknown kms_provider: None

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 762, in create_encrypted_collection
    encrypted_fields["fields"][i]["keyId"] = self.create_data_key(
                                             ~~~~~~~~~~~~~~~~~~~~^
        kms_provider=kms_provider,  # type:ignore[arg-type]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        master_key=master_key,
        ^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 858, in create_data_key
    with _wrap_encryption_errors():
         ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/alexclark/.pyenv/versions/3.13.3/lib/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 130, in _wrap_encryption_errors
    raise EncryptionError(exc) from exc
pymongo.errors.EncryptionError: unknown kms_provider: None

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/alexclark/Developer/django-mongodb-cli/qe.py", line 34, in <module>
    encrypted_collection = client_encryption.create_encrypted_collection(
        encrypted_database, "encrypted_collection", encrypted_fields
    )
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 767, in create_encrypted_collection
    raise EncryptedCollectionError(exc, encrypted_fields) from exc
pymongo.errors.EncryptedCollectionError: unknown kms_provider: None

Seems like I may have missed passing kms_providers somewhere.

I ruled out crypted_shared_lib_path being the cause. If you configure an invalid /path/to/libmongocrypt.so you get:

pymongocrypt.errors.MongoCryptError: A crypt_shared override path was specified [/path/to/libmongocrypt.so], but we failed to open a dynamic library at that location. Load error: [Error while opening candidate for crypt_shared dynamic library [/path/to/libmongocrypt.so]: dlopen(/path/to/libmongocrypt.so, 0x0005): tried: '/path/to/libmongocrypt.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/path/to/libmongocrypt.so' (no such file), '/path/to/libmongocrypt.so' (no such file)]

So… yeah.

timgraham · 2025-06-19T14:38:07Z

            parse_uri("cluster0.example.mongodb.net")
+
+
+# TODO: This can be moved to `test_features` once transaction support is merged.


test_features.py would only be for testing the logic of DatabaseFeatures.supports_queryable_encryption.

timgraham · 2025-06-19T14:41:25Z

+        try:
+            import pymongocrypt  # noqa: F401
+
+            has_pymongocrypt = True
+        except ImportError:
+            has_pymongocrypt = False


It's inappropriate to check for this package here. Instead the ImportError should be surfaced to the user if they try to use a feature that requires it. I can imagine a couple of ways to do so, but I think this concern should be deferred until the implementation is ironed out.

I'll move it to get_auto_encryption_options

timgraham · 2025-06-19T14:49:42Z

+def get_auto_encryption_options(crypt_shared_lib_path=None):
+    key_vault_database_name = "encryption"
+    key_vault_collection_name = "__keyVault"
+    key_vault_namespace = f"{key_vault_database_name}.{key_vault_collection_name}"
+    kms_providers = {"local": {"key": os.urandom(96)}}
+    return AutoEncryptionOpts(
+        key_vault_namespace=key_vault_namespace,
+        kms_providers=kms_providers,
+        crypt_shared_lib_path=crypt_shared_lib_path,
+    )


It's still obvious to me from the design doc to what extent it's appropriate for this library to provide helpers to generate AutoEncryptionOpts.

From the design doc

Take less steps than manual configuration of QE

We can expand that bullet into a section about configuration and it does not have to be a helper like get_auto_encryption_options but the tutorial takes this approach and I like it so far.

timgraham · 2025-06-19T14:54:18Z

+    )
+
+
+def parse_uri(uri, *, auto_encryption_options=None, db_name=None, test=None):


I'd rather than options=None than auto_encryption_options=None so that we're not blowing up the signature of parse_uri() with every kwarg of MongoClient. (If it's even in scope to continually expand parse_uri() rather than to guide more advanced users toward a dictionary DATABASES). I think all the changes in this file could be discussed/implemented in a follow up.

To enable this feature in PyMongo and start development in Django, an auto_encryption_options object has to be passed to MongoClient in base.py. As someone who opposed db_name in parse_uri I can tell you this is not a casual addition and this implementation is based on your suggestion to merge the options. We could rename, but I'm not convinced yet. I'll shorten it to auto_encryption_opts to match PyMongo for now.

I'm suggesting:

def parse_uri(uri, *, ... options=None)`: ... if options: options = {**uri.get("options"), options}

Usage:

parse_uri(uri, options={"auto_encryption_options": ...})

Otherwise, we're headed down the road of adding a new keyword to parse_uri() for each MongoClient keyword that the user wants to customize.

Ah! Got it, thanks

timgraham · 2025-06-19T14:59:17Z

+
+# TODO: This can be moved to `test_features` once transaction support is merged.
+class ParseUriOptionsTests(TestCase):
+    @skipUnlessDBFeature("supports_queryable_encryption")


Probably we just make installing pymongo[encryption] a required part of running our test suite. In that case, this test wouldn't need a skip since it doesn't interact with the server.

Not sure, but as you said above we shouldn't check imports in features and so this check is for enterprise_or_atlas and we'd skip it on anything that is not that (similar to transaction support checks) and that has nothing to do with encryption libraries.

aclark4life added 6 commits June 18, 2025 07:46

INTPYTHON-527 Add test

a58edbc

Update test

09fd8ec

Update test

ea923dc

Update test

eff8f8b

"Fix" kms provider

7d5e39b

Add encrypted model stubs

ef49313

timgraham reviewed Jun 19, 2025

View reviewed changes

aclark4life closed this Jun 23, 2025

This was referenced Jun 23, 2025

INTPYTHON-527 Add Queryable Encryption support #323

Closed

INTPYTHON-527 Add Queryable Encryption support #329

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

INTPYTHON-527 Add queryable encryption support#319

INTPYTHON-527 Add queryable encryption support#319
aclark4life wants to merge 6 commits intomongodb:mainfrom
aclark4life:INTPYTHON-527

aclark4life commented Jun 19, 2025 •

edited

Loading

Uh oh!

timgraham Jun 19, 2025

Uh oh!

aclark4life Jun 23, 2025

Uh oh!

timgraham Jun 19, 2025

Uh oh!

aclark4life Jun 23, 2025

Uh oh!

timgraham Jun 19, 2025

Uh oh!

aclark4life Jun 23, 2025

Uh oh!

timgraham Jun 19, 2025

Uh oh!

aclark4life Jun 23, 2025 •

edited

Loading

Uh oh!

timgraham Jun 23, 2025

Uh oh!

aclark4life Jun 23, 2025

Uh oh!

timgraham Jun 19, 2025

Uh oh!

aclark4life Jun 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		parse_uri("cluster0.example.mongodb.net")


		# TODO: This can be moved to `test_features` once transaction support is merged.

		)


		def parse_uri(uri, *, auto_encryption_options=None, db_name=None, test=None):

Conversation

aclark4life commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Question

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aclark4life Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aclark4life commented Jun 19, 2025 •

edited

Loading

aclark4life Jun 23, 2025 •

edited

Loading