Big Picture

This document explains all of the concepts used in Django URL Filter in context hence providing a “big picture” of how it works.

Basics

In order to filter any data, this library breaks the process into 3 phases:

  1. Parse the URL querystring into LookupConfig
  2. Loop through all the configs and generate FilterSpec when possible
  3. Use the list of specs to actually filter data

And here is a bit more information about each phase.

Parsing

Fundamentally a querystring is a collection of key-pairs. As such, this data is naturally flat and is usually represented as a simple dictionary:

?foo=bar&happy=rainbows => {
    'foo': 'bar',
    'happy': 'rainbows',
}

Note

Technically, this is not 100% true since key can be repeated which is why Django uses QueryDict but for the purposes of this discussion, let’s assume no duplicate keys are present.

The filtering however is not flat. Each querystring key can be nested when using nested FilterSet and in addition it can optionally contain lookup. For example:

?foo=bar
?foo__contains=bar
?foo__nested=bar
?foo__nested__contains=bar
?foo__nested__othernested=bar
?foo__nested__othernested__contains=bar

In order to accomodate the nested structure of querystring keys, Django URL Filter parses all querystring key-value pairs into nested dictionaries. For example:

?foo__nested__othernested=bar => {
    'foo': {
        'nested': {
            'othernested': 'bar'
        }
    }
}
?foo__nested__othernested__contains=bar => {
    'foo': {
        'nested': {
            'othernested': {
                'contains': 'bar'
            }
        }
    }
}

That is essentially what LookupConfig stores. Since these dictionaries are flat (each dictionaty has at most one key), it also provides some utility properties for dealing with such data. You can refer to the LookupConfig API documentation for more information.

Filter Specification

As mentioned in README, Django URL Filter decouples parsing of querystring and filtering. It achieves that by constructing filter specifications which have all necessary information to filter data without actually filtering data. Thats what FilterSpec is. It stores 3 required pieces of information on how to filter data:

  • Which attribute to filter on. Since models can be related by attributes of related models, this actually ends up being a list of attributes which we call components.
  • Lookup to use to filter data. This specifies how the value should be compared while doing filtering. Example is exact, contains. By default only lookups from Django ORM are supported however custom CallableFilter can be used to define custom lookups.
  • If the filter is negated. For example, to filter when username is 'foo' or filter when username is not 'foo'.

Filtering

Since filtering is decoupled from the FilterSet, the filtering honors all go to a specified filter backend. The backend is very simple. It takes a list of filter specifications and a data to filter and its job is to filter that data as specified in the specifications.

Note

Currently we only support a handful of backends such as Django ORM, SQLAlchemy and plain Python interables filter backends but you can imagine that any backend can be implemented. Eventually filter backends can be added for more exotic sources like Mongo, Redis, etc.

Steps

Above information hopefully puts things in perspective and here is more detailed step-by-step guide what Django URL Filter does behind the scenes:

  1. FilterSet is instantiated with querystring data as well as queryset to filter.
  2. FilterSet is asked to filter given data via filter method which kicks in all the steps below.
  3. FilterSet finds all filters it is capable of Filtering via get_filters. This is where custom filtersets can hook into to do custom stuff like extracting filters from a Django model.
  4. FilterSet binds all child filters to itself via bind. This practically sets parent and name attributes.
  5. Root FilterSet loops through all querystring pairs and generates LookupConfig for all of them.
  6. Root FilterSet loops through all generated configs and attemps to find appropriate filter to use to generate a spec fo the given config. The matching happens by the first key in the LookupConfig dict. If that key is found in available filters, that filter is used and otherwise that config is skipped. This is among the reasons why LookupConfig is used since it allows this step to be very simple.
  7. If appropriate filter is found, it is passed nested config to the child filter which then goes through very similar process as in previous step until it gets to a leaf filter.
  8. Leaf Filter gets the config. In then checks if the config is still nested. For example if the config is simply a value (e.g. 'bar') or is still a dictionary (e.g. {'contains': 'bar'}). If the config is just a value, it then uses a default lookup for that filter as provided in default_lookup parameter when instantiating Filter. If the config is a dictionary, it makes sure that it is a valid lookup config (e.g. its not {'rainbows': {'contains': 'bar'}} since it would not know what to do with rainbows since it is not a valid lookup value).
  9. Now that Filter validated the lookup itself, it cleans the actual filter value by using either form_field as passed as parameter when instantiating Filter or by using lookup overwrite. Overwrites are necessary for more exotic lookups like in or year since they need to validate data in a different way.
  10. If the value is valid, then the leaf filter constructs a FilterSpec since it has all the necessary information to do that - 1) all filter component names from all ancestors (e.g. all attributes which should be accessed on the queryset to get to the value to be filtered on); 2) the actual filter value and 3) if the filter is negated.
  11. At this point, root FilterSet will get the FilterSpec as bubbled up from the leaf filter. If any ValidationError exceptions are raised, then depending on strict_mode, it will either ignore errors or will propagate them up to the caller of the filterset.
  12. Once all specs are collected from all the querystring key-value-pairs, root FilterSet instantiates a filter backend and passes it all the specs.
  13. Finally root FilterSet uses the filter backend to filter given queryset and returns the results to the user.

Some important things to note:

  • Root FilterSet does all the looping over querystring data and generated configurations.
  • Children filters of a root FilterSet are only responsible for generating FilterSpec and in the process of validating the data.