Skip to content

Instantly share code, notes, and snippets.

@rrei
rrei / problem.py
Created July 29, 2020 14:26
Basic formulation of the problem
def do_stuff(obj):
"""Performs some simple processing on `obj`."""
obj.y = obj.related_obj.x ** 2 + 1 # DB query to fetch `.related_obj`
obj.save() # DB query to update `obj`
# Obtain a large queryset for this example.
qs = MyFurstModel.objects.all()
# Simply iterate over the queryset and do some processing on each element.
@rrei
rrei / example.py
Last active April 12, 2025 18:33
Different attempts at solving memory usage problems caused by large Django querysets
def do_stuff(obj):
"""Performs some simple processing on `obj`."""
obj.y = obj.related_obj.x ** 2 + 1 # DB query to fetch `.related_obj`
obj.save() # DB query to update `obj`
# Obtain a large queryset for this example.
qs = MyFurstModel.objects.all()
# Attempt no. 1: NO GOOD. This direct/naive approach has the advantage of describing
@rrei
rrei / chunked_qs.py
Created July 29, 2020 08:40
Break a large Django queryset into an equivalent set of smaller querysets (caveats apply)
def chunked_queryset(queryset, chunk_size=10000):
"""Slice a queryset into chunks. This is useful to avoid memory issues when
iterating through large querysets.
Code adapted from https://djangosnippets.org/snippets/10599/
"""
if not queryset.exists():
return
queryset = queryset.order_by("pk")
pks = queryset.values_list("pk", flat=True)
@rrei
rrei / settings.py
Created December 26, 2018 11:41
Django settings for reverse-proxy scenarios
USE_X_FORWARDED_HOST = True
USE_X_FORWARDED_PORT = True
SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
@rrei
rrei / proxy-headers.conf
Last active April 12, 2025 18:35
Nginx proxy headers
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # allows django to get the client's IP address (see the ipware module)
proxy_set_header X-Forwarded-Proto $scheme; # allows django to determine that the request is secure (see SECURE_PROXY_SSL_HEADER and request.is_secure())
proxy_set_header X-Forwarded-Host $host; # allows django to determine the name/addr of the server that the client originally connected to (see request.get_host())
proxy_set_header X-Forwarded-Port $server_port; # same idea as above, but for port number (see request.get_port())
/path/to/logs/*.log {
# Effective uid/gid used when running this pattern
# NOTE: this **is not** the same as the owner user and group of the rotated files
su root root
# Rotate once every day
daily
# Keep a history of 5 rotations
rotate 5
# Rotate even if the log file is empty
ifempty