Django-Syzygy is a Django companion that makes database migrations safer in real production environments. It focuses on one specific, painful problem: migrations that are technically valid, but unsafe during rolling deployments.

Syzygy works with Django’s existing migration system and workflow. It doesn’t replace it, and it doesn’t require you to learn a new mental model. It quietly adds structure where Django is intentionally flexible, so deployments stop being a source of stress.

What Makes It Different?

Consider a common change: removing a database field that’s no longer needed.

In development, this is trivial. In production, during a rolling deploy, it can break running instances that still expect that column to exist. Django doesn’t protect you from that class of failure.

Syzygy does.

The natural workflow:

1. You change your model (for example, remove a field)

2. Run makemigrations as usual

3. Syzygy detects a potentially unsafe change and asks for minimal input

4. It automatically splits the change into safe deployment stages

You keep using Django’s commands. Syzygy handles the staging.

Why Django-Syzygy Exists

The Problem

Django migrations are excellent at describing what should change, but they don’t encode when a change is safe to apply relative to your application code.

A classic failure mode looks like this:

Without Syzygy:

• You remove a field from a model

• A migration drops the column

• You deploy new code

• Old application instances are still running

• Migration runs → database column disappears → runtime errors

This is not a rare edge case. It’s a structural issue with rolling deployments.

With Syzygy:

• The migration is automatically split into deployment-safe stages

• Changes that old code can tolerate run first

• Code is deployed

• Breaking changes run only after old code is gone

The result is not “perfect safety,” but the elimination of an entire category of deployment-time failures.

What Syzygy Actually Does

Syzygy integrates with Django’s migration autodetector and migration system:

• It analyzes generated migrations for deployment safety

• It assigns or infers a stage (pre-deploy or post-deploy)

• When necessary, it splits migrations automatically

• It enforces consistency via Django system checks

It does not replace Django’s migration engine or alter database behavior. All migrations are still standard Django migrations.

How It Works in Practice

Removing a Field

# models.py
class MyModel(models.Model):
    name = models.CharField(max_length=100)
    # old_field removed

python manage.py makemigrations

Syzygy detects that removing a field is unsafe if old code is still running, and asks for a temporary default value so existing rows remain compatible.

It generates two migrations:

# 0001_pre_deploy.py
operations = [
    syzygy.operations.AlterField(
        model_name='mymodel',
        name='old_field',
        field=models.CharField(default='safe_default'),
    ),
]

# 0002_remove_field.py
operations = [
    migrations.RemoveField(
        model_name='mymodel',
        name='old_field',
    ),
]

The first migration keeps old application instances working.
The second removes the field only after deployment.

Adding a Field Without a Default

class MyModel(models.Model):
    name = models.CharField(max_length=100)
    new_field = models.CharField(max_length=200)

Syzygy ensures the database can accept writes from old code:

# 0001_add_field.py (pre-deploy)
operations = [
    migrations.AddField(
        model_name='mymodel',
        name='new_field',
        field=models.CharField(
            max_length=200,
            db_default='temp_default',
        ),
    ),
]

# 0002_drop_db_default.py (post-deploy)
operations = [
    syzygy.operations.AlterField(
        model_name='mymodel',
        name='new_field',
        field=models.CharField(max_length=200),
        stage=syzygy.constants.Stage.POST_DEPLOY,
    ),
]

The database default exists only to bridge the deployment window. Syzygy removes it once the new code is fully live.

(PostgreSQL is the recommended database; support for other backends depends on their DDL capabilities.)

Pre-Deploy vs Post-Deploy

Syzygy classifies operations based on one rule:

Old code must survive pre-deploy changes.
New code must survive post-deploy changes.

Typical classification:

Pre-Deploy:

• Adding columns or tables

• Adding indexes

• Adding database defaults

• Creating new constraints

Post-Deploy:

• Removing or renaming fields

• Dropping constraints

• Destructive schema changes

If Syzygy can’t determine this safely, it fails early.

Installation & Setup

pip install django-syzygy

INSTALLED_APPS = [
    # ...
    'syzygy',
]

Optional configuration for edge cases:

from syzygy import Stage

MIGRATION_THIRD_PARTY_STAGES_FALLBACK = Stage.PRE_DEPLOY

MIGRATION_STAGES_OVERRIDE = {
    'problematic_app.0001_bad_migration': Stage.POST_DEPLOY,
}

Daily Workflow

Development

Nothing changes:

python manage.py makemigrations
python manage.py check
python manage.py migrate

Production Deployment

# Apply safe changes
python manage.py migrate --pre-deploy

# Deploy application code

# Apply remaining changes
python manage.py migrate

If you forget --pre-deploy, migrate still works. The flag enables phased deployments; it doesn’t break existing pipelines.

CI/CD Integration

The most valuable part of Syzygy in CI is simple:

Migration staging mistakes become CI failures instead of production incidents.

script:
  - python manage.py check
  - python manage.py makemigrations --dry-run --check

Syzygy’s system checks ensure migrations are internally consistent and safely staged.

When Syzygy Complains

“Cannot automatically determine stage”

This means a migration mixes safe and unsafe operations in a way that requires human intent.

You can:

• Split the migration

• Explicitly assign a stage

• Override it in settings

The important part: the failure happens early, not during deployment.

Best Practices

Syzygy doesn’t replace good judgment:

• Review generated migrations

• Test migrations in staging

• Prefer additive changes over destructive ones

• Document complex transitions

Syzygy enforces structure; it doesn’t remove responsibility.

Who This Is For

Django-Syzygy is most useful for:

• Teams running rolling or blue-green deployments

• Kubernetes/ ArgoCD / Docker environments

• Applications with real uptime requirements

It’s probably unnecessary for:

• Solo projects

• Development-only environments

• Apps that can afford downtime

Conclusion

Django-Syzygy makes deployment-unsafe migrations explicit and enforceable. It doesn’t promise perfection. It removes an entire class of avoidable production failures by encoding deployment reality directly into Django migrations.

You keep Django’s workflow.
You gain deployment safety.
And migration anxiety becomes a lot rarer.

That’s the whole point.

What is Apache Kafka?

Apache Kafka is a distributed streaming platform built for high-throughput, fault-tolerant messaging. Imagine a massive, continuous conveyor belt: producers place messages onto it, and consumers retrieve them, each at their own pace.

Core Concepts: Topics & Partitions

Topics: The Communication Channels

In Kafka, a topic is a named channel or stream where producers publish messages and consumers subscribe to read them. Think of a topic as:

• A database table: Each message is an immutable, append-only row.
• A persistent queue: Unlike traditional queues, Kafka topics are durable and replayable. Messages are not removed after being read, allowing multiple consumers to process the same data independently and at their own pace.

Examples: user-signups, payment-transactions, order-updates

Partitions: The Key to Parallelism

Each Kafka topic is divided into partitions, which are ordered, immutable sequences of messages. Partitions are fundamental to Kafka's scalability and parallelism.

graph TD
    A[Topic: user-events] --> B[Partition 0]
    A --> C[Partition 1]
    A --> D[Partition 2]

    B --> B1[Message 1]
    B --> B2[Message 2]
    B --> B3[Message 3]

    C --> C1[Message 1]
    C --> C2[Message 2]

    D --> D1[Message 1]
    D --> D2[Message 2]
    D --> D3[Message 3]
    D --> D4[Message 4]

Every message within a partition has a unique, sequential offset. Kafka guarantees message order only within a single partition, not across the entire topic.

How Kafka Assigns Messages to Partitions: The Message Key

Producers can include an optional key with each message. This key determines the target partition using a consistent hashing strategy:

• With a key: partition = hash(key) % num_partitions
• Without a key: Kafka uses a round-robin approach or a custom partitioner.

This mechanism ensures:

• All messages with the same key are directed to the same partition.
• Message order is preserved only for messages sharing the same key.

Crucial Point: If strict message order is vital for related events (e.g., all actions by a specific user), always use the same key for those messages.

Keyed Message Example

Consider processing user-events using user_id as the key:

graph TD
    A[Producer] -->|user_id: 101| P0[Partition 0]
    A -->|user_id: 102| P1[Partition 1]
    A -->|user_id: 103| P2[Partition 2]
    A -->|user_id: 101| P0
    A -->|user_id: 102| P1
    A -->|user_id: 103| P2

Here, all events for user_id: 101 consistently land in Partition 0, user_id: 102 in Partition 1, and so on. This preserves order for individual users while still achieving parallelism across different users.

Design Tip: Use keys intentionally to group related events and plan your partition count based on expected key distribution.

Consumers and Consumer Groups

Why One Consumer Per Partition?

Kafka's design dictates that only one consumer within a consumer group can read from a given partition at any time. This strict rule is crucial for preserving message order within partitions.

If multiple consumers could read from the same partition concurrently, message ordering would break, especially for keyed messages where sequential processing is critical.

graph TD
    subgraph Partition 0
        M1[Message 1 - user 101]
        M2[Message 2 - user 101]
        M3[Message 3 - user 101]
    end

    subgraph Consumer A
        A1[Receives M1]
        A2[Receives M3]
    end

    subgraph Consumer B
        B1[Receives M2]
    end

    M1 --> A1
    M2 --> B1
    M3 --> A2

Note: The above diagram illustrates a hypothetical scenario that is impossible in Kafka. It demonstrates why Kafka enforces the one-consumer-per-partition rule: to prevent out-of-order processing of messages within a partition.

Kafka's design ensures:

• ✅ One partition → One active consumer (per group)
• ✅ Guaranteed in-order delivery for messages within that partition

What is a Consumer Group?

A consumer group is a collection of consumers that cooperate to read data from a topic. Kafka distributes the topic's partitions among the consumers in the group, ensuring each partition is processed by exactly one consumer.

• Multiple consumer groups can read the same topic independently, each receiving a full copy of all messages.
• Within a single consumer group, partitions are exclusively assigned to individual consumers.

graph TD
    A[Topic: payments<br/>1 Partition] --> B[Partition 0]

    subgraph Consumer Group
        C1[Consumer 1] --> B
        C2[Consumer 2] -.-> |Idle| B
        C3[Consumer 3] -.-> |Idle| B
    end

    style C1 fill:#4CAF50,stroke:#388E3C
    style C2 fill:#9E9E9E,stroke:#616161
    style C3 fill:#9E9E9E,stroke:#616161

Partition Count vs. Consumer Count: Maximizing Parallelism

To fully leverage consumer parallelism, you should aim for:

number_of_partitions >= number_of_consumers_in_group

• If you have fewer partitions than consumers, some consumers will remain idle as partitions cannot be shared.
• If you have more partitions than consumers, some consumers will process multiple partitions.

Why? Partitions are Kafka's unit of parallelism. More partitions allow for more parallel consumers, leading to higher throughput.

Multiple Partitions = Parallel Consumers

Increasing the number of partitions directly enables parallel processing.

graph TD
    A[Topic: orders] --> B[Partition 0]
    A --> C[Partition 1]
    A --> D[Partition 2]
    A --> E[Partition 3]

    F[4 Partitions] --> A

    subgraph Consumer Group
        C1[Consumer 1] --> B
        C1 --> C
        C2[Consumer 2] --> D
        C2 --> E
    end

    style C1 fill:#4CAF50
    style C2 fill:#4CAF50

This design allows Kafka to scale linearly with the number of partitions.

Rebalancing Consumers

When consumers join or leave a group, Kafka automatically rebalances partition ownership among the remaining or new consumers.

sequenceDiagram
    participant T as Topic (3 partitions)
    participant C1 as Consumer 1
    participant C2 as Consumer 2
    participant C3 as Consumer 3
    participant C4 as Consumer 4

    T->>C1: Assign Partition 0
    T->>C2: Assign Partition 1
    T->>C3: Assign Partition 2
    Note right of C4: Idle - no partitions left

    Note over C1,C3: Rebalancing occurs when consumers join/leave

    C1->>T: Consumer 1 leaves
    T->>C2: Assign Partition 0
    T->>C4: Assign Partition 1
    T->>C3: Keep Partition 2

Impact: Rebalancing can cause temporary processing delays (lag spikes) as partitions are reassigned.

Scaling with Partitions: A Critical Consideration

You Can Only Increase Partitions

Kafka allows you to increase the number of partitions for a topic, but never decrease them. This limitation exists due to:

1. Offset Mismatch Risks: Decreasing partitions could lead to inconsistencies in message offsets.
2. Potential Data Loss: Messages might be lost during a partition reduction.
3. Consumer Confusion: Rebalancing becomes chaotic and unreliable.

# ✅ Increase partitions
kafka-topics --alter --topic user-events --partitions 8 \
  --bootstrap-server localhost:9092

# ❌ This will fail
# kafka-topics --alter --topic user-events --partitions 2

Planning Your Partition Count: Start Small, Scale Up

It's generally best to start with a lower number of partitions and increase them as your needs grow. Starting with too many can introduce unnecessary overhead. Since you can only increase, not decrease, beginning small allows for smoother, iterative scaling.

Monitor throughput and consumer utilization to determine the optimal time to add more partitions. Also consider:

• Expected throughput: How much data will flow through this topic?
• Number of consumers: How many consumers will process this data?
• Future growth: Anticipate your system's scaling needs.

Kafka Topic Migrations: Navigating the Challenges

Moving Kafka topics between clusters or environments presents unique challenges.

The “Unreadable Message” Problem

This common issue often arises with Avro-based data:

graph LR
    A[Source Cluster] --> B[Messages with<br/>Schema ID 42]
    B --> C[Migration Tool]
    C --> D[Target Cluster]
    D --> E[Messages with<br/>Schema ID 42]
    E -.-> F[Schema Registry<br/>ID 42 ≠ Different Schema!]

The problem occurs when the message's embedded schema ID in the new environment points to a different schema than intended, rendering messages unreadable.

Tombstone Messages: Don't Lose Them!

Tombstone messages are special messages with null values used for:

• Deleting keys in compacted topics.
• Triggering data cleanup.
• Representing logical deletions.

Ensure these critical messages are handled correctly during migration to avoid data inconsistencies.

Best Practices for Topic Migration

1. Test thoroughly: Always test migrations with sample data first.
2. Schema Registry compatibility: Verify that schema registries in source and target environments are compatible.
3. Tombstone handling: Confirm your migration strategy correctly handles tombstone messages.
4. Monitor rebalancing: Keep a close eye on consumer group rebalancing during and after migration.
5. Tool caution: Use tools like MirrorMaker2 or Confluent Replicator with a deep understanding of their behavior and limitations.

Practical CLI Examples

Scenario 1: Ordered, Low-Volume Events

For topics requiring strict ordering and handling low throughput (e.g., user actions where sequence is critical):

kafka-topics --create --topic user-actions \
  --partitions 1 --replication-factor 3 \
  --bootstrap-server localhost:9092

Outcome: Only one consumer will be active, ensuring strict ordering.

Scenario 2: High Throughput Event Streams

For topics demanding high parallelism and real-time processing (e.g., web events for analytics):

kafka-topics --create --topic web-events \
  --partitions 16 --replication-factor 3 \
  --bootstrap-server localhost:9092

Outcome: Up to 16 consumers can process messages in parallel, ideal for real-time analytics or event sourcing.

Key Takeaways

• Partitions = Parallelism: The fundamental unit for scaling Kafka.
• One Partition → One Consumer (per group): Ensures in-order processing within a partition.
• Partition Count is Immutable Downward: Plan carefully; you can only increase partitions.
• Migration is Tricky: Pay close attention to schema compatibility and tombstone messages.
• Rebalance = Temporary Lag Spikes: Be aware of processing delays during consumer group rebalancing.

Understanding Kafka’s partition model and consumer mechanics is paramount for building resilient, scalable distributed systems. As the saying goes, "With great partitioning comes great responsibility." 🧠

Final Thoughts

This guide provided a solid foundation in Kafka's core concepts. However, Kafka is a vast ecosystem with many advanced features, including:

• Kafka Streams for real-time stream processing
• Exactly-once semantics and transactions for data integrity
• Schema Registry and various serialization formats
• Comprehensive security, monitoring, and operational tooling

Consider this your starting point. Continue exploring Kafka's broader capabilities to unlock its full potential in your distributed applications!

Deploying Django apps with zero downtime is essential in modern production systems, especially when rolling out updates on Kubernetes or deploying worker-based systems incrementally. However, even a simple migration, like adding a new field, can silently disrupt your system if not handled carefully.

In this post, we’ll explore how django-migration-linter helps detect backward-incompatible migrations before they reach production—and how to effectively integrate it into your Django project and CI pipeline.

🚨 Migrations Are More Dangerous Than They Look

Django makes database migrations easy—but that simplicity can be deceptive. Let’s say you add a new field like this:

# models.py
class Order(models.Model):
    status = models.CharField(max_length=50, default="draft")

You run makemigrations, commit, and deploy. Everything seems fine, right?

Wrong.

Django's default= only applies at the Python level, not in the database itself. If your application uses old code that tries to write to this model without populating the new field, and there's no database-level default (db_default), your code can crash with a NOT NULL constraint failed.

This becomes especially dangerous during rolling deployments where both the old code and new schema run simultaneously.

Scenario:

• Migration is applied, a new column is created without a DB default
• A user tries to create a new row via the backend (still running old code)
• The backend doesn’t know about the new field and doesn’t send it
• The DB rejects the insertion → crash

⚠️ Common Pitfalls in Zero-Downtime Migrations

Here are a few common operations that seem innocent but can cause serious trouble during deployments:

• ❌ Adding a NOT NULL field without a db_default
• ❌ Renaming a field or table (breaks old code)
• ❌ Dropping a field still used by old workers
• ❌ Changing field types or constraints directly

To see a full list of potentially incompatible operations, check out this fantastic incompatibility reference.

🔧 What is django-migration-linter?

django-migration-linter is a Django app that scans your migrations and flags operations that are not backward compatible. It acts as a safety net to ensure that every migration you write won’t break live systems when old and new code run side-by-side.

✅ What it catches:

• Dangerous field additions
• Dropped fields or indexes
• Renames and type changes
• Constraint modifications

It’s especially useful in CI pipelines and rolling deployments.

📦 Installation

Install django-migration-linter as part of your project dependencies (not just in CI):

pip install django-migration-linter

Then, add it to your INSTALLED_APPS in settings.py:

INSTALLED_APPS += ["django_migration_linter"]

Optional Configuration

You can further customize the linter in settings.py:

MIGRATION_LINTER_OPTIONS = {
    # Only check migrations added since this commit (optional for legacy projects)
    "git_commit_id": "SOMECOMMITHASH",
    "project_root_path": BASE_DIR,
}

🧪 Linting Migrations in Your Project

Run the migration linter using Django's management command:

python manage.py lintmigrations

This will lint all migrations unless you configure it to only check those added since a specific Git commit (useful in legacy codebases).

🚀 Why This Matters in Kubernetes and Rolling Deployments

Let’s visualize a common deployment pattern:

sequenceDiagram
    participant Dev as Developer
    participant CI as CI/CD
    participant K8s as Kubernetes
    participant DB as Database

    Dev->>CI: Push code with migration
    CI->>DB: Apply migration
    CI->>K8s: Start rolling update
    K8s->>App: Some pods use old code
    K8s->>App: Some pods use new code
    App->>DB: Old code reads/writes old schema
    App->>DB: New code reads/writes new schema
    Note over DB: Must support both code versions

Of course, this is a simplified view (not involving GitOps), but our main goal is to demonstrate the possible issues.

Also note the time gap between the migration being applied and the start of the deployment. Even without rolling updates, your system may run old code against the new schema for some time. This is why every schema change must be backward compatible—django-migration-linter helps enforce that rule.

⚙️ CI Integration: GitHub Actions and GitLab CI

You can block unsafe migrations in your CI by running the linter after installing your Django app.

✅ GitHub Actions

name: Check Django Migrations

on: [pull_request]

jobs:
  lint-migrations:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: 3.13
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
      - name: Run Migration Linter
        run: |
          python manage.py lintmigrations --no-cache

✅ GitLab CI

lint_migrations:
  image: python:3.13
  stage: lint
  script:
    - pip install -r requirements.txt
    - python manage.py lintmigrations --no-cache
  interruptible: true

This ensures that only new migrations are linted on pull requests or merge requests. You can also configure your pipeline to skip the job if no migration files were added or updated.

📚 Resources

• 🔗 GitHub Repo: 3YOURMIND/django-migration-linter
• 📖 Incompatibility List: See all unsafe patterns

🔚 Conclusion

Schema changes are deceptively simple in Django, but their impact can be critical in a live environment. django-migration-linter helps you enforce safety by detecting migration operations that aren’t backward compatible—before you deploy.

✅ Add it to your project
✅ Run it locally
✅ Enforce it in CI

Your users won’t even notice you migrated.

🧩 Have you used django-migration-linter in production? Share your thoughts or lessons learned in the comments!

In this guide, you’ll learn how to set up this powerful combination and build reusable, maintainable form components.

✅ Why This Combination Works

• django-tailwind Integrates Tailwind CSS into your Django project with hot reloading and easy configuration.

• django-widget-tweaks Lets you dynamically add classes and attributes to form fields right in your templates — no need to modify the form definitions themselves.

• Custom CSS Classes You can group frequently-used Tailwind utilities into semantic, reusable class names, simplifying your markup and promoting a consistent design system.

⚙️ Setting Up

1️⃣ Install dependencies

pip install django-tailwind django-widget-tweaks

2️⃣ Update your Django settings

In settings.py:

INSTALLED_APPS = [
    # your other apps...
    'tailwind',
    'theme',  # this will be your Tailwind app
    'widget_tweaks',
]

TAILWIND_APP_NAME = 'theme'

3️⃣ Initialize Tailwind

python manage.py tailwind init

This will create a theme app containing Tailwind’s configuration and build pipeline.

🎨 Creating Custom CSS Classes

Inside theme/static_src/src/styles.css, define semantic classes that bundle complex Tailwind utilities. For example:

@import "tailwindcss";

/* Dark mode variant helper */
@custom-variant dark (&:where(.dark, .dark *));

/* Example: Form Fields */
.form-field-element {
  @apply block w-full p-2 border border-gray-300 dark:border-gray-700
         rounded-md focus:ring-blue-500 focus:border-blue-500
         dark:focus:ring-blue-400 dark:focus:border-blue-400
         dark:bg-gray-800 dark:text-gray-300;
}

.form-field-label {
  @apply block text-sm font-medium text-gray-700 dark:text-gray-300;
}

⚠️ Note on Using @apply

Tailwind’s @apply directive is very handy for grouping repetitive utility classes into a single, meaningful class name. However, it’s not a replacement for Tailwind’s utility-first philosophy.

Best practice: ✅ Use @apply for logical, reusable patterns — like form elements, cards, or buttons that appear consistently throughout your project. 🚫 Avoid creating a custom class for every possible utility combo — this defeats Tailwind’s purpose and can cause CSS bloat and specificity headaches.

In short: be intentional — use custom classes to encapsulate genuine design patterns, not every tiny variation.

🧩 Reusable Form Components

Example: Basic field component

templates/components/forms/field.html

{% load widget_tweaks %}

{% if field.is_hidden %}
  {{ field }}
{% else %}
  <div class="mb-6">
    <label for="{{ field.id_for_label }}" class="form-field-label">
      {{ field.label }}
      {% if field.field.required %}<span class="text-red-500 ml-1">*</span>{% endif %}
    </label>

    {% if field.errors %}
      {% render_field field class="form-field-element border-red-300 dark:border-red-600 focus:border-red-500 focus:ring-red-200" placeholder=field.label %}
      {% for error in field.errors %}
        <p class="mt-2 text-sm text-red-600 dark:text-red-400">{{ error }}</p>
      {% endfor %}
    {% else %}
      {% render_field field class="form-field-element" placeholder=field.label %}
    {% endif %}

    {% if field.help_text %}
      <p class="mt-2 text-sm text-slate-500 dark:text-slate-400">{{ field.help_text|safe }}</p>
    {% endif %}
  </div>
{% endif %}

✅ Key Benefits

✔️ 1. Semantic Names

Create meaningful class names like .form-field-element instead of repeating long utility strings.

✔️ 2. Consistent Design

Reusable styles guarantee your forms look cohesive everywhere.

✔️ 3. Built-in Dark Mode

Dark mode support works out of the box by leveraging Tailwind’s dark: variant.

✔️ 4. Easy Maintenance

Update your design system in a single CSS file — no messy search-and-replace in templates.

✔️ 5. Fully Responsive

Tailwind utilities and custom breakpoints ensure your forms adapt beautifully to all screen sizes.

🚀 Development Workflow

• Start Tailwind’s build process during development:

    python manage.py tailwind start
  

• Build a minified version for production:

    python manage.py tailwind build
  

📁 Recommended Template Structure

templates/
├── base.html
├── components/
│   └── forms/
│       ├── field.html
│       ├── password_field.html
│       └── checkbox_field.html
└── your_app/
    └── form.html

💡 Best Practices

✅ Use semantic, purposeful custom classes for true design patterns

✅ Keep related styles together in your Tailwind CSS build

✅ Use Tailwind’s @apply wisely — don’t recreate utility classes unnecessarily

✅ Always test your forms in both light and dark modes

✅ DRY: reuse components instead of duplicating markup

🎉 Conclusion

Combining django-tailwind, django-widget-tweaks, and a few well-chosen custom CSS classes is a powerful, maintainable way to build beautiful Django forms.

Start small, build a library of reusable components, and enjoy the flexibility and power of Tailwind’s utility-first design — without sacrificing clean, readable templates.

✨ Happy coding — and beautiful forms ahead!

HTMX is a powerful tool that allows developers to create highly interactive and dynamic user interfaces without the need for heavy JavaScript frameworks. At its core, HTMX uses HTML attributes to define behaviour, making it an ideal companion to traditional server-side frameworks such as Django.

Key features of HTMX:

Simplicity: HTMX simplifies the process of adding dynamic functionality to web applications by using familiar HTML attributes such as hx-get, hx-post, and hx-trigger.

Minimal JavaScript: Unlike complex JavaScript frameworks, HTMX requires minimal JavaScript code, reducing the overhead associated with client-side scripting.

Server-side rendering: HTMX integrates seamlessly with server-side rendering frameworks such as Django, allowing developers to leverage the power of both front-end and back-end technologies.

Progressive enhancement: HTMX follows the principle of progressive enhancement, ensuring that basic functionality is available to all users, while improving the experience for those with JavaScript-enabled browsers.

Compatibility: HTMX is compatible with all modern browsers and works well with existing web technologies, making it easy to integrate into existing projects.

How HTMX works with Django:

In a Django application, HTMX can be used to create dynamic UI elements within Django templates, enabling real-time updates without reloading pages. By using Django's powerful template language alongside HTMX attributes, developers can easily create dynamic and responsive web applications.

Benefits of using HTMX with Django include

• Simplified development: HTMX simplifies the development process by allowing developers to focus on writing HTML and Python code rather than complex JavaScript logic.

• Improved performance: With its minimal JavaScript footprint, HTMX helps improve the performance of Django applications by reducing the amount of client-side processing required.

• Improved user experience: By enabling real-time updates and interactions, HTMX improves the overall user experience, resulting in higher user engagement and satisfaction.

In the next section, we'll dive into the process of setting up a Django project with HTMX, using Poetry for dependency management.

Setting up the Project

No interest in reading step by step guide, here is the full example in Github.

Poetry is a dependency management tool for Python that simplifies the process of managing project dependencies and virtual environments. In this section, we'll guide you through setting up a new Django project with Poetry and installing the necessary dependencies for integrating HTMX.

1. Install Poetry: First, make sure you have Poetry installed on your system. You can install Poetry using the following command:

curl -sSL https://install.python-poetry.org | python3 -

For alternative installation methods, refer to the Poetry documentation.

2. Create a New Folder: Create a new folder for your Django project. You can do this using your system's file explorer or the command line:

mkdir my_django_project
cd my_django_project

3. Initialize Poetry: Initiate a new Poetry project within your project folder by running the following command:

poetry init

Follow the interactive prompt to provide details about your project, such as its name, version, and dependencies.

4. Install Dependencies: Once the Poetry project is initialized, you can start installing the required dependencies. Run the following command to install the core dependencies:

poetry add django django-htmx django-crispy-forms crispy-bootstrap5 django-render-block

This command will add Django, django-htmx, django-crispy-forms, crispy-bootstrap5, and django-render-block to your project's dependencies.

5. Add Development Dependencies: Additionally, you may want to add development dependencies for tasks such as code formatting, linting, and debugging. You can install these dependencies using the following command:

poetry add black ruff djlint ipdb ipython --group dev

This command will add black, ruff, djlint, ipdb, and ipython as development dependencies.

Setting up the Django Project

Now that we have installed the necessary dependencies with Poetry, let's proceed to set up the Django project. In this section, we'll guide you through initializing a new Django project and configuring it to integrate with HTMX.

1. Initialize Django Project: Run the following command to initialize a new Django project in the current directory:

django-admin startproject django_htmx .

This command creates a new Django project named django_htmx in the current directory. The dot (.) at the end specifies that the project should be created in the current directory.

2. Configure Settings: Navigate to the django_htmx directory and open the settings.py file within your favorite text editor. Update the INSTALLED_APPS setting to include the installed dependencies:

INSTALLED_APPS = [
    ...
    'django_htmx',
    'crispy_forms',
    'crispy_bootstrap5',
    'django_render_block',
    'htmx',
]

Make sure to replace 'app_name' with the name of your Django app.

3. Configure Middleware: In the same settings.py file, add the HTMX middleware to the MIDDLEWARE setting:

MIDDLEWARE = [
    ...
    'htmx.middleware.HtmxMiddleware',
]

This middleware is essential for HTMX to intercept and process AJAX requests.

Django-htmx adds extra parameters to the request based on the recognised headers, making it easy for us to use them later in our views. E.g. the request.htmx which we can use to know if the request is coming from htmx or not.

4. Configure Crispy Forms: If you plan to use Crispy Forms for styling your forms, add the following settings to settings.py:

CRISPY_ALLOWED_TEMPLATE_PACKS = 'bootstrap5'
CRISPY_TEMPLATE_PACK = 'bootstrap5'

This configures Crispy Forms to use Bootstrap 5 for styling.

5. Run Migrations: Before you can start using your Django project, you need to apply the initial database migrations. Run the following command:

python manage.py migrate

This command creates the necessary database tables for your Django project.

6. Verify Setup: To verify that the project setup is successful, you can start the Django development server by running:

python manage.py runserver

Open a web browser and navigate to http://127.0.0.1:8000. If you see the Django welcome page, congratulations! Your Django project is set up and ready to integrate with HTMX.

Great! Let's set up the todos app with the provided model, URLs, and admin configurations.

Todos app

1. Create thetodosApp: If you haven't already created the todos app, you can create it using the following command:

python manage.py startapp todos

This command will create a new app named todos.

2. Define the Model: Inside the models.py file of the todos app, define the TodoItem model as provided:

# todos/models.py

from django.db import models

class TodoItem(models.Model):
    title = models.CharField(max_length=100)
    completed_on = models.DateTimeField(null=True, blank=True)

    class Meta:
        ordering = ["-id"]

    def __str__(self):
        return self.title

3. Define the URLs: Create a urls.py file inside the todos app directory and define the URLs as provided:

# todos/urls.py

from django.urls import path
from . import views

app_name = "todos"

urlpatterns = [
    path("", views.TodoItemListView.as_view(), name="todoitem-list"),
    path("create/", views.TodoItemCreateView.as_view(), name="todoitem-create"),
    path("<int:pk>/mark-complete/", views.TodoMarkComplete.as_view(), name="todoitem-mark-complete"),
    path("<int:pk>/delete/", views.TodoItemDeleteView.as_view(), name="todoitem-delete"),
]

4. Create Views (to be implemented): You'll need to create the views mentioned in the URLs (views.py) for TodoItemListView, TodoItemCreateView, TodoMarkComplete, and TodoItemDeleteView.

5. Register Model with Admin: In the admin.py file of the todos app, register the TodoItem model as provided:

# todos/admin.py

from django.contrib import admin
from .models import TodoItem

@admin.register(TodoItem)
class TodoItemAdmin(admin.ModelAdmin):
    list_display = ("title", "completed_on")
    search_fields = ("title",)

6. Configure Settings: Make sure the todos app is included in the INSTALLED_APPS list in your project's settings.py file.

Integrating HTMX with Class-Based Views in Django

In this section, we'll explore how to integrate HTMX with Django's Class-Based Views (CBVs) using a custom HtmxTemplateResponseMixin. We'll examine the provided views.py file, which contains CBVs for handling todo items in our Django application, and explain how each view contributes to creating dynamic user interfaces with HTMX.

# todos/views.py

# Import necessary modules and classes
from typing import Any
from django import forms
from django.http import HttpResponse
from django.shortcuts import redirect
from django.template.response import TemplateResponse
from django.urls import reverse_lazy
from django.utils import timezone
from django.views.generic import CreateView, DeleteView, ListView, View
from django_htmx.http import retarget
from render_block import render_block_to_string
from todos.forms import TodoItemForm
from todos.models import TodoItem

# Define a custom TemplateResponse class for HTMX integration
class HtmxResponseClass(TemplateResponse):
    def __init__(self, htmx_template_block: str | None = None, *args, **kwargs):
        self.htmx_template_block = htmx_template_block
        super().__init__(*args, **kwargs)

    @property
    def rendered_content(self):
        context = self.resolve_context(self.context_data)
        if self._request.htmx and self.htmx_template_block:
            return render_block_to_string(
                self.template_name,
                block_name=self.htmx_template_block,
                context=self.context_data,
                request=self._request,
            )
        template = self.resolve_template(self.template_name)
        return template.render(context, self._request)

# Define a mixin for HTMX template responses
class HtmxTemplateResponseMixin:
    response_class = HtmxResponseClass
    htmx_template_block = None

    def render_to_response(self, context, **response_kwargs):
        response_kwargs.setdefault("content_type", self.content_type)
        return self.response_class(
            request=self.request,
            template=self.get_template_names(),
            context=context,
            using=self.template_engine,
            htmx_template_block=self.htmx_template_block,
            **response_kwargs,
        )

# Define a view for listing todo items
class TodoItemListView(HtmxTemplateResponseMixin, ListView):
    model = TodoItem
    template_name = "todos/todoitem_list.html"
    context_object_name = "todoitems"
    ordering = ["-id"]
    htmx_template_block = "object_list_block"

    def get_context_data(self, **kwargs: Any) -> dict[str, Any]:
        context = super().get_context_data(**kwargs)
        context["form"] = TodoItemForm()
        return context

# Define a view for creating todo items
class TodoItemCreateView(HtmxTemplateResponseMixin, CreateView):
    form_class = TodoItemForm
    template_name = "todos/todoitem_list.html"
    htmx_template_block = "form_block"
    http_method_names = ["post"]
    success_url = reverse_lazy("todos:todoitem-list")

    def form_invalid(self, form: forms.ModelForm) -> HttpResponse:
        response = super().form_invalid(form)
        return retarget(response, "this")

# Define a view for deleting todo items
class TodoItemDeleteView(DeleteView):
    model = TodoItem
    success_url = reverse_lazy("todos:todoitem-list")
    http_method_names = ["post"]

# Define a view for marking todo items as complete
class TodoMarkComplete(View):
    http_method_names = ["post"]

    def post(self, request, *args, **kwargs):
        todoitem = TodoItem.objects.get(pk=kwargs["pk"])
        todoitem.completed_on = timezone.now()
        todoitem.save()
        return redirect("todos:todoitem-list")

Understanding the Views

Now, let's break down what each view does:

1. TodoItemListView: This view lists all todo items in the database. It utilizes the HtmxTemplateResponseMixin to handle HTMX requests, so that we return the whole content when accessing the page via the browser and only the list when accessing it via HTMX requests.

   The get_context_data method adds a form (TodoItemForm) to the context to enable the creation of new todo items.

2. TodoItemCreateView: This view handles the creation of new todo items. It uses the HtmxTemplateResponseMixin to handle HTMX requests and overrides the form_invalid method to return an HTMX response with validation errors.

3. TodoItemDeleteView: This view deletes todo items from the database. It specifies the model (TodoItem) and the success URL to redirect to after deletion.

4. TodoMarkComplete: This view marks a todo item as complete by updating its completed_on field with the current timestamp. It redirects the user to the todo item list view (TodoItemListView) after marking the item as complete.

By using CBVs with the HtmxTemplateResponseMixin, we can seamlessly integrate HTMX functionality into our Django application, providing users with a dynamic and interactive user experience.

Django Templates

In this section, we'll define a base HTML template (base.html) that serves as the foundation for our HTMX-powered Django application. This template provides the necessary structure and includes links to Bootstrap for styling and HTMX for seamless AJAX interactions.

1. Add Template Directory to Settings: Before creating the base template, ensure that the templates directory is configured in your Django project's settings.py file:

# settings.py

TEMPLATES = [
    {
        ...
        'DIRS': [BASE_DIR / "templates"],
        ...
    },
]

This configuration specifies the location of the templates directory relative to the project's base directory.

2. Define Base Template (base.html):

<!-- templates/base.html -->
{% load static %}
{% load django_htmx %}

<!DOCTYPE html>
<html lang="en" data-bs-theme="dark">
    <head>
        <meta name="description"
              content="HTMXANGO is a Django application that demonstrates how to use HTMX with Django.">
        <meta name="keywords" content="HTMXANGO, HTMX, Django, Python">
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        <title>
            {% block title %}
                HTMXANGO
            {% endblock title %}
        </title>
        <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.2/dist/css/bootstrap.min.css"
              rel="stylesheet"
              integrity="sha384-T3c6CoIi6uLrA9TneNEoa7RxnatzjcDSCmG1MXxSR1GAsXEV/Dwwykc2MPK8M2HN"
              crossorigin="anonymous">
        <link rel="stylesheet"
              href="https://cdn.jsdelivr.net/npm/bootstrap-icons@1.11.0/font/bootstrap-icons.css">
        {% block styles %}
        {% endblock styles %}
    </head>
    <body hx-headers='{"X-CSRFToken": "{{ csrf_token }}"}'>
        <div class="container">
            {% block content %}
            {% endblock content %}
        </div>
        {% block scripts %}
        {% endblock scripts %}
        <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.2/dist/js/bootstrap.bundle.min.js"
                integrity="sha384-C6RzsynM9kWDrMNeT87bh95OGNyZPhcTNXj1NW7RuBCsyN/o0jlpcV8Qyq46cDfL"
                crossorigin="anonymous"></script>
        <script src="https://unpkg.com/htmx.org@1.9.10" defer></script>
        {% django_htmx_script %}
    </body>
</html>

Understanding the Base Template:

• Load Static Files: The {% load static %} tag allows us to reference static files such as CSS and JavaScript within our templates.

• Load Django-HTMX Tags: The {% load django_htmx %} tag enables the usage of HTMX-specific template tags and attributes.

• HTML Structure: The template follows standard HTML5 structure with <head> and <body> sections.

• Title Block: The {% block title %} tag allows child templates to define their own titles while providing a default title ("HTMX Django").

• Bootstrap Integration: The template includes links to Bootstrap CSS and JavaScript files for styling and interactivity.

• CSRF Token: The hx-headers attribute ensures that HTMX requests include the CSRF token for security purposes.

• Content Block: The {% block content %} tag defines a placeholder for child templates to inject their content.

• Scripts Block: The {% block scripts %} tag defines a placeholder for additional JavaScript scripts.

• HTMX Integration: The template includes HTMX JavaScript library (htmx.min.js) and the {% django_htmx_script %} tag for HTMX-specific configurations django-htmx third party app.

  Understanding the Use of CSRF Token in HTMX Requests

  In the base HTML template (base.html) provided for our HTMX-powered Django application, you may have noticed the inclusion of the hx-headers attribute with the CSRF token in the <body> tag:

    <body hx-headers='{"X-CSRFToken": "{{ csrf_token }}"}'>
  

  This addition serves a crucial security purpose when making HTMX requests within our application.

  Why include a CSRF token in HTMX requests?

  Django protects against Cross-Site Request Forgery (CSRF) attacks by requiring the inclusion of a CSRF token in requests that change state (e.g. POST requests). This token is typically included in forms as a hidden field and is validated by Django to ensure that the request comes from the same site and is not forged by a malicious third party.

  If we are using HTMX to make AJAX requests within our application, we need to ensure that these requests include the CSRF token so that Django can validate them properly. Including the CSRF token in HTMX requests helps prevent CSRF attacks and ensures the security of our application.

  How it works

  By including the hx-headers attribute with the CSRF token in the tag, we instruct HTMX to include this token in all AJAX requests made within our application. This ensures that each request contains the necessary CSRF token for Django to validate.

Implementing Todo List Template with HTMX

In this section, we'll define the todoitem_list.html template, which is responsible for displaying the list of todo items in our HTMX-powered Django application. This template extends the base template (base.html) and includes HTMX-specific attributes and functionality for dynamic interactions.

1. Define Todo List Template (todoitem_list.html):

<!-- templates/todos/todoitem_list.html -->

{% extends "base.html" %}
{% load crispy_forms_tags %}

{% block content %}
    <div class="row align-items-center">
        <div class="row">
            <h1>Todo Item</h1>
            <div class="spinner-border htmx-indicator" role="status" id="spinner">
                <span class="sr-only">Loading...</span>
            </div>
            {% block form_block %}
                <form method="post"
                      hx-post="{% url 'todos:todoitem-create' %}"
                      hx-swap="innerHTML"
                      hx-trigger="submit"
                      hx-target="#todo-list"
                      hx-indicator="#spinner">
                    {% crispy form %}
                    <button type="submit" class="btn btn-primary">Save</button>
                </form>
            {% endblock form_block %}
        </div>
        <div class="row">
            <h1>Todo List</h1>
            <div id="todo-list">
                {% block object_list_block %}
                    {% if todoitems %}
                        <table class="table table-striped">
                            <thead>
                                <tr>
                                    <th scope="col" class="col-1">#</th>
                                    <th scope="col" class="col-5">Title</th>
                                    <th scope="col" class="col-6">Completed</th>
                                </tr>
                            </thead>
                            <tbody>
                                {% for todo in todoitems %}
                                    <tr>
                                        <th scope="row">{{ todo.id }}</th>
                                        <td>{{ todo.title }}</td>
                                        <td>
                                            <input class="form-check-input"
                                                   type="checkbox"
                                                   value=""
                                                   id="completed-checkbox"
                                                   {% if todo.completed_on %}checked{% endif %}
                                                   hx-post="{% url 'todos:todoitem-mark-complete' todo.id %}"
                                                   hx-swap="outerHTML"
                                                   hx-target="closest table"
                                                   hx-trigger="click"
                                                   {% if todo.completed_on %}disabled{% endif %}>
                                            <label class="form-check-label" for="completed-checkbox">
                                                {% if todo.completed_on %}
                                                    {{ todo.completed_on }}
                                                {% else %}
                                                    Not Completed
                                                {% endif %}
                                            </label>
                                            <button hx-post="{% url 'todos:todoitem-delete' todo.id %}"
                                                    hx-confirm="Are you sure?"
                                                    hx-swap="outerHTML"
                                                    hx-target="closest table"
                                                    hx-trigger="click"
                                                    class="btn btn-danger btn-sm float-end"
                                                    tooltip="Delete Todo Item">
                                                <i class="bi bi-trash"></i>
                                            </button>
                                        </td>
                                    </tr>
                                {% endfor %}
                            </tbody>
                        </table>
                    {% else %}
                        <p>No todo items found.</p>
                    {% endif %}
                {% endblock object_list_block %}
            </div>
        </div>
    </div>
{% endblock content %}

Understanding the Todo List Template:

• Extending Base Template: The template extends the base.html template to inherit its structure and functionality.

• Load Crispy Forms Tags: The {% load crispy_forms_tags %} tag enables the usage of Crispy Forms for rendering forms in a Bootstrap-friendly manner.

• Content Block: The {% block content %} tag defines the main content area of the template.

• Todo Item Form: The form block includes a form for creating new todo items. The form is submitted via HTMX with AJAX, and the response replaces the content of the todo list (#todo-list).

• Todo List Display: The object_list_block block iterates over todo items and displays them in a table format. Each todo item includes options to mark as complete or delete using HTMX-powered interactions.

• HTMX Attributes: HTMX attributes (hx-post, hx-swap, hx-target, hx-trigger, hx-indicator, hx-confirm) are used to define AJAX behavior, such as posting data, swapping HTML content, and indicating loading state.

In this section, we've defined the todoitem_list.html template to display the list of todo items in our HTMX-powered Django application. This template uses HTMX attributes and functionality to enable dynamic interactions such as creating, marking as complete, and deleting todo items, all without refreshing the page.

Testing HTMX Views in Django

We'll write the test cases test_views.py to ensure the functionality of our HTMX-powered views in the Django application. These test cases cover various scenarios, including rendering templates, handling form submissions, and processing AJAX requests.

1. Define Test Cases (test_views.py):

# todos/tests/test_views.py

from django.test import TestCase
from django.urls import reverse
from todos.forms import TodoItemForm
from todos.models import TodoItem

class TodoItemListViewTests(TestCase):
    # Test case to check rendering of the todo item list view
    def test_get(self):
        response = self.client.get(reverse("todos:todoitem-list"))
        self.assertEqual(response.status_code, 200)
        self.assertIn("form", response.context_data)
        self.assertIsInstance(response.context_data["form"], TodoItemForm)

    # Test case to check rendering of the todo item list view with HTMX
    def test_get_with_htmx(self):
        response = self.client.get(
            reverse("todos:todoitem-list"), headers={"hx-request": "true"}
        )
        self.assertEqual(response.status_code, 200)
        self.assertContains(response, "No todo items found.")
        self.assertNotContains(response, "form")

# Additional test cases for other views (TodoItemCreateView, TodoItemDeleteView, TodoMarkCompleteTests) are omitted for brevity.

Understanding the Test Cases:

• TestCase Class Structure: Each test case class inherits from django.test.TestCase, providing access to Django's testing utilities.

• TodoItemListViewTests: This class contains test cases for the todo item list view (TodoItemListView).

  • test_get: Tests the rendering of the todo item list view and ensures that the form is present in the context.

  • test_get_with_htmx: Tests the rendering of the todo item list view with HTMX enabled and verifies that the form is not included in the response.

• Additional Test Cases: Similar test cases are defined for other views (TodoItemCreateView, TodoItemDeleteView, TodoMarkCompleteTests) to cover their respective functionalities. (check the github link at the end for the full example)

Conclusion

In this article, we've looked at integrating HTMX with Django to create dynamic and interactive web applications. HTMX provides a lightweight and intuitive way to add AJAX functionality to Django projects, enabling seamless user experiences without the need for complex JavaScript frameworks.

We started by setting up a Django project with the necessary dependencies and configuring the HTMX middleware. We then defined models, views, templates and tests for a todo application, showing how HTMX simplifies the implementation of dynamic features such as form submissions, real-time updates and interactive user interfaces.

Through examples such as the todo item list view and form submissions, we demonstrated the power and flexibility of HTMX in extending Django applications. With HTMX, developers can leverage the simplicity of Django's templates and views while adding rich client-side interactions.

Testing HTMX views ensures the reliability and correctness of our application, allowing us to deploy features into production with confidence. By writing comprehensive test cases, we can verify that our HTMX-powered views behave as expected under various conditions.

To further explore and experiment with the codebase discussed in this article, you can access the full example on GitHub. Feel free to fork, clone, and contribute to the project, and unlock the full potential of HTMX in your Django applications.

With HTMX, developers can deliver modern, responsive, and user-friendly web applications while leveraging the simplicity and robustness of the Django framework. Whether you're building a todo app, a dashboard, or an e-commerce platform, HTMX enables you to create delightful experiences for your users.

Harness the power of HTMX and take your Django projects to new heights of interactivity and engagement. Happy coding!

However, manually managing software updates can be a tedious and time-consuming task, especially when you have to deal with multiple packages and dependencies.

This is where Renovate comes in - an automated software update tool that simplifies the process of managing software updates and helps developers stay up-to-date with the latest releases.

Automating boring stuff like package updates can help developers save time and reduce the risk of errors caused by manual intervention. By using automated update tools, developers can focus on more important tasks like writing code and improving functionality.

Gitlab use case

Unfortunately, GitLab users have been left out in the cold when it comes to dependable automation tools like Dependabot, which is only available on GitHub. There are some libraries built on top of dependabot-core but the release cycle doesn't seem reliable. This is where Renovate comes in as a great alternative.

Support for many packages managers and tools

Renovate is a reliable and feature-packed tool that automates software updates across multiple package managers, including npm, Docker, and Git submodules. It supports multiple languages and tools like pre-commit, Docker, and Docker Compose, and is highly configurable to suit your specific needs. The tool is designed to be highly customizable, allowing you to configure updates for specific packages, control update frequency, and apply custom version ranges.

Community and support

One of the best things about Renovate is the frequency of its releases. The tool is updated weekly with new features and bug fixes, ensuring that it stays up-to-date with the latest software releases. Additionally, the tool has an active community of developers who contribute to its development and support, providing a reliable and responsive support system for users.

Running Renovate from a Docker image

Renovate also offers a Docker image, making it easy to run as a container in your GitLab environment. By running Renovate as a Docker image, you can automatically create merge requests in GitLab, reducing the manual work required to manage software updates. The tool can be configured to run on a schedule or triggered by changes to your codebase, ensuring that your software is always up-to-date.

Gitlab Token setup

To use Renovate with GitLab, you'll need to create a GitLab group token with developer access that you can use to authenticate with Renovate. Here's how you can create a GitLab group token with developer access:

1. Log in to your GitLab account and navigate to the group or sub-group that you want to use with Renovate.

2. Click on the Settings button in the sidebar menu, and then click on Access Tokens under the Access Control section.

3. Click on the Create a token button to create a new token.

4. Enter a name and expiration date for the token, and then select the scopes that you want to grant the token. In this case, you'll want to grant the token the "API", "read_repository", and "write_repository" scopes.

5. Make sure that the "Developer" role is selected for the token's "Role" field.

6. Click on the Create personal access token button to create the token.

7. Copy the token that GitLab generates and store it somewhere secure, as you won't be able to access it again once you leave the page.

Now that you have created a GitLab group token with the necessary scopes and roles, you can use it with Renovate to automate software updates in your GitLab environment. Remember to set the token as an environment variable called RENOVATE_TOKEN when running Renovate as a Docker container, as described in the previous answer.

GitHub token // Yes also when using Gitlab

Renovate supports fetching release notes from GitHub for some package managers and languages, such as Node.js and Python. To fetch release notes, Renovate requires a GitHub personal access token with the read_only scope, which allows access to public repositories.

Here's how you can set up a GitHub token to be used only for fetching release notes:

1. Log in to your GitHub account and navigate to the "Settings" page.

2. Click on "Developer settings" in the sidebar menu, and then click on "Personal access tokens."

3. Click on the "Generate new token" button to create a new token.

4. Enter a name for the token in the "Token description" field.

5. Under the "Select scopes" section, click on the "repo" dropdown menu and select "public_repo."

6. Click on the "Generate token" button to create the token.

7. Copy the token that GitHub generates and store it somewhere secure, as you won't be able to access it again once you leave the page.

Now that you have created a GitHub token with the public_repo scope, you can use it with Renovate to fetch release notes for packages. To use the token with Renovate, you'll need to set it as an environment variable called GITHUB_TOKEN when running Renovate as a Docker container or via other deployment methods.

Running with Docker

docker run -it --rm \
  -e RENOVATE_TOKEN=${RENOVATE_TOKEN} \
  -e GITHUB_COM_TOKEN=${GITHUB_TOKEN}
  -e RENOVATE_REPOSITORIES=mygitlabgroup/gitlab-repo
  -e RENOVATE_PLATFORM=gitlab \
  renovate/renovate:latest

In the first run, Renovate will create an onboarding merge request which will create renovate.json config file.

For details about the possible configuration check out the documentation.

An example of renovate.json

{
    "$schema": "https://docs.renovatebot.com/renovate-schema.json",
    "labels": [
        "dependencies",
        "renovate"
    ],
    "separateMajorMinor": true,
    "pre-commit": {
        "enabled": true
    }
}

You can customize Renovate with such config, for example, to create a single MR for dev dependencies, to auto-merge specific changes, to exclude some packages by default or to assign a specific person.

To run Renovate for all repositories to which the GitLab token has access, you can run this instead of the previous command:

docker run -it --rm \
  -e RENOVATE_TOKEN=${RENOVATE_TOKEN} \
  -e GITHUB_COM_TOKEN=${GITHUB_TOKEN}
  -e RENOVATE_AUTODISCOVER=true
  -e RENOVATE_PLATFORM=gitlab \
  renovate/renovate:latest

You can run such a script in a cronjob or set it up properly in your Kubernetes cluster.

Worth mentioning

The full version of the Renovate Docker image is typically easier to use than the slim version, as it includes support for Node and uses a sidecar to provide other necessary components. You can find more information on this topic in the Running Renovate section of the official documentation.

If you have many projects and expect to Renovate to make a large number of calls, it's important to be aware of Dockerhub's rate limit, as this could impact your Dockerhub usage.

In general, running Renovate every week should be sufficient for most use cases.

If you have multiple outdated repositories, it may be easier to update them manually before starting any experimentation with Renovate.

This can help prevent an overwhelming number of Merge requests from being created, which can be difficult to manage and review. Maybe start each project type at a time until you find the best configuration (React app, Python Backend, Java application etc...)

Conclusion

In conclusion, Renovate is an excellent tool for automating software updates and keeping your software up-to-date with the latest releases.

Its support for multiple package managers, languages, and tools, along with its frequent updates and active community, make it a reliable and flexible solution for developers. By using Renovate, you can save time and reduce the risk of errors caused by manual intervention, ensuring that your software remains performant, secure, and up-to-date.

What is Ruff?

Ruff is a static code analysis tool for Python that provides a fast and comprehensive analysis of your code. It checks your code against a set of predefined rules and best practices, helping you to identify and fix problems early in the development cycle. Ruff has a large number of built-in rules covering a wide range of coding standards and best practices, making it a comprehensive tool for ensuring the quality of your code.

Advantages of Ruff

Fast performance: Ruff is written in Rust, a programming language renowned for its performance and efficiency. This makes Ruff much faster than other code analysis tools such as Flake8. Using Rust allows Ruff to quickly analyze even the largest Python projects, giving you fast and accurate results.

Comprehensive analysis: Ruff provides a comprehensive analysis of your code, covering a wide range of best practices and coding standards. This makes it easier to identify and fix issues early in the development cycle before they become bigger problems.

Easy configuration: Ruff allows you to configure your analysis in the pyproject.toml file. This makes it easier to set up and customize your analysis as all the configuration is stored in a single file.

Auto-fix feature: Ruff also has a unique auto-fix feature that allows you to automatically fix common problems with a single command. The --fix option can be used to automatically fix problems that Ruff has identified in your code. This saves you time and effort, allowing you to focus on more important tasks.

Getting started with Ruff

Getting started with Ruff is simple. You can install Ruff using pip, the Python package manager. Just run the following command in your terminal

pip install ruff

Once Ruff is installed, you can configure your analysis in the pyproject.toml file. Here is an example configuration:

[tool.ruff]
# Docs: https://beta.ruff.rs/docs/
# Rules: https://beta.ruff.rs/docs/rules/
select = ["B", "C4", "EXE", "F", "E", "ISC", "ICN", "INP", "PIE", "SIM", "W", "T20", "UP", "T10", "G", "C90", "ERA"]
ignore = ["B008", "SIM102"]
fixable = ["F", "E", "B", "C4", "EXE", "ISC", "ICN", "INP", "PIE", "SIM", "W", "T20", "UP"]
unfixable = []

# Exclude a variety of commonly ignored directories.
exclude = [
    ".git",
    ".mypy_cache",
    ".pre-commit-cache",
    ".ruff_cache",
    ".tox",
    ".venv",
    "venv",
    "docs",
    "__pycache",
    "**/migrations/*",
]

# Same as Black.
line-length = 100

# Allow unused variables when underscore-prefixed.
dummy-variable-rgx = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$"
# Assume Python 3.11.
target-version = "py311"

[tool.ruff.mccabe]
# Unlike Flake8, default to a complexity level of 10.
max-complexity = 10

For a full list of Ruff rules check it out here.

To run Ruff, simply navigate to your project directory in the terminal and run the following command:

ruff check .

Ruff will analyze your code and display any errors or warnings it has found. If you want to automatically fix common issues, you can use the --fix option:

ruff check . --fix

Ruff will automatically fix any problems it finds, saving you time and effort.

VSCode extension

Ruff also has a Visual Studio Code (VSCode) extension that makes it even easier to use. The VSCode extension provides real-time analysis of your code, allowing you to quickly identify and fix problems.

Pre-commit integration

- repo: https://github.com/charliermarsh/ruff-pre-commit
  # Ruff version.
  rev: 'v0.1.13'
  hooks:
    - id: ruff

The bottom line

In conclusion, Ruff is a fast and powerful static code analysis tool for Python. Its fast performance, comprehensive analysis and easy configuration make it an excellent choice for any Python project looking to improve code quality and avoid common pitfalls.

Install Docker and Docker Compose

To get started, you will need to have Docker and Docker Compose installed on your machine. If you don't have them installed, you can download them from the Docker website. Once you have them installed, you can proceed to the next step.

Create a Docker Compose file

Create a new directory on your machine, and within that directory, create a new file called "docker-compose.yml". This file will be used to define the services that make up our Kafka cluster.

Define the Kafka Service

In the "docker-compose.yml" file, add the following code to define the Kafka service:

version: '3'
services:
  kafka:
    image: confluentinc/cp-kafka:7.3.0
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,CONNECTIONS_FROM_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONNECTIONS_FROM_HOST:PLAINTEXT
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_LOG_FLUSH_INTERVAL_MESSAGES: 1
    ports:
      - 9092:9092

This code defines a single service called "kafka" that uses the Confluent Kafka image version 5.5.1. It also sets the environment variables that are used to configure the Kafka service.

Define the Zookeeper Service

Add the following code to the "docker-compose.yml" file to define the Zookeeper service:

  zookeeper:
    image: confluentinc/cp-zookeeper:7.3.0
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
    ports:
      - 2181:2181

This code defines a single service called "zookeeper" that uses the Confluent Zookeeper image version 5.5.1. It also sets the environment variable "ZOOKEEPER_CLIENT_PORT" which is used to configure the Zookeeper service.

Start the Services

Now that we have defined our services, we can start them by running the following command in the same directory as the "docker-compose.yml" file:

docker-compose up

This command will start the Kafka and zookeeper services defined in the "docker-compose.yml" file.

Install the Confluent Kafka library

Make sure to install librdkafka which is used by the confluent kafka python package.

To do so in macOS: brew install librdkafka

To interact with the Kafka cluster, we will use the python Confluent Kafka library. To install it, run the following poetry command (you can use pip too):

poetry add confluent-kafka

Creating topics with the python client

create_topic.py

import sys
from confluent_kafka.admin import AdminClient, NewTopic

def create_topic(topic_name):
    conf = {'bootstrap.servers': 'localhost:9092'}
    admin_client = AdminClient(conf)
    new_topic = admin_client.create_topics([NewTopic(topic_name, num_partitions=1, replication_factor=1)])
    for topic, future in new_topic.items():
        try:
            future.result()
            print("Topic {} created successfully".format(topic))
        except Exception as e:
            print("Failed to create topic {}: {}".format(topic, e))

if __name__ == '__main__':
    if len(sys.argv) != 2:
        print("Usage: python {} <topic_name>".format(sys.argv[0]))
        sys.exit(1)
    topic_name = sys.argv[1]
    create_topic(topic_name)

This script uses the AdminClient class from the Confluent Kafka package to create a new topic. It takes a topic name as input, creates a new topic with that name, 1 partition and replication factor 1. If the topic is created successfully, it will print "Topic topic_name"

Create a producer

Create a new python file and import the confluent_kafka library, then create a function that will be responsible for producing messages to the Kafka topic.

`produce_messages.py`

import sys
from confluent_kafka import Producer

def produce_message(topic_name):
    conf = {'bootstrap.servers': 'localhost:9092'}
    producer = Producer(conf)

    def delivery_callback(err, msg):
        if err:
            print('Message delivery failed: {}'.format(err))
        else:
            print('Message delivered to {} [{}]'.format(msg.topic(), msg.partition()))

    for i in range(10):
        message = 'This is message number {}'.format(i)
        producer.produce(topic_name, value=message.encode('utf-8'), callback=delivery_callback)
        producer.poll(0)
        # flush is only needed if you want to synchronously ensure the message was received, but the producer is async
        #producer.flush()

if __name__ == '__main__':
    if len(sys.argv) != 2:
        print("Usage: python {} <topic_name>".format(sys.argv[0]))
        sys.exit(1)
    topic_name = sys.argv[1]
    produce_message(topic_name)

This function takes a topic name as an input and creates a producer object with the bootstrap servers configuration set to "localhost:29092" (which is the default port for Kafka on a single node setup). It then creates a for loop to produce 10 messages to the topic with a simple string value.

Create a consumer

consume_messages.py

import sys
from confluent_kafka import Consumer, KafkaError

def consume_message(topic_name):
    conf = {'bootstrap.servers': 'localhost:9092',
            'group.id': 'mygroup',
            'auto.offset.reset': 'earliest',
            }
    consumer = Consumer(conf)
    consumer.subscribe([topic_name])
    try:
        while True:
            msg = consumer.poll(1.0)
            if msg is None:
                continue
            if msg.error():
                if msg.error().code() == KafkaError._PARTITION_EOF:
                    print('Reached end of topic {} [{}] at offset {}'.format(msg.topic(), msg.partition(), msg.offset()))
                else:
                    print('Error occured: {}'.format(msg.error()))
            else:
                print('Received message: {}'.format(msg.value()))
    except KeyboardInterrupt:
        print("Exiting...")
    finally:
        consumer.close()

if __name__ == '__main__':
    if len(sys.argv) != 2:
        print("Usage: python {} <topic_name>".format(sys.argv[0]))
        sys.exit(1)
    topic_name = sys.argv[1]
    consume_message(topic_name)

This script uses the Consumer class from the Confluent Kafka package to consume messages from a Kafka topic. It takes a topic name as input, creates a consumer for that topic, and enters a while loop to poll for new messages. If a message is received, it is printed to the console.

This script will only display the messages, you can add some logic to handle them like storing them in a database for example.

Also, you can run this script after running the previous script of producing messages to see the messages being consumed.

Run the examples

Now that we have created our producer and consumer functions, you can run them and see the messages being produced and consumed.

Create the test topic:

poetry run python create_topic.py testtopic

Run the consume message script

poetry run python consume_messages.py testtopic

Keep it running and produce some message using:

poetry run python produce_messages.py testtopic

You should see the message output in the consumer command:

Conclusion

In conclusion, setting up Kafka via Docker and Docker Compose is a straightforward process. By using the Confluent Kafka library, it is also easy to interact with the Kafka cluster and build useful producers and consumers. With this setup, you can quickly spin up a Kafka cluster for testing and development purposes and easily add more nodes to the cluster when needed.

Full source code on GitHub.

You don't know what is does? In simple words it will create required services to make a specific project up and running.

I will not cover basic knowledge about docker and docker compose, there are plenty of tutorials/videos doing deep introduction for this.

Objectif

We will have a VSCode + Docker compose setup which will run the project in VSCode dev containers, this will make sure that all services are up and running (DB, Django's runserver, Cache ...) We will also setup useful VSCode extensions which will boost productivity and will be available for anyone using the project with VSCode.

Requirements

• Linux (docker + VSCode + VSCode Dev Containers extension)
• Mac OS (Docker Desktop + VSCode Dev Containers extension)

• Windows (Docker Desktop + VSCode Dev Containers extension)

• Docker

• Docker Desktop
• VSCode
• Dev Containers extension

// Mac M1/M2 users please use Apple Silicon version for Docker Desktop and VSCode for a better performance

VSCode dev container (Developing inside a Container)

Why?

• How many time did you have to install virtualenv in your machine to enable full features of the Django project you're working on?
• Wasn't you annoyed of how you will keep them up to date if your team use Docker or/and Docker compose and sync them.
• What about installing system dependencies needed for compilation for some scenarios (python dev, imaging libraries, librdkafka, libmemcached-dev, postgresql headers if not using psycopg2-binary etc)?
• And how much time did you already spend on this and you will spend on each new install of your OS or some new upgrades?

What about the time spent by your team or team members on that and specially when using different OS (Mac, Linux, Windows)?

What about opening VSCode (No terminal is needed) and then everything is setup for you out of the box, you will only need git, docker/docker desktop and that's all !

This will also keep your machine clean from different dependencies, you don't have to switch between dependencies version or system packages version, Dockerize everything!

For a full documentation on setting up dev containers please check this official link. I will later on refer to some section to it.

Dummy Django project

We will start by cloning a dummy Django project which don't do much and try to add Dockerfile, docker-compose.yml and dev container setup to it.

git clone --branch initial_project_setup https://github.com/MounirMesselmeni/django-vscode-dev-containers.git

This a simple Django project containing the required setup with Dockerfile and docker-compose.yml, we use postgres image for database

Project structure

Dockerfile

FROM python:3.10

ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
ENV LC_ALL=C.UTF-8

WORKDIR /app

COPY requirements.txt /app/

RUN pip install -r requirements.txt

docker-compose.yml

version: "3.9"

services:
  django:
    build:
      context: .
      dockerfile: ./Dockerfile
    command: python manage.py runserver 0.0.0.0:8000
    ports:
      - 8000:8000
    volumes:
      - .:/app:cached
    depends_on:
      - postgres
  postgres:
    image: postgres:15.0-alpine
    user: postgres
    environment:
      - POSTGRES_USER=app
      - POSTGRES_PASSWORD=app
      - POSTGRES_DB=app
    volumes:
      # with named volume we will persist database state E.g after restarting docker/machine
      - postgres-data:/var/lib/postgresql/data
    restart: unless-stopped

volumes:
  postgres-data:

Add a .env file under django_vscode_dev_containers folder

DEBUG=yes
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY=django-insecure-qk0&tbbj(&c9x^t6wu6wkwwg@xox^t0*@(dbo3et03x*+1-avy
ALLOWED_HOSTS=localhost,127.0.0.1,::1,0.0.0.0
DATABASE_URL=psql://app:app@postgres:5432/app

Setting up dev containers

We can do that using VSCode command palette and dev containers extension helper but for the sake of simplicity we will do this by hand.

Create a .devcontainer/devcontainer.json file in the main directory

// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
// https://github.com/microsoft/vscode-dev-containers/tree/v0.245.2/containers/docker-existing-docker-compose
// If you want to run as a non-root user in the container, see .devcontainer/docker-compose.yml.
{
    "name": "Django VSCode Dev Containers",

    // Update the 'dockerComposeFile' list if you have more compose files or use different names.
    // The .devcontainer/docker-compose.yml file contains any overrides you need/want to make.
    "dockerComposeFile": [
        "../docker-compose.yml",
        "docker-compose.yml"
    ],

    // The 'service' property is the name of the service for the container that VS Code should
    // use. Update this value and .devcontainer/docker-compose.yml to the real service name.
    "service": "django",

    // The optional 'workspaceFolder' property is the path VS Code should open by default when
    // connected. This is typically a file mount in .devcontainer/docker-compose.yml
    "workspaceFolder": "/app",

    "runServices": [
        "django",
        "postgres"
    ]

    // Uncomment the next line if you want to keep your containers running after VS Code shuts down.
    // "shutdownAction": "none",

    // Uncomment the next line to run commands after the container is created - for example installing curl.
    // "postCreateCommand": "apt-get update && apt-get install -y curl",

    // Uncomment to connect as a non-root user if you've added one. See https://aka.ms/vscode-remote/containers/non-root.
    // "remoteUser": "vscode"
}

Note on runServices, we define here which services we want VSCode to make sure they are up and running, we could have a redis service which we don't need, or it's essential to the proper functioning of the application so we would like VSCode to take care of running it with docker/docker-compose.

And devcontainers/docker-compose.yml

version: '3.9'
services:
  # Update this to the name of the service you want to work with in your docker-compose.yml file
  django:
    # If you want add a non-root user to your Dockerfile, you can use the "remoteUser"
    # property in devcontainer.json to cause VS Code its sub-processes (terminals, tasks, 
    # debugging) to execute as the user. Uncomment the next line if you want the entire 
    # container to run as this user instead. Note that, on Linux, you may need to 
    # ensure the UID and GID of the container user you create matches your local user. 
    # See https://aka.ms/vscode-remote/containers/non-root for details.
    #
    # user: vscode

    # Overrides default command so things don't shut down after the process ends.
    command: /bin/sh -c "while sleep 1000; do :; done"

Open the Command Palette (Ctrl/Cmd+shift+p) and search for Rebuild and reopen in Container

Now VSCode will open the project inside the container, we can see the docker building command and output by clicking on Starting Dev container show logs

The logs would look like this

We know that we are developing inside a container in the bottom left corner we can see the name we provided in devcontainer.json file

Setting up python extensions

Now we need more productivity by setting up proper python extensions to use code completion etc with Pylance.

Let's add required extensions list into the .devcontainers/devcontainer.json and some custom vscode setting (Yes you can define VSCode settings in dev container and share them with others)

  "settings": {
        "python.terminal.activateEnvironment": false,
        "python.terminal.activateEnvInCurrentTerminal": true,
        "python.pythonPath": "/usr/local/bin/python",
        "python.defaultInterpreterPath": "/usr/local/bin/python",
        "python.languageServer": "Pylance",
        // file formatting options
        "files.trimTrailingWhitespace": true,
        "files.insertFinalNewline": true,
        // files to exclude from all checks
        "files.exclude": {
            "**/*.pyc": true,
            "**/.git": false,
            "**/migrations/*": false
        },
        "python.analysis.extraPaths": [
            "/usr/local/lib/python3.10/site-packages/"
        ],
        "python.analysis.useImportHeuristic": true,
        "python.analysis.autoSearchPaths": true
    },
   "extensions": [
        "ms-python.python",
        "ms-python.vscode-pylance",
        "aaron-bond.better-comments",
        "visualstudioexptteam.vscodeintellicode",
        "mikestead.dotenv"
    ]

Command Palette and search for Rebuild to rebuilt the container and restart:

Now we can see VSCode is installing the extensions inside the dev container

A common problem to this, that sometime Pylance does not work properly from the first time and you need to restart or rebuild, and everytime you add a requirement or change dockerfile VSCode will re-install all the extensions. To fix this we can avoid VSCode extensions reinstalls by adding a persisted volume for those extensions.

Let's modify our Dockerfile to create required extensions folder

FROM python:3.10

ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
ENV LC_ALL=C.UTF-8

# Yes we are using the root user now, it's not encouraged but this is only for demo purposes
RUN mkdir -p /root/.vscode-server/extensions /root/.vscode-server-insiders/extensions

WORKDIR /app

COPY requirements.txt /app/

RUN pip install -r requirements.txt

And add the new docker volume in the .devcontainer/docker-compose.yml

version: '3.9'
services:
  # Update this to the name of the service you want to work with in your docker-compose.yml file
  django:
    # If you want add a non-root user to your Dockerfile, you can use the "remoteUser"
    # property in devcontainer.json to cause VS Code its sub-processes (terminals, tasks, 
    # debugging) to execute as the user. Uncomment the next line if you want the entire 
    # container to run as this user instead. Note that, on Linux, you may need to 
    # ensure the UID and GID of the container user you create matches your local user. 
    # See https://aka.ms/vscode-remote/containers/non-root for details.
    #
    # user: vscode

    # Overrides default command so things don't shut down after the process ends.
    command: /bin/sh -c "while sleep 1000; do :; done"
    volumes:
      - vscode_extensions:/root/.vscode-server/extensions

volumes:
  vscode_extensions:

Now rebuild again using commands palette

We can re-run Rebuild without Cache just to test out if the extensions changes are working properly

Now we don't re-install extensions as they are persisted in the docker volume

We should be able now to have autocompletion working and navigating/going to third party packages like environ in settings.py with Cmd/Ctrl click

To go inside the container (Our case the django/python one) you click in the + button as shown bellow

We can run our runserver from there

./manage.py runserver 0:8000

And access localhost:8000

End

Nice what do you think?

This might looks like a lot of work, but you will set it up once and after it's mostly copy paste and you have your environment easy to work with your colleagues/teammates and simplify the process.

Final result could be find in the github repo:

Part 2 (Next)

• I will add a part 2 soon in which we will cover how to use git from inside the container, this is handy if you want to have some pre-commit scripts and you have no interest to install that on the host machine or if you like using the git integration inside VSCode.
• Setup automatic formatting using black - On save
• Setup automatic imports sorting using isort - On save
• Pylint/Flake8 previews inside VSCode

Pre-commit is a python based tool to enable easy integration of git hooks, and it's supported by plenty of tools like ruff, black, ...

While coding a Django project in a team, you would like to have unified rules and styles to avoid unnecessary changes on each Pull Request another team member will review. After setting up for the first time, this is for me a must for every project to ensure consistency and code quality.

After installing pre-commit, you will need to create a .pre-commit-config.yaml file and run pre-commit install inside the project to activate the git hooks.

Default pre-commit hooks

repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
    -   id: check-yaml
    -   id: end-of-file-fixer
    -   id: trailing-whitespace
-   repo: https://github.com/psf/black
    rev: 24.1.1
    hooks:
    -   id: black

Check this link for a full list of builtins hooks from pre-commit.

isort

The initial example contains configuration of python Black formatting, on top of it I strongly recommend using consistent sorted import via isort

  - repo: https://github.com/pycqa/isort
    rev: 5.13.2
    hooks:
      - id: isort

If you use isort with black you will need to use black profile as follow in pyproject.toml in your root project directory.

[tool.black]
line-length = 88
include = '\.pyi?$'

[tool.isort]
profile = "black"
py_version=311
multi_line_output = 3
line_length = 88
default_section = "THIRDPARTY"
skip = ["migrations"]

Pyupgrade

Another amazing tool you might need to use is pyupgrade, this one will ensure you will use new python feature like new style classes, removing unicode literals, use the python3 super() calls etc.

- repo: https://github.com/asottile/pyupgrade
    rev: v3.15.0
    hooks:
      - id: pyupgrade
        args: ["--py3-plus", "--py311-plus"]

If you use python 3.10 you can change --py39-plus with --py310plus

Djangoupgrade

Next similar tools is djangoupgrade, it will fix deprecated features of Django and make it easy to upgrade your Django projects, unfortunately is does only check against python files, for Django templates it's not doing anything, but I still find it useful.

- repo: https://github.com/adamchainz/django-upgrade
    rev: "1.15.0"
    hooks:
      - id: django-upgrade
        args: [--target-version, "5.0"]

Note the target-version, this depend on the current supported Django version in the project.

Bandit

Another great tool to use is Bandit, Bandit is designed to find common security issues in Python code. To do this Bandit processes each file, builds an AST from it, and runs appropriate plugins against the AST nodes. Once Bandit has finished scanning all the files it generates a report.

- repo: https://github.com/pycqa/bandit
    rev: 1.7.7
    hooks:
      - id: bandit
        args: ["-iii", "-ll"]

Ruff

Ruff is a powerful Python tool that consolidates various code quality checkers like pycodestyle, pyflakes, and mccabe, along with third-party plugins. It ensures the style and quality of Python code through a unified interface.

- repo: https://github.com/charliermarsh/ruff-pre-commit
    # Ruff version.
    rev: "v0.2.1"
    hooks:
      - id: ruff

Now we will need to setup Ruff config in pyproject.toml

[tool.ruff]
line-length = 88
target-version = "py311"
exclude = [
    ".git",
    ".mypy_cache",
    ".pre-commit-cache",
    ".ruff_cache",
    ".tox",
    ".venv",
    "venv",
    "docs",
    "__pycache",
    "**/migrations/*",
]
dummy-variable-rgx = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$"

[tool.ruff.lint.mccabe]
max-complexity = 10

To run ruff manually, once installed (e.g pip install ruff)

ruff check .

Final result

The final .pre-commit-config.yaml

repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
    -   id: check-yaml
    -   id: end-of-file-fixer
    -   id: trailing-whitespace
-   repo: https://github.com/psf/black
    rev: 22.8.0
    hooks:
    -   id: black
- repo: https://github.com/pycqa/isort
    rev: 5.13.2
    hooks:
      - id: isort
- repo: https://github.com/asottile/pyupgrade
    rev: v3.15.0
    hooks:
      - id: pyupgrade
        args: ["--py3-plus", "--py39-plus"]
- repo: https://github.com/adamchainz/django-upgrade
    rev: "1.15.0"
    hooks:
      - id: django-upgrade
        args: [--target-version, "4.0"]
- repo: https://github.com/pycqa/bandit
    rev: 1.7.7
    hooks:
      - id: bandit
        args: ["-iii", "-ll"]
- repo: https://github.com/charliermarsh/ruff-pre-commit
    # Ruff version.
    rev: "v0.2.1"
    hooks:
      - id: ruff

Bonus: CI integration

To make sure no one skip the pre-commit setup, either by intention of if someone forget to install the hooks locally after cloning the project for the first time, you can run pre-commit run --all in your CI.

Caching might be tricky and useful as initial environment setup take some time and therefore you can use the PRE_COMMIT_HOME for specifying which directory is used for the cache

Gitlab CI

precommit:
  image:
    name: python:3.10
  stage: test
  variables:
    PRE_COMMIT_HOME: ".cache/.pre-commit-cache/"
  script:
    - pip install pre-commit
    - pre-commit run --all
  rules:
    - exists:
        - ".pre-commit-config.yaml"
  interruptible: true
  cache:
    key: $CI_PROJECT_NAME-precommit-cache-v1
    paths:
      - .cache/.pre-commit-cache/

Github Actions

name: CI

on:
  create:
    tags:
      - "*" # run on all tags
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2
      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: 3.11
      - name: Install pre-commit
        run: pip install pre-commit
      - name: Cache pre-commit hooks
        id: pre-commit-cache
        uses: actions/cache@v2
        with:
          path: $HOME/.cache/pre-commit
          key: ${{ runner.os }}-pre-commit-hooks-${{ hashFiles('**/.pre-commit-config.yaml') }}
          restore-keys: |
            ${{ runner.os }}-pre-commit-hooks-
      - name: Run pre-commit
        run: |
          pre-commit run --all-files

Project setup

To begin, we will use pipenv to set up our environment. First, we will clone the base repository and then run a few additional commands. One of the benefits of using pipenv is that it automatically creates a virtual environment for us.

git clone -b minimal https://github.com/MounirMesselmeni/restframework-tutorial.git rest_tuto
cd rest_tuto
# installing pipenv as system package
pip install pipenv
# installing project dependencies, use --python 3.6 if it's installed
pipenv install --python3
# activating virtualenv via pipenv
pipenv shell
# run migration
./manage.py migrate
# create a superuser to play with your models via admin
./manage.py createsuperuser
# run django server
./manage.py runserver

Now, you can examine the structure of the models in the courses/models.py file. Then, proceed to the admin area and add some data.

Starting the API

Now, let's create our first API endpoint and ensure it is functioning properly. To start, create a new Python module named api.

mkdir api
touch api/__init__.py
touch api/serializers.py
touch api/views.py
touch api/urls.py

Next, we will create a serializer for the Teacher model in "api/serializers.py". The serializer will instruct Django REST framework on how to present our data in various formats (such as HTML, JSON, or XML) and how to handle data parsing between these formats and Python.

from rest_framework import serializers

from courses.models import Teacher


class TeacherSerializer(serializers.ModelSerializer):
    class Meta:
        model = Teacher
        fields = [
            'id',
            'name',
            'speciality',
        ]

We will be using ModelSerializer which is quite similar to Django's ModelForm, it's a serializer for a model, that helps us to write less code and automatically generates fields based on the model fields including validation and field types.

Now, let's utilize Django REST framework viewsets. ViewSets allows us to perform CRUD operations using a single endpoint:

• Create via POST requests

• Read via GET requests

• Update via PUT requests

• Delete via DELETE requests

We can choose to disable some of them if not needed, but for now, we will use all of them.

Let's create our first viewset in api/views.py.

from rest_framework import viewsets

from . import serializers
from courses import models as courses_models


class TeacherViewSet(viewsets.ModelViewSet):
    queryset = courses_models.Teacher.objects.all()
    serializer_class = serializers.TeacherSerializer

We need to specify a queryset for the ViewSet, which will also be used for listing teachers. If we want to display only teachers with specific filters, we can make that change here.

Now, we will inform Django REST framework to publish our first endpoint or viewset. This can be done manually by adding the viewset to the urls.py file in the traditional Django way, but it's better to use Django REST framework routers for this. They are useful as they will automatically generate URLs with an ID parameter for retrieving a specific teacher, for example.

Let's add this to api.urls.py.

from django.conf.urls import url, include
from rest_framework.routers import DefaultRouter

from . import views

router = DefaultRouter()
router.register(r'teachers', views.TeacherViewSet)

urlpatterns = [
    url(r'^', include(router.urls)),
    url(r'^api-auth/', include('rest_framework.urls', namespace='rest_framework'))
]

Now we have to include api.urls inside our project main urls, rest_tuto/urls.py

from django.contrib import admin
from django.urls import path, include
import api.urls

urlpatterns = [
    path('admin/', admin.site.urls),
    path('api/', include(api.urls)),
]

The api-auth is used for API authentication, we will cover that later.

Now, you can open your browser and navigate to "http://localhost:8000/api/". The HTML displayed is generated by Django REST framework and lists all available endpoints. You're currently viewing the HTML format, but you can also view the JSON format by adding ?format=json to the URL.

Go to the teachers API and experiment with it. Add some data using curl or the Django REST framework HTML form.

# Create teacher
curl -d '{"name": "John", "speciality": "Physics"}' -H "Content-Type: application/json" -X POST http://localhost:8000/api/teachers/

# List teachers
curl -X GET http://localhost:8000/api/teachers/

# Get one teacher
curl -X GET http://localhost:8000/api/teachers/1/

# Delete teacher
curl -X DELETE http://localhost:8000/api/teachers/1/

# Create teacher another time
curl -d '{"name": "John", "speciality": "Physics"}' -H "Content-Type: application/json" -X POST http://localhost:8000/api/teachers/

# Update one teacher
curl -d '{"name": "John2", "speciality": "Physics"}' -H "Content-Type: application/json" -X PUT http://localhost:8000/api/teachers/2/

# Create teacher with missing data
curl -d '{"speciality": "Physics"}' -H "Content-Type: application/json" -X POST http://localhost:8000/api/teachers/
# Will complain and show errors in json

Now, let's explore relationships further, based on the models we have already set up.

Custom serializer method

t can be useful for example, for calculated fields. In the case of students, we have their birthdate and we want to return their calculated age for each student. To achieve this, we can use the Django REST framework serializer method field SerializerMethodField. The default method name will be get_fieldname where fieldname is the name of your field.

class StudentSerializer(serializers.ModelSerializer):
    age = serializers.SerializerMethodField()

    def get_age(self, obj):
        today = datetime.date.today()
        return (
            today.year - obj.birth_date.year - ((today.month, today.day) < (obj.birth_date.month,
                                                                            obj.birth_date.day))
        )
    class Meta:
        model = Student
        fields = [
            'id',
            'name',
            'birth_date',
            'current_level',
            'age',
        ]

Note that obj will be an instance of Student model.

Relationships

By default, Django REST framework will display related fields as IDs. If you want to include the entire related object, you will need to specify which serializer should be used. In the case of many-to-many relationships, you must pass the serializer with "many=True"

Let's take a look at how the "CourseSerializer" should be implemented:

class CourseSerializer(serializers.ModelSerializer):
    teacher = TeacherSerializer()
    students = StudentSerializer(many=True, read_only=True)

    class Meta:
        model = Course
        fields = [
            'id',
            'name',
            'teacher',
            'level',
            'students',
        ]

We used read_only=True to tell drf to use that field only while reading data, for creation this field will be ignored, if disabled drf will raise an exception when trying to post data complaining that you should deal on your own with m2m fields in the perfom_create method.

Here an overview of how api app looks like right now, as it's time to play around with it.

api/serializers.py

import datetime
from rest_framework import serializers

from courses.models import Teacher, Student, Course


class TeacherSerializer(serializers.ModelSerializer):
    class Meta:
        model = Teacher
        fields = [
            'id',
            'name',
            'speciality',
        ]


class StudentSerializer(serializers.ModelSerializer):
    age = serializers.SerializerMethodField()

    def get_age(self, obj):
        today = datetime.date.today()
        return (
            today.year - obj.birth_date.year - ((today.month, today.day) < (obj.birth_date.month,
                                                                            obj.birth_date.day))
        )
    class Meta:
        model = Student
        fields = [
            'id',
            'name',
            'birth_date',
            'current_level',
            'age',
        ]


class CourseSerializer(serializers.ModelSerializer):
    teacher = TeacherSerializer()
    students = StudentSerializer(many=True, read_only=True)

    class Meta:
        model = Course
        fields = [
            'id',
            'name',
            'teacher',
            'level',
            'students',
        ]

api/views.py

from rest_framework import viewsets

from . import serializers
from courses import models as courses_models


class TeacherViewSet(viewsets.ModelViewSet):
    queryset = courses_models.Teacher.objects.all()
    serializer_class = serializers.TeacherSerializer


class StudentViewSet(viewsets.ModelViewSet):
    queryset = courses_models.Student.objects.all()
    serializer_class = serializers.StudentSerializer


class CourseViewSet(viewsets.ModelViewSet):
    queryset = courses_models.Course.objects.all()
    serializer_class = serializers.CourseSerializer

api/urls.py

from django.conf.urls import url, include
from rest_framework.routers import DefaultRouter

from . import views

router = DefaultRouter()
router.register(r'teachers', views.TeacherViewSet)
router.register(r'students', views.StudentViewSet)
router.register(r'courses', views.CourseViewSet)

urlpatterns = [
    url(r'^', include(router.urls)),
    url(r'^api-auth/', include('rest_framework.urls', namespace='rest_framework'))
]

Filtering

To help with quick filtering implementation, we will use the django-filter package. It's already installed, but let's add it to our INSTALLED_APPS.

INSTALLED_APPS = [
    ...
    'django_filters',
]

Now, we can configure Django REST framework to use django-filter as our global filter backend. This can also be done on a per-view level. Add the following to your settings.py file.

REST_FRAMEWORK = {
    'DEFAULT_FILTER_BACKENDS': (
        'django_filters.rest_framework.DjangoFilterBackend',
    ),
}

Create a api/filters.py where we will add filtering for the teacher endpoint

import django_filters
from django_filters import rest_framework as filters

from courses import models 

class TeacherFilter(filters.FilterSet):
    name = django_filters.CharFilter(lookup_expr='icontains')

    class Meta:
        model = models.Teacher
        fields = ['name']

The last thing we need is to specify the filter_class in the TeacherViewSet

class TeacherViewSet(viewsets.ModelViewSet):
    queryset = courses_models.Teacher.objects.all()
    serializer_class = serializers.TeacherSerializer
    filter_class = filters.TeacherFilter

The lookup_expr setting specifies to django-filter which Django ORM lookup it will use. Let's test it out using curl and see the results.

curl -X GET http://localhost:8000/api/teachers/\?name\=john

Now, we want to filter for students based on their age, we will need to use a custom method for filtering as we will calculate the right year to look for. To calculate the difference in years, we will need to install python-dateutil.

pipenv install python-dateutil

Then modify our filters.py:

import datetime
from dateutil.relativedelta import relativedelta
import django_filters
from django_filters import rest_framework as filters

from courses import models 

class TeacherFilter(filters.FilterSet):
    name = django_filters.CharFilter(lookup_expr='icontains')

    class Meta:
        model = models.Teacher
        fields = ['name']


class StudentFilter(filters.FilterSet):
    age = django_filters.NumberFilter(method='age_filter')

    def age_filter(self, queryset, name, value):
        today = datetime.date.today()
        age = (today - relativedelta(years=value)).year
        return queryset.filter(birth_date__year=age)

    class Meta:
        model = models.Student
        fields = ['age']

Let's try it out, maybe you will need to add some students, via curl, drf forms, Django admin or Django shell

curl -X GET http://localhost:8000/api/students/?age=20

Pagination

Configure pagination in settings.py, pagination can help when dealing with a lot of data, imagine we have a list of 10000 students, is will not be good to return this big list, we can return 100 per 100.

Be careful as pagination will modify the returned format:

{
    "count": 1023
    "next": "https://api.example.org/accounts/?limit=100&offset=500",
    "previous": "https://api.example.org/accounts/?limit=100&offset=300",
    "results": [
       …
    ]
}

REST_FRAMEWORK = {
    'DEFAULT_FILTER_BACKENDS': ('django_filters.rest_framework.DjangoFilterBackend',),
    'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.LimitOffsetPagination',
    'PAGE_SIZE': 100,
}

Authentication

It's best to use a session if you would like to use DRF build in HTML for testing and trying out the API endpoints. There are many authentication backends supported by DRF, there are also some other third-party apps that provide more backends like JWT (You can check)

We will now configure session and token authentication, let's configure DRF in settings.py

REST_FRAMEWORK = {
    'DEFAULT_FILTER_BACKENDS': ('django_filters.rest_framework.DjangoFilterBackend',),
    'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.LimitOffsetPagination',
    'PAGE_SIZE': 100,
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.SessionAuthentication',
        'rest_framework.authentication.TokenAuthentication',
    ),
}

Now we have to add DRF auth app to INSTALLED_APPS

INSTALLED_APPS = (
    ...
    'rest_framework.authtoken',
)

We can also expose an obtain token endpoint, where users can send a username/password and have their token back. To do so we have to modify api/urls.py

from django.conf.urls import url, include
from rest_framework.routers import DefaultRouter
from rest_framework.authtoken.views import obtain_auth_token

from . import views

router = DefaultRouter()
router.register(r'teachers', views.TeacherViewSet)
router.register(r'students', views.StudentViewSet)
router.register(r'courses', views.CourseViewSet)

urlpatterns = [
    url(r'^', include(router.urls)),
    url(r'^api-auth/', include('rest_framework.urls', namespace='rest_framework')),
    url(r'^api-token-auth/', obtain_auth_token),
]

Now we have to run migrate to create token app tables:

./manage.py migrate

Let's get the token via the obtain token view:

curl -X POST http://localhost:8000/api/api-token-auth/ \
-H 'Content-type: application/json' \
-d '{"username": "admin", "password": "yourAdminPassword"}'

{"token":"c98d345c4eeeb5529e184526b4f3effbc8957251"}%

We can now turn on permission, we can set up permission per view/viewset level or globally via settings.py

REST_FRAMEWORK = {
    'DEFAULT_FILTER_BACKENDS': ('django_filters.rest_framework.DjangoFilterBackend',),
    'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.LimitOffsetPagination',
    'PAGE_SIZE': 100,
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.SessionAuthentication',
        'rest_framework.authentication.TokenAuthentication',
    ),
    'DEFAULT_PERMISSION_CLASSES': ('rest_framework.permissions.IsAuthenticated',),
}

We still can override permission per view level if we want. Let's see if the curl call will return a 403 status code

curl -X GET http://localhost:8000/api/teachers/\?name\=john                                                                        
{"detail":"Authentication credentials were not provided."}%

Now we should provide the token which we got as header: Authorization: Token c98d345c4eeeb5529e184526b4f3effbc8957251 Do not forget to replace c98d345c4eeeb5529e184526b4f3effbc8957251 by your local token

curl -H 'Authorization: Token c98d345c4eeeb5529e184526b4f3effbc8957251' -X GET http://localhost:8000/api/teachers/\?name\=john

{"count":4,"next":null,"previous":null,"results":[{"id":2,"name":"John2","speciality":"Physics"},{"id":3,"name":"John","speciality":"Physics"},{"id":4,"name":"John","speciality":"Physics"},{"id":5,"name":"John","speciality":"Physics"}]}%

Nested API

We will use drf-extension to add nested support. We will be able to list all courses of a teacher directly via the teacher detail endpoint /api/teacher/1/courses/ I'm not using this too often as I prefer to use flat endpoints, but sometime it may be requested and I will show how it's possible and easy.

We will install drf-extensions which have a bunch of helpers, we will need right now the nested router.

pipenv install git+https://github.com/chibisov/drf-extensions.git@master#egg=drf_extensions

PS: Until the time of writing this, drf-extension official release does not support Django 2.0. that's why we are using the master branch directly from Github. You may not need to use git link as soon as a new version will be released (> 0.3.1)

Let's modify api.urls.py

from django.conf.urls import url, include
from rest_framework_extensions.routers import ExtendedDefaultRouter
from rest_framework.authtoken.views import obtain_auth_token

from . import views

router = ExtendedDefaultRouter()
router.register(r'teachers', views.TeacherViewSet).register(
    r'courses', views.CourseViewSet, base_name='courses', parents_query_lookups=['teacher'],
)
router.register(r'students', views.StudentViewSet)

urlpatterns = [
    url(r'^', include(router.urls)),
    url(r'^api-auth/', include('rest_framework.urls', namespace='rest_framework')),
    url(r'^api-token-auth/', obtain_auth_token),
]

As you see, we can chain router registrations, keep in mind this is only supported for reading, for creation you will need to do a bit more to get for example the current teacher while creating a new course.

And then api/views/py to add NestedViewSetMixin to our viewsets

from rest_framework import viewsets
from rest_framework_extensions.mixins import NestedViewSetMixin

from . import serializers
from . import filters
from courses import models as courses_models


class TeacherViewSet(NestedViewSetMixin, viewsets.ModelViewSet):
    queryset = courses_models.Teacher.objects.all()
    serializer_class = serializers.TeacherSerializer
    filter_class = filters.TeacherFilter


class StudentViewSet(NestedViewSetMixin, viewsets.ModelViewSet):
    queryset = courses_models.Student.objects.all()
    serializer_class = serializers.StudentSerializer
    filter_class = filters.StudentFilter


class CourseViewSet(NestedViewSetMixin, viewsets.ModelViewSet):
    queryset = courses_models.Course.objects.all()
    serializer_class = serializers.CourseSerializer

Now you can try /api/teachers/1/courses/

curl -H 'Authorization: Token c98d345c4eeeb5529e184526b4f3effbc8957251' -X GET http://localhost:8000/api/teachers/1/courses/

{"count":1,"next":null,"previous":null,"results":[{"id":1,"name":"c1","teacher":{"id":1,"name":"Mounir Messelmeni","speciality":"Python"},"level":2,"students":[{"id":1,"name":"John Wick","birth_date":"1990-02-14","current_level":3,"age":27},{"id":2,"name":"JN","birth_date":"2017-12-15","current_level":2,"age":0}]}]}%

I will demonstrate some simple examples on how to order different type of list. You will realise how simple it's.

A list of integer

There are two ways to do so, using the sort method of the list object or using python builtin method sorted. The main difference is that the list's sort method will change the list, but sorted will creates a new list from the old and returns the new one will.

  In [1]: my_list1 = [4, 8, 0, 3, 7, 2]

  In [2]: my_list1.sort()

  In [3]: my_list1
  Out[3]: [0, 2, 3, 4, 7, 8]

  In [4]: my_list2 = [4, 8, 0, 3, 7, 2]

  In [5]: sorted(my_list2)
  Out[5]: [0, 2, 3, 4, 7, 8]

  In [6]: my_list2
  Out[6]: [4, 8, 0, 3, 7, 2]

A list of characters or strings

This apply the same to a list of strings or characters which will be ordered by the ascii code of the first characters. But pay attention that it will be case-sensitive sorting, to do so we have to use the key parameter to define what the sorting should be based on.

  In [1]: my_list1 = ['z', 'Da', 'dC', 'b', 'a']

  In [2]: my_list1.sort()

  In [3]: my_list1
  Out[3]: ['Da', 'a', 'b', 'dC', 'z']

  In [4]: my_list1.sort(key=str.lower)

  In [5]: my_list1
  Out[5]: ['a', 'b', 'Da', 'dC', 'z']

  In [6]: my_list2 = ['z', 'Da', 'dC', 'b', 'a']

  In [7]: sorted(my_list2, key=str.lower)
  Out[7]: ['a', 'b', 'Da', 'dC', 'z']

A list of dictionnary based on value

Here we need to use lambda functions, there are some other way like using attrgetter from the builtin operator module from python but I prefer lambda as they are inline and more readable (in this case)

  In [1]: my_list1 = [dict(name='Tom', age=25), dict(name='John', age=50), dict(name='Anna', age=20)]

  In [2]: my_list1.sort(key=lambda x: x['age'])

  In [3]: my_list1
  Out[3]: 
  [{'age': 20, 'name': 'Anna'},
   {'age': 25, 'name': 'Tom'},
   {'age': 50, 'name': 'John'}]

  In [4]: my_list2 = [dict(name='Tom', age=25), dict(name='John', age=50), dict(name='Anna', age=20)]

  In [5]: sorted(my_list2, key=lambda x: x['age'])
  Out[5]: 
  [{'age': 20, 'name': 'Anna'},
   {'age': 25, 'name': 'Tom'},
   {'age': 50, 'name': 'John'}]

A list of object based on some attribute

Here we need can also to use lambda functions, it's very similar to the dictionary ordering but instead of accessing the dictionary per key we will access the object attribute

  In [1]: class Person(object):
     ...:     def __init__(self, name, age):
     ...:         self.name = name
     ...:         self.age = age

  In [2]: my_list1 = [Person(name='Tom', age=25), Person(name='John', age=50), Person(name='Anna', age=20)]

  In [3]: my_list1.sort(key=lambda x: x.age)

  In [4]: my_list1
  Out[4]: 
  [<__main__.Person at 0x7fa49abdaf50>,
   <__main__.Person at 0x7fa49abda590>,
   <__main__.Person at 0x7fa49abda250>]

Reverse a list or order a list in a descending order

Both of the method we used (list.sort and sorted) support a boolean parameter called reverse, when set to True the order will be descending.

  In [1]: my_list1 = [dict(name='Tom', age=25), dict(name='John', age=50), dict(name='Anna', age=20)]

  In [2]: my_list1.sort(key=lambda x: x['age'], reverse=True)

  In [3]: my_list1
  Out[3]: 
  [{'age': 50, 'name': 'John'},
   {'age': 25, 'name': 'Tom'},
   {'age': 20, 'name': 'Anna'}]

  In [4]: my_list2 = [dict(name='Tom', age=25), dict(name='John', age=50), dict(name='Anna', age=20)]

  In [5]: sorted(my_list2, key=lambda x: x['age'], reverse=True)
  Out[5]: 
  [{'age': 50, 'name': 'John'},
   {'age': 25, 'name': 'Tom'},
   {'age': 20, 'name': 'Anna'}]

Reverse a list without sorting it

We have the same logic between list.sort and sorted as for list.reverse and reversed, EXCEPT that reversed return a listreverseiterator object instead of a list.

  In [1]: my_list1 = [dict(name='Tom', age=25), dict(name='John', age=50), dict(name='Anna', age=20)]

  In [2]: my_list1.reverse()

  In [3]: my_list1
  Out[3]: 
  [{'age': 20, 'name': 'Anna'},
   {'age': 50, 'name': 'John'},
   {'age': 25, 'name': 'Tom'}]

  In [4]: my_list2 = [dict(name='Tom', age=25), dict(name='John', age=50), dict(name='Anna', age=20)]

  In [5]: list(reversed(my_list2))
  Out[5]: 
  [{'age': 20, 'name': 'Anna'},
   {'age': 50, 'name': 'John'},
   {'age': 25, 'name': 'Tom'}]

  In [6]: my_list2
  Out[6]: 
  [{'age': 25, 'name': 'Tom'},
   {'age': 50, 'name': 'John'},
   {'age': 20, 'name': 'Anna'}]

I used casted the return of the reversed as I mentioned above that it's returned result is not a list but an iterator. I recommend to cast the return of reversed if you will not just iterate over it or to use list comprehension.

  In [1]: my_list2 = [dict(name='Tom', age=25), dict(name='John', age=50), dict(name='Anna', age=20)]
  In [2]: reversed(my_list2)
  Out[3]: <listreverseiterator at 0x7fe9508b5b50>

If you are new to Python and you don't know about such kind of usage, I believe you will wonder how great it's to use such a cool language. All the examples were simple and there are plenty of other ways to do each of them, some could be more efficient regarding performance when dealing with larger list's elements and some could be more readable, I took the most simple ways I'm aware of.

The solution to this problem is either to redefine the logic to use prefetched data or more clean to make these methods now if there already prefetched objects, that's the best solution in my cases:

A basic models example models.py:

    class Teacher(models.Model):
        name = models.CharField(max_length=255)

        def students_count(self):
            return self.student_set.filter(age__gte=20).count()

    class Student(models.Model):
        name = models.CharField(max_length=255)
        teacher = models.ForeignKey(Teacher)
        age = models.IntegerField(default=20)

    for t in Teacher.objects.all():
        print(teacher.students_count())
    # This will results into multiple DB queries equal to the number of teachers + 1
    # If we have 100 teachers in the DB, this will be causing 101 queries.

Let's use prefetch_related:

    for t in Teacher.objects.all().prefetch_related('student_set'):
        print(teacher.students_count())
    # Our prefetch have no effect as the method students_count is using filter which
    # will ignore the prefetched objects

We have to fix the method students_count to consider checking for prefetched objects:

    class Teacher(models.Model):
        name = models.CharField(max_length=255)

        def students_count(self):
            if hasattr(self, '_prefetched_objects_cache') and 'student' in self._prefetched_objects_cache:
                return len([x for x in self.student_set.all() if x.age >= 20])
            return self.student_set.filter(age__gte=20).count()

This way even calling students_count in a single instance without calling prefetch_related on the queryset will use the filter instead of the prefetched.

The solution to this problem is either to redefine the logic to use prefetched data or more clean to make these methods now if there already pre-feteched objects, that's the best solution in my cases:

A basic models example models.py:

    class Teacher(models.Model):
        name = models.CharField(max_length=255)

        def students_count(self):
            return self.student_set.filter(age__gte=20).count()

    class Student(models.Model):
        name = models.CharField(max_length=255)
        teacher = models.ForeignKey(Teacher)
        age = models.IntegerField(default=20)

    for t in Teacher.objects.all():
        print(teacher.students_count())
    # This will results to the number of teachers + 1 SQL queries

Let's use prefetch_related:

    for t in Teacher.objects.all().prefetch_related('student_set'):
        print(teacher.students_count())
    # Our prefetch have no effect as the method students_count is using filter which
    # will ignore the prefetched objects

We have to fix the method students_count to consider checking for prefetched objects:

    class Teacher(models.Model):
        name = models.CharField(max_length=255)

        def students_count(self):
            if hasattr(self, '_prefetched_objects_cache') and 'student' in self._prefetched_objects_cache:
                return len([x for x in self.student_set.all() if x.age >= 20])
            return self.student_set.filter(age__gte=20).count()

This way even calling students_count in a single instance without calling prefetch_related on the queryset will use the filter instead of the prefetched, this can be useful in many cases but I better to use this only when you need to.

Thanks for reading

Here we will see how to use annotate to add this field and how to order it. You only have to define admin_order_field as in the following.

A basic models example models.py:

    class Teacher(models.Model):
        name = models.CharField(max_length=255)

    class Student(models.Model):
        name = models.CharField(max_length=255)
        teacher = models.ForeignKey(Teacher)

And the admin.py

    @admin.register(Teacher)
    class TeacherAdmin(admin.ModelAdmin):
        list_display = (
            'id',
            'students_count',
        )

        def get_queryset(self, request):
            return Teacher.objects.annotate(students_count=Count('student'))

        def students_count(self, obj):
            return obj.students_count

        students_count.short_description = _('Students count')
        students_count.admin_order_field = 'students_count'

UPDATE (20/10/2022): With Django >= 3.2 you can use the new display decorator

    @admin.register(Teacher)
    class TeacherAdmin(admin.ModelAdmin):
        list_display = (
            'id',
            'students_count',
        )

        def get_queryset(self, request):
            return Teacher.objects.annotate(students_count=Count('student'))

        @admin.display(
            ordering='students_count',
            description=_('Students count'),
        )
        def students_count(self, obj):
            return obj.students_count

I found this awesome snippets that's helped me to achieve this, just in my case I needed to add a line of code to show only options with parent.

Example

Here's a model example:

    class Category(models.Model):

        name = models.CharField(max_length=100)
        parent = models.ForeignKey('self', null=True, blank=True)

        def __unicode__(self):
            return u'{0}'.format(self.name)

I created a form that use this model:

    class ExampleForm(forms.Form):

        category = GroupedModelChoiceField(
            label=_('Category'),
            group_by_field='parent',
            queryset=Category.objects.all(),
        )

The snippet show categories with None label, I don't need to make user able to select Categories, only childs can be selected. So, I just edited the line 50 of the snippet to remove group categories.

    from itertools import groupby

    from django.forms.models import ModelChoiceIterator, ModelChoiceField


    class GroupedModelChoiceField(ModelChoiceField):

        def __init__(self, group_by_field, group_label=None, *args, **kwargs):
            """
            group_by_field is the name of a field on the model
            group_label is a function to return a label for each choice group
            """
            super(GroupedModelChoiceField, self).__init__(*args, **kwargs)
            self.group_by_field = group_by_field
            if group_label is None:
                self.group_label = lambda group: group
            else:
                self.group_label = group_label

        def _get_choices(self):
            """
            Exactly as per ModelChoiceField except returns new iterator class
            """
            if hasattr(self, '_choices'):
                return self._choices
            return GroupedModelChoiceIterator(self)
        choices = property(_get_choices, ModelChoiceField._set_choices)


    class GroupedModelChoiceIterator(ModelChoiceIterator):

        def __iter__(self):
            if self.field.empty_label is not None:
                yield (u"", self.field.empty_label)
            if self.field.cache_choices:
                if self.field.choice_cache is None:
                    self.field.choice_cache = [
                        (self.field.group_label(group), [
                         self.choice(ch) for ch in choices])
                        for group, choices in groupby(
                            self.queryset.all(),
                            key=lambda row: getattr(row, self.field.group_by_field))
                    ]
                for choice in self.field.choice_cache:
                    yield choice
            else:
                for group, choices in groupby(
                        self.queryset.all(),
                        key=lambda row: getattr(
                            row, self.field.group_by_field)):
                    if group is not None: #Line added
                        yield (
                            self.field.group_label(group),
                            [self.choice(ch) for ch in choices])

Okay, now this custom field work, but what if I need a grouped select multiple with ModelMultipleChoiceField ?

We can just add a new field like GroupedModelChoiceField that inherits from ModelMultipleChoiceField

    from django.forms.models import ModelMultipleChoiceField


    class GroupedMultipleModelChoiceField(ModelMultipleChoiceField):

        def __init__(self, group_by_field, group_label=None, *args, **kwargs):
            """
            group_by_field is the name of a field on the model
            group_label is a function to return a label for each choice group
            """
            super(GroupedMultipleModelChoiceField, self).__init__(*args, **kwargs)
            self.group_by_field = group_by_field
            if group_label is None:
                self.group_label = lambda group: group
            else:
                self.group_label = group_label

        def _get_choices(self):
            """
            Exactly as per ModelChoiceField except returns new iterator class
            """
            if hasattr(self, '_choices'):
                return self._choices
            return GroupedModelChoiceIterator(self)
        choices = property(_get_choices, ModelMultipleChoiceField._set_choices)

You have a model that contains a ManyToMany field without through model, Django in this case take care itself for the join table, it create a join table that contains three columns: ID, pk of model1 and pk of model2 with a unique together constraint. At a specific time you want to add new column to the join table. Alright, with South installed you can do it easly but South doesn't import your old data it's just delete old table. How to import data from old schema to the new one?

Let's make a concret example to be able to understand this case briefly.

• Create a Django project: django-admin.py startproject migration_tutorial
• Go into project: cd migration_tutorial
• Create a Django app named studies: python manage.py startapp studies
• In migration_tutorial/migration_tutorial/settings.py change database config to sqlite and add studies to INSTALLED_APPS
• We add also South to our INSTALLED_APPS

You should read South documentation if you are not familiar with.

Here's our changes in settings.py:

    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.postgresql_psycopg2',
            'NAME': 'migration_tutorial',
            'USER': 'postgres',
            'PASSWORD': '',
            'HOST': '',
            'PORT': '',
        }
    }

    INSTALLED_APPS = (
        'django.contrib.auth',
        'django.contrib.contenttypes',
        'django.contrib.sessions',
        'django.contrib.sites',
        'django.contrib.messages',
        'django.contrib.staticfiles',
        'django.contrib.admin',

        'south',

        'studies',
    )

Let's create our model

We are going to create a simple model for Professor, Student and Course application.

• A Professor teach many Students
• A Student is taught by many Professors

So we have a ManyToMany relation between Students and Professors. Here's our Django model under the studies app

    from django.db import models


    class Professor(models.Model):

        first_name = models.CharField(max_length=50)
        last_name  = models.CharField(max_length=50)

        def __unicode__(self):
            return u"%s %s" %(self.first_name, self.last_name)


    class Student(models.Model):

        first_name = models.CharField(max_length=50)
        last_name  = models.CharField(max_length=50)
        professors = models.ManyToManyField(Professor)

        def __unicode__(self):
            return u"%s %s" %(self.first_name, self.last_name)

Run the project

Alright, let's run our project:

• Init South Migration: python manage.py schemamigration studies --init
• Sync Database: python manage.py syncdb --noinput
• Migrate our app: python manage.py migrate, in this way South take care of schema creation

Now, we are going to insert some data using shell:

Enter in shell mode: python manage.py shell

    >>> from studies.models import Professor, Student
    >>> professor1 = Professor(first_name='Joshua', last_name='Smith')
    >>> professor1.save()
    >>> professor2 = Professor(first_name='Ethan', last_name='Brown')
    >>> professor2.save()
    >>> student1 = Student(first_name='Justin', last_name='Martin')
    >>> student1.save()
    >>> student2 = Student(first_name='Tom', last_name='Taylor')
    >>> student2.save()

    #Let's add professor 1 to Student one
    >>> student1.professors.add(professor1)

    #Check if data is persisted to database
    >>> student1 = Student.objects.get(first_name='Justin')
    >>> student1.professors.all()

    #Let's add professor2 to student1 too
    >>> student1.professors.add(professor2)

    #Professor2 to student2
    >>> student2.professors.add(professor2)

Note that when working with Django ManyToManyField without through, Django provide some useful method like add method we just use.

Our migration case

Now think if we had this model in production, we have many Students and Professors relation stored in our database. At a specific time we want to add course name taught by Professor to a student, this is a join table column. We are going no to use a through model to add this attribute.

Here's our updated models:

    from django.db import models


    class Professor(models.Model):

        first_name = models.CharField(max_length=50)
        last_name  = models.CharField(max_length=50)

        def __unicode__(self):
            return u"%s %s" %(self.first_name, self.last_name)


    class Student(models.Model):

        first_name = models.CharField(max_length=50)
        last_name  = models.CharField(max_length=50)
        professors = models.ManyToManyField(Professor, through='Course')

        def __unicode__(self):
            return u"%s %s" %(self.first_name, self.last_name)


    class Course(models.Model):

        name      = models.CharField(max_length=50)
        professor = models.ForeignKey(Professor)
        student   = models.ForeignKey(Student)

        class Meta:
            unique_together = ['professor', 'student']

        def __unicode__(self):
            return u"%s" %self.name

Now, we are going to use South to migrate our schemas. Run python manage.py schemamigration studies --auto.

South create a migration script under studies/migrations named 0002_auto__add_course__add_unique_course_professor_student.py

Here's it:

    # -*- coding: utf-8 -*-
    import datetime
    from south.db import db
    from south.v2 import SchemaMigration
    from django.db import models


    class Migration(SchemaMigration):

        def forwards(self, orm):
            # Adding model 'Course'
            db.create_table(u'studies_course', (
                (u'id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
                ('name', self.gf('django.db.models.fields.CharField')(max_length=50)),
                ('professor', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['studies.Professor'])),
                ('student', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['studies.Student'])),
            ))
            db.send_create_signal(u'studies', ['Course'])

            # Adding unique constraint on 'Course', fields ['professor', 'student']
            db.create_unique(u'studies_course', ['professor_id', 'student_id'])

            # Removing M2M table for field professors on 'Student'
            db.delete_table(db.shorten_name(u'studies_student_professors'))


        def backwards(self, orm):
            # Removing unique constraint on 'Course', fields ['professor', 'student']
            db.delete_unique(u'studies_course', ['professor_id', 'student_id'])

            # Deleting model 'Course'
            db.delete_table(u'studies_course')

            # Adding M2M table for field professors on 'Student'
            m2m_table_name = db.shorten_name(u'studies_student_professors')
            db.create_table(m2m_table_name, (
                ('id', models.AutoField(verbose_name='ID', primary_key=True, auto_created=True)),
                ('student', models.ForeignKey(orm[u'studies.student'], null=False)),
                ('professor', models.ForeignKey(orm[u'studies.professor'], null=False))
            ))
            db.create_unique(m2m_table_name, ['student_id', 'professor_id'])


        models = {
            u'studies.course': {
                'Meta': {'unique_together': "(['professor', 'student'],)", 'object_name': 'Course'},
                u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
                'name': ('django.db.models.fields.CharField', [], {'max_length': '50'}),
                'professor': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['studies.Professor']"}),
                'student': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['studies.Student']"})
            },
            u'studies.professor': {
                'Meta': {'object_name': 'Professor'},
                'first_name': ('django.db.models.fields.CharField', [], {'max_length': '50'}),
                u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
                'last_name': ('django.db.models.fields.CharField', [], {'max_length': '50'})
            },
            u'studies.student': {
                'Meta': {'object_name': 'Student'},
                'first_name': ('django.db.models.fields.CharField', [], {'max_length': '50'}),
                u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
                'last_name': ('django.db.models.fields.CharField', [], {'max_length': '50'}),
                'professors': ('django.db.models.fields.related.ManyToManyField', [], {'to': u"orm['studies.Professor']", 'through': u"orm['studies.Course']", 'symmetrical': 'False'})
            }
        }

        complete_apps = ['studies']

There's two main methods:

• forwards: used to apply new changes
• backwards: used to revert new changes

We are going now to add custom code to import existing relation into the new schema Course. For the attribute name in Course I'm going to insert mathematics, in some case we need some details to know what we are going to put in the added column.

We insert our custom code in the forwards method, it load data from old schema generated by Django and push it into the new schema Course.

    # -*- coding: utf-8 -*-
    from south.utils import datetime_utils as datetime
    from south.db import db
    from south.v2 import SchemaMigration
    from django.db import models

    from studies.models import Student, Professor, Course


    class OldJoinMock(models.Model):

        class Meta:
            db_table = u'studies_student_professors'

        student = models.ForeignKey(Student)
        professor = models.ForeignKey(Professor)


    class Migration(SchemaMigration):

        def forwards(self, orm):
            # Adding model 'Course'
            db.create_table(u'studies_course', (
                (u'id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
                ('name', self.gf('django.db.models.fields.CharField')(max_length=50)),
                ('professor', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['studies.Professor'])),
                ('student', self.gf('django.db.models.fields.related.ForeignKey')(to=orm['studies.Student'])),
            ))
            db.send_create_signal(u'studies', ['Course'])

            # Adding unique constraint on 'Course', fields ['professor', 'student']
            db.create_unique(u'studies_course', ['professor_id', 'student_id'])

            #Our custom code here, before delete_table and after create Course table
            Course.objects.bulk_create([
                Course(
                    student=data.student,
                    professor=data.professor,
                    name='mathematics')
                for data in OldJoinMock.objects.all()
            ])

            # Removing M2M table for field professors on 'Student'
            db.delete_table(db.shorten_name(u'studies_student_professors'))


        def backwards(self, orm):
            # Removing unique constraint on 'Course', fields ['professor', 'student']
            db.delete_unique(u'studies_course', ['professor_id', 'student_id'])

            # Deleting model 'Course'
            db.delete_table(u'studies_course')

            # Adding M2M table for field professors on 'Student'
            m2m_table_name = db.shorten_name(u'studies_student_professors')
            db.create_table(m2m_table_name, (
                ('id', models.AutoField(verbose_name='ID', primary_key=True, auto_created=True)),
                ('student', models.ForeignKey(orm[u'studies.student'], null=False)),
                ('professor', models.ForeignKey(orm[u'studies.professor'], null=False))
            ))
            db.create_unique(m2m_table_name, ['student_id', 'professor_id'])


        models = {
            u'studies.course': {
                'Meta': {'unique_together': "(['professor', 'student'],)", 'object_name': 'Course'},
                u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
                'name': ('django.db.models.fields.CharField', [], {'max_length': '50'}),
                'professor': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['studies.Professor']"}),
                'student': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['studies.Student']"})
            },
            u'studies.professor': {
                'Meta': {'object_name': 'Professor'},
                'first_name': ('django.db.models.fields.CharField', [], {'max_length': '50'}),
                u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
                'last_name': ('django.db.models.fields.CharField', [], {'max_length': '50'})
            },
            u'studies.student': {
                'Meta': {'object_name': 'Student'},
                'first_name': ('django.db.models.fields.CharField', [], {'max_length': '50'}),
                u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
                'last_name': ('django.db.models.fields.CharField', [], {'max_length': '50'}),
                'professors': ('django.db.models.fields.related.ManyToManyField', [], {'to': u"orm['studies.Professor']", 'through': u"orm['studies.Course']", 'symmetrical': 'False'})
            }
        }

        complete_apps = ['studies']

Run migration: python manage.py migrate

Let's check our data now:

    >>> from studies.models import Professor, Student
    >>> student1 = Student.objects.get(first_name='Justin')
    >>> student1.professors.all() #should be professor1 and professor2
    ... [<Professor: Joshua Smith>, <Professor: Ethan Brown>]
    >>> student2 = Student.objects.get(first_name='Tom')
    >>> student2.professors.all() #should be professor2
    ... [<Professor: Ethan Brown>]

Here's it, our relation are imported to our Course model easily. You can add also a custom code in backwards method needed for revert.

Just don't forget to update your code because methods add and remove are not valid now:

• Add:

    >>> student1.professors.add(professor1)
    #Replace with
    >>> Course(student=student1, professor=professor1, name='Django')
  

• Remove:

    >>> student1.professors.remove(professor1)
    #Replace with
    >>> Course.objects.get(student=student1, professor=professor1).delete()
  

Last thing, use South in your Django projects, you must use it from project start else from now!

Good luck with your migrations !

So it come in my mind to write a simple html/javascript application, without any backend server.

How to read a file via browser ?

The unique constraint was how can I read the CSV file in browser.

HTML5 finally provides a standard way to interact with local files, via the File API

We are going to use FileReader

First, let’s check if the browser support FileReader API

    function handleFiles(files) {
      // Check for the various File API support.
      if (window.FileReader) {
          // FileReader are supported.
      } else {
          alert('FileReader are not supported in this browser.');
      }
    }

How to read a file via browser ?

To select a file we’re going to use a simple html <input type="file">

We just call handleFile to make JavaScript returns the list of selected File objects but we are going to use only one.

    <script type="text/javascript" src="static/js/read-csv.js"></script>
      <input type="file" id="csvFileInput" onchange="handleFiles(this.files)"
            accept=".csv">

##Reading the File

After we’ve obtained a File reference, we instantiate a FileReader object to read its contents into memory. When the load finishes, the reader’s onload event is fired and its result attribute can be used to access the file data.

FileReader includes four options for reading a file, asynchronously:

• FileReader.readAsBinaryString(Blob|File) – The result property will contain the file/blob’s data as a binary string. Every byte is represented by an integer in the range [0..255].
• FileReader.readAsText(Blob|File, opt_encoding) – The result property will contain the file/blob’s data as a text string. By default the string is decoded as ‘UTF-8’. Use the optional encoding parameter can specify a different format.
• FileReader.readAsDataURL(Blob|File) – The result property will contain the file/blob’s data encoded as a data URL.
• FileReader.readAsArrayBuffer(Blob|File) – The result property will contain the file/blob’s data as an ArrayBuffer object.

For our example, we want to read a CSV file, so we can read it as text. And perfom data processing after in processData(csv) function.

    function handleFiles(files) {
      // Check for the various File API support.
      if (window.FileReader) {
          // FileReader are supported.
          getAsText(files[0]);
      } else {
          alert('FileReader are not supported in this browser.');
      }
    }

    function getAsText(fileToRead) {
      var reader = new FileReader();
      // Read file into memory as UTF-8      
      reader.readAsText(fileToRead);
      // Handle errors load
      reader.onload = loadHandler;
      reader.onerror = errorHandler;
    }

    function loadHandler(event) {
      var csv = event.target.result;
      processData(csv);
    }

    function processData(csv) {
        var allTextLines = csv.split(/\r\n|\n/);
        var lines = [];
        for (var i=0; i<allTextLines.length; i++) {
            var data = allTextLines[i].split(';');
                var tarr = [];
                for (var j=0; j<data.length; j++) {
                    tarr.push(data[j]);
                }
                lines.push(tarr);
        }
      console.log(lines);
    }

    function errorHandler(evt) {
      if(evt.target.error.name == "NotReadableError") {
          alert("Canno't read file !");
      }
    }

That’s it.

Resources

• Source code for this article.