GSoC Task 3 -- Declarative syntax for Formsets

About the Task

This task is the highlight of my project, something I’m quite proud that I was able to implement. The job was to add declarative syntax support for the formset class. The declarative syntax is something which you regularly see in the Model or Form class. Something similar to the below example

class Musician(models.Model):
    first_name = models.CharField(max_length=50)
    last_name = models.CharField(max_length=50)
    instrument = models.CharField(max_length=100)

The way formsets were declared was using formset_factory(), which when passed the desired inputs created a FormSet class and returned it. But if you lets say need to have an extra function in your MyFormSet then you would first have to derive a class from the BaseFormSet then pass it as the argument to formset_factory which will then return the desired class with desired methods. Short story it was a little bit clumsy and a good clean implementation for declaring a Formset was needed.

Overall breakdown of the milestones I completed

Since this was a big task, I started with reading and learning about the things that were required for this task. As advised by my mentor, I started reading about metaprogramming in python. It took me a while to absorb metaprogramming concepts. After that, I started working on the code and tests. Once that was done, I started by writing the documentation and the references for the new classes. It took me the whole of July and August to finish this. Still, the docs may need some edits here and there, but the overall task is done. After the first two weeks, I realized that I could hold on with this one task, because review and updating were taking too long, so I started working on a different task in parallel from the second week of July. This decision was indeed quite good in the long I would say. Here is a brief detail of every milestone.

  1. Learning about Metaprogramming in python
  2. Writing code
  3. Peicing together PR
  4. Never forget the documentation
  5. Documentation continued

Learning about Metaprogramming in python

Metaprogramming is synonymous to black magic in the world of coding. One few people can understand and do unbelievable things with it. At the same time, it scares the sh!t out of other people. Its a branch of coding that if are not sure you need it or not you don’t need it.

I know, I have done the worst job of explaining it, so please check out this tutorial from GeeksForGeeks which is incredible and does a way better job of explaining what metaprogramming is with some beautiful examples. Here are a few quotes from the tutorial.

In a nutshell, we can say metaprogramming is the code which manipulates code. As quoted by Tim Peters: “Metaclasses are deeper magic that 99% of users should never worry about. If you wonder whether you need them, you don’t (the people who actually need them know with certainty that they need them, and don’t need an explanation about why).” This whole meta thing can be summarized as – Metaclass create Classes and Classes creates objects.

I needed metaprogramming because I had to verify at the time of the creation of FromSet class whether or not the required parameters are provided to the class. Earlier it was achieved by keeping some parameters compulsory to the factory() function and after that checking the validity of the parameters before the creation of the FormSet instance.

Writing code

So finally, after gaining all the required skills, I started coding. And frankly, this was the first task where I was actually writing code and not just reading or documenting. Hence I was very excited about this. So first I started by enabling the normal formsets with the declarative syntax. So started editing formsets.py to add the FormSet class, which can be used to make Formset classes by just deriving the classes from this class. Then while doing it, I realized that I would first need to have a metaclass from the FormSet which will mostly do all the checks that are done in the formset_factory function usually. Doing some changes here and there and was able to do it successfully.

Once that was done, I had to repeat the same thing for the ModelFormSet and InlineFormSet. But one problem they had was of complicated inheritance structure, and I needed to figure out the inheritance structure of these two classes. So I came up with one got it verified from my mentor and bam!! I had declarative syntax on my plate.

Peicing together PR

Even though I was able to generate Formset using the declarative syntax, the next hurdle was to ensure that this new Formset was identical to the one created by the formset_factory(). And a simple answer to this was to rerun the same test on as we run on the older Formset, instead create the Formset using the declarative approach. So what I did was to edit the earlier tests to include support for Formsets to be built in different ways and passed.

Once this was done, and all the tests were passing, I was quite confident. Also, my mentor had to see my work, because it was like more than 3-4 weeks, and he had no idea what’s going on, apart from the weekly updates. So finally I cleaned my code a little bit and created a Pull Request. It was one of the happiest days of my whole GSoC :)

Never forget the documentation

The happiness didn’t last long though. The first thing that happened after I push my PR was that some of the online tests fail. It turns out it was just simple errors like my imports weren’t in the alphabetical order. And in some flake8 errors.

But there was still a significant milestone yet to be complicated – DOCUMENTATION. By then, I had done some documentation once or twice. But it was more like adding one or two lines. This was something huge. So as suggested by my mentor, I started editing the topics and ref pages for the formsets. But the problem was the content of these pages was far from ideal. Since very few people work on formset, this was not edited as per the standards. And the problem was I was unable to understand why my mentor has an issue with the simple documentation I did by simply fitting my work with the template which was already present.

Documentation continued

As I said, the docs were far from perfect. So finally, after many reviews from Carlton (my mentor), I requested him to tell me exactly what he wants in baby words. And he literally told me the bullet points of where should I write.

It took me around 2 to 3 weeks for writing code and tests. But it took me one month to write the documentation for this ticket. So the lesson I learned was never underestimating the documentation part in development. As of now, i.e., at the time of writing this blog the documentation still may need some final touch-ups, but surely this feature will be a part of Django 3.0 release and it makes me proud.

Technical description of the Task

Code

The description of this task is simple provide a declarative syntax for FormSet, ModelFormSet and InlineFormSet. The end goal was to have something similar to this;

class AuthorForm(forms.ModelForm):
    class Meta:
        model = Author

class DeclarativeAuthorFormSet(forms.models.ModelFormSet):
    form = AuthorForm
    model = Author
    extra = 3

class BookForm(forms.ModelForm):
    class Meta:
        model = Book

class DeclarativeAuthorBooksFormSet(forms.models.InlineFormSet):
    parent_model = Author
    model = Book
    form = BookForm

To achieve this, I made three new classes namely FormSet, ModelFormSet and InlineFormSet. Each class had its own custom metaclass and was inherited form the respective base class. Here is a flow chart of the inheritance in the formsets.

Tests

For testing the new formset which was formed using the declarative approach to be valid, it had to pass all the tests which the formset made using formset_factory will do. Hence each test case was changed to call a method which will return a formset: And then that method can make the formset using two methods i.e. formset_factory or declarative approach. This drastically reduced the efoorts to write tests for the new FormSet class. Here is a small example of the approach

class TestFormSet(Testcase):
  def make_formset(self, ...):
    # Do some stuff
    return formset_factory(...)

  def some_test(self):
    formset = self.make_formset(...)
    # do some test


class TestDeclarativeFormSet(Testcase):
  def make_formset(self, ...):
    # Do some stuff
    class SomeFormSet(formsets.FormSet):
      form = someform

    return  SomeFormSet

Here the some_test would ran twice with different make_formset moethods each time, hence even if some new tests are added in the normal formset it in the future it will be automatically be tested for the declarative formsets.