pydanny

Core Concepts of Django ModelForms

2013-06-13T15:33:00-07:00

In my opinion, the concepts behind Django's model forms can be listed in just six (6) bullets. The bullets I've marked in bold at the top are the topic of this blog post, while the two of these that were covered in a previous blog post on Django forms are at bottom of my list.

ModelForms render Model fields as HTML.
ModelForms select validators based off of Model field definitions.
ModelForms don't have to display/change all available fields.
ModelForms save dictionaries to SQL tables.
Forms are "just" Python constructs. (covered previous)
Forms validate Python dictionaries. (covered previous)

ModelForms render Model fields as HTML.

If I create a Django model:

# myapp/models.py
from django.db import models

class MyModel(models.Model):

    title = models.CharField(max_length=100)

Then attach it to a ModelForm:

# myapp/forms.py
from django import forms

from .models import MyModel

class MyModelForm(forms.ModelForm):

    class Meta:

        model = MyModel

I can render it in a template, or for better clarity in this post, the Python REPL:

>>> from myapp.forms import MyModelForm
>>> mf = MyModelForm()
>>> mf
<__main__.MyForm object at 0x1023c8bd0>
>>> print(mf)
<tr><th><label for="id_title">Title:</label></th>
<td><input id="id_title" name="title" maxlength="100" type="text" /></td></tr>

ModelForms select validators based off of Model field definitions.

One of the nice things about Django is that its forms library protects your models. It does this by assigning one or more of Django's many built-in validators to the form fields it generates, and using them to check incoming data. Let's dive in:

>>> from myapp.forms import MyModelForm
>>> mf = MyModelForm()
>>> mf
<__main__.MyForm object at 0x1023c8bd0>
>>> mf.fields
{'title': <django.forms.fields.CharField object at 0x102474bd0>}
>>> field = mf.fields['title']
>>> field.max_length
100
>>> field.validators
[<django.core.validators.MaxLengthValidator object at 0x102403b10>]

Each individual field contains a list of validators (in this case, just one validator) supplied by Django and any ModelForm customizations that might have been done.

If you want to add more validators to a ModelForm (perhaps we want our title field to require at least 20 characters) one way to do it is by overriding the field definition in the ModelForm class's __init__ method. That's a mouthful, so I'll just demonstrate in code:

# myapp/forms.py
from django import forms
from django.core.validators import MinLengthValidator

from .models import MyModel

class MyModelForm(forms.ModelForm):

    def __init__(self, *args, **kwargs):
        super(MyModelForm, self).__init__(*args, **kwargs)
        self.fields["title"].min_length = 20
        self.fields["title"].validators.append(MinLengthValidator)

    class Meta:

        model = MyModel

If we stop/start the shell, we now see some new elements added to the form object:

>>> # Don't forget to stop/start the Django shell!
>>> from myapp.forms import MyModelForm
>>> mf = MyModelForm()
>>> mf
<__main__.MyForm object at 0x1023c8bd0>
>>> mf.fields
{'title': <django.forms.fields.CharField object at 0x1023ee810>}
>>> field = mf.fields['title']
>>> field.max_length
100
>>> field.min_length
20
>>> field.validators
[<django.core.validators.MaxLengthValidator object at 0x10240c7d0>, <django.core.validators.MinLengthValidator object at 0x1023eef90>]

Now we have two validators for the field!

There are other ways to override the title field validators. The easiest but not necessarily the best way is to replicate the ModelForm definition of the field in the form like so:

# myapp/forms.py
from django import forms

from .models import MyModel

class MyModelForm(forms.ModelForm):

    title = forms.CharField(max_length=100, min_length=20)

    class Meta:

        model = MyModel

I don't like this technique. This makes it so we are defining the title field in two places, once in the model and once in the form. I go into more of the details and problems of this approach in my previous blog post at Overloading Django Form Fields.

ModelForms don't have to display/change all available fields.

Before we dive into this section, let's increase our model to have two fields as shown below:

# myapp/models.py
from django.db import models

class MyModel(models.Model):

    title = models.CharField(max_length=100)
    slug = models.SlugField()

Let's say that we don't want to allow users the ability to change slugs on existing content, otherwise URLs will be broken. In this case, we rely on the fields attribute of ModelForm.Meta to make it so we only display what we want to display:

# myapp/forms.py
from django import forms

from .models import MyModel

class MyModelForm(forms.ModelForm):

    class Meta:

        model = MyModel
        fields = ('title', )

Easy!

But what about ModelForm.Meta.excludes?

We advocate strongly against using ModelForm.Meta.excludes.

In fact, when we were writing Two Scoops of Django the majority of our technical reviewers as well as our security reviewer fervently insisted that we advocate against use of ModelForm.Meta.excludes. We provide numerous warnings about it's usage, and go in-depth as to why in section 21.12. For reference, Django's own documentation is now including a rather mild warning (no warning box) on the subject at selecting the fields to use. I might try and get that addressed in the next few days...

In any case, the problem with ModelForm.Meta.excludes is similar to but worse than duplicating field functionality. It means that changes to models (new fields for example) will display in associated forms unless you remember to modify the associated forms. Since a single model can have multiple forms, and we developers forget or leave projects, you can understanding what sort of security nightmare this can cause.

Do yourself a favor and stay away from ModelForm.Meta.excludes.

ModelForms save dictionaries to SQL tables

In my previous post of Django forms I covered forms validate dictionaries. Well, ModelForms do the same thing AND give us the power to save that validated dictionary to SQL tables. We don't even need to involve web pages!

This is really useful because it means that we can take data from any source, be it user input from the web, JSON data fetched from an API, and even CSVs generated from Excel reports and transform that into data that resides in SQL.

Let's go over using our ModelForm with title/slug fields used with all those methods. In our samples (web page, json, csv), we'll use generating a timestamp to demonstrate how we can modify the model data before it's saved, and we'll base all three examples off the model and ModelForm combination listed below.

# myapp/models.py
from django.db import models


class MyModel(models.Model):

    title = models.CharField(max_length=100)
    timestamp = models.DateTimeField()

# myapp/forms.py
from django import Forms

from .models import MyModel


class MyModel(forms.ModelForm):

    class Meta:

        model = MyModel
        fields = ('title', )

And now to our three examples!

Example #1 Web Page

This should look pretty familiar to many Django developers. it's the traditional Django view pattern of processing simple model forms.

# myapp/views.py
from django.core.shortcuts import render, redirect
from django import forms
from django.utils import timezone

from .forms import MyModelForm


def add_model(request):

    if request.method == "POST":
        form = MyModelForm(request.POST)
        if form.is_valid():

            # commit=False means the form doesn't save at this time.
            # commit defaults to True which means it normally saves.
            model_instance = form.save(commit=False)
            model_instance.timestamp = timezone.now()
            model_instance.save()
            return redirect('victory')
    else:
        form = MyModelForm()

    return render(request, "my_template.html", {'form': form})

Example #2 API/JSON

In this example, we're validating the output of a RESTful API before letting it touch our database. It's critical that such APIs are not allowed to touch our systems without proper validation - don't make my mistakes! Also, even internally within a project it's really important to validate all data coming from different databases. And Django makes it easy!

# myapp/api/reitz.py
from django.utils import timezone

import requests  # You are using requests-python, right?

from .forms import MyModelForm


class ReitzApiException(Exception):
    pass


def fetch_reitz_data(target_url):
    response = requests.get(target_url)
    if response.status_code == "200":

        # generate the form from the response
        form = MyModelForm(response.json())
        if form.is_valid():
            model_instance = form.save(commit=False)
            model_instance.timestamp = timezone.now()
            model_instance.save()
            return model_instance

        # Simplistic exception handling
        raise ReitzApiException(form.errors)

    # Simplistic exception handling
    raise ReitzApiException(response.status_code)

Example #3 CSV Import

I'll admit my mistake again: I've written my own validation tools to handle data coming from CSVs and Excel documents into Django projects. My validation scripts always seem fragile, and they are. What I'm doing going forward is I'm leaning on form libraries to do the hard work of validating data and saving it to models.

import csv

from django.utils import timezone

from .forms import MyModelForm


def import_csv(filename):
    rows = open(filename)
    records_added = 0
    errors = []
    # Generate a dict per row, with the first CSV row being the keys.
    for row in csv.DictReader(rows, delimiter=","):

        # Bind the row data to the MyModelForm
        form = MyModelForm(row)
        if form.is_valid():
            model_instance = form.save(commit=False)
            model_instance.timestamp = timezone.now()
            model_instance.save()
            records_added += 1
        else:
            errors.append(form.errors)

    return records_added, errors

Closing Thoughts

I can think of three things:

Forget the HTML: For nearly every sort of occasion where new data is coming into your system, form libraries save you from doing extra work and cover your behind. While my experience is with Django forms, there are many form libraries out there. The patterns explored in this post are certainly available in other web frameworks (see http://flask.pocoo.org/snippets/category/forms/ for proof).

HTML Rendering Issues: No form library is going to do everything, and because of evolving standards, decent HTML rendering is a pain for form library authors. For example, Django's default form HTML rendering remains stuck in 2005 because if they had kept up with modern trends of HTML form layout we would have many different flavors of forms in Django core (a testing nightmare). Which means, as a developer, it's important when using a new form library to learn how to override the default form HTML rendering.

What about AJAX?: Whoops! Does this mean I have to write another blog post? Not at all. In Django, AJAX is just another view, either function- or class-based. The secret is to validate the incoming data the same way as you would any other view request by using forms.

The Easy Form Views Pattern Controversy

2013-05-10T12:00:00-07:00

In the summer of 2010 Frank Wiles of Revsys exposed me to what I later called the "Easy Form Views" pattern when creating Django form function views. I used this technique in a variety of places, including Django Packages and the documentation for django-uni-form (which is rebooted as django-crispy-forms). At DjangoCon 2011 Miguel Araujo and I opened our Advanced Django Forms Usage talk at DjangoCon 2011 with this technique. It’s a pattern that reduces the complexity of using forms in Django function-based views by flattening the form handling code.

How the Easy Form Views pattern works

Normally, function-based views in Django that handle form processing look something like this:

def my_view(request, template_name="my_app/my_form.html"):

    if request.method == 'POST':
        form = MyForm(request.POST)
        if form.is_valid():
            do_x() # custom logic here
            return redirect('home')
    else:
        form = MyForm()
    return render(request, template_name, {'form': form})

In contrast, the Easy Form Views pattern works like this:

def my_view(request, template_name="my_app/my_form.html"):

    form = MyForm(request.POST or None)
    if form.is_valid():
        do_x() # custom logic here
        return redirect('home')
    return render(request, template_name, {'form': form})

The way this works is that the django.http.HttpRequest object has a POST attribute that defaults to an empty dictionary-like object, even if the request’s method is equal to "GET". Since we know that request.POST exists in every Django view, and os at least as an empty dictionary-like object, we can skip the request.method == 'POST' by doing a simple boolean check on the request.POST dictionary.

In other words:

If request.POST dictionary evaluates as True, then instantiate the form bound with request.POST.
If the request.POST dictionary evaluates as False, then instantiate an unbound form.

Great! Faster to write and shallower code! What could possibly be wrong with that?

The Controversy

Before you jump to convert all your function based forms to this pattern, consider the following argument raised against it by a good friend:

This one of those things where "empty dictionary and null both evaluate as false" can bite you.

There's a difference between "There is no POST data", and "This wasn't a POST".

—by Russell Keith-Magee (paraphrased)

The problem he is talking about is data besides multipart/form-data or application/x-www-form-urlencoded would still end up in the request.POST dictionary-like attribute.

Where is the controversy? Well, I didn't write a retraction until now. Arguably I should have done it earlier. However, since I never ran into the edge case, I didn't see the need. Yet when it comes down to it, the "Easy Forms" approach has an implicit assumption about the incoming object, which in Python terms is not a good thing.

Getting bit by the Easy Form Views method

Here's how it happens:

Before Django 1.5 HTTP methods such as DELETE or PUT would see their data placed into Django's request.POST attribute. The form would fail, but it might not be clear to the developer or user why. HTTP GET and POST methods work as expected.

For Django 1.5 (and later) if a non-POST comes in then the form fails because request.POST is empty. HTTP GET and POST methods also work as expected.

Conclusion

Going forward, I prefer to use Django's class-based views or Django Rest Framework which make the issue of this pattern moot. When I do dip into function-based views handling classic HTML forms, I'm leery of using this pattern anymore. Yes, it is an edge case, but to inaccurately paraphrase Russell, "edge cases are where you get bit".

What I'm not going to do is rush to change existing views on existing projects. That's because personally I've yet to run into an actual problem with using this pattern. As they say, "If it ain't broke, don't fix it." While I'm not saying my code isn't broken, I'm also aware that 'fixing' things that aren't reporting errors is a dangerous path to tread.

Also, next time I get called on something by a person I respect, I'll respond more quickly. Nearly two years is too long a wait.

Update: Changed some of the text to be more succinct and took out the leading sentence.

Tools we used to write Two Scoops of Django

2013-05-03T10:00:00-07:00

Because of the ubiquitousness of reStructuredText in the lives of Python developers and the advocacy of it, it's not uncommon for people to assume we used it to write our book. However, that's not really the case.

The short Answer is we used:

reStructuredText (RST)
Google Documents
Apple Pages
LaTeX

The long answer is the rest of this posting. Since writing the book was broken up into three major stages 'alpha', 'beta', and 'final', so have I broken up blog article.

Alpha Days

Some of the original alpha material was written in rough draft form as RST since it was what we were used to using. Unfortunately, the PDF generation wasn't to our liking, so we immediately began looking at other options. Since she enjoyed using it at MIT and because it gave us greater individual control, Audrey wanted to switch to LaTeX. I was worried about the challenges of learning LaTeX, so we compromised and moved to Google Documents.

For the most part, Google Documents was great in the early stages. The real-time collaborative nature was handy, but the gem was the comment system. It gave us the ability to have line-by-line written dialogues with our technical reviewers. However, Google Documents makes it nigh-impossible to use WYSIWYG editor styles, add in better print fonts, forced us to cut-and-paste code examples, and finally the PDF export system was flakey on our massive document.

Our original thought was to convert the Google Document output to PDF and then modify it with Adobe InDesign. Upon trying it, we found it had a lackluster user interface that had a steep learning curve and was prohibitively expensive ($550-$700). Our friend and reviewer, Kenneth Love of Getting Started with Django fame, offered to do the conversion work, but we wanted to be able to update our work at will. Awesome as Kenneth might be, we couldn't expect him to drop what he was doing to update the final output of our work whenever we wanted.

Therefore, what we did in the week of January 10th-16th was convert the book to Apple Pages, which is the word processor in Apple iWorks. This was as painful as it sounds. We also discovered the day before launch that Apple Pages doesn't create a sidebar PDF table of contents, which a lot of people enjoy (including ourselves). Tired and exhausted from weeks of 16 hour days, we launched anyway on January 17th with the book weighing in at 5.1 MB.

Beta Experiences

People were so positive it really gave us a boost. Hundreds of people sent us feedback and we were delighted beyond words, with a significant portion sending us commentary/corrections about our writing and code. I'll admin did get tired over a certain 'moat' mistake since I got corrected on it over 50 times. However, the number of code corrections we were getting was higher than expected. It was clear we needed to be able to import the code modules from testable chunks of real code. We had so many kindle/epub requests we also needed the ability to render the text attractively across multiple formats.

After stumbling through RST, Google Documents, and Apple Pages different tools, I finally agreed with Audrey that the challenges of learning LaTeX was worth it. While we could have used RST, we would have had to use LaTeX anyway for our customizations since when RST is converted to PDF it actually uses an interim step of LaTeX!

So while I handled the corrections and feedback from thousands, Audrey built the fundamentals of the LaTeX file structure. Audrey really got her hands dirty by teaching me LaTeX, since my brain is slow and thick. Here's a sample of what I've learned how to do, taken from Chapter 6, Section 1, Subsection 5 (6.1.5):

\subsection{Model Inheritance in Practice: The TimeStampedModel}
It's very common in Django projects to include a \inlinecode{created} and \inlinecode{modified} timestamp field on all your models. We could manually add those fields to each and every model, but that's a lot of work and adds the risk of human error. A better solution is to write a \inlinecode{TimeStampedModel} \index{TimeStampedModel} to do the work for us:

\goodcodefile{chapter_06/myapp/core/timestampedmodel.py}

Take careful note of the very last two lines in the example, which turn our example into an abstract base class: \index{abstract base classes}

\goodcodefile{chapter_06/myapp/core/class_meta.py}

By defining \inlinecode{TimeStampedModel} as an abstract base class \index{abstract base classes} when we define a new class that inherits from it, Django doesn't create a \inlinecode{model\_utils.time\_stamped\_model} table when syncdb is run.

Once I got the hang of LaTeX, then began the hard work of converting the book's current content from Apple Pages That was a couple weeks of grueling effort on my part. Daily I would request a new LaTeX customizations, which Audrey would address. However, as she was working on literally rewriting the content of a dozen chapters including templates, testing, admin, and logging my interruptions became an issue. So we enlisted the help of Italian economist and LaTeX expert Laura Gelsomino. Thanks to her the desired text formatting was achieved.

During the conversion process we also rewrote every single code example, putting them into easily testable projects, and pull them into via use of custom LaTeX commands called \goodcodefile{} and \badcodefile{}.

Eventually I joined Audrey on rewriting and reviewing chapters and on February 28th, the beta was launched. LaTeX generates lean PDFs so the book came in at just 1.6 MB while adding a whopping 50 pages (25% more) of content.

Final Efforts

The final effort was focused on cleanup, new formats, presentation, and art.

For cleanup, our amazing readers gave us so much feedback we could barely keep up. We fought to keep our dialogues with them personal yet brief. With reader oversight we corrected many of the 'quirks' of my writing style (Audrey is a stickler for Strunk and White, I am not). We also made numerous corrections based on feedback and our own observations.

With the guidance of fellow Python author Matt Harrison I wrote scripts that took the archaic HTML generated by LaTeX module tex4ht and rendered it into something that Kindlegen could use to generate Kindle .mobi files. At first the results looked awesome on modern kindles and other new ebook readers, but was terrible on older devices. So I toned back the fancy stuff to what you see today. Getting technical books to look nice on all readers is really, really hard - and unfortunately some publishers take shortcuts that hurt the efforts of the authors. If you have problem with an e-book's format, please consider that before writing a negative review about the final output.

Speaking of mobile editions, we also wrote a second version of each Python example to deal with the smaller format. While libraries exist to do the work for you, since I did a lot of it from scratch (albeit coached by Matt) I had to dig into the lackluster .mobi/.epub documentation to figure out things like .ncx files.

note: If you want to be the self-published author of a technical book I strongly recommend you read Matt's Ebook Formatting: KF8, Mobi & EPUB. Also check out his rst2epub2 library for converting RST files to various formats.

While I worked on the mobile editions, Audrey focused on the print version and adding more art and tiny bit of new content. She focused on clarity and flow, and the result is that the book feels even lighter to read and yet is dense with useful information. To test how the book launched, she would order a copy from the printer and wait several days for it to arrive. Then she would inspect the cover and interior with her incredibly exacting eye. It's a slow process, but Audrey wanted to make absolutely certain our readers would enjoy and use the print edition.

On April 10th we launched the final in PDF, Kindle, and ePub form. The PDF weighs in at 2.7 MB, and the Kindle file is a bit heaver. At some point we'll do the work to reduce file size, but for now we're working on other things.

A week later we announced the launch of the print version of the book. People seem to really like the design and feel of the physical book, and we've even had requests for t-shirts.

Thoughts

Writing a technical book was really hard. Crazy hard. Also very satisfying. We could have made more money doing just client work, but this was a dream come true. Sometimes money doesn't matter.

Whither Two Scoops of Django?

Two Scoops of Django: Best Practices for Django 1.5 will still receive periodic corrections, but won't see new content unless it's security related for Django 1.5. Don't worry though, for when Django 1.6 comes nigh, we'll commence work on Two Scoops of Django: Best Practices for Django 1.6 (TSD 1.6). The plan is to update practices as needed and hopefully add more content on testing, logging, continuous integration, and more. Like it's predecessor TSD 1.6 will be written using LaTeX.

That said, if I ever fulfill my dream of writing fiction I'll just use Matt Harrison's rst2epub2 library.

Concerns about Open Sourcing

We've considered open sourcing our current book generation system, but installation is rather challenging and requires serious Audrey/Laura-level LaTeX knowledge combined with my experience with Python. Unfortunately, from our experience on managing other open source projects, dealing with requests for documentation and assistance would take up a prohibitive amount of our time. Honestly, we would rather write another book or sling code.

Book Generation as a Service?

Another option is turning our system into a service, which would convert existing RST or even MarkDown to LaTeX so it could generate books in the Two Scoops format. Doing this would require at least a month of full-time work on both of our parts, and we have no idea as to the interest level. We think it would be a low amount of interest, but then again, hasn't leanpub done pretty well using this model of business?

In any case we're working on other projects. Maybe even a new technical book...

Filepicker.io and South

2013-04-23T10:00:00-07:00

I've heard good things about filepicker.io, which is a service that makes file uploading a much better experience. Unfortunately, the Django package for filepicker.io doesn't work with South. When I try to create a migration using the filepicker.io field using code like the following...

# products/models.py
from django.db import models
from django_filepicker.models import FPFileField

class Product(models.Model):
    title = models.CharField(max_length=255)
    file = FPFileField(upload_to='uploads')

...when I try to run the command:

(tsd)$ python manage.py schemamigration products --initial

It results in this unpleasant looking response:

(tsd)$ python manage.py schemamigration products --initial
Creating migrations directory at '/Users/danielgreenfeld/code/tsp/tsp/products/migrations'...
Creating __init__.py in '/Users/danielgreenfeld/code/tsp/tsp/products/migrations'...
 ! Cannot freeze field 'products.product.fpfile'
 ! (this field has class django_filepicker.models.FPFileField)
 ! Cannot freeze field 'products.release.fpfile'
 ! (this field has class django_filepicker.models.FPFileField)

 ! South cannot introspect some fields; this is probably because they are custom
 ! fields. If they worked in 0.6 or below, this is because we have removed the
 ! models parser (it often broke things).
 ! To fix this, read http://south.aeracode.org/wiki/MyFieldsDontWork

The last line in the error report is important. I'll repeat it to illustrate it more clearly:

! To fix this, read http://south.aeracode.org/wiki/MyFieldsDontWork

Experience working on other projects has taught me I can simply add two lines of code to products/models.py and everything should just work:

# South migration rules for the FPFileField field
from south.modelsinspector import add_introspection_rules
add_introspection_rules([], ["django_filepicker\.models\.FPFileField"])

In case it's not clear, here's my new products/models.py file:

# products/models.py
from django.db import models
from django_filepicker.models import FPFileField

# South migration rules for the FPFileField field
from south.modelsinspector import add_introspection_rules
add_introspection_rules([], ["django_filepicker\.models\.FPFileField"])

class Product(models.Model):
    title = models.CharField(max_length=255)
    file = FPFileField(upload_to='uploads')

Now I can create South migrations and they'll just work.

Unfortunately, the problem is that for any model where I need to use filepicker's FPFileField I need to add those two lines of code. I don't like this approach, since it violates Don't Repeat Yourself (DRY).

At some point I'll demonstrate how to fix this violation of DRY with an easy fix. In fact, I plan submit that fix as a pull request to django-filepicker.

Off to Europe!

2013-04-20T09:00:00-07:00

I feel like I'm dreaming, but we're on our way to Europe!

A few weeks Audrey and I decided to move. While looking at possible rentals or even mortgages, we were idly (dreaming really... still paying off debts incurred from last year) looking at how much a room/apartment would cost to rent in Europe. To our surprise, we found a lot of good options that cost less then it would to pay monthly rent/mortgage in Los Angeles.

o_O

Then, we discovered an amazing travel deal. It combined a very cheap red-eye flight from Los Angeles to Fort Lauderdale, and a slow trans-atlantic ship from Port Miami to Barcelona, about as much as a cheap hotel but with free food! If we avoid alcohol and any extra fees, it's actually extremely inexpensive. Being mostly disconnected from the internet would mean we could really focus on a couple of critical projects. If only we had a business justification for going to Europe...

It just so happens that we have business in Europe.

Our plan is to spend a few months in Europe before returning back to the USA. We'll work hard on the ship, work hard when we arrive, and just get stuff done. It's not a vacation, but it is Europe, and there is such a thing as weekends.

We'll be spending a few days in early May in Barcelona, then making our way to Poland for several weeks. Yes, we're attending Django Circus, plus I want to investigate some family roots. After Poland, we're thinking either France, Italy, Hungary, England, Belgium, Scotland, Ireland, and other options. We'll see where the winds and affordable AirBNB places take us.

When do we go back? Well, we plan to stay for at least two months. Even if we go back to the USA, we would have at least been in Europe for May and June. How cool is that?

We could have talked ourselves out of going. We nearly did. However, even with our workload this is the chance of a lifetime.

Finally, for the next two weeks I'll only be able to access the internet for a few minutes per day. Please expect my responses to emails and tweets to be slower than normal.

Update: In the middle of all of this, I was invited to keynote Django Circus: http://blog.djangocircus.com/post/48439600279/meet-daniel-greenfeld

Two Scoops of Django is in print!

2013-04-17T09:00:00-07:00

Since I was a child I wanted to be a published author. I've dreamt of people reading my book. While one could say that dream was fulfilled when we launched the e-book version in January, it's not the same as seeing the printed copy. Today I got to see my dream come true, a printed book is out there with my name on it.

Also, I have a confession to make. When it comes to technical books, I really prefer the print version. For me, nothing beats being able to flip through pages of a hardcopy next to a keyboard, or simply reading while I'm on a bus, train, or plane. So this adds to my excitement when I get to say that the book that Audrey Roy and I wrote together is in print.

Note: As of April 17th it's only available in the US. However, it's in the queue for being available for Amazon Europe. Enough people are asking that we are also looking into getting it bulk shipped to Django Circus.

While it's been an amazing journey, it's also been a long hard road. Starting with a late night dinner with Randall Degges, his wife Samantha, and Sean Bradley, to spending hours upon hours writing and revising, to taking painfully honest critiques from mentors, to alpha launch, then rewriting the book followed by the beta launch. It wasn't easy, but the growing volume of positive feedback and suggestions from readers kept us going.

Between the Alpha and Beta releases we switched over to LaTeX. Audrey taught it to me in February, and I'm proud to say I've progressed from absolute neophyte to beginner. We chose it because of it's fine-grained control over layout for the PDF and print versions. Creating the Kindle and ePub versions from LaTeX was harder thanks to using LaTeX, but everything's automated now so making updates is easy. Future editions of this book (1.6, 1.7, etc) or other technical books I hope to write will continue to be in LaTeX, but if I publish fiction I'll probably go with RestructuredText.

While working on the final version we were greatly saddened by the death of Malcolm Tredennick, to whom the book is dedicated. We miss him terribly. I've tried to write about Malcolm's volume of his contribution to the book, but it's still too hard. For now, let's just say that he gave and taught us so much and asked us for nothing in return, which is why we continue to give free copies of the PDF edition of the book to developers in need, as well as try to stay in contact with all recipients of the free PDF.

In fact, if you are a developer in need, and can't afford a copy, drop us a note at 2scoops@cartwheelweb.com explaining your situation. Our quick rule of thumb is that if you can afford to buy games for your favorite device, then you can afford to buy our book. Also, please do your friends a favor and don't just give them a copy without having them ask us. Why not? Well, we update the book frequently, and have a strong commitment for adding security related content even now that the final edition is out.

What now? Well, we aren't certain. We're maintaining a couple client projects but aren't taking on new ones. We've got a couple projects we're working on and I'm talking employment with a couple companies who do good things for the community.

Generating NCX files with Python

2013-04-15T09:00:00-07:00

With the help of fellow Python developer Matt Harrison's excellent Ebook Formatting: KF8, Mobi & EPUB, we managed to create pretty decent looking Kindle and ePub versions of Two Scoops of Django.

One of many things we did was focus on providing an excellent table of contents. Of course we provided one inside the content of the book, but much like the PDF version we also provided one that various ebook readers can display in sidebars or drop down menus. Unfortunately, building this navigation isn't well documented (except for Matt's book), and I've yet to see any good ways to generate it via code.

Which is why I present the following code. It looks at the HTML that KindleGen and ePub generators demand and pulls from it a chapter-based table of contents. Then constructs a .ncx file, which is what ebook readers use to generate the sidebar/dropdown table of contents.

Our requirements:

Jinja2
Django
BeautifulSoup4

And now the code:

#!/usr/bin/python
# -*- coding: utf-8 -*-


from bs4 import BeautifulSoup
from django.utils.text import slugify
from jinja2 import Template

TEMPLATE = Template("""<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE ncx PUBLIC "-//NISO//DTD ncx 2005-1//EN" "http://www.daisy.org/z3986/2005/ncx-2005-1.dtd">
<ncx xmlns="http://www.daisy.org/z3986/2005/ncx/" version="2005-1" xml:lang="en">
<head>

<!-- The content of dtb:uid must be exactly the same as the uuid specified in the OPF file. -->

<meta name="dtb:uid" content="urn:uuid:BLAH-BLAH-BLAH"/>
<meta name="dtb:depth" content="1"/>
<meta name="dtb:totalPageCount" content="0"/>
<meta name="dtb:maxPageNumber" content="0"/>
</head>

<docTitle><text>Two Scoops of Django: Best Practices for Django 1.5</text></docTitle>
<docAuthor><text>Greenfeld, Daniel</text></docAuthor>
<docAuthor><text>Roy, Audrey</text></docAuthor>
<navMap>
</navPoint>
{% for chapter in chapters %}
<navPoint id="{{ chapter.slug }}" playOrder="{{ loop.index }}">
<navLabel><text>{{ chapter.string.strip() }}</text></navLabel>
<content src="{{ chapter.href }}" />
</navPoint>
{% endfor %}
</navMap>
</ncx>
""")


def main(filename):

    # Grab the base file for review
    with open(filename) as f:
        text = f.read()

    # load the text into a bs4 object
    soup = BeautifulSoup(text)

    # grab the nav element
    nav = soup.find("nav")

    # loop through the TOC for chapters.
    # Sections/Subsections can't be displayed, so don't worry about them
    # li.chapter is how we constructed our TOC. Your mileage may vary.
    chapters = []
    for li in nav.find_all("li", "chapter"):
        chapters.append(dict(
            href=li.a['href'],
            string=li.a.text,
            slug=slugify(li.a.string)
        ))

    # Render the template
    template = TEMPLATE.render(chapters=chapters)

    # convert to ASCII
    template = template.encode("ascii", "xmlcharrefreplace")

    # Save to the toc.ncx
    with open("toc.ncx", "w") as f:
        f.write(template)


if __name__ == '__main__':
    main('book.html')

There is more to adding a table of contents then just this simple module. You also have to construct the .opf file, which is another undocumented mess that I'll blog about.

Core Concepts of Django Forms

2013-03-29T11:00:00-07:00

In my opinion, the concepts behind Django's non-model forms can be listed in just three (3) bullets:

Forms render HTML.
Forms are "just" Python constructs.
Forms validate dictionaries (Python's Key/Value structure).

Let's dig in!

Forms render HTML.

If I construct a Django form:

# myapp/forms.py
from django import forms

class MyForm(forms.Form):

    title = forms.CharField(required=True)

I can render it in a template, or for better clarity in this post, the Python REPL:

>>> from myapp.forms import MyForm
>>> f = MyForm()
>>> f
<__main__.MyForm object at 0x1016c6990>
>>> print(f)
<tr><th><label for="id_title">Title:</label></th>
<td><input id="id_title" name="title" type="text" /></td></tr>

You can even see this done with initial values in the Django docs: https://docs.djangoproject.com/en/1.5/ref/forms/api/#django.forms.Form.initial

Forms are "just" Python constructs.

I believe it was Alex Gaynor who said back in 2008 that Django forms were "just" Python constructs. He's right:

>>> from myapp.forms import MyForm
>>> # class
>>> MyForm
<class 'myapp.forms.MyForm'>
>>> # object
>>> form = MyForm()
>>> form
<myapp.forms.MyForm object at 0x1023f1450>
>>> # iterable
>>> [x for x in form]
[<django.forms.forms.BoundField object at 0x102495990>]
>>> [x for x in form.fields]
['title']
>>> # dictionary-like
>>> form.fields['title']
<django.forms.fields.CharField object at 0x1024a17d0>

Understanding the structure of Django forms is really useful. This structure is what allows the modification mechanism that I described in my previous post.

We don't have to stop in just the forms.py module. You can also modify forms in views (either the classic views.py module or in whatever API library you might be using):

from django import forms
from django.shortcuts import redirect
from django.views.generic import FormView

class MyFormView(FormView):

    form_class = MyForm

    def get_form(self, form_class):
        form = form_class(**self.get_form_kwargs())
        form.fields['favorite_icecream'] = forms.ChoiceField(
            label="What is your favorite flavor from this list?",
            choices=((0, "Chocolate"), (1, "Vanilla"), (2, "Berry")),
            widget=forms.RadioSelect,
            required=True
        )
        return form

    def form_valid(self, form):
        # Get user's favorite ice cream.
        # You can do anything you want with it
        favorite_icecream = form.cleaned_data['favorite_icecream']

        # return the anticipated redirect
        return redirect("home")

As you can see, with an understanding of basic Python types and some experience with Django forms you can become very creative in applications of forms. Please keep in mind that the devil is in the details, and overly creative use of forms (or anything) is a road you should carefully tread. It's always good to remember that simplicity is best and that the goal isn't to just write code, but to write maintainable code.

Forms validate dictionaries.

One of the primary functions of any HTTP-friendly form libraries is validating dictionary-like data objects. HTTP query strings are key/value structures and in order to avoid corruption in the persistence layer of any project, regardless of framework or language, validation needs to occur.

During it's request/response cycle Django converts HTTP POST (and HTTP GET) objects into something called a QueryDict, which is an merely an extended Django dictionary. See the comments in the code example below for proof:

import logging

from django.http import HttpResponse
from django.http.request import QueryDict
from django.utils.datastructures import MultiValueDict

logger = logging.getLogger(__main__)

def my_form_view(request):

    logging.debug(
        # logs True because request.POST is an instance of QueryDict
        isinstance(request.POST, QueryDict)
    )
    logging.debug(
        # logs True because QueryDict is a dictionary
        issubclass(QueryDict, dict)
    )

    return HttpResponse()

This is all fine and good, but what does it mean for developers trying to solve problems? Well, it means that Django forms serve quite handily as a means for validation of dictionaries:

>>> from myapp.forms import MyForm
>>> good_form = MyForm({"title": "Two Scoops of Django"})
>>> good_form.is_valid()
True
>>> good_form.errors
{}
>>> bad_form = MyForm({})
>>> bad_form.is_valid()
False
>>> bad_form.errors
{'title': [u'This field is required.']}

The power of this can't be understated. In fact, I'll be exploring this particular facet of Django forms more in at least one upcoming blog post.

Epilogue

ModelForms adds at least three more bullets...

ModelForms render Model fields as HTML
ModelForms automatically choose validators based off of Model field definitions.
ModelForms save dictionaries to SQL tables.

... and I'll be touching on them in the future.

Overloading Django Form Fields

2013-03-27T09:00:00-07:00

One of the patterns we get positive feedback for mentioning in our book is overloading form fields.

The problem this pattern handles is the use case of when we have a model with a field(s) that allows for blank values, how do we force users to enter values?

For example, assuming the following model:

# myapp/models.py
from django.db import models

class MyModel(models.Model):

    name = models.CharField(max_length=50, blank=True)
    age = models.IntegerField(blank=True, null=True)
    profession = models.CharField(max_length=100, blank=True)
    bio = models.TextField(blank=True)

How do we make all those fields (name, age, profession, bio) required without modifying the database?

This is the way I used to do it:

# myapp/forms.py
from django import forms

from .models import MyModel

class MyModelForm(forms.ModelForm):

    name = forms.CharField(max_length=100, required=True)
    age = forms.IntegerField(required=True)
    profession = forms.CharField(required=True)
    bio = forms.TextField(required=True)

    class Meta:
        model = MyModel

See the problems with this approach?

MyModelForm is nearly a copy of MyModel, and was in fact created by copy/pasting model and then modifying it. In software engineering parlance, it violates the principal of Don't Repeat Yourself (DRY) and is fertile ground for introducing bugs.

`MyModelForm` has a bug!

Can you spot the bug?

The code example below illuminates where I purposefully/gleefully placed an error:

class MyModel(models.Model):

    # 50 character database field
    name = models.CharField(max_length=50, blank=True)

class MyModelForm(forms.ModelForm):

    # Most people don't write tests to check for field length.
    # 100 character form field - probably not spotted until deployed.
    # Easy error to make when violating DRY since the model can change
    #   and leave the form definition behind.
    name = forms.CharField(max_length=100, required=True)

Bugs like this happen either because developers are human and make mistakes, or because the model evolves over time and the forms are left behind. This is a serious maintenance issue, and one that will bite you or the developers who end up maintaining code you've written.

Can you spot the second bug? ;-)

How do we fix this?

A Better Way

In instantiated Django forms, fields are kept in a dict-like object. Which means, instead of writing forms in a way that duplicates the model, a better way is to explicitly modify only what we want to modify:

from django import forms

from .models import MyModel

class MyModelForm(forms.ModelForm):

    def __init__(self, *args, **kwargs):
        super(MyModelForm, self).__init__(*args, **kwargs)
        # Making name required
        self.fields['name'].required = True
        self.fields['age'].required = True
        self.fields['bio'].required = True
        self.fields['profession'].required = True

    class Meta:
        model = MyModel

Other field attributes

This isn't just limited to the required attribute. It can also be applied to help_text, label, choices, widgets, or any other form field attribute:

from django import forms

from .models import MyModel

class MyModelForm(forms.ModelForm):

    def __init__(self, *args, **kwargs):
        super(MyModelForm, self).__init__(*args, **kwargs)
        # snip the other fields for the sake of brevity
        # Adding content to the form
        self.fields['profession'].help_text = "Job title here"

    class Meta:
        model = MyModel

Try it with Inheritance!

We can even do this with inheritance:

from django import forms

class BaseEmailForm(forms.Form):
    email = forms.EmailField("Email")
    email2 = forms.EmailField("Email 2")

    def clean(self, *args, **kwargs):
        email = self.cleaned_data['email']
        email2 = self.cleaned_data['email2']
        if email != email2:
            raise forms.ValidationError("Emails don't match")
        return self.cleaned_data

class ContactForm(BaseEmailForm):
    message = forms.CharField()

    def __init__(self, *args, **kwargs):
        super(ContactForm, self).__init__(*args, **kwargs):
        self.fields['email2'].label = "Confirm your email"
        self.fields['email2'].help_text = "We want to be sure!"

Summary

From the perspective of general software development, it's always a good thing to avoid repeating yourself. This might seem like as much or in some cases even more typing, but it's a lot better than making an embarrassing/costly mistake.

From the perspective of a Python developer our approach more closely matches the Zen of Python. This is because we only modify the field properties that need to be modified, the approach specified is more explicit.

Today's reading is Matt Harrison's Guide to Learning Iteration and Generators in Python

Beginner's Guide to PyCon 2013 Part III

2013-03-10T23:50:00-07:00

This is Part III in a series of blog posts about PyCon US 2013. The goal is to provide a handy reference guide for first time attendees of the world's largest Python conference. Part I was mostly about tutorials, Part II was mostly about the first day of talks, this third post in the series will be about the second day of talks.

Early Saturday Morning

The PyCon 5K Charity Fun Run begins at 7 AM. Registration for this event is seperate from PyCon, and all proceeds go to the John Hunter Memorial Fund and the American Cancer Society.

Saturday Morning

After breakfast ends at 8:30 am, don't miss 30 minutes of lightning talks!

Then get ready as noted Python experts and proven speakers Jessica McKellar and Raymond Hettiger deliver keynotes to remember. They've changed communities and lives with the speeches given around the world.

10:50 AM talks

Getting started with automated testing (Carl Meyer) - This talk will get you moving into good test automation practices, and is presented by one of the maintainers of Django, pip, and virtualenv.
5 powerful pyramid features (Carlos de la Guardia) - Pyramid is a minimalist, modular web framework that encourages excellent coding patterns. Carlos will give you a tour of some of the great features that makes this such a powerful tool for application development.
So Easy You Can Even Do It in JavaScript: Event-Driven Architecture for Regular Programmers (Glyph) - Asynchronous frameworks are said to be the new hotness, but Python's Twisted framework has been around for over 11 years. The creator of the project isn't just a master developer, but an illuminating speaker capable of explaining the most sophisticated concepts in a beginner friendly way.

11:30 AM talks

Who’s there? - Home Automation with Arduino/RaspberryPi (Rupa Dachere) - Automate your home with Python! Learn how to use a Arduino or RaspberryPi to notify you by SMS that packages have arrived.
Copyright and You (Frank Siler) - Using and writing open source software involves a basic understanding of licensing and understanding of copyright law. The presenter, a licensed attorney, will cover the fundamentals needed to work safely in the open source world.
Scaling community diversity outreach (Asheesh Laroia, Jessica McKellar, Dana Bauer, Daniel Choi) - Leaders of community groups around the USA will explain how they get people of all backgrounds not just coding, but helping each other in order to grow the diversity of the technical community.

12:10 PM talks

Why you should use Python 3 for text processing (David Mertz) - The presenter literally wrote the book on text processing using Python (many of his techniques have since made their way into core Python). In this often technical talk he explains why Python 3 is the tool you should be using.
What teachers really need from us (Selena Deckelman) - Noted speaker and developer, Selena Deckelman, explains what the what the Python community can do to help K-12 teachers need to forward programming education.

Saturday Lunch (12:40 PM)

Food! Chow down! Don't forget the meal tickets you picked up during registration.

PyLadies Lunch at PyCon (12:30 PM)

PyLadies have organized to host a lunch for women attending the conference who love Python, or who want to learn more about Python or PyLadies. While the event is filled up, you can register to get on the waitlist.

Saturday Afternoon

1:40 PM talks

Designers + Developers: Collaborating on your Python project (Julia Elman, Mark Lavin) - Like many developers I've had good and bad experiences with designers. We both play our critical roles in projects, and it will be good to hear their presentation on how to collaborate and work in tandem as part of a team.

1:55 PM talks

Teaching with the IPython Notebook (Matt Davis) - The IPython Notebook is an unbelievably incredible tool. Imagine a Python shell that allows you to embed images and graphs right into the REPL. Designed by and for scientists, the IPython Notebook is useful for developers of all levels. You have to see it in action to believe it!

2:35 PM talks

This is a rough spot because there are four really beginner good talks by four great speakers happening at the same time. Since all of this are going to be wonderful presentations, I'm labeling them all as 'must-see'.

Who are we? A sociological analysis of the indigenous Python tribe (Jackie Kazil) - Journalist and computer scientist at the Library of Congress, Jackie Kazil uses her amazing skills to analyze the Python community's code bases across thousands of projects to determine where we've been as a people and we are going in the future. If you want to see how an expert analyzes trends in the Python-verse, this is a talk that can't be missed!
Building full-stack scientific applications in Python (Luke Lee) - Did you know that Python is used in high performance scientific applications? Tools like SciPy and NumPy are built on top of very robust, optimized C and Fortran libraries. Luke will cover their use and also how to present the data using tools like PyQt. If you are into big data, you can't miss this talk!
Customizing the Django Admin: The How and the Why (Lakshman Prasad) - Following my talk is a deep dive into Django's famous admin interface. I was fortunate enough to get a sneak preview of the talk and I can say with confidence this is going to be an incredible presentation. Don't miss it!
Location, Location, Location (Julia Grace) - Django, via the GeoDjango toolkit has amazing support for building GIS based applications. Julia's an experienced developer and great speaker, making this a can't miss talk!

3:15 PM talks

There isn't a beginner-focused talk in this slot, so I'm going to make this the roller-coaster slot. In other words, I'm going to steer you to the most intensely advanced talk at this time. Attend, buckle in the safety harness, and drink in the extreme knowledge!

Making Apache suck less for hosting Python web applications (Graham Dumpleton) - It is not hard to find developers who will tell you that Apache sucks for running Python web applications. Is there a valid basis to such claims or have they simply been misguided by the views of others? Well, Graham understands how Python serves up web pages and won't hold back on technical detail.
Numba: A Dynamic Python compiler for Science (Travis Oliphant, Siu Kwan Lam, Mark Florisson) - Numba is a compiler for Python syntax that uses the LLVM library and llvmpy to convert specifically decorated Python functions to machine code at run-time. It allows Python syntax to be used to do scientific and numerical computing that is blazing fast yet tightly integrated with the CPython run-time.

4:15 talk

For this time period there is a single talk specifically aimed at beginners. There are some other useful talks around this time, but they are aimed at educators and integrators over beginners.

Crypto 101 (Laurens Van Houtven) - Cryptography is a tricky subject, and the goal of this 4:15 PM talk isn't to turn attendees into cryptography experts. Instead, you'll have a basic understanding of how some common systems compare, and also a sense for detecting and exposing snake oil.

4:30 PM talk

Getting Python and Django Through Your Java Shop Front Door (with Jython) (Frank Wierzbicki) - Work in a Java shop and want to bring in Python? Learn how to do it using Jython, the Python implementation that works in the JVM!

5:10 PM talks

MTO On Blast: Using Python's Natural Language Toolkit to Model Gossip Blogs (Robert Elwell) - Python's powerful Natural Language Toolkit will be featured as the presenter gives an overview of Natural Language Processing.
What is the Python Software Foundation? (Brian Curtin) - This talk aims to engage us in thinking about what it takes to further Python. Whether it's an idea of code, community, or otherwise, it will jumpstart us into helping the very positive community that has brought us together for the amazing event that is PyCon.
Asset Management in Python (Robert Kluin, Beau Lyddon) - Coffeescript, Less, SASS, and all those other exciting new front end asset tools can be a challenge to deploy when doing web development in tools like Django, Flask, and Pyramid. The presenters will introduce webassets, a library designed to make deployment of assets quick and easy.

Evening Activities

Saturday night will be the evening of a huge number of fantastic fun. Two items of note (more to come):

The PyCon Charity Auction (6:30 PM+) - Along with Disney Animation Studios and many others, Audrey and I are contributing! In fact, we're submitting a one of a kind, preview copy of the upcoming print edition of Two Scoops of Django: Best Practices for Django 1.5!
The famously unrecorded Testing in Python Birds of a Feathers event.

Part IV

Stay tuned for Part IV of this series where I cover the third day of talks best suited for new Python developers!

PyCon Tutorial: Wiring Up Django Packages

2013-03-08T11:00:00-08:00

You just finished the Django tutorial. What now?

You learn how to Wire Up Django Packages!

This is a 3+ hour PyCon tutorial on March 14th at 1:20pm, in Santa Clara California at PyCon US.

In alphabetical order, the instructors:

Audrey Roy

Daniel Greenfeld (myself)

Kenneth Love

Tutorial Description

Django is part of an ecosphere of over 20,000 packages, which can be leveraged to great effect. This tutorial will teach the evaluation, use, and extension of third party Python and Django applications in your projects. This tutorial will be a lecture with a lot of detailed and annotated code examples.

This tutorial does much more than instruct on how to wire in third party packages, it also instructs on Django/Python best practices such as PEP-8, loose coupling, good database design, clarity of concept, and workable patterns.

Material covered:

Evaluating packages using Django Packages, PyPI, Read the Docs, and other sources.
Must-include packages for every project
Class-based Views
Forms
Models
Caching
Comments
Templates
Admin

Format

This will be a lecture with detailed and annotated code examples. Attendees will have a chance per section to ask questions and submit alternative packages of their own.

Prerequisites

Practical or professional experience with Django is recommended, but the material covered will be useful to anyone familiar with Python who has completed the Django tutorial.

If you are a new Python developer, we suggest in addition to our course you sign up for some of the introductory Python tutorials taught at PyCon next week. If that is not possible or you wish reinforcement, please dive into one of the following resources before you attend:

http://learnpythonthehardway.org/
Treading on Python
Think in Python
Python for Kids (meant for kids but good for beginning developers of all ages)

Registration

Space is filling up quickly, hurry up and register!

Registration for this event is $150 and occurs at https://us.pycon.org/2013/registration/register/.

Note: Registration for the conference part of PyCon is closed, but tutorials are still open.

We are not using PayPal

2013-03-02T09:00:00-08:00

In January Audrey Roy and I launched a book about Django called Two Scoops of Django: Best Practices for Django 1.5. We decided to not use PayPal. Here's why:

Open Source Events Get Burned By PayPal

PayPal has a long, sordid history of freezing the accounts of Python related conferences and events around the world. In fact, this article was born out of the fact DjangoCon Europe 2013 had its PayPal account frozen. In the past, DjangoCon Europe 2012, Plone Conferences 2008, 2011, and at least one PyCon Australia dealt with the same PayPal problem (DjangoCon 2013 was forewarned and took measures to protect itself). We also have unconfirmed reports of other Python and Django events also running into problems with PayPal freezing accounts. Going with just confirmed conferences having issues with PayPal, this is a combined total of assets in excess of over US$100,000 dollars.

It's not just a Python issue either, it's an issue that strikes other open source languages and tools. It's at the point now where conference organizers don't trust PayPal and make a point telling each other to use alternative payment gateways.

Fear, Shame, and PayPal

The terrifying thing to consider is that I suspect that the number of technical conferences affected by PayPal freezes is much, much larger. My reasoning is that most conferences keep quiet about it because they're afraid that raising a fuss will annoy PayPal's anti-fraud division. Let's also face the fact that most people feel ashamed when bank accounts they are responsible for get frozen, so probably don't publicize the issue.

The usual way conferences deal with these lockouts is conference organizers beg and borrow from friends, family, take second mortgages, local banking institutions, and pray that PayPal will eventually free their account. When you deal with a hostile, inaccessible payment gateway who won't let you provide for the hundreds or even thousands of people who paid you their hard earned money, it's the only way to get by.

The 'Needs' of PayPal's Anti-Fraud Division

While I could respect the needs of PayPal's anti-fraud division when dealing with non-fungible products like ticket sales, it's simply unacceptable that prominent conferences for open source projects are treated this way.

The software represented by these conferences drives the modern e-commerce world, including the myriad of systems that use PayPal to process sales. Yet PayPal continues to burn open source conferences year after year, and we've never heard of any conference outreach by their so-called 'developer evangelists' when a conference's account is frozen.

Legal recourse against PayPal is folly

Ask any lawyer and they'll basically say against PayPal you have no options. PayPal has an army of lawyers and in most places isn't a bank, meaning your course of action is constrained from the beginning by your agreeing to PayPal's Terms of Service (TOS). Also, in the United States, their TOS prevents you from joining a class action lawsuit against them. Whether or not that TOS clause is enforceable in court, the fact that it is in their TOS greatly reduces any faith I might have had in them because it paints a picture of a company hostile to my needs.

In talking to authors and entrepreneurs, we've just heard (and read) too many horror stories. For merchants of non-fungible goods such as digital goods like the e-book I co-authored, PayPal seems even less trustworthy.

Which means using PayPal places us at an unacceptable risk. We simply don't have the deep pockets to deal with PayPal freezing our funds for sales lost from a more reliable distribution system such as Stripe that powers our sales through Gumroad.

It is Wrong to Use PayPal

Considering PayPal's unacceptable behavior in regards to the open source community I love and merchants who try to work in their system, I feel it is wrong to support PayPal. Audrey agrees, and so our policy of not using PayPal to sell the book is set.

PayPal is at Risk

There was a day when Microsoft had what seemed to be an unassailable lock on the commercial software world. On many levels, Microsoft is a shadow of its former self, and I contend it wasn't just Apple's competition. Instead, Microsoft's contempt for their own customers and the developer community hurt them just as much.

PayPal is on the same path of self-destruction. The've gone from the scrappy company helping people grow their business to the monolithic overlord that kills businesses and well-meaning events.

PayPal's demise won't happen this year, or the next, but every time they damage their customer base and the developer community it's another nail in the coffin. I submit that unless PayPal changes its ways, within 5 years PayPal will be a shadow of its former self as the army of growing competitors such as Stripe, Balanced Payments, wepay, and Payoneer expands their availability and options around the world.

What PayPal Can Do for Conferences

PayPal does have to worry about ticket sales for bogus events, since that separates people from their money, but identifying real conferences is easy:

PayPal developer evangelists and community managers need to track every valid developer event in the world. It's the job of people in these roles to have the connections and subject matter expertise to identify real events from fake ones.
PayPal needs to sponsor these events. Why? See point #3.
PayPal's anti-fraud division needs to be informed that any PayPal sponsored event is off-limits.

What PayPal Can Do for Small Business

PayPal has its developer evangelists, community managers, and marketing departments working hard. However, at the end of the day, if you treat your customers with disrespect and a lack of trust, none of that matters. Bad press and market forces will see their revenues drop as customers will migrate to solutions that are more trustworthy and less antagonistic.

I believe that PayPal needs to revise how its anti-fraud division communicates with people who have frozen accounts. They need to change the adversarial pose they take with their own customers to one that is collaborative.

Note

If this makes you angry as it did me, take a deep breath and step back. I've found this book recommended by my friend Randall Degges useful in getting back on track and staying productive.

Two Scoops of Django 1.5 Beta Released

2013-02-28T15:00:00-08:00

After thirty four days of hard work, Audrey Roy and I are proud to announce that we've released the beta of Two Scoops of Django: Best Practices for Django 1.5. It's been a monumental effort, a labor of love, and we hope you find our efforts worthy. You can buy it right now in e-book (PDF) form on the website: http://django.2scoops.org

Two Scoops of Django: Best Practices for Django 1.5 is a compilation of best practices for the Django framework. Like any framework, Django has tips, tricks, and pitfalls that aren't documented in one place. Experienced developers know this stuff, but gleaning it off the Internet takes a lot of time. We decided to take everything we know and write it down.

We stand of the shoulders of giants: We received advice from Django experts including a BDFL, Django core developers, and subject matter experts from around the world. Reader contributions from advanced and beginning developers have also shaped the book, and we can't thank enough those who gave us their insights.

What is different about the beta release?

Added 50+ pages of content while reducing the file size from 5.1 MB to 1.5 MB.
Added list of tables, list of figures, and an index.
Dramatically improved the chapters on security, testing, logging, admin, and many more.
Added so many more code samples that even with the new example numbering system we lost count.
Speaking of code examples, most of the code samples in the book are loaded directly out of fully tested projects.
Incorporated the feedback of dozens of readers. Thank you so much for your contributions!

What does buying the beta get me?

Access the PDF version of the book.
Access to the Kindle and ePub formats when we release the final.
Access to final release of the book.

Why aren't there Kindle or ePub versions yet?

We want to support more formats, but won't do it poorly. It's going to take us at least several days to render the book in an alternative format that looks good. Since we both have multiple presentations at the upcoming PyCon US, we simply don't have time to get it done.

If I bought the alpha do I have to buy the beta?

Alpha buyers get full access the beta. In fact, you should have already received an email with a link to the latest version. If that hasn't arrived, please email us at 2scoops@cartwheelweb.com.

Is this book good for beginners?

The response from beginning Django developers reading it has been unbelievably, overwhelmingly positive. This is the book we wanted to have when we learned Django, and we're honored by the response the book has received.

Since the book isn't a tutorial, we ask that before beginners purchase it, they learn the fundamentals of Python and take the official Django tutorial.

When beginners find sections of the book above their head, they do what everyone does when reading a technical reference: skip over challenging sections until they have enough grounding to understand.

If any beginners are confused by our explanations, or have difficulties, please email us at 2scoops@cartwheelweb.com. We'll do our best to answer your questions. Your issues provide invaluable insight into what we need to improve.

Is this book good for experienced developers?

We've done our best to provide worthwhile material for developers of all levels. In fact, a lot of intermediate to advanced developers have said they've found good things to use in our book. Many have also found that by using our book as a supplementary manual, it helps them their team work better.

Is this book going to get updated?

Absolutely!

During the alpha we updated the book multiple times before knuckling down on the beta rewrite. Even after the final version comes out, we'll continue to update the book. One of the advantages of self-published e-books is we have total control of content.

When can I get a print version?

After a lot of consideration, we've decided to hold off on the print version of the book until what we call the final release, which won't be released until after PyCon US. We're very finicky and don't want buyers of the print version to have an unfinished book.

Tutorial? PyCon? Are you going?

Yes! We're going to be at PyCon next month! We're giving the Wiring Up Django Packages tutorial, and Kenneth Love (of Getting Started with Django fame) is part of our team.

While tickets to PyCon itself are sold out, you can still register for the tutorials.

When is the book's final release?

The final release won't occur until after PyCon.

Our plan for the final release of Two Scoops of Django: Best Practices for Django 1.5 is to focus on errata, alternative electronic formats, and the print version.

Can we purchase the book in bulk?

We're working on a program to provide organizations the capability to purchase many e-copies and keep the recipients of the book updated constantly. We hope to have this ready in a few days.

Can I buy the book via PayPal?

No. I've answered in full here.

We're considering other non-credit card options to handle payment.

Known Issue Warnings

Copy/Paste

There are problems with copy/pasting of code from the PDF. The new build process is turning the code examples into some sort of image type of object, which explains the strange characters. We're working on it, but aren't sure how long the fix will take.

Android

On Android, if you try to open the PDF directly from the Gumroad download, you might get a "Cannot open file" error; however, if you navigate to the download location on your phone/tablet with a file browser (e.g. Samsung's "My Files"), you should be able to open the file from there. Gumroad is currently investigating the issue.

Where should I send errata or suggestions?

Please send errata to 2scoops@cartwheelweb.com. We try to respond personally to every request.

Beginner's Guide to PyCon 2013 Part II

2013-01-23T16:00:00-08:00

This is Part II in a series of blog posts about PyCon US 2013. The goal is to provide a handy reference guide for first time attendees of the world's largest Python conference. Part I was mostly about tutorials, this post will be about registration and the first day of talks.

Registration

If you haven't done so yet, please register now. Last year PyCon sold out way in advance, and hundreds missed the event. Which is a shame, because the conference is awesome!

Keep in mind that the money you spend on registration will go towards a very good thing. You see, after the costs for running PyCon are subtracted, the rest will go to the non-profit Python Software Foundation (PSF). In addition to supporting the Python language itself, the PSF provides financial aid and grants for aspiring developers around the world. Which means the money you spend registering for PyCon will literally change people's lives. This isn't an idle exaggeration, this outreach has made a difference for you, me, and arguably the world. I would love to say more, but that's an article for another day...

Alright, on to the talks!

Talk Attendance Guidelines

At PyCon, talks are either 30 or 45 long. They represent the best and brightest in both old hands in the community and rising stars. Some quick guidelines:

Ask Questions! If the speaker leaves time for questions at the end, go and ask!
Shut the Laptop and Turn Off the Device. It's disconcerting to give a speech to hundreds of people staring at their portable electronics and not responding to your banter. Unless you are using electronics to actively take notes on the talk, consider turning them off to look at the speaker.
Don't Heckle. It's nerve wracking going up in front of hundreds of people live and tens of thousands on streaming video. Heckling is never funny and it's a good way to lose friends and make enemies. Unless the speaker asks for commentary during the talk, wait until the end and then ask your questions.

Alright, that out of the way, let's take a look at what Friday, March 15th, has to offer in terms of beginner friendly talks...

Friday Morning

The registration desk opens at 7:00 AM. Breakfast begins at 8:00 AM. From 9:00 AM to 10:20 AM is the welcome and initial keynote speeches. PyCon keynotes aren't boring - they are crazy awesome!.

In selecting these talks, I've tried to focus on the more technical ones.

10:50 AM talks

How to Except When You're Excepting (Esther Nam) - One of the rising stars of the community, Esther works for a group that demands 100% test coverage. She's going to cover the fundamentals of writing tests and why tests are a good thing.
How the Internet works (Jessica McKellar) - Want to know how the internet actually works? Jessica McKellar (also giving two tutorials and a keynote speech) will use Python to help you understand what's really going on.
Gittip: Inspiring Generosity (Chad Whitacre) - Gittip is an open source Python platform for sustaining open source development via small, anonymous donations. This is a technical case-study about the story of community.

11:30 AM talks

Scrapy: it GETs the web (Asheesh Laroia) - In 2009 I took a web scraping tutorial by the author. He impressed me with his energy and knowledge. While technically this is an intermediate talk, I know everyone will get something out of this talk

12:10 AM talks

API Design for Library Authors (Chris McDonough) - Well designed APIs are critical for how other developers interact with your code. Chris is one of the talents in the community when it comes to creating easy-to-use but extremely powerful tools. I'm delighted he's made this an introductory talk.

Friday Lunch

Quick notes:

Don't forget to keep your PyCon meal tickets you got when you registered in the morning or they won't let you into the lunch room!
Sit down to people you don't know and introduce yourself. Every time I do this I don't just get to meet interesting people, I get to meet amazing people. PyCon is full of brilliant minds and you'll never get to know any of them unless you try.

Friday Afternoon

Got lunch? Time for more talks!

1:40 PM talks

You can be a speaker at PyCon! (Anna Ravenscroft, Lynn Root) - PyCon loves new blood. Get some pointers on how to get your talks ready for submission next year!
Python 3.3: Trust Me, It's Better than 2.7 (Brett Cannon) - The world of Python is heading towards Python 3. Want to know why? Attend this talk!

1:55 PM talk

Twisted Logic (Ashwini Oruganti) - Twisted is a mature, stable asynchronous framework written in Python that's been around for over a decade. New to the framework, the presenter explains why you shouldn't be scared of Twisted.

2:35 PM talks

This slot is going to be hard because all three beginner talks are things you don't want to miss!

Loop like a native: while, for, iterators, generators (Ned Batchelder) - One of the really awesome features of Python is that you can write custom looping classes. Factor in generators and you'll be amazed by one of Python's most powerful features.
Visualizing Github, Part I: Data to Information (Dana Bauer, Idan Gazit) - How can you go wrong with visualizing GitHub data using Python? This first part of the talk is on gathering and processing of data.
Encapsulation with descriptors (Luciano Ramalho) - Python has no private fields, but the property decorator lets you replace public attributes with getters and setters without breaking client code. It's amazing what Python can do when you delve into it's subtleties!

3:15 PM talk

Music Theory and Performance Analysis with Sebastian and Czerny (James Tauber) - An amazing speaker talking about one of his favorite subjects, this will be a great beginner talk. James mentored me in my early days of Django and JQuery, and I only have good things to say about him.
Visualizing Github, Part II: Information to Meaning (Idan Gazit, Dana Bauer) - How can you go wrong with visualizing GitHub data using Python? This second part of the talk is on visualizing the data into something meaningful.

4:15 PM talks

Transforming Code into Beautiful, Idiomatic Python (Raymond Hettinger) - This is the talk that's going to bring down the house, because Raymond covers the basics in such a way that even the most advanced developers pick up new tricks. This talk will be standing room only, filled with beginners to the most senior developers on the planet. Get to this one early or you'll miss it!
Deploy your Python app in 5 min with a PaaS (Nate Aune) - Long time presenter Nate Aune doesn't just explain and demonstrate various Platforms as a Service, he also gives tips on evaluating which one is best for your needs. For someone getting into the Python web this talk is an invaluable service.

4:30 talk

If You Code, You Should Write (Brian Jones) - Python has an amazing culture of documentation. We believe it's our civic duty to document our work on private and public projects. Brian explains why this is so and how to get across the concepts of your work in the best way possible.

5:10 talks

SimpleCV - Computer Vision using Python (Katherine Scott) - This is a crash course on computer vision using the amazing ipython notebook along with NumPy and SciPy.
Planning and Tending the Garden: The Future of Early Childhood Python Education (Kurt Grandis) - After his PyCon talk last year about using Python to fend off squirrels, Kurt suddenly found himself in the world of kid's education. This talk goes over the current state of the art for kids learning programming in Python and other tools.
Write the Docs (James Bennett) - Have you heard of a certain well-documented web framework called 'Django'? James Bennett is one of the people responsible for it's amazingly high documentation standard. In this talk he's going to coach on how to write prose that inspires and invigorates people to use your tools.

Evening Activities

On friday night the social scene will kick into high gear. There will be dinners, parties, Starcraft II contests, and much more.

Part III

Stay tuned for Part III of this series where I cover the second day of talks best suited for new Python developers!

Beginner's Guide to PyCon 2013 Part I

2013-01-22T16:00:00-08:00

New to Python and thinking about going to the upcoming PyCon US for the first time? You know, that big Python conference taking place near San Francisco in March?

Or perhaps you signed up already and are getting worried about being overwhelmed by hundreds of tutorials, talks, and activities?

No worries! This multi-part guide will aid you in getting the most out of attending one of the best technical conferences on the planet. I'm going to share how to get through the whole event in good shape and cover some incredible beginner friendly events.

The Basics

First off, register already! PyCon US sold out last year around this time. Don't be left out in the cold. Buy your ticket now before it's too late!

Now that you've got that done, keep in mind that Python has two days of tutorials, three days of talks, and four days of sprints. Nine days of Python! The trick is to not get exhausted or sick. Therefore, do the following every day of the conference:

Take at least one shower. Please.
Eat at least two good meals. Make a point of sitting with different people at each meal and introducing yourself. Make new friends!
Get at least three hours of sleep. More if possible because your brain is going to go into learning overdrive.

In addition, PyCon US also has a code of conduct. I like to sum it up as, "Act professionally." Whether or not you agree with codes of conduct for conferences, PyCon US has one and it will be enforced.

Alright, enough basics, let's get onto tutorials!

Tutorials

Tutorials take place on March 13th and 14th. They each cost $150 but are well worth the price. The following are the talks I believe are best suited for beginners.

Wednesday Morning

For novice programmers start with Jessica McKellar's A hands-on introduction to Python for beginning programmers. I can vouch that the instructor is amazing, and will jumpstart you on the road to being a Python developer.
Are you an experienced developer new to Python? I suggest Going from web pages to web apps with Python by Asheesh Loroia and Karen Rustad. In 2009 I took a class by Asheesh and was impressed by his energy, enthusiasm, and depth of knowledge.
Want to learn Python web that isn't Django? Try out Paul Everitt's Pyramid for Humans class.

Wednesday Afternoon

You simply can't go wrong with David Beazley's Learn Python Through Public Data Hacking. He's the author of Python Essential Reference, an amazing instructor, and delivered a PyCon keynote last year.

If you're interested in the web, check out Nathan Yergler's Effective Django. I can assure you that he knows his Django material and is going to deliver a great experience.

Thursday Morning

If you're new to Python, you need to learn the Python environment. This means version control, PyPI, test automation, and more. Matt Harrison covers this in Scripting: from Hard-drive to Github to PyPI
Getting back to the web, Fully Test-Driven Web Development with Django and Selenium is another run through the Django tutorial, but done from the test driven perspective.
A fascinating tutorial offered at this time is A Gentle Introduction to Computer Vision. It's a great way to learn how Python is used in many creative and unexpected ways.

Thursday Afternoon

Jessica McKellar returns to instruct on how to give back to the community with Contribute with me! Getting started with open source development.
Want to learn how to program games? Look no further than Richard Jones' Introduction to PyGame. He's taught this class for years, and I can't say enough good things about him.
This talk is not meant for beginning Django developers, but it's useful anyway. Kenneth Love and Jacob Kaplan-Moss are presenting Wiring Up Django Packages.

Evening Activities

Opening Reception

Part II

Check out Beginner's Guide to PyCon 2013 Part II where I cover the first day of talks best suited for new Python developers!

Two Scoops of Django FAQ

2013-01-21T16:00:00-08:00

The launch of the Two Scoops of Django: Best Practices for Django 1.5 book has gone pretty well. The response has been almost entirely positive and sales have been pretty brisk. We've gotten a ton of great, constructive editorial feedback, which we're sorting through as we race towards the BETA release. Here are answers to the commonly asked questions:

When can I buy the book via PayPal?

We will support PayPal in either the BETA or FINAL release of the book.

In order to handle PayPal we need to find a reputable e-publisher that supports it that gives us similar rates, customer service, and product control (updates at any time, uploads of PDF, no surprise charges, etc) as Gumroad. If you know of an e-publisher who is willing to provide that same level of service, please let me know.

For the record, we aren't leaving Gumroad. We just want to support developers and students who don't have credit cards.

note: While we could certainly write our own publishing platform, that would delay work on the book. ;-)

When can I get a Kindle (mobi) or ePub version?

Our plan is to support those formats in either the BETA or FINAL releases. In fact, I really love my Kindle. I can't wait to proof my own book on my favorite reading device.

When can I get a print version?

We hope to have a print version in about 30 days. We need to ensure that the cover and layout actually looks nice, and that involves ordering copies from the Lulu, CreateSpace, whatever and reviewing them and making corrections. Then ordering again to make sure the corrections look okay. That simply takes a lot of time.

We are hoping for a mid-February release. Also, keep in mind we aren't sure yet if we'll do a print version of anything but a FINAL release.

When are the BETA and FINAL releases?

As soon as possible.

We can't make promises, but at the latest, the FINAL should be done by the start of March so people have them in hand at PyCon US.

Is this book going to get updated?

Absolutely!

In fact, we've already sent out three minor updates for grammar and spelling issues, and some bug fixes. Even after the FINAL version comes out, we'll continue to update the book. One of the advantages of self-published books is we have control of content.

How much work went into the book?

Audrey and I put in hundreds of hours writing and testing the content. We worked 12-16 hour days for weeks. It was madness, but a glorious madness. We became like hermits and toiled on the book while ignoring distractions like sleep, food and consistent exercise.

Our amazing technical reviewers also worked very hard to provide us with critical commentary, advice and corrections.

Where should I send errata?

Please send errata to 2scoops@cartwheelweb.com. The first person who reports an issue or makes a suggestion that we adopt will receive credit in the book. If not in the changelog, then in the dialogue of the book itself.

Are you going to PyCon US?

Absolutely! Wouldn't miss PyCon US for the world! Audrey and I met and fell in love at PyCon US 2010! We'll be there for at least the second day of tutorials, all the talks, and all the sprints. I've even got a blog post coming that should serve as a guide for beginning Python and Django developers attending the event.

Shameless Plug: We're both speaking and giving a tutorial. Sign up for the tutorial or attend our talks!

Wiring Up Django Packages (Tutorial)

You just finished the Django tutorial. What do you do now? You wire in Django Packages! Django is part of an ecosphere of over 20,000 packages, which can be leveraged to great effect. This tutorial will teach the evaluation, use, and extension of third party Python and Django applications in your projects. This tutorial will be a lecture with a lot of detailed and annotated code examples. Instructors are Audrey Roy, Daniel Greenfeld, and Kenneth Love (of Getting Started with Django fame).

Advanced Django Forms Usage (Talk)

Django forms are really powerful but there are edge cases, especially with class based views), that can cause a bit of anguish. This talk will go over how to handle many common solutions not currently described in the core documentation. It will also cover useful third-party libraries. Speaker is Daniel Greenfeld.

Using Python To Generate Art And Sound (Talk)

The trick to navigating the overwhelming Python audio/imaging landscape is understanding how the fundamentals work, using common data processing/visualization libraries, in-depth code samples, and simple math operations. I use the stdlib, NumPy, SciPy, Matplotlib, PIL, and PyCairo to create building blocks, which I then combine to demonstrate advanced sound and image generation techniques. Speaker is Audrey Roy.

Are you going to Django Circus?

We hope so. Django Circus, a.k.a. DjangoCon Europe, is one of the important industry events for us and we're trying to figure out logistics.

Our problem is that 2012 was a very hard year - the majority of our clients have not paid for the work that we did. Collecting is turning out to be extremely difficult, time consuming, and unpleasant. This is impacting everything we want to do in 2013.

How does the ALPHA work?

The ALPHA is a preview version that we are selling at an introductory price of $12. When you purchase the ALPHA, you get access to the BETA and FINAL versions when they are released.

In case you're wondering, we are planning to increase the price of the book when we release the BETA and FINAL versions.

Our Django Book Has Launched

2013-01-17T10:00:00-08:00

We (Audrey Roy and I) wrote a book on Django! It's called Two Scoops of Django: Best Practices for Django 1.5, and you can buy it right now in e-book (PDF) form on the website: http://django.2scoops.org.

Django, like any framework, has tips, tricks, and pitfalls that aren't documented in one place. Experienced developers know this stuff, but gleaning it off the Internet takes a lot of time. We decided to take everything we know and write it down. This book reflects a portion of what we documented, and if it does well we plan to follow it up with other references.

If you've followed this blog or watched our talks you've know we've explained tons of Django and Python related examples. We enjoy using Django and Python to build stable, fast web applications quickly and efficiently. This is the book we would have wanted while learning the intricacies of our tools, and then kept it handy for reference.

In the book we cover everything from customizing the Django 1.5 User model, forms, views, templates, security, bottleneck analysis and so much more. We tie in third-party packages from the Django and Python community. We discuss the positive patterns that allow for rapid development, and point out the anti-patterns that cause problems. The result isn't a walk-through or tutorial, but a reference guide for anyone with at least a basic understanding of Django.

We tried to make the code copy/pastable, but we are constrained by the limits of PDF. While the code examples display nicely, when pasting from Adobe Reader the leading white spaces vanish. We apologize and ask if anyone knows how to address this issue to let us know at 2scoops@cartwheelweb.com.

Even with an amazing cadre of technical reviewers, there is always the possibility of errors. We are going to be actively maintaining the book, so if you find a mistake, let us know and we'll update it and send out an updated copy of the PDF to all buyers.

If you'd like to buy the book (or learn more about it), you can do so here: http://django.2scoops.org

New Year's Resolutions for 2013

2012-12-31T10:45:00-08:00

I'm one of those people who not only likes to make New Year's Resolutions, I like to blog about them.

Resolution

Go to at least one technical conference in North America, South America, Europe, Asia, Africa, and Australia.
Visit at least one new nation. It's hard to come up with preferences since the whole world is so exciting.
Go back to at least one of the nations I've visited before.
Take a road trip across the USA.
See the Grand Canyon.
Keep to a 32 waist for the whole year.
Visit friends and family back east.
Blog once a week. That is at least 52 blog entries!
Visit the beach for more than one day.
Take a fun class with Audrey.
Teach some Python or Django.
Build something real using Twisted as a critical component.
Take a high level Python class from the likes of Raymond Hettinger or David Beazley.
Commit to core Python
Visit my Son.
Upload all my outstanding pictures to Flickr!
Pull off an Aú sem Mão (no-handed cartwheel).
Attend my first Capoeira Batizado (formal gathering).
Work out at least three times a week.
Learn how to surf or snowboard.
Have a beer with Thomas, Andy, Andy, Tony, Garrick, Bernd, and the rest of Ye Aulde Gange.
See my old DC area friends such as Eric, Chris, Steve, Beth, Sarah, Daye, Renee, Kenneth, Leslie, Whitney, Dave, and many others.

Accomplished

Publish as least one book. See Two Scoops of Django
Update my blog to Pelican 3.
Implement a custom theme design so my blog looks more unique. ;-)

New Year's Python Meme 2012

2012-12-28T18:00:00-08:00

Tarek Ziade has a habit of ending the year with a Python-themed meme. I've matched his meme the times he previousstarted it, and as you can tell from the title of this blog post I'm matching him yet again.

1. What’s the coolest Python application, framework or library you have discovered in 2012?

This question took some thought. It was a toss-up between these three choices:

Django Class Based Views (CBVs) allow developers do amazing things with Django, but needed some polish (improved documentation, missing functionality) to be able to shine.
ReportLab for generating PDF allowed me to create some impressive results, but the API needed updating. The other Python PDF libraries might be better, but getting images to work trivially in them
If I didn't completely agree with Armin Ronacher on the subject, I might have gone with a combination of PyMongo, MongoEngine, and MongoKit. Just like Armin, I've learned through working with schemaless databases to know that schemas are awesome.

The winner?

Django Class Based Views

In 2012 what I managed to accomplish with Django CBVs was incredible. From early self-instructional work I did for django-mongonaut, to client efforts and personal projects where I honed my craft, plus examples I blogged about or helped get into Django core, it made for a great year. Also I wasn't just productive personally, I helped increased the productivity of others around the world.

And you ain't seen nothing yet!

2. What new programming technique did you learn in 2012?

I thought I understood multiple inheritance.

I really did.

However, since the start of this year I've delved really deep into it, only to discover just how much I didn't know. While that didn't do my ego any favors, it was a nice refreshing reminder not to get arrogant about one's skills.

3. Which open source project did you contribute to the most in 2012? What did you do?

The answer to this is Django.

After using Django professionally for 2.5 years, I finally began contributing to the core framework at the DjangoCon Eu 2012 sprints. I joined a group of other dedicated people who decided to improve the Django CBV documentation, our goal being setting a new standard for documentation. I'm not sure if we set a new bar in documentation, but we did improve on the existing material.

I branched out into some other areas of core Django development with mixed results, which played out that way because I just didn't have the time to do more.

4. Which Python blog or website did you read the most in 2012?

As always, http://planet.python.org.

5. What are the three top things you want to learn in 2013?

Really advanced Python as taught by Raymond Hettiger or David Beazley.
I really want to learn Twisted.
How to do an Aú sem Mão.

6. What is the top software, application or library you wish someone would write in 2013?

I've got a couple:

A Python SDK that produces results that work perfectly for both modern iOS and Android. Think Corona SDK but with Python.
A modern PyGame release that installs trivially on Mac OS X.

Want to do your own list? here's how:

copy-paste the questions and answer to them in your blog
tweet it with the #2012pythonmeme hashtag

Developer Time

2012-12-04T12:00:00-08:00

This blog post got started with a tweet.

That tweet got retweeted a lot by developers. And system administrators. And database administrators. And any creative type.

As of December 7th, 2012, it had been retweeted over 500 times, a personal best. Obviously I struck a chord that resonated with a lot of people.

Developers should have 4-6 hours of uninterrupted activity each day. Each 3-5 minute interruption costs more than you can imagine.
— Daniel Greenfeld (@pydanny) December 3, 2012

Why did this tweet resonate with so many people?

What I said in that tweet was not new - dozens if not hundreds of others have tweeted similar thoughts before and gotten many retweets. Heck, it's been written about in blogs and articles for years, either as a huge rant or often as an effort to politely educate others on how to set up a developer/operations/creative shop.

I think this is because developers/engineers/creatives (and good managers) know that even a tiny distraction to someone "in the zone" takes more than just the 3-5 minutes it takes for you to finish a question and get an answer. Plus, because you are trying to remember the pieces in your head, you won't be able to give this issue your full attention. Which causes a whole other set of issues.

We know this. Good managers know this. Good companies live off of it.

And yet this remains a huge problem for many of us because the distractions keep happening. Maybe it's because your manager doesn't understand this issue, or you have a team mate who can switch in-and-out of the zone with impunity. Or because you foolishly leave the chat/IRC/phone on and you get pinged.

How do we fix it?

We can't.

I know this sounds depressing, but I'm being realistic. Here's some reasons that apply:

Unless you are fortunate enough to be working on a solo effort, you are working with other developers/engineers/creatives. And if they can't ask you questions or inform you about critical stuff then you or they can waste hours.
Meetings, short or long, online or in person, are a necessary part of any operation. If not with your boss or underlings, then with clients or users.
Your boss/co-workers/underlings/kids/pets/parents don't get it that you can't be interrupted.
As a freelancer, you can't be inaccessible to existing or potential clients.

No really, how do we fix it?

Doesn't matter how frequently you ask the the question (or rant about it), I don't believe this problem can be really solved. However, here are some possible mitigations:

1. Maker's Day

My friend, Craig Kerstiens describes how Heroku gives engineers a full day each week (Thursday) to focus on getting stuff done. No meetings, no standup, just 100% uninterrupted quiet time to focus on making stuff.

Suggest this at your company and see how it goes!

2. Blocks of Time

As of 2010, Eldarion broke up their work day into at least two sizable chunks of uninterrupted activity. Between the chunks you communicated with co-workers. The advantage of this approach is if you went down the wrong path or someone had a critical question, the period between chunks addressed this issue.

In theory this is how a lot of places work (lunch being the break). The reality is that the distractions often still pile in. For Eldarion it worked because everyone was remote.

3. New Desk Location

If working at an office, asked to be moved to a place with less foot traffic. The downside to this is you can end up in noisy, cold, dark places.

4. New Job

Switch to a new job that promises less distractions.

Maybe I'm wrong

I'm more than happy to be proven wrong.

Maybe there is a way to get 4-6 hours a day of uninterrupted maker's time every work day.

If you've got any ideas, let me know!

Stay with the Django CBV defaults!

2012-11-27T16:00:00-08:00

One virtue of Django Class Based Views (CBVs) is that they come with pretty good default settings. The virtue of this is you can really pare your code down in size and complexity.

For example, here is an implementation of CBVs based on a straight-forward Django model , stuffage.models.Stuff, that has a get_absolute_url method:

stuffage/views.py:

from django.views import generic

from stuffage.models import Stuff

class StuffDetailView(generic.DetailView):
    model = Stuff

class StuffListView(generic.ListView):
    model = Stuff

class StuffCreateView(generic.CreateView):
    model = Stuff

class StuffUpdateView(generic.UpdateView):
    model = Stuff

stuffage/urls.py:

from django.conf.urls.defaults import patterns, url, include

from stuffage import views

urlpatterns = patterns("",

    url(
        regex=r"^create/$",
        view=views.StuffCreateView.as_view(),
        name="stuff_create",
    ),
    url(
        regex=r"^update/(?P<pk>\d+)/$",
        view=views.StuffUpdateView.as_view(),
        name="stuff_update",
    ),
    url(
        regex=r"^(?P<pk>\d+)/$",
        view=views.StuffDetailView.as_view(),
        name="stuff_detail",
    ),
    url(
        regex=r"^$",
        view=views.StuffListView.as_view(),
        name="stuff_list",
    ),
)

These four CBVs will default to the following three templates without any action on my part:

stuffage/stuff_detail.html (StuffDetailView)
stuffage/stuff_form.html (StuffCreateView, StuffUpdateView)
stuffage/stuff_list.html (StuffListView)

So easy I use a simple script to render all this code!

What about doing this all in the urls.py?

Yes, I could do this all in the urls.py, but real Django code involves doing some logic in views, no matter how skinny you try to make said views. While I'm a huge proponent of logic in fat models, invariably I'm adding to the view context, or doing something else that requires tweaking of CBV settings.

The problem

One trait of developers is we like to tinker.

Unfortunately, I keep seeing developers tinkering on the settings for Django CBVs without any reason besides tinkeringWhich means you get things like:

unfortunately tinkered stuffage/views.py

# 1. Using template_name means extra code and extra developer lookup time.
# 2. Changing the context_object_name means extra code  and extra developer
#       lookup time.
class StuffDetailView(generic.DetailView):
    model = Stuff
    template_name = "stuffage/stuffs.html"
    context_object_name = 'stuff'

unfortunately tinkered stuffage/urls.py

# 1. Logic into your URLConf should be kept to a minimum
# 2. Unless you are using the same view more than once, specifying the
#       template_name here is a waste of code. And makes it harder to
#       understand the view.
url(
    regex=r"^$",
    view=views.StuffListView.as_view(template_name="stuffage/stuffs.html"),
    name="stuff_list",
),

# No matter how fat your models get, you always end up extending all views,
#   so this will have to be moved into the formal views.py at some point. So
#   why not start with it in the views.py in the first place.
url(
    regex=r"^$",
    view=ListView.as_view(
        model=Stuff,
        template_name="stuffage/stuffs.html"),
    name="stuff_list",
),

Don't forget you can also tinker/tweak formats and slug/pk identifier defaults, and a ton of other things that are part of Django CBVs. While this gives you great power, if misused that power can cause grief in terms of code obfuscation and the need for additional testing.

My opinion is that these defaults were meant as a standard for the CBV to operate, upon which developers familiar with the Django CBV API could extend and expand their code for minimal effort. Yes, you can tweak them to match your preferred patterns, but that's extra work. Work you shouldn't be doing if you can avoid it.

My Advice

Stick with the defaults and only modify behavior that actually needs to be modified. For example, if you want to show multiple versions of a ListView you might do something like:

stuffage/urls.py with a pydanny approved use of template_name

url(
    regex=r"^$",
    view=views.StuffListView.as_view(),
    name="stuff_list",
),

# Same view but with a template designed to show larger list items.
url(
    regex=r"^large/$",
    view=views.StuffListView.as_view(template_name="stuffage/stuff_list_large.html"),
    name="stuff_list_large",
),

Summary

This is the pattern I follow when I build projects. I stick to the framework standard as much as possible. Since many systems rely on convention over configuration, this makes it easier and faster to develop projects, be it Django, Twisted, or some other tool.

It's the work you can see in my recent public projects, and what I want to port to long existing sites like Django Packages.

Case Study: URL Design for petcheatsheets.com

2012-11-21T22:00:00-08:00

Backstory: On Saturday, November 17, 2012 Audrey Roy and I decided to participate in the Petcentric hackathon, a Los Angeles area Pet-themed product/coding contest held at Amplify. We arrived a bit late, but armed with Audrey's idea of creating a pet based reference sheet for owners, pet sitters, vets, and anyone else needing a card with data on pets, we got to work. About eight hours later we toggled a DNS switch and petcheatsheets.com was live.

Update: Pet Cheatsheet's owner's pet information is private, because it includes emergency contact information that often includes phone numbers, email addresses, and even physical addresses of family members and friends. Maintaining the privacy of pets and their owners was also a consideration in implementation.

URL Design Thoughts

During development, one of the things I considered carefully was URL design of the primary feature, which was pets. The obvious choice was to go with a design that identified owners with pets:

/<owner_username>/<pet_name>/

However, upon reflection, this didn't sit well with me. What if a pet changed owners? Identifying a pet with a particular owner in the URL meant that if we ever added a 'transfer ownership' feature, there would be extra work. Also, if we ever implemented a sharing feature, changing URLs on a pet going to the same veterinarian their whole life might make the veterinarian's list of pets and their URLs invalid.

With that in mind, I decided to go with an identifier and pet name, where the pet name was actually not used in the lookup:

/<pet_id>/<pet_name:not_required>/

One more thing, rather than just use the pet table's primary key as <pet_id> I decided to go with base36 (0-9 and a-z) encoding. It's not unlike what URL shortening services do, and if we gained any traction, it makes recitation of a pet's URL easier. So the final result was actually:

/<pet_id:base_36_encoded>/<pet_name:not_required>/

Implementation

Here's a simplified view of the final implementation, starting with the model:

# pets/models.py
from django.db import models
from django.utils.translation import ugettext_lazy as _

class Pet(models.Model):

    name = models.CharField(_("Pet's name"), max_length=100)
    identifier = models.CharField(_("identifier"), max_length=50,
        null=True, blank=True, db_index=True)
    # More fields...

Then the form:

# pets/forms.py
from django import forms

from pets.models import Pet

class PetForm(forms.ModelForm):

    class Meta:
        model = Pet
        fields = ('name', ) # more fields

With model and form, we build the views:

# pets/views.py
from django.shortcuts import get_object_or_404
from django.utils.baseconv import base36
from django.views.generic import CreateView, DetailView, UpdateView

from braces.views import LoginRequiredMixin

from pets.forms import PetForm

class PetCreateView(LoginRequiredMixin, CreateView):

    model = Pet
    form_class = PetForm

    def form_valid(self, form):
        pet = form.save()
        pet.identifier = base36.encode(pet.pk)
        pet.owner = self.request.user
        # Save again - it's not taking THAT many server cycles AND we needed
        #    the pet.pk in advance to generate the pet.identifier
        pet.save()
        return super(PetCreateView, self).form_valid(form)


class GetPetMixin(object):
    """ Any view that needs to get a Pet object can use this Mixin

        Pet Cheatsheet's owner's pet information is private, because it
            includes emergency contact information that often includes phone
            numbers, email addresses, and even physical addresses of family
            members and friends.
    """

    def get_object(self):
        pet = get_object_or_404(Pet, identifier=self.kwargs['identifier'])

        if pet.owner != self.request.user:
            # Rather than a 'forbidden' result, we want to show a 'Pet Not
            #    Found' page so we can educate site users.
            raise Http404
        return pet

class PetDetailView(LoginRequiredMixin, GetPetMixin, DetailView):
    pass

class PetUpdateView(LoginRequiredMixin, GetPetMixin, UpdateView):
    model = Pet
    form_class = PetForm

class PetPDFView(LoginRequiredMixin, GetPetMixin, DetailView):
    model = Pet

    # snip: lots of code for rendering the PDFs.

Then we wire up the views into the urls:

from django.conf.urls.defaults import patterns, url

from pets import views

urlpatterns = patterns("",

    url(
        regex=r"^build-cheatsheet/$",
        view=views.PetCreateView.as_view(),
        name="pet_create",
    ),
    url(
        regex=r"^update/(?P<identifier>[\w\d]+)/(?P<slug>[\w\d\-\_]+)/$",
        view=views.PetUpdateView.as_view(),
        name="pet_update",
    ),
    url(
        regex=r"^(?P<identifier>[\w\d]+)/(?P<slug>[\w\d\-\_]+)/$",
        view=views.PetDetailView.as_view(),
        name="pet_detail",
    ),
    # snip: a lot of other views
)

Result

In the image below you can see how Marko's URL has his own unique identifier, along with his name. I can change the name in the URL or even in the database, but so long as I don't modify the identifying part of the URL (1m), his information always shows up.

Django GetOrCreateView

2012-10-16T22:00:00-07:00

Today I decided to use the Django class based view (CBV) CreateView, but I wanted to avoid duplications and submit to the view from the front page of a site. The reason was I needed a simple newsletter signup form. This is what I cooked up and should work for Django 1.3, 1.4, and the forthcoming 1.5 release. Here is what I did:

1. Installed dependencies

This version requires the following package to be pip installed into your virtualenv.

django-extensions so we can have easy timestamps on models.

This also needs to be added to your list of INSTALLED_APPS:

INSTALLED_APPS += (
    'django_extensions',
)

2. Defined the model

The model is really simple, and inherits from TimeStampedModel so we know when people signed up:

from django.db import models

from django_extensions.db.models import TimeStampedModel

class NewsLetterSignup(TimeStampedModel):

    email = models.EmailField("Email")

    def __unicode__(self):
        return self.email

3. Wrote the view

Here's the somewhat challenging part that forced me to dive into Django's source code. Even with the documentation work we've done over the past few months, it's clear we've got a long way to go.

Because of that source code diving, for this blog post I really did my best to document why I did things in the NewsLetterSignupView.form_valid() method.

from django.http import HttpResponseRedirect
from django.views.generic import CreateView

from .models import NewsLetterSignup

class NewsLetterSignupView(CreateView):
    """ Signs up users to a newsletter """

    model = NewsLetterSignup
    success_url = '/newsletter-signed-up/'  # replace with reverse

    def form_valid(self, form):
        """
        If the form is valid, save the associated model.
            (django.views.generic.edit.ModelFormMixin)
        If the form is valid, redirect to the supplied URL.
            (django.views.generic.edit.FormMixin)
        """

        # Get the email from the form.cleaned_data dictionary
        email = form.cleaned_data.get("email", "")

        # Get or create the signup. We don't need to do anything with the
        #   model instance or created boolean so we don't set them.
        NewsLetterSignup.objects.get_or_create(email=email)

        # Don't use super() to inherit as it will do a form.save()
        # You could call the FormMixin's form_valid() method but I think
        #   using a HttpResponseRedirect() much more explicit.
        return HttpResponseRedirect(self.success_url)

4. Wired it together

In urls.py:

from django.conf.urls import patterns, url
from django.views.generic import TemplateView

from .views import NewsLetterSignupView

urlpatterns = patterns('',
    url(regex=r'^newsletter-signed-up/$',
        view=TemplateView.as_view(
            template_name="pages/newsletter_signed_up.html"
        ),
        name='newsletter_signedup',
    ),
    url(regex=r'^newsletter-signup/$',
        view=NewsLetterSignupView.as_view(),
        name='news_letter_signup',
    ),
)

Closing thoughts

First off, you'll notice I didn't include the pages/newsletter_signed_up.html because for this case it's too trivial.

Second, this is one of those very clear cases where a functional view would have been so much easier compared to the effort I spent writing this as a class based view. The line count would have been about the same, but the mental bandwidth involved in figuring this would have been a fraction of the effort I spent.

Third, this is probably better served with an implementation django.views.generic.FormView. Oh well...

Fourth, I want to see a configurable version of this in the next release of django-braces. ;-)

Los Angeles Open Source Sprint on Nov 4th!

2012-10-14T14:00:00-07:00

Yet again, myself, Audrey Roy, and a small but elite cadre of volunteers are feverishly at work putting together another Los Angeles open source event. Our last effort went smashingly well, and like any good engineer, we're ready to scale up - this time to 150 attendees!

LA Open Source #5 is a day long coding event for Open Source developers of all languages and skill levels to come and code like fiends. They'll be joined by dozens of either really smart coders or nice people like me. Sponsors are providing food, drinks, venue, and more!

RSVP at http://www.meetup.com/LA-Hackathons/events/85658952/

I'll be there to:

Co-lead the event with the assistance of the awesome Los Angeles technical community!
Code like a fiend. I plan to work on core Django. Maybe some ember.js stuff too.

And now to open the floor to questions...

When and where?

Where:

Cross Campus (crosscamp.us)
820 Broadway
Santa Monica, CA

When:

November 4th, 2012
10 AM to 9 PM

Is this like a Hackathon?

Yup. See http://en.wikipedia.org/wiki/Hackathon#Sprints

What if I don't have a project of my own to bring? Should I come?

Heck yeah! There will be a number of projects around that you can join and contribute to in order to make the world a better place. The current list of projects include things like:

Django - in fact, this time we've got at least one core Django developer!
Node.js
Python
Rails
Ruby
Pyramid
Salt Stack
Clojure
PHP

I'm just starting as a developer, should I come?

It depends.

If you've never coded before, this isn't the right place. Instead, you might consider one of the local coding workshops or classes. In fact, here's a good bi-weekly hack night / study group for you.

If you've done a tutorial or two, sprints can be a great way to learn new skills or hone your technique by sitting alongside experienced developers who actually need your help. A lot of projects have what are called 'low hanging fruit', which are 'simpler' tasks saved for beginner developers to wet their teeth on. Things I've learned at events like these include Git, Mercurial, JQuery, and a hundred other things that have made me a better coder.

Can I come and do one of the following:

Recruit.
Get people to work on my private project.
Get people to work on my start-up idea.

No.

This is not a job fair or way to get free developers services, and we don't want unnecessary distractions.

On the other hand, if you want to help sponsor we mention you on the official event description, in social media and many other perks. Please contact us at sponsors@laopensource.org.

In addition, we're considering future events that mix in business into our efforts. If you're interested in that sort of thing then stay tuned for more announcements.

What about sponsorship?

In return for sponsorship we mention your organization or person on the official event description, and provide many other perks including mention in social media.

In fact, our attendees are well-noted for their feverish support of sponsors. Developer-only open source events are rare in Los Angeles, so attendees are incredibly grateful to those who make the event possible.

If interested, please contact us at sponsors@laopensource.org.

What do I need to bring?

Your own functioning laptop with power cord. Neither event organizers, the venue, or sponsors are providing equipment. We also encourage you to bring a power strip labeled with your name.

I'm sold! How much does it cost and where do I register?

The event is $10, covers food and drink for the day, and you can RSVP by clicking on the image below

We need more PyCon US 2013 submissions!

2012-09-27T18:00:00-07:00

The PyCon US 2013 call for papers (CFP) ends tomorrow, September 28th, 2012. We need more talk and tutorial submissions. Talks are 30 or 45 minute efforts in front of the PyCon audience and are recorded for posterity. Tutorials are three hours long and are given to attendees who have paid an additional fee in order to slurp in knowledge from the masters.

On the 3 hour tutorial side of things, we especially need more Intro to Python level submission. That means getting beginners up to speed on basic Python techniques, so they can then exploit the other tutorials, conference, and sprints to their full advantage.

Now onto some questions...

1. I would like to give a tutorial but it's so much work to put together 3 hours of quality content.

The organizers of PyCon recognize that putting together a quality tutorial is an amazing amount of work. Which is why tutorial presenters are compensated for their effort.

2. What is the most likely length talk to be accepted? 30 minutes or 45 minutes?

The vast majority of PyCon sessions are 30 minutes long, so 45 minute slots are rare and valuable commodities. So if your talk needs to be 45 minutes long your proposal has to really speak to the PyCon talk reviewers.

Not only that, if reviewers send you information requests for any duration talk or tutorial, you dramatically increase your odds of talk acceptance with timely responses.

And, as said before, PyCon really needs more introductory level Python tutorial submissions.

3. I would like to present a talk or tutorial but I can't afford to come to PyCon.

PyCon's financial aid program is said to favor accepted speakers to PyCon. They really want you to come!

4. I would like to present but I can't come up with a good idea!

I had this problem as well! Then I looked at the Suggested Tutorial Topics and got some ideas.

5. Ack! I've got a talk idea but it's going to take me too much time to put it together!

Submit the talk anyway before the talk and in fields you aren't ready to fill in, simply put 'TBD'. Then over the course of the next few days replace TBD with real material. Don't wait too long though to fix those TBDs, no more than a week!

6. I'm a beginner/nobody in the community, is there any chance my proposal will get accepted?

Absolutely!

The PyCon talk/tutorial reviewers love to see new people present. While experienced/proven speakers have an edge, good talk/tutorial proposals from promising candidates can make it into the conference. Carefully double-check your submission, be responsive to reviewers, and stay positive. I'm not saying you will get in, but I am saying it's worth the effort.

Good luck!

PyCon 2013 is going to be bigger and better than any previous year. That's because we pull together as a community to run an amazing event that is known to jumpstart careers and cause amazing life changes. We can't do this without you, so hurry up on your PyCon US 2013 submissions!

Thoughts on my stack

2012-09-04T10:30:00-07:00

I'm an open source developer. I use Python, Django, PostgreSQL, JQuery, MongoDB, Memcached, and Redis. I push production code to Linux servers.

And yet:

My laptop runs Apple's Mac OS X.
My primary editor is Sublime Text.
My production servers are provided mostly by Heroku.
All my sites are hosted on Amazon EC2.

I've got nothing against Apple, Heroku, or Amazon, but their tools are not open source.

And that's food for thought.

A Public Service Message to the Python Community

2012-08-20T14:00:00-07:00

Hi, I'm Daniel Greenfeld. You might know me from my blog. I'm here to talk to you about a very import subject: Submitting your talk early to PyCon US.

Last year there were hundreds of talks were submitted for just a very few speaking slots. Unpaid volunteers labored for hours before and after their normal jobs reviewing and debating which talks were to get in. The volume of talks combined with simple reviewer fatigue means that earlier submitted talks got more attention.

This year, following the pattern of previous years, we're going to have 25% more talk submissions.

Please don't leave your poor talk or tutorial out in the cold.

Submit early.

Help us fix the problem in one of two ways:

Production notes for this service message

I've always wanted to do one of those TV public service messages done by B or C grade actors. As a D-grade blogger I thought I might be a pretty good match for the role. :-)

Django Requirements 2012-08-15

2012-08-15T18:00:00-07:00

A little over three months ago I blogged about my preferred requirements list. It's now nearly the eve of Django Dash, and I feel it's time to update the list. I'm going to bump the versions on some of the existing packages and add some new ones to the list.

New Packages

django-braces==0.1.3

Want to use Django Class Based Views but unhappy with the missing components like LoginRequiredMixin, SelectRelatedMixin, and even StaffuserRequiredMixin? Not to worry, as this library will make Django CBVs 134% easier to use.

django-secure==0.1.2

Django is rather secure, but there is a checklist of things that the security experts want you to do. Save yourself forgetting something and use this library to do all those little things.

django-profiletools==0.1.3

Have you ever used the django-debug-toolbar and noticed that you did that same request.user.get_profile() call a dozen times? Ever want to just call that once? This library, by yours truly, resolves the issue. It loads the user's profile object once, and then passes it down the request chain.

Existing Packages

Django==1.4.1

If you need sessions, forms, templates, and relational database models, then I can argue you've got the ideal Django project. Make certain you are running the latest Django version (1.4.1). If you have any reason to stick to the Django 1.3 series, I advise bumping it up to Django 1.3.2.

psycopg2==2.4.5

This is the database connector to PostgreSQL, which is what you should be using. Django is known for playing 'nicer' with PostgreSQL than say... MySQL.

django-extensions==0.8

Because amongst other things this library gives you, I never want to write my own TimeStampedModel ever again. :-)

South==0.7.6

Django gives you the freedom to migrate data in the way you want. The way I want to do it is via South.

django-registration==0.8.0

The common go-to tool for non-Social registration.

This is a very solid tool, but you do have to make your own templates or find someone's fork that has a copy of templates that match.

django-floppyforms==1.0

An excellent tool for making your forms HTML5-ish out of the box. It allows full control of form rendering in the templates.

django-crispy-forms==1.1.4

The child of my own django-uni-forms, this will let me create forms using div-based controls super fast, and do layout customizations if I need them.

django-heroku-postgresify==0.2

This tool makes getting the PostGreSQL settings out of Heroku trivial.

django-heroku-memcacheify==0.3

This tool makes getting the memcache settings for Heroku trivial.

gunicorn==0.14.6

All the cool kids who play in devops swear by Gunicorn.

Installing the above packages

Never copy/paste these libraries directly into your projects. Do it the right way: use proper Python dependency management.

Create a requirements.txt file and install them as proper dependencies. The file should contain the following text:

Django==1.4.1
South==0.7.5
django-braces==0.1.3
django-crispy-forms==1.1.4
django-debug-toolbar==0.9.4
django-extensions==0.8
django-floppyforms==1.0
django-social-auth==0.7.4
django-heroku-memcacheify==0.3
django-heroku-postgresify==0.2
django-profiletools==0.1.3
django-registration==0.8.0
django-secure==0.1.2
gunicorn==0.14.2
psycopg2==2.4.5

Once you have that, you install them thus in your virtualenv:

pip install -r requirements.txt

Now that I have all this, it's time to code!

Curiosity has landed!

2012-08-06T08:00:00-07:00

I'm terribly proud of what was accomplished last night: A car-sized robot weighing a whopping 8,580 pounds (3,893 kg) was landed on Mars. It is packed with scientific packages that will expand our knowledge of Mars, and has the power to use them for at least 14 years.

Simply put, this is an incredible step forward.

As an American, I'm proud that our nation is capable of putting together such amazing feats of engineering and science. People from a diverse arrangement of backgrounds and origins came together to make something incredible happen.

As a human being, I'm proud that we have yet again proven that we are still capable of working together to make amazing things happen.

As someone who grew up in the 1970s, and someone who is proud to have worked at NASA for over 5 years, this is my American dream.

Going Forward

Curiosity is about to do amazing things and will tell a story to us that we'll never forget.

Alas, NASA is a terribly easy target for politicians. It's budget has shrank to a small fraction of the it's early 1970s high, and it remains a target to this day. While the politicians congratulate the JPL team on one hand, they are also cutting the JPL budget in a short-sighted attempt to make themselves look good. History will not be kind to such politicians.

The answers to our questions and our problems will be found in the heavens. Let's increase NASA funding and populate the world with miracles of science.

Deadline for DjangoCon Financial Aid requests is tomorrow!

2012-08-02T21:30:00-07:00

For 2012, DjangoCon US has moved to Washington, DC, and everyone is invited! If you need financial support to get there, the Django Software Foundation and PyLadies have paired with a number of forward thinking sponsors to help get you there. All genders are eligible for assistance, so don't hesitate to submit your application.

The deadline for financial aid requests is tomorrow, Friday, August 3rd by 11:59pm Eastern Time.

Apply while you still have time!

Attaching custom exceptions to functions and classes

2012-08-02T09:30:00-07:00

Having too many custom exceptions on a project can be a pain, but a few choices ones are really nice. The problem is that in complex libraries having to import both functions and exceptions becomes a drag. To mitigate having to remember to import custom exceptions, this is a handy pattern you can use in a project and can be done on both functions and classes.

Attaching a custom exception to a function

This works because Python functions are first-class objects. They can be passed around as things, and in this case, have things assigned to them.

# logic.py
class DoesNotCompute(Exception):
    """ Easy to understand naming conventions work best! """
    pass

def this_function(x):
    """ This function only works on numbers."""
    try:
        return x ** x
    except TypeError:
        raise DoesNotCompute

# Assign DoesNotCompute exception to this_function
this_function.DoesNotCompute = DoesNotCompute

Now I can import the function, and it won't just through DoesNotCompute exceptions, it will also carry the function along with the import:

>>> from logic import this_function
>>> this_function(5)
3125
>>> this_function(4.5)
869.8739233809259
>>> this_function('will throw an error.')
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "logic.py", line 10, in this_function
    raise DoesNotCompute
DoesNotCompute

Alright, that doesn't seem like much, but let's add in some exception handling:

>>> try:
...     this_function('is an example')
... except this_function.DoesNotCompute:
...     print('See what attaching custom exceptions to functions can do?')
...
...
See what attaching custom exceptions to functions can do?

Attaching the custom exception to a class

All we have to do is enhance our existing logic.py file by adding ThisClass:

# logic.py
class DoesNotCompute(Exception):
    """ Easy to understand naming conventions work best! """
    pass

# removed the function example for clarity

class ThisClass(object):
    # Since the DoesNotCompute exception exists, let's just assign it
    # as an attribute of ThisClass
    DoesNotCompute = DoesNotCompute

    def this_method(self, x):
        """ This method only works on numbers."""
        try:
            return x ** x
        except TypeError:
            raise DoesNotCompute

Now to demonstrate in the shell (Python REPL for the semantic purists):

>>> from t import ThisClass
>>> this_class = ThisClass()
>>> this_class.this_method(3.3)
51.415729444066585
>>> this_class.this_method('Jack Diederich warned against custom exceptions')
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "logic.py", line 24, in this_method
    raise DoesNotCompute
DoesNotCompute
>>> try:
...     this_class.this_method('I need to write a follow-up on my OAuth post')
... except ThisClass.DoesNotCompute:
...     print('Waiting to see how the OAuth stuff pans out')
...
...
Waiting to see how the OAuth stuff pans out

Admonition: Don't go crazy

Rather than use this trick all over the place, considering using it in a few places to powerful effect. For example, Django uses it only in a few places, and publicly only on MyModelClass.DoesNotExist and MyModelClass.MultipleObjectsReturned. By limiting Django's use of this technique, Django libraries are that much easier to comprehend. In this case, less complexity means more.

I say this because this pattern lends itself to creating custom exceptions to the point of effectively replacing Python's stock exceptions with your own. This makes for harder-to-maintain and harder-to-learn projects.

Not that I've ever done that. Ahem.

Django Update View without slug in the url

2012-07-28T09:30:00-07:00

Today I wanted to use the Django Class Based View (CBV) UpdateView but without a slug identifier in the URL. For example, instead of /profiles/pydanny/ I would go to /my-crazy-profile/. Also, I needed to force authentication.

I've done this with Django functional views a few times times, but today I did it in Django. This is what I did:

1. Added django-braces to my project

Kenneth Love and Chris Jones' awesome django-braces package has some very handy mixins for working with Django CBVs. Kenneth and Chris really understand CBVs, specifically on how to extend them, and have provided a bunch of really useful utility methods in the django-braces library. Yeah, I could figure this stuff out on my own, but since those guys already did the hard work I might as well just lean on them.

pip install django-braces==0.1.3

# settings.py
INSTALLED_APPS = (
...
'braces',
...
)

2. Wrote the view

Assuming a very simple profile Model and Form (which they weren't - but that's not what this post is about), this is how I implemented the view:

# profiles/views.py
from django.views.generic import UpdateView

from braces.views import LoginRequiredMixin  # handles authentication

from profiles.forms import ProfileForm
from profiles.models import Profile

class ProfileUpdateView(LoginRequiredMixin, UpdateView):

    form_class = ProfileForm
    success_url = "/my-crazy-profile/"  # You should be using reverse here

    def get_object(self):
        return Profile.objects.get(user=self.request.user)

3. Wrote the URLconf

The URL pretty much wrote itself:

from django.conf.urls.defaults import patterns, url

from profiles import views

urlpatterns = patterns("",
    url(regex=r'^my-crazy-profile/$',
        view=views.ProfileUpdateView.as_view(),
        name='profile_update'),
)

Closing Thoughts

For a while, I've used django-braces for anything that involves CBVs. I can't imagine working on projects using CBVs without them. In fact, some of the mixins such as LoginRequiredMixin are things that I could argue ought to be in core Django.

This pattern really nails the sweet spot of Django CBVs. Thanks to the use of mixins and model forms, I get an amazing amount of stuff done in a 5 line CBV.

July 15th, 2012 LA Open Source Recap

2012-07-16T18:00:00-07:00

On July 15th, 2012, at the July LA Hackathons Open Source event, over 60 Python, Django, Ruby, Ruby on Rails, PHP, JavaScript, Node.js, Perl, and Clojure developers arrived to work on a variety of projects. We went from 10 AM to 10 PM, and there was much coding and learning done by all. There was a very powerful vibe shared by nearly everyone there, and it was great to see so many people from different personal and open source backgrounds working together and having a great time as a group.

Filtering out distractions

We made some changes to our proceedings for this event, entirely because last time with 'distractions' that caused some issues. In a nutshell, here is 50% of what we did:

A clear statement in the event description and follow up emails that this event was for developers and designers only.
Laptops were an absolute requirement for entry into the event.

We also rolled in a couple other tricks I'm not going to write down. If you see me in person, ask away. The results were near perfect, and we won't hesitate to use these techniques again.

Code/Projects worked on

Next time I'm going to be a lot more careful about getting down the details of what people did. Maybe a signup sheet or an online form? Heck, sounds like a great quick project for next time!

Django (see below for details)
Dynasaur (Node.js/DynamoDB ORM)
Salt Stack
meteordraw
pyit
Ruby on Rails
DBIx::Config
Clojure Tutorials
engulf (Clojure load testing tool that got 2 more tests)
HTML5
PHP
Lots more!

Django Sprints

My conservative estimate is that we had 12 people working on Django or Django related projects. That's at least 20% of attendees. Unfortunately, because I'm not a Django core developer, I wasn't reviewing commits; so I am not entirely sure what was being worked on. I do know that I wasn't the only person to commit to core Django.

As for me, I continued my efforts started at DjangoCon Europe to refactor the Class Based Views documentation. I got in two pull requests and finally managed to badly teach myself how to use git rebase.

note: I want to make it exceedingly clear that I'm not the only one working the CBV doc refactor. I'm just one of the fold.

Socializing

During meals we tried to keep people's hands off the keyboard. Specifically everyone got a chance to state their name, toolset they were working with, and something interesting/embarrassing about themselves. Three of the more memorable ones:

An 8 year old attendee wrote his first 4 PHP programs!
sym_kat is invited to speak at the next London Perl Workshop.
Someone employed at SpaceX took Zed Shaw's LPTHW class taught by the man himself at Cartwheel Web HQ in 2011 got himself moved from the C# to the Python team. Go open source!

Closing thoughts

There were a few glitches, but nothing major. Most of those glitches stem from us coming off two months of helping organize multiple events while traveling literally around the world. I think things would have been better if I had delegated more to the right people.

For the most part, there was this electrifying energy that had people focused and working on what they wanted all day. Yet it wasn't all heads-down-coding, their was some good coder-to-coder discussion and fun. The result is at the end of the day people seemed refreshed and empowered, wanting another day as soon as possible. We hope to be able to catch this feeling at future events.

Which means we're plotting the next event now. If you are interested in providing a venue or sponsoring the event, let me know at pydanny@cartwheelweb.com.

Update: Corrected the details about the SpaceX guy.

Simple HTTP Basic Auth Wall

2012-07-09T12:00:00-07:00

I have a client who wanted their entire unlaunched public content site quickly but temporarily blocked for a short period of time. He wanted a universal password so he could send the site to reviewers, done quickly, and nothing else. In a few days the site will launch, and even if someone got through the authentication, nothing bad will happen except for an early visitor. So we determined this was a job for a very simple Basic access authentication implementation.

I asked around and Jacob Kaplan-Moss gave me this awesome snippet using barrel that I pasted right into the bottom of the Django 1.4-style application's wsgi.py file.

# Add to the bottom of your wsgi.py file
# Don't forget to add barrel to your requirements!
from barrel import cooper

REALM = "PRIVATE"
USERS = [('spam', 'eggs')]

application = cooper.basicauth(users=USERS, realm=REALM)(get_wsgi_application())

This took all of 5 minutes to implement and launch. The result is that the first time you visit the site the login prompt appears. If you enter 'spam' and 'eggs' then you can see the site fine.

It worked and the customer was happy.

Will this block a concerted penetration attempt? Of course not. If the site has/had critical or identifying information it would be implemented with HTTPS. Implementing a Django site with HTTPS is something I've done many times now, but this use case was 'do it fast, easy, and make it temporary'.

Moral of the story: Pay attention to your requirements.

Note : As this is just adding in some WSGI middleware, this should work without much modification in Flask, Pyramid, and other WSGI compliant web frameworks.

PyCon Philippines 2012 Day 2

2012-07-05T12:00:00-07:00

The second day of PyCon Philippines 2012 had a really good turnout. My unofficial estimate is that we had about 90% of the attendees from the first day attend (with the unfortunate exception of most of the faculty and students of Agoo Computer College).

The day started with...

Lightning Talks

Unfortunately I don't normally take notes on lightning talks. I'm kicking myself now. Really hard. I'll try and relate some of the talks:

Import this and that, a parody talk by Daniel Greenfeld (me). Malcolm comment on it.
Getting into space using weather balloons by Frank Pohlmann
Ponystrap by Audrey Roy (video from DjangoCon Europe)
Call for Tagalog and other Philippine languages translation of Django by Malcolm Tredinnick
The obligatory PyCon lighting talk on Vim by Bryan Bibat
OpenERP by Frank Pohlmann and Ann Tan-Pohlmann
A parody talk by Sony Valdez

Then it was on to...

Sprints and Tutorials

Encouraged at #pyconph seeing so many people declaring themselves beginners and then diving in. Enthusiasm was a bit contagious.

—Malcolm Tredinnick on twitter

Primary funding for the sprints came from the Django Software Foundation and Python Sprints. That funding went towards food, drinks, and various logistics for handling the estimated 200+ attendee turnout. Things sprinted on:

Django

Django had it's own dedicated and very full room (no more chairs could fit so I sat on the floor), with Malcolm Tredinnick leading the effort, with Marconi Moreto helping out. It turned into mostly an effort to get a lot of people kick-started in Django. What was very heartening was watching people race through the tutorials on their own, and then work on projects using the framework.

I suspect next year the sprints will see a lot of submissions to Django core.

I had hoped to continue my work on Class Based View documentation, but it was wisely pointed out that I should focus more on helping others get moving in Django and other projects.

Ansible

Rodney Quillo led the sprint on the Ansible tool of which he is so fond. About seven people fond of engineering joined him. They seemed very productive and happy by day's end.

pymongo

A number of developers working on embedded devices gathered to work out an edge case issue with pymongo. I wish I knew more about their efforts, but I can tell you that this sort of very serious engineering effort is very common amongst professional Filipino developers. Because of geography, telecommunications and embedded devices are much more prevalent in the island nation.

Other efforts

Again I'm kicking myself. There were a ton of other development going on and I didn't document any of it. Certainly I was busy here and there helping out, but this is the sort of thing you need to document at the moment. Next time I assure you I will be very diligent about recording the efforts of so many.

Tutorial Room

About half the people stayed in what I dubbed the tutorial room. This is where speakers took turns going over material we had brought to the conference. Attendees followed along, and the diligence and focus in the room was very impressive. The tutorials/talks included:

Intro to Python using Turtle by Sony Valdez
Introduction to git and github by Bryan Bibat
21 Cool Things You Can Do With Python by Daniel Greenfeld (me)

Food and drinks

I mean to mention that the breakfast, lunch, and snacks at PyCon Philippines 2012 were pretty good. The logistic chairs had decided rather than gamble on the usual sort of over-priced and not-so-good professional catering service that most conferences rely on to the dismay of... well... everyone, to go with local restaurants.

For dinner people went out on their own. We were lucky enough to go to Pino in Quezon City both times. The Kare-Kareng Bagnet was really tasty.

Conference End

The conference chair, Frank Pohlmann, gave us a few well spoken words about how wonderful the event was to run and host. How gratifying it was to see the enthusiasm and dedication of the attendees, and how inspired he felt.

Then we cheered him.

After that it was a lot of pictures and goodbyes to many new friends. Which meant a lot of sad feelings about a great event coming to an end. I felt inspired and changed. I wasn't the same person going in as I was coming out. I'm more grateful for what I have, and deeply honored that I had this chance to bring PyCon to so many wonderful people.

More thoughts on PyCon Philippines in a forthcoming blog post.

Coming soon

The Story of PyCon Philippines

PyCon Philippines 2012 Day 1

2012-07-03T12:00:00-07:00

PyCon Philippines 2012 (PyCon PH) happened just this past weekend at the University of the Philippines Diliman (UP Diliman) campus in Quezon City, which is part of Metro Manila.

I can assure you that PyCon PH was an wonderful, amazing, humbling experience. I'm hoping that this post and others will get across to you why.

Arrival

Frank Pohlmann, chairman of PyCon Philippines picked us up and drove us to the event. He had worked day and night getting the event ready, all the while running his company, working a job, and being a father to his lovely 21 month old daughter.

As we arrived at the venue, Audrey and I were met by our good friend, Filipino-American Bryan Veloso of Github, who was to provide the closing keynote. As soon as Bryan heard that there was going to be a PyCon in the Philippines he gave up his trip to Europython to be a speaker at PyCon PH.

Inside the Institute of Electrical and Electronics Engineering, we were greeted Ann Tan-Pohlman, logistics co-chair and volunteer organizer. She had over a dozen volunteers getting ready for the oncoming flood of registrants.

More on the herculean efforts of the organizers and volunteers of this event in a forthcoming blog post.

Inspiration

We noticed a large group gather by a side corridor and went to say hello. There we met Sony Valdez, President of Agoo Computer College, and forty of his faculty and students. They were buzzing with energy and excitement, belying the fact that they had left at midnight the night before in order to arrive on time (they traveled many hours by bus). Audrey, Bryan, and I were floored by their effort to educate themselves, and were determined in our speeches not to disappoint.

Unfortunately, the majority of the Agoo Computer Science had to go in the middle of the afternoon. We didn't know they had to go until it was time to leave. Next year I plan to do whatever is possible to help them come for the duration of the conference.

PyCon Begins

After the national anthem, the Dean of Computer Science, Adrian Roy Valdez, started the event with a few words. Frank Pohlmann, PyCon PH chair thanked everyone involved including the sponsors, speakers, and especially the attendees. I gave a keynote speech I'll blog about shortly, and then it was time for the talks.

Intro to Python [1]

by Paolo Barazon

Paulo gave the Intro to Python talk. He was burdened by some equipment issues, but kept going undeterred. It reminded me of the first time I spoke at PyCon 2010, when for the first 10 minutes of my talk they couldn't get the projector to work. In any case, Paolo did a great job under very trying conditions.

Python Tricks you can't live without [2]

By Audrey Roy

Audrey summarized a lot of the really important things you need to know in Python but aren't part of an introductory talk. Things like pip and virtualenv, how to construct packages, basic use of PyPI, and hammered home the need for good documentation. I may be partial, but I think this was the best version of this kind of talk ever given. :-)

Django Quickstart [3]

By Marconi Moreto

Marconi built off what Audrey provided, and added to it by giving a good introduction to Django. He also contributed a lot during the sprints the next day by pitching in with beginners. His talk featured code for a simple TODO app, and you can see the working example at http://quickstart.marconijr.com/.

Game Programming with Python [4]

Are you afraid of math? Too bad! In this tutorial you will learn how to math! Scary, isn't it?

—Sony Valdez

Sony Valdez, who I mentioned earlier, gave a talk on pygame and math while wearing the Barong Talalog. Not just any talk on pygame, but the best one I've ever seen. He's a natural speaker and educator, and extremely funny. And while giving this amazing talk on no sleep (remember - he was on a bus with 40 of his students and faculty), he broke many classic presenter rules:

1. Live code demo

He wrote a functioning game while giving this talk. He made his mistakes part of the talk.

2. Used notepad

That's right. He didn't use Vim, Emacs, Sublime Text, or PyCharm. He used notepad.

3. Someone else's laptop

He used an unfamiliar computer for his talk. Ever try and type on a keyboard you aren't familiar with? This is what he was dealing with while after a 5+ hour bus journey with probably no sleep the night before.

I'm in awe.

Functional Programming in Python [5]

by Malcolm Tredinnick

Sony was a hard act to follow, but Malcolm gave an amazing rendition of his classic talk on functional programming. I've seen Malcolm give this talk before, but this instance was clearly the best.

Ansible [6]

by Rodney Quillo

What's a python conference without one talk about system engineering? Fortunately for us, Rodney gave a talk on Ansible, a pretty new Python based deployment tool. What makes Ansible different is it's range of options, and simplicity of use.

I didn't know it's Python: Python Advocacy [7]

"PyCon: More fun in the Philippines"

—Variant of the Philippines Department of Tourism slogan

Paulo Barazon returned to talk about the big users of Python, how to promote it's use, and why it's so awesome for... well.. lots of things.

Maps of Imaginary Lands [8]

Malcom came back to give us a talk on how to assemble the components necessary to do a mashup of an imaginary place he cooked up.

Closing Keynote: Design your open source project [9]

by Bryan Veloso

Bryan closed out the day with an inspirational and funny keynote speech instructing beginners on how to make your project used and loved by developers. His slides were incredible, as you should see for yourself.

Coming soon

Day 2 of PyCon Philippines
The Story of PyCon Philippines
More...

[1]	http://pydanny-event-notes.readthedocs.org/en/latest/PyconPH2012/basic_python.html

[2]	http://pydanny-event-notes.readthedocs.org/en/latest/PyconPH2012/python_tricks.html

[3]	http://pydanny-event-notes.readthedocs.org/en/latest/PyconPH2012/django_quickstart.html

[4]	http://pydanny-event-notes.readthedocs.org/en/latest/PyconPH2012/game_programming.html

[5]	http://pydanny-event-notes.readthedocs.org/en/latest/PyconPH2012/functional_programming.html

[6]	http://pydanny-event-notes.readthedocs.org/en/latest/PyconPH2012/ansible.html

[7]	http://pydanny-event-notes.readthedocs.org/en/latest/PyconPH2012/python_advocacy.html

[8]	http://pydanny-event-notes.readthedocs.org/en/latest/PyconPH2012/closing-keynote.html

[9]	http://pydanny-event-notes.readthedocs.org/en/latest/PyconPH2012/closing-keynote.html

Announcing PyCon Philippines!

2012-06-09T12:00:00-07:00

PyCon Philippines 2012, set to occur on June 30 and July 1, is the first Python programming conference held in the Philippines. PyCon is a volunteer run effort that brings together Python developers from a variety of backgrounds and skill levels into a friendly, cooperative environment in order to educate, inspire, and work together - building relationships that transcend the event and can turn into lifelong friendships or even very impressive business alliances.

This event is possible due to the hard work and contributions of members of the Manila Python users group, the support of the Python Software Foundation, various members of the Python community, our gracious sponsors, and many others.

I'll be there, along with Audrey Roy and Django core developer Malcolm Tredinnick. Hope to see you there!

What is Python?

Python is an open source programming language used for science, engineering, robotics, entertainment, web development, and more. It is used by organizations such as Google, NASA, Instagram, Pinterest, Mozilla, Walt Disney Animation Studios, WETA Digital, and many more.

Interested in attending?

Then head to the registration page. The early bird discount ends on June 15th, so buy your tickets now!

Interested in sponsoring?

Please go to the sponsorship page.

Your sponsorship would go a long way towards making PyCon Philippines 2012 a memorable event. As this is an entirely volunteer-run event, 100% of funds go toward actual event costs: food, drinks, chairs, tables, signage, nametags, flyers, advertising, etc. Per custom with PyCons around the world, the organizers receive no payment for their efforts, and any funds left over will be either placed in a Philippine based non-profit corporation or donated back to the Python Software Foundation.

Django Class Based View: email form with CAPTCHA

2012-05-23T09:30:00-07:00

Yesterday I showed how to implement a simple email form for Django using Class Based Views. Today I'm going to extend yesterday's work to use the excellent RECAPTCHA service to help reduce spam content.

This version requires pip installing the following into your virtualenv.

pip install django-crispy-forms so we can do Python driven layouts.
pip install django-floppyforms so we get HTML5 elements for free.
pip install django-recaptcha to do the RECAPTCHA work.

Don't forget to add the app to your INSTALLED_APPS in settings.py:

INSTALLED_APPS += (
    'crispy_forms',
    'floppyforms',
    'captcha',
)

Generate your KEYs from the RECAPTCHA site and add them in settings.py:

RECAPTCHA_PUBLIC_KEY = '6LcVu9ESAAAAANVWwbM5-PLuLES94GQ2bIYmSNTG'
RECAPTCHA_PRIVATE_KEY = '6LcVu9ESAAAAAGxz7aEIACWRa3CVnXN3mFd-cajP'

In myapp.forms.py:

from captcha.fields import ReCaptchaField  # Only import different from yesterday
from crispy_forms.helper import FormHelper
from crispy_forms.layout import Submit
import floppyforms as forms

class ContactForm(forms.Form):

    name = forms.CharField(required=True)
    email = forms.EmailField(required=True)
    subject = forms.CharField(required=True)
    message = forms.CharField(widget=forms.Textarea)
    captcha = ReCaptchaField()  # Only field different from yesterday

    def __init__(self, *args, **kwargs):
        self.helper = FormHelper()
        self.helper.add_input(Submit('submit', 'Submit'))
        super(ContactForm, self).__init__(*args, **kwargs)

In myapp.views.py:

# Unchanged from yesterday. :-)
from django.conf import settings
from django.core.mail import send_mail
from django.views.generic import FormView

from myapp.forms import ContactForm

class ContactFormView(FormView):

    form_class = ContactForm
    template_name = "myapp/email_form.html"
    success_url = '/email-sent/'

    def form_valid(self, form):
        message = "{name} / {email} said: ".format(
            name=form.cleaned_data.get('name'),
            email=form.cleaned_data.get('email'))
        message += "\n\n{0}".format(form.cleaned_data.get('message'))
        send_mail(
            subject=form.cleaned_data.get('subject').strip(),
            message=message,
            from_email='contact-form@myapp.com',
            recipient_list=[settings.LIST_OF_EMAIL_RECIPIENTS],
        )
        return super(ContactFormView, self).form_valid(form)

In templates/myapp/email_form.html:

{# Also unchanged from yesterday. :-)  #}
{% extends 'base.html' %}
{% load crispy_forms_tags %}

{% block title %}Send an email{% endblock %}

{% block content %}
    <div class="row">
        <div class="span6">
            <h1>Send an email</h1>
            {% crispy form form.helper %}
        </div>
    </div>
{% endblock %}

{% block extrajs %}
<script src="{{ STATIC_URL }}js/jquery-1.7.1.min.js"></script>
<script type="text/javascript">
$(function() {
    $('#id_name').focus()
});
</script>
{% endblock %}

What I did

Using pip I installed three packages into my Python environment.
Added those three packages into the INSTALLED_APPS setting.
Set the RECAPTCHA keys for my site.
Modified the forms.py file from yesterday to include the RECAPTCHA field.
Reduced spam content.

What I could do

Pin the app versions for a particular release. This is what you should be doing in normal development and in production, but for a blog entry I'm avoiding it because release numbers become quickly dated.
Rather than change the ContactForm from yesterday, I could have extended it via inheritance.

Want to learn more?

If you live in the Los Angeles area and want to learn more about Django, everything from the basics to setting up a Content Management System or E-Commerce system, check out our Django (and Python) training at Cartwheel Academy.

Simple Django email form using CBV

2012-05-22T09:30:00-07:00

Here's a simple FormView Class Based Views for Django. Here is a sample of how to do one as a simple email form. There is no CAPTCHA in this example, that's the topic of a future blog post.

This version requires the following packages pip installed into your virtualenv.

django-crispy-forms so we can do Python driven layouts.
django-floppyforms so we get HTML5 elements for free.

They also need to be added to your list of INSTALLED_APPS:

INSTALLED_APPS += (
    'crispy_forms',
    'floppyforms',
)

In myapp.forms.py:

from crispy_forms.helper import FormHelper
from crispy_forms.layout import Submit
import floppyforms as forms

class ContactForm(forms.Form):

    name = forms.CharField(required=True)
    email = forms.EmailField(required=True)
    subject = forms.CharField(required=True)
    message = forms.CharField(widget=forms.Textarea)

    def __init__(self, *args, **kwargs):
        self.helper = FormHelper()
        self.helper.add_input(Submit('submit', 'Submit'))
        super(ContactForm, self).__init__(*args, **kwargs)

In myapp.views.py:

from django.conf import settings
from django.core.mail import send_mail
from django.views.generic import FormView

from myapp.forms import ContactForm

class ContactFormView(FormView):

    form_class = ContactForm
    template_name = "myapp/email_form.html"
    success_url = '/email-sent/'

    def form_valid(self, form):
        message = "{name} / {email} said: ".format(
            name=form.cleaned_data.get('name'),
            email=form.cleaned_data.get('email'))
        message += "\n\n{0}".format(form.cleaned_data.get('message'))
        send_mail(
            subject=form.cleaned_data.get('subject').strip(),
            message=message,
            from_email='contact-form@myapp.com',
            recipient_list=[settings.LIST_OF_EMAIL_RECIPIENTS],
        )
        return super(ContactFormView, self).form_valid(form)

In templates/myapp/email_form.html:

{% extends 'base.html' %}
{% load crispy_forms_tags %}

{% block title %}Send an email{% endblock %}

{% block content %}
    <div class="row">
        <div class="span6">
            <h1>Send an email</h1>
            {% crispy form form.helper %}
        </div>
    </div>
{% endblock %}

{% block extrajs %}
<script src="{{ STATIC_URL }}js/jquery-1.7.1.min.js"></script>
<script type="text/javascript">
$(function() {
    $('#id_name').focus()
});
</script>
{% endblock %}

Tomorrow's blog post

In tomorrow's post I'll show how to add CAPTCHA into your project to help reduce spam messages.

Want to learn more?

May 12th, 2012 LA Open Source Recap

2012-05-21T09:30:00-07:00

On May 12th, 2012, over 50 Python, C++, Ruby, PHP, JavaScript, and Node.js developers arrived to code on a variety of projects. It was awesome! Tons of open source projects saw contributions, and people across languages and frameworks worked together.

Event Background

Less then two weeks before May 12, a bunch of us Los Angeles area Python developers were hanging out and wishing we had a local sprint to attend that was just about developers working on open source projects. It was then that Audrey Roy and I, along with an army of hardworking volunteers, decided to stop wishing and make it happen on May 12th.

We lined up a venue, contacted awesome sponsors Spire.io, Heroku, Github, Cars.com, and Cartwheel Academy. As we did that, we also invited people from the many Los Angeles programming communities in Los Angeles to join us. The result of everyone's hard work? We filled up all sixty spots in less than 96 hours!

Some of the projects worked on included:

Salt Stack: https://github.com/saltstack/salt
A node.js-powered streaming terminal, allowing for shared input at a terminal among several participants.
A JavaScript powered astrolabe.
Settlers of Catan analytics in JavaScript.
OpenFrameworks, a cross-platform toolkit for creative coding in C++.
My own time at the sprint was spent with Audrey Roy and Randall Degges on engineering cleanup and fixing bugs on OpenComparison.

More open source sprinting on July 15

There's going to be another Los Angeles open source event on July 15 at Originate. Instead of less then two weeks to plan, we have nearly two months - so it's going to be better!

RSVP here: http://www.meetup.com/LA-Hackathons/events/64542582/

If you want to sponsor or volunteer, email me at pydanny (at) cartwheelweb.com or audreyr (at) cartwheelweb.com. We go out of our way to ensure that sponsors and volunteers feel appreciated.

10 reasons to go to DjangoCon Europe

2012-05-14T19:30:00-07:00

You should go to DjangoCon Europe in lovely Zurich, Switzerland. Here are 10 reasons why:

1. Chocolate

So much of what we like about chocolate comes from Switzerland. For example, Milk Chocolate was invented in Switzerland.

2. Keynote speaker: Jacob Kaplan-Moss

Always a great speaker and fun to be around, he's one of the BDFL's of Django.

3. Cheese

I grew up thinking that Swiss Cheese was just about holes. It's so much more. I can't wait to try fresh European cheese made by master craftsmen from the freshest ingredients.

4. Keynote speaker: Jessica McKellar

In a word, Jessica is incredible. She's a Twisted core developer, PSF board member, part of the trio responsible for the gigantic Boston Python User Group's massive size explosion, and a talented speaker. She's used her incredible talents and skills to increase diversity in the community and generally help other people.

5. Breakfast

Muesli was invented in Switzerland. I love Muesli. I was floored by how much better it was in New Zealand. I can't wait to try it in it's homeland.

6. Web Site

The DjangoCon Europe site is crazy. I mean, look at all those animations!

7. Talks

This is a single track event with proven speakers like Zachary Voase and Andrew Godwin, yet balances that with bringing in new blood to spice things up. And dare I say I'm giving a technical talk with Audrey Roy? ;-)

8. Mountains

With all the incredible food, you would think you would gain umpteen kilograms. Fortunately there are mountains all around to climb and hike.

9. Sprints

Want to sprint on Django itself? Look no further because there will be Django core developers around! There will also be notable Python developers like Kenneth Reitz and others around working hard on a lot of different projects. It's going to intense and fun!

10. Castles

Living in the USA, we just don't have anything like castles. DjangoCon Europe will be near a small horde of stone fortifications. Which means if the Zombie Apocalypse happens during the conference, we'll have many secure places to go. They also make lovely tourist destinations. :-)

What are you waiting for?

DjangoCon Europe has a cap on attendance. Tickets for Python events have been selling out, not just for PyCon US. Don't miss out!

It's all about me

Yup.

Call me selfish but I want you there because I haven't haven't met all our European friends yet in person. Hope to see you next month in Zurich!

CSS Hacking to make my code samples legible

2012-05-11T08:30:00-07:00

I've been very happy with Pelican as a blog engine so far, and haven't even moved off the sample theme. There's just been one problem: Myself and others have had a lot of trouble reading the code snippets.

I didn't have time to cook up a full Pelican theme, so instead I just hacked the local CSS files. The problem with this hack is that every time I regenerate the blog I have to copy the right CSS files into place. So next week when I have time I'll do a proper Pelican theme.

In the meantime, enjoy!

from random import shuffle

class Meal(object):
    def __init__(self):
        self.food_type = ['Beef', 'Fish', 'Vegetarian', 'Chicken']
        shuffle(self.food_type)

Choosing an API framework for Django

2012-05-10T08:00:00-07:00

First off, out of the box, Django lets you construct API responses with a little work. All you need to do is something like this:

# Copied from https://docs.djangoproject.com/en/1.4/topics/class-based-views/#more-than-just-html
from django import http
from django.utils import simplejson as json

class JSONResponseMixin(object):
    def render_to_response(self, context):
        "Returns a JSON response containing 'context' as payload"
        return self.get_json_response(self.convert_context_to_json(context))

    def get_json_response(self, content, **httpresponse_kwargs):
        "Construct an `HttpResponse` object."
        return http.HttpResponse(content,
                                 content_type='application/json',
                                 **httpresponse_kwargs)

    def convert_context_to_json(self, context):
        "Convert the context dictionary into a JSON object"
        # Note: This is *EXTREMELY* naive; in reality, you'll need
        # to do much more complex handling to ensure that arbitrary
        # objects -- such as Django model instances or querysets
        # -- can be serialized as JSON.
        return json.dumps(context)

Once you get that mixin, use it in your views like so:

# modified from djangoproject.com sample code
from django.utils import simplejson as json

class JSONDetailView(JSONResponseMixin, MyCustomUserView):
    def convert_context_to_json(self, context):

        context['objects'] = User.objects.values('first_name','last_name','is_active')
        return json.dumps(context)

This works pretty well in a number of simple cases, but doing things like pagination, posting of data, metadata, API discovery, and other important things ends up being a bit more work. This is where the resource oriented API frameworks come in.

What makes a decent API Framework?

These features:

pagination
posting of data with validation
Publishing of metadata along with querysets
API discovery
proper HTTP response handling
caching
serialization
throttling
permissions
authentication

Proper API frameworks also need:

Really good test coverage of their code
Decent performance
Documentation
An active community to advance and support the framework

If you take these factors, at this time there are only two API frameworks worth using, django-tastypie and django-rest-framework.

Which one is better? django-tastypie or django-rest-framework?

I say they are equal.

You simply can't go wrong with either one. The authors and communities behind both of them are active, the code is solid and tested. And here are my specific thoughts about both of them:

django-tastypie

Using django-tastypie is like playing with pure Python while using the Django ORM. I find it very comfortable. Seems really fast too. The documentation is incredible, and I rarely have any problems figuring anything out. It also supports OAuth 1.0a out of the box, which is mighty awesome these days.

In fact, I wrote a custom OAuth2 handler for django-tastypie for consumer.io that I'm working to extract for publication.

django-rest-framework

As it's based off Django 1.3 style Class Based Views (CBVs), it has a very familiar pattern. Actually, because of the quality of the documentation, I really prefer using django-rest-framework CBVs more than using Django's actual CBVs.

Maybe I should make an HTML renderer for django-rest-framework? :-)

But what about django-piston?

Don't use django-piston.

I don't want to say anything negative, but let's face it: django-piston is dead. Besides a critical security release last year, nothing has been done for it in about 3 years. The documentation is weak, the code mostly untested, and the original author left. He has gone on to do some amazing things. Django-piston was amazing in its time, but its time has passed and so should you.

The only reason for using django-piston for years has been that it supported OAuth, but django-tastypie now addresses that use case. I've used django-tastypie's basic OAuth class and rolled custom Authentication modules to support some extra OAuth flavors and found it wonderful.

Use django-tastypie or django-rest-framework instead. You'll be much, much happier for it.

Discuss this on Hacker News

Django Requirements for a project

2012-05-09T08:00:00-07:00

Today I'm starting a new project. I'm working as fast as I can and hope to launch on Friday. What are my package dependencies?

Django==1.4

Unlike my last quick project which was Flask, this effort really falls into Django's sweet spot. I need sessions, forms, templates, and models to do things in an ideal Django pattern.

psycopg2==2.4.5

I need transactions and hard-type validation in the database, which means PostgreSQL. If I didn't need transactions or the hard-type validation I would consider MongoDB instead.

django-extensions==0.8

Because amongst other things this library gives you, I never want to write my own TimeStampedModel ever again. :-)

South==0.7.5

Django gives you the freedom to migrate data in the way you want. The way I want to do it is via South.

django-registration==0.8.0

Normally django-social-auth is my go-to tool for registration, but in this case I need simple username/password registration. This is a very solid tool, but you do have to make your own templates or find someone's fork that has a copy of templates that match.

django-floppyforms==0.4.7

An excellent tool for making your forms HTML5-ish out of the box.

django-crispy-forms==1.1.3

The child of my own django-uni-forms, this will let me create forms using div-based controls super fast, and do layout customizations if I need them.

django-heroku-postgresify==0.2

This tool makes getting the PostGreSQL settings out of Heroku trivial.

django-heroku-memcacheify==0.1

This tool makes getting the memcache settings for Heroku trivial.

gunicorn==0.14.2

All the cool kids who play in devops swear by Gunicorn. I use it because Heroku seems to recommend it for Django deployments.

Installing the above packages

Never copy/paste these libraries directly into your projects. If you do that, you'll end up hating yourself later as your local instances become unmaintained forks of the real project. Also, unless you are really careful in your copy/pasting, you'll be in violation of various open source licenses. Odds are the FOSS police aren't going to find you, but I can assure you that when you bring in one of the authors of these packages to help you fix a problem he/she is going to be mighty annoyed at the lack of attribution.

Do it the right way: do proper Python dependency management.

Create a requirements.txt file and install them as proper dependencies. The file should contain the following text:

Django==1.4
South==0.7.5
django-crispy-forms==1.1.3
django-debug-toolbar==0.9.4
django-extensions==0.8
django-floppyforms==0.4.7
django-heroku-memcacheify==0.1
django-heroku-postgresify==0.2
django-registration==0.8.0
gunicorn==0.14.2
psycopg2==2.4.5

Once you have that, you install them thus in your virtualenv:

pip install -r requirements.txt

Now that I have all this, it's time to code!

Los Angeles Open Source Sprint on May 12th!

2012-05-01T09:20:00-07:00

This is a day long coding event in Los Angeles for Open Source developers of all languages and skill levels to come and code like fiends. They'll be joined by dozens of either really smart coders or nice people like me. Sponsors are providing food, drinks, venue, and more!

RSVP at http://www.meetup.com/LA-Hackathons/events/62796642/ before it fills up! It's free.

I'll be there to:

Organize the event with the assistance of the awesome Los Angeles technical community!
Code like a fiend. I want to work on django-mongonaut and could use some GraphViz and JavaScript help.

And now to open the floor to questions...

Where and when?

Where:

Spire.io
7257 Beverly Blvd #210
Los Angeles, CA 90036

When:

May 12, 2012
10 AM to 10 PM

Is this like a Hackathon?

Yup. See http://en.wikipedia.org/wiki/Hackathon#Sprints

Will there be Wifi?

Yes!

I'm just starting as a developer, should I come?

It depends.

What if I don't have a project of my own to bring? Should I come?

Heck yeah! There will be a number of projects around that you can join and contribute to in order to make the world a better place. There isn't a list up yet, but I'm hoping by Saturday there will be one.

What if I want to come and recruit people?

Absolutely not.

This is not a job fair and we don't want unnecessary distractions.

On the other hand, if you want to help sponsor we'll happily mention you on the meetup.com description.

Are there going to be any presentations or lightning talks?

No.

This is a sprint, not a conference or demonstration. We'll try and limit announcements and interruptions as much as possible, the only exception being for letting you know food has arrived.

What should I bring?

Your own functioning laptop with power cord. Neither event organizers, the venue, or sponsors are providing equipment. We also encourage you to bring a power strip labeled with your name.

I'm sold! How much does it cost and where do I register?

The event costs you nothing and you RSVP at http://www.meetup.com/LA-Hackathons/events/62796642/.

Join us at DjangoCon Europe!

2012-04-10T12:00:00-07:00

This year in June, the European edition of DjangoCon is going to be held in lovely Zurich, Switzerland. It is my great fortune and delight to announce that Audrey and I will be attending. Hooray!

Needless to say, we're extremely excited. My trip to Europe in 2007 was an amazingly fun and educational experience, and I can't wait to share a similar experience with Audrey. We can't wait to see all our old European friends, make new ones, and be dazzled by what Europe has to offer.

Also, let's face it, the conference is going to be incredible. They've lined up some awesome venues, the food (cheese, chocolate, muesli, etc) in Switzerland is world-renowned, the website looks fantastic, and they've got Jacob Kaplan-Moss to keynote. It looks so good I'm surprised there are any sponsor slots left!

Talks

While attending the conference, we submitted and hope to give a presentation on the positives (speed, flexibility, community, etc) and negatives (a few) of using MongoDB as a Django datastore. I'm also considering giving a second talk. Some ideas for that include:

Zen of Django
Django isn't for beginners (A Tredennick inspired talk explaining why teaching beginning developers Django isn't a good idea)
Advanced Django Template Usage
Advanced Django Models Usage (An intimidating talk to put together)
Django Packages Thunderdome II

Any thoughts?

While I'm at it, I'm going to poke any prospective conference goers to submit their own talks. I love attending conference talks and at PyCon US this year I barely got to see any. Show me what you got!

Calendar About Nothing

2012-03-28T11:30:00-07:00

On October 16th, 2011, which was one hundred and sixty-four days before I published this post, I resolved to get myself onto the Longest Streak list of Calendar About Nothing. Today, with this blog post, I've managed to do just that - get on the Longest Streaks.

Calendar About Nothing tracks your open project commits on Github. It tallies your total number of days, records your longest streak, and lets you compare yourself against other committers. The idea comes from an article where Jerry Seinfeld says his secret to productivity is to get something done every day, because if you skip one day then skipping another day is easier.

I feel a great sense of accomplishment.

The majority of those days I would like to think I did real commits. I pledged to myself to do more than just white space tweaks or write a script to do the work for me. I wanted to accomplish things and make a difference. I started projects, wrote code, cleaned up old code, added tests, documented a lot, and moved my blog here. It's been an awesome time, and keeping my fingers constantly in projects has been eye opening.

I feel a great sense of relief.

I've really enjoyed this but there were days it was hard to get to the internet or days I was busy on other things. I discovered early on my phone was not a good interface with Github's text editor. Some days I had no desire to code or lacked the creativity to write. Then I remembered that my ability to work openly on Github is a gift, and I should just do it. Now I can relax a bit and if I miss a day, it's not such a big deal. I've made it to the Longest Streak list!

Going for a longer streak

We'll see what happens now. I just feel happy that I've captured the moment and made my mark on the board. I invite you to do the same.

Launching our API at PyCon 2012

2012-03-09T07:30:00-08:00

A few months ago me and my fiancee, Audrey Roy, launched our start up, Consumer Notebook. It's a Python powered product comparison site that combines the best features of Open Comparison, Yelp, Consumer Reports, and Pinterest. We've worked day and night to make it better, with countless members of the Python community using the site and giving us invaluable feedback.

All of that brings us to PyCon. We're not just here as attendees and participants, but also to promote our startup. It used to be the cool thing to launch your startup at SXSW, but times have changed. Now it's the cool thing to launch at PyCon!

Like Twilio, Twitter, Facebook, and Google, we've got an API we want developers to use. And as an upcoming startup, we've got to really be creative in how we gain your attention, so here is what we're doing for PyCon 2012:

1. Demos at our PyCon Startup Row booth.

Thanks PyCon!

We've got a booth on Saturday. We've got banners and bright red track jackets. We're giving out handy API reference cards, as well as 10 different flavors of Oreo cookies (see our site for a complete list of every Oreo cookie flavor, by the way).

So stop on by, sign up for the BETA, have a cookie, and talk to us about our API. For that matter, feel free to talk to me or Audrey about the API any time you see us. We're wearing bright red jackets with 'Ask me about our API' on our back.

Speaking of the API, the next item on our plan is...

2. A big kickoff API contest.

We've put together the Consumer Notebook API, which lets you get product list and comparison data out of our site via a simple REST interface.

We're kicking off our PyCon 2012 API launch with an API contest.

You have from now until April 22 to find new and exciting ways to use our API. If you win we'll give you a Kindle Fire or equivalent gift certificate valid in your nation of origin.

API Contest Details

3. We're hosting the Saturday night PyCon Capoeira Open Space

Want to see some awesome Capoeira moves? Want to learn some of the basics? Want to just have fun?

We've hired professional Hollywood stuntman and Capoeira instructor to the stars, Contra Mestre Xingu, to come up from Los Angeles to lead this event. In addition to appearing in movies, videos, and commercials, he's got first hand experience with snakes. In his own words:

Capoeira saved my life. What? How? Well it was a sunny afternoon, I was shooting a film in the Hollywood hills off Coldwater Canyon at a house. I was running back to set from the bathroom and out of nowhere a rattlesnake jumped up from the ground and tried to bite me. I quickly jumped out of the way and did a fast negative rolé away from the snake. Then I called the other people on set and we caught it. I really didn't think much of it afterwards until after the shoot. My friend told me the rattlesnake was a young one and I'm lucky I didn't get bitten. When a younger rattler bites you, it usually unloads all of it's venom in you because it cannot control the amount it releases.

—Contra Mestre Xingu

Over 30 people have signed up up for this event, including Python core developers.

Details:

When: Saturday, 8pm - 10pm
What to bring: Camera!
What to bring if you are participating: Workout clothes, water bottle

Location: The PyCon upstairs central open space

See you around!

The sorry state of Python OAuth providers

2012-03-05T07:30:00-08:00

This is one of those challenging posts to write. The people whose projects I'm going to describe have put in a lot of dedicated, hard work to overcome a challenging subject. Writing an OAuth consumer is a hard problem and writing an OAuth provider is an even harder problem. The efforts put in by the authors of these projects has been nothing short of incredible. The problem, however, is that the existing projects are not usable as-is, and need the support of the community in order to improve.

The terrible thing is that this is a solved problem within our community. Python based projects are successfully implementing OAuth providers, and often using internally hacked versions of the efforts I'm about to describe. However, they aren't giving this back to the community. It might be that they want to protect their competitive edge, but I'm going to be nice and say that it's because their too busy to find time to send pull requests back.

In any case, let me present our use case. For Consumer Notebook we want an API. We want to be able to track usernames, passwords, and the application using our API - which is the OAuth use case. Much as BasicAuth or DigestAuth is the easier way to go in terms of implementation, OAuth was designed for our use case: allowing third-party developers to build apps using our API without having to store credentials. In fact, it's a critical security issue: Twitter dealt with malicious "Twitter apps" stealing usernames and passwords before they switched to OAuth. As an API provider, being an OAuth provider might be more challenging, but it's the responsible thing to do.

Existing OAuth Providers

Time to get into the meat of the issue. Let's look at the current implementations of OAuth providing within the Python community. Again, I wish I didn't have to be negative, but I'm up against the wall:

OAuth2app (Django)

https://github.com/hiidef/OAuth2app

OAuth version: 2.0

Strange URL construction that might be a security hole.
Bitwise operators in the logic making it harder to debug. Security is hard. Don't complicate your security code because your mistakes will cost.
Uncommented code. Security is hard. Comment your code.
Documentation outdated and insufficient.
Doesn't work without serious hacking and adding of undocumented parameters. Which means I have to worry if I'm breaking anything.
We managed to get it working with GET requests. Then we realized that we were using GET requests, which seems like a bad idea.

django-piston (Django)

https://bitbucket.org/jespern/django-piston

OAuth version: 1.0

Stalled project.
Documentation insufficient.

django-oauth-plus (Django)

http://code.larlet.fr/django-oauth-plus

OAuth version: 1.0a

Tutorial doesn't work.
Documentation insufficient.
Doesn't work without serious hacking. Which means I have to worry if I'm breaking anything.
We could not get it to work.

lastuser (Flask)

https://github.com/hasgeek/lastuser

OAuth version: 2.0

No documentation
No tests to serve as documentation
Lack of documentation means I'm not sure if it is actually a OAuth provider.

python-oauth2 (Python)

https://github.com/dgouldin/python-OAuth2 (best example)

OAuth version: 1.0

Called 'OAuth2' but only works with OAuth 1? Really? WTF? This needs to fixed.
Documentation insufficient.
Provides only a skeleton of a provider. Not a turnkey solution.
Doesn't work as a provider without serious hacking. Which means I have to worry if I'm breaking anything.
Many, many forks of the project, with various blog posts advising people to use various particular forks rather than the main one.

How about a solution?

Alright, I've ranted and laid out out a bunch of bullets identifying a problem. Time to try and fix the problem.

For starters, a production-usable OAuth provider should meet certain standards:

Near turnkey solution
Working code (duplicates above bullet but I'm making a point)
Working tutorials
Documentation
Commented code
Linted code
Test coverage > 80%

This is my specification. If your project for any Python framework matches it, I'll list it on a forthcoming website that also covers Python based OAuth consumers.

For what it's worth, Idan Gazit has been working on something to help address the problem, specifically https://github.com/idan/oauthlib. It also is intended to cover the Python OAuth consumption issue I didn't cover in this article. It and related efforts need a lot of work, so...

The PyCon US 2012 sprints start on March 12. I think as a community, we Pythonistas should band together and make things right. I think we'll have the brainpower and enough eyes on the problem to make serious headway on the issue, either by fixing existing solutions or creating new ones. Right now I've got interest from people to join in and help, including Idan Gazit, Audrey Roy, George Hickman, and others.

We're willing to put in the time to make OAuth in Python better, how about you?

Join us at the PyCon US sprints either in person or on-line. Details of the sprint are near the bottom of this PyCon Sprint page.

Updates

03/05/2012 - Removed Velruse from the list of providers as it's lead, Michael Merickel, clarified that it is not a provider. 03/06/2012 - Added a link to the PyCon OAuth sprints.

Discuss this post on Hacker News

You should Heroku

2012-02-28T12:45:00-08:00

In mid-November me and my fiancee, Audrey Roy began our startup. We had been frustrated with trying to do on-line product research and came up with an idea to take the lessons learned from Django Packages / Open Comparison and apply them to a commercial effort. The result has been Consumer Notebook, and it's been a steadily growing success.

We've been bootstrapping the project. That means supporting it with consulting and grinding away on it during our free time. That means 12-16 hour days of Python, Django, and Javascript coding, marketing, system administration, graphic design, communicating with users and vendors, and a thousand other tasks. Since we've had to explore new techniques for making things work on the backend and front end, that means we've needed to have a robust system that is trivial to deploy and certain to never go down. Which, of course, requires serious sys admin skills.

The Big Problem

I hate system administration work.

Sys admin is boring. I find it tedious and dull. Devops doesn't make it easier/faster, it just makes it possible to do it at a large scale.

Fortunately for me, my fiancee likes the sys admin side of things. However, she's got serious programming skills in Python/Javascript, understands CSS, is an excellent illustrator, and has good business skills to boot. Which means I needed Audrey not to be doing sys admin.

Solution: Platform as a Service

Platform as a Service, or PaaS, is where someone else does the majority of work involved in system administration. There are now dozens of companies edging into the Python capable PaaS space. We've been leery of using any of them but finally settled on Heroku after a long period of evaluation.

Why Heroku?

We choose Heroku for a number of reasons:

We competed in a Los Angeles area Hacking contest with Randall Degges. He was responsible for the sys admin and went with Heroku. He got it up and it was out of the way for the competition. He spent his time coding, adding features, and fixing templates instead of tweaking knobs on something in the cloud. We saw other people not deliver products at the contest because of this issue.
Heroku doesn't lock you in. If I wanted to, I could take all the pieces out in about 10 minutes, then go old school and host it myself on my own closet server.
Heroku has very good PostgreSQL support. Our web framework is Django, which has an ORM that works best with PostgreSQL.
Heroku has staff. At least seventy of them. Odds are they would have people around 24/7 to deal with issues.
The add-on system means they've got many other people adding great new features. Want MongoDB? No problem! How about something to handle video? You got it!
Heroku scales up trivially. If we get an upswell of users, I just type heroku ps:scale web=50 and I've got 50 web server things handling the load.
When I think of Heroku I think of Puffer Fish. Which is awesome because Puffer Fish are awesome.

Creative Commons: Some rights reserved by Saspotato

Things that we really liked about using Heroku

As we progressed down the journey of building our site, we discovered even more nice features about Heroku. Here are some of the things that really make me smile:

Releases and especially rollbacks means we deploy with a lot more confidence.
Logging and other diagnostic add-ons like Sentry and New Relic means we know what's going on.
During one huge data migration effort I scaled up the workers so a 6 hour task became a 5 minute task. Cost was less then 10 cents for workers instead of me losing hours of labor.
In case we go viral, we don't have to worry about load balancers and all that high performance stuff.

What does that mean?

It means I'm doing the deployments. I'm the sys admin. And I'm happy with my role because it takes minutes out of my day. Me and Audrey team up on everything else and the results so far have been great. If you've ever worked with me, the fact that Consumer Notebook is administered and deployed by me is going to be a shock.

We've been able to really focus on development of the project. And when I mean development, I mean a lot of things. I mean:

Python
Django
HTML
CSS
JavaScript
Data Modeling
Documenting the API
Marketing: blogging on Consumer Notebook, Tweeting, and working with other groups
Trying out public tickets
Iterating through the user experience by communicating to users
All the boring legal and business stuff

What you don't see is anything about sys admin issues. That's because what could have been a huge sink in time and resources is pretty much gone. We deploy staging servers with a bit of code I copy/pasted from a bash history into a Fabric script:

from fabric.api import local

commands = """
heroku create --stack cedar
heroku addons:add memcache
heroku config:add S3_KEY=HAHAHAHAHAHA S3_SECRET=NOTGIVINGITOUT
heroku addons:add redistogo
heroku addons:add sendgrid:starter
heroku addons:add mongolab:starter
heroku addons:add sentry:test
heroku addons:add pgbackups
heroku addons:add custom_domains:basic
heroku addons:add zerigo_dns:basic
heroku domains:add staging.consumernotebook.com
heroku addons:add ssl:piggyback
git push heroku master
heroku scale web=1
heroku addons:add heroku-PostgreSQL:ronin
heroku pg:wait
"""

def build_staging():
    for command in commands.strip().split('\n'):
        local(command)

How awesome is that?

How much does Heroku really cost?

You can do Heroku for free. A lot of people do. More power to them.

But let's face it, beyond a certain point, every PaaS, including Heroku, is going to be more expensive then getting your own EC2, Rackspace, Dreamhost, or Linode hosted server. For a fraction of the cost, you can provision a server, install all the bits, configure the database, http server, load balancers, and even write Chef/Puppet/Fabric scripts so you can do it repeatedly at scale. Cheap!

So why pay more for Heroku? Why not just do it ourselves? For example, right now we're on dedicated PostgreSQL hosting which Heroku charges us $200/month. That's a lot, right?

Wrong.

Right now we're seeing a 50% increase in visits every day. So if we ran our own servers, Chef/Puppet/Fabric or not, odds are we would be spending at least 10 hours a month doing server work. And I can assure you that when we consult that we make more than $20/hour.

$200 < 10 hours of us doing consulting work to bootstrap the project.

Until you hit a certain point, these days the real cost of servers is labor. If you're a developer or small effort, and you think going with a cheap hosting provider is the way to go, think again. Think about the hours you're losing monkeying around with servers and databases instead of getting code done.

Heroku saves us money.

The Takeaway

One of the problems Django and other Python web frameworks has had is the difficulty of deployment. I can't tell you how many projects I didn't do because of the thought of handling the sys admin side of things. Let's face it, one of the great ongoing successes for PHP is that deploying the majority of sites is trivial.

With the rise of devops we've seen a lot of developers across languages and frameworks dive into Chef and Puppet. It's been sadly amusing watching people muck around with these great tools to make the deployment of 1-2 servers 'easier', when the real benefit of those tools has been to do things at scale. Things like deployments of fifty servers at once or deployment abstractions for hundreds of people (my fancy talk for PaaS).

In any case, things have changed. Deploying Python web apps is as trivial as deploying PHP code.

For developers I see great times ahead.

Discuss this post on Hacker News

My PyCon 2012 Schedule

2012-02-22T09:00:00-08:00

Here I was thinking that this year's PyCon wasn't going to be so busy because I didn't submit a talk or tutorial. Ha! What the heck was I thinking?

Here's what I've already got in the works.

Wednesday, March 7th

Me and Audrey are driving up from Los Angeles. I've wanted to do this drive for a while, so this is very exciting. We'll arrive in the evening and hopefully tag up with friends old and new.

Thursday, March 8th

I'm moderating the Code Reuse panel of the Python Web Summit. Have you submitted a question yet?

In the evening we'll be helping assemble bags for the conference. That's always a blast.

Friday, March 9th

PyCon really begins! I'll be at the keynotes, and then the talks begin. These are some of the talks I'm really leaning towards watching:

The problem, of course, is that all the talks look awesome. Missing some of these talks is going to hurt.

In the evening we're going to the New Relic/Loggly/Skull Candy party and hang out at the (in)famous TIP BOF.

Saturday, March 10th

I'll be at the Consumer Notebook Startup Row booth all day! Come and hang out with us and all the other bright, new companies.

In the evening I'm hosting the PyCon Capoeira Open Space. We've got a seriously large room assigned for the event, and over twenty five people currently signed up to participate (there's still room for signups). Even if you don't want to play a musical instrument or learn/play at Capoeira, come anyway to watch. Some of the players there have some seriously amazing skills - I can assure you've seen them in the movies.

Sunday, March 11th

More talks!

Monday March 12 - Thursday, March 15

Sprints! Nothing set in stone yet, but some ideas I've had include:

Launch python.opencomparison.org for all of Python.
OpenComparison integration of PyPI OpenID, SourceForge OAUTH, and BitBucket OAUTH.
Improved OpenComparison integration of the Launchpad API.
Implement Gitorious API for OpenComparison.
Implement Text Based search for OpenComparison.
Hack on Mongo related technologies such as django-mongonaut and some other planned projects.

Selected for PyCon StartUp Row

2012-02-21T01:00:00-08:00

Me and Audrey have been working hard on Consumer Notebook, a Python/Django based project. We submitted it to PyCon StartUp row and found out this morning we've been accepted. Hooray!

From the PyCon StartUp Row winner's post, here is a little bit of what we're about:

Have you tried researching products online recently? Search engines are gamed by scammy marketers. Product review sites overwhelm you with ads, have unreliable reviews, or dryly compare raw product specs. Consumer Notebook is working to solve this problem. It is like Yelp for products, with product comparison grids inspired by the founders' open-source work on Django Packages and Open Comparison.

Just so you know, PyCon in North America is the biggest gathering for the international Python community. The conference this year is in Santa Clara, California, and is sold-out at 1500+ attendees, but could have easily sold out several times that number. The talks, tutorials, and events are going to be incredible, and I'm delighted to be part of the PyCon experience!

We look forward to sharing the StartUp Row with friends like Transifex, Python Packages, and Wedding Lovely.

Please check out Consumer Notebook right now and also visit our PyCon booth on Saturday, March 10th and we'll show you what we're doing.

django-uni-form end of life

2012-02-18T01:00:00-08:00

I started on django-uni-form in January 2009. In order to use Pinax on an internal social network for NASA HQ, we had to render all content, including forms, Section 508 compliant. Rather than rewrite the html for all 50+ forms that existed in the Pinax 0.5.x framework at that time, I decided to minimize my work and automate things. James Tauber gave guidance and insight, my co-workers were supportive, and Jannis Leidel suggested the Uni-form library. The name Django Uni-Form was obvious, and lo the project was named.

Looking at the old, extremely deprecated Google Code site for django-uni-form, I see that the first commit happened on January 7th, 2009. That was for version 0.1, with some core code that was literally a merger between the Django form example on how to integrate divs into forms and the simplest template tag I could figure out.

The python code in uni_form/templatetags/uni_form.py:

from django.template import Context, Template
from django.template.loader import get_template
from django import template


register = template.Library()

@register.filter
def as_uni_form(form):
    template = get_template('templates/uni_form.html')
    c = Context({'form':form})

    return template.render(c)

The template tag code was nearly exactly copy/pasted from the starter Django docs on forms:

{% for field in form %}
    <div class="ctrlHolder {% if field.errors %}error{% endif %}">
        {% for error in field.errors %}
            <p class="errorField">
                <strong>{{ error }}</strong>
            </p>
        {% endfor %}
        {{ field.label_tag }}
        {{ field }}
        {% if field.help_text %}
        <p class="formHint">
            {{ field.help_text }}
        </p>
        {% endif %}

    </div>
{% endfor %}

Using it was trivial, you just wrote out:

{% load uni_form %}

<form>
    {{ form|as_uni_form }}
</form>

Days of django-uni-form

Leading up to PyCon 2009 James Tauber suggested some things that lead to the FormHelpers, and we hammered out the API on IRC. We knew it crossed the rigid lines between Model-View-Template, but sometimes it's advantageous to break a few rules and abstractions in order to get better things down the road

At PyCon 2009 with the mentoring of Jannis, James, and Brian Rosner, I moved the project from google code to Github. Jannis released it on PyPI and I followed the pattern he showed me for two years. Yeah, I learned tons under those guys.

After PyCon 2009 a pull request with the Layout helper was provided. It took some work to make it pass all the tests and use cases, but the end result was definition of form layout in the Python. This broke the rigid battle lines of Model-View-Template and left purists screaming in agony, but it certainly made working with forms in Django trivially easy.

Lots of people started to use the project across projects like Pinax and organizations like NASA, PBS, Discovery Channel, various newspapers and many others. Lots of pull requests came in and the features grew.

In 2010, Alice Rowland submitted the first Sphinx docs, and it was her work that really helped get me started on doing lots of Sphinx work.

And, all the way into 2011, pull requests for Django Form Sets started to come rolling in, and almost none were of acceptable quality. They never came with documentation, tests, and almost always broke existing tests really hard. Since I'm not a huge fan of Django FormSets, I didn't want to put in a ton of effort making them work. I believe one of them was pretty good, but life was crazy busy at the time and I let it slide. Apologies to whoever it was gave me a working FormSet pull request with documentation, tests, and working code.

Long periods were going by without new versions. I admit I often slow about accepting pull requests. Life was busy and reviewing the incoming code took a lot of time. Browser cross-checking, running tests, and more was really time consuming. I tried to get others to become co-leads on the project, but invariably they didn't have time to do it. Note: If someone asks you to co-lead something, respond in 24 hours.

Enter Miguel Araujo

After PyCon 2011, when there was some unpleasant stress in my life, I woke up cranky one morning and mouthed off on twitter to this guy who asked me to accept a pull request on django-uni-form. This guy tweeted back to me saying I ought to be nicer since I had a library people liked.

He was right.

I apologized to the guy (Miguel Araujo) and remembered my manners. Over the next couple of months we chatted via Twitter and Github's messaging system. He was smart, trustworthy, and passionate about everything he did. I knew I had found my co-lead. He responded promptly and I gave him commit rights.

Working together (with him doing the vast majority of the work), we moved the project into new releases. The architecture and design changed, driven by discussions we had together. The code was cleaned up, gnarly bits in there to support old versions of Python and Django kicked out, and the documentation revised. The project had new life!

The only blip I saw with Miguel is my own fault of sometimes being too nice as a project leader when it comes to accepting pull requests. I believe pull requests should be really atomic - for one thing and one thing only with support tests and documentation. Otherwise it becomes nigh impossible to incorporate them and these days I reject multi-purpose pull requests. One pull request in particular took a huge amount of debate and discussion to work in. I think after that Miguel is much better at being upfront at the beginning about rejecting pull requests with giant scopes.

During all this I asked Miguel to take over the project, he accepted, and I even blogged my announcement his role as project leader. Miguel is indeed very nice and after that fact he asked me to remain on board as co-lead.

We finally met in September of 2011 and co-presented on Advanced Django Form Usage at DjangoCon 2011. The deepest technical material we presented was authored by Miguel. During our research he uncovered at least one bug in Django and got an ancient bug closed. It was a great experience and I hope he'll co-present with me in the future.

django-uni-form is dead, long live django-crispy-forms

The upside of django-uni-form is that it grew in features organically thanks to my own needs and general community effort. The downside of django-uni-form is that it grew in features organically thanks to my own needs and general community effort. In any long running project there is cruft and weird patterns that start to hurt after a while. django-uni-form was no different.

So I'm making this absolutely official as of now. django-uni-form is at it's end of life. It's done and kaput. No more pull requests will be accepted and the issue tracker will be turned off shortly. Just so no one is mistaken:

django-uni-form is deprecated. Use django-crispy-forms

Miguel asked if he could start the project anew, under a different name. We both had been uncomfortable with the name django-uni-form for some time, especially since it had almost nothing to do with Uni-form anymore. In fact, I often using template overrides to avoid the Uni-form HTML layout - the most common alternative being Twitter Bootstrap.

We tossed around names for the project, but all of them were stupid, especially mine. We are both huge fans and users of django-floppyforms (HTML5 form widget app), so my fiancee, Audrey Roy, suggested django-crispy-forms. And lo, the project was named.

Right now django-crispy-forms has an improved API, better performance, and supports both Twitter bootstrap forms 2.0 and UniForm. Adding new form layouts will be easier, and feature controls will be better.

What you should be using now is django-crispy-forms. Don't worry about changing over as there are migration instructions on the excellent documentation.

Try it. You'll like it. :-)

pydanny

Core Concepts of Django ModelForms

ModelForms render Model fields as HTML.

ModelForms select validators based off of Model field definitions.

ModelForms don't have to display/change all available fields.

But what about ModelForm.Meta.excludes?

ModelForms save dictionaries to SQL tables

Example #1 Web Page

Example #2 API/JSON

Example #3 CSV Import

Closing Thoughts

The Easy Form Views Pattern Controversy

How the Easy Form Views pattern works

The Controversy

Getting bit by the Easy Form Views method

Conclusion

Tools we used to write Two Scoops of Django

Alpha Days

Beta Experiences

Final Efforts

Thoughts

Whither Two Scoops of Django?

Concerns about Open Sourcing

Book Generation as a Service?

Filepicker.io and South

Off to Europe!

Two Scoops of Django is in print!

Generating NCX files with Python

Core Concepts of Django Forms

Forms render HTML.

Forms are "just" Python constructs.

Forms validate dictionaries.

Epilogue

Overloading Django Form Fields

MyModelForm has a bug!

A Better Way

Other field attributes

Try it with Inheritance!

Summary

Beginner's Guide to PyCon 2013 Part III

Early Saturday Morning

Saturday Morning

10:50 AM talks

11:30 AM talks

12:10 PM talks

Saturday Lunch (12:40 PM)

PyLadies Lunch at PyCon (12:30 PM)

Saturday Afternoon

1:40 PM talks

1:55 PM talks

2:35 PM talks

3:15 PM talks

4:15 talk

4:30 PM talk

5:10 PM talks

Evening Activities

Part IV

PyCon Tutorial: Wiring Up Django Packages

You just finished the Django tutorial. What now?

Tutorial Description

Format

Prerequisites

Registration

We are not using PayPal

Open Source Events Get Burned By PayPal

Fear, Shame, and PayPal

The 'Needs' of PayPal's Anti-Fraud Division

Legal recourse against PayPal is folly

It is Wrong to Use PayPal

PayPal is at Risk

What PayPal Can Do for Conferences

What PayPal Can Do for Small Business

Note

Two Scoops of Django 1.5 Beta Released

What is different about the beta release?

What does buying the beta get me?

Why aren't there Kindle or ePub versions yet?

If I bought the alpha do I have to buy the beta?

Is this book good for beginners?

Is this book good for experienced developers?

`MyModelForm` has a bug!