Using IPython in your development workflow can be a massive time saver, and great for TDD, investigating APIs or legacy code or just for saving time performing some manual task.

I’m currently working on a legacy Django codebase that’s deployed to a few different unique servers.

I’m needing to do a lot manual inspection, and sometimes manual alterations, to deployments via the Django management shell.

With IPython installed I’ve found this a really conveniant way to quickly inspect, manipulate and experiment with code that can be a little rough around the edges.

IPython is also really tweakable, and I’ve made some tweaks that I find really useful.

Setup

To start with make sure to install IPython in the venv in use.

pip install -U ipython

Django will then make use of it when running;

DJANGO_SETTINGS_MODULE="config.foo" python manage.py shell

The DJANGO_SETTINGS_MODULE is likely unnecessary for most folks but I need it for the moment.

Magics

IPython has a concept of magic commands which allow it to add a range of functionality to the default python shell.

There are many default magics, and extensions can add to them too.

One that’s quite handy is;

%history -opf some_file.doctest

Which will output the history of one’s session in the doctest format which is useful for sharing.

There’s also %pdb which will make IPython drop automatically to pdb if something raises an exception.

One can also run shell code in line which is great for demos and documentation;

%%bash
ls -la | tee foo

There’s many more too. I’m still learning useful ones all the time. See the docs for a complete list.

There’s also the %save magic which I haven’t had a chance to try yet, and can’t find in the docs, but it would seem it allows the saving of a session including all the variables for reloading later. Could be handy!

Also it’s not really a magic but when debugging troublesome scripts, especially when the scripts aren’t importable due to a missing __name__ == "__main__" and other import side-effects I’ve been using;

sys.argv = [...]  # Set args to something sensible.
execfile("somefile.py")

There’s probably a better magic to do it though!

Creating a config profile

IPython creates minimal config by default. Create a full profile by running the following on the command line;

ipython profile create

This will create a profile somewhere in the home directory, this is platform specific but for me it was created in ~/.ipython/profile_default/

Storing in homeshick

I use homeshick to manage my home environmental config.

In this case I use it to track my config files;

• ~/.ipython/profile_default/ipython_config.py
• ~/.ipython/profile_default/ipython_kernel_config.py

Although I’ve only changed the former from the defaults.

I also track the contents of ~/.ipython/profile_default/startup/ which can contain python files to be executed on startup (some docs here).

Finally, I also track the contents of ~/.ipython/extensions. Strictly speaking the use of this directory is deprecated and I should create proper modules for the extensions I use but right now it’s convenient for me to stick them in my homeshick repo.

Logging everything

One nice thing about IPython is it makes it easy to export what’s been done in a way that’s useful for documentation or converting into a script.

This can be done manually using the %logstart magic and similar, but I wanted it to be done by default so I can quickly see what I was doing a few days ago.

I found this really nice script that when put in the startup directory will automatically create dated logs in the profile directory for everything done in the IPython shell including output. These logs can later be executed like any normal python script, either by manually copying into an IPython shell or by converting it to a proper script.

The script is here. It’s basically the same as the cookbook recipe that it’s based on.

Just stick it in the ~/.ipython/profile_default/startup directory and it’ll start working.

It’d be nice to turn it into a proper extension module, or maybe the same behaviour is possible via config alone, but this is what’s working for me right now.

Here’s an example of what the log files look like. Here’s what might be in ~/.ipython/profile_default/log/log-default-2018-07-05.py;

#!/usr/bin/env python
# log-default-2018-07-05.py
# IPython automatic logging file
# =================================
# 15:37:16
# =================================
from git import Git
g = Git()
help(g.execute)
example_output = list(range(10)); example_output
#[Out]# [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
get_ipython().run_line_magic('load_ext', 'ipy_reprrequests')
# =================================
# 15:07:37
# =================================
example_output = list(range(10)); example_output
#[Out]# [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Notice how the magic %load_ext is converted to executable python. Super cool!

Extensions

Apart from startup scripts, IPython has good support for extensions.

Requests

The first one I used I stole from minkr‘s Github repo. It will automatically prettify and add context to the responses from the requests module when IPython outputs the result in the shell.

It looks like this;

In [1]: %load_ext ipy_reprrequests
In [2]: import requests
In [3]: requests.get("https://jsonplaceholder.typicode.com/posts/1")
Out[3]:
200 https://jsonplaceholder.typicode.com/posts/1
headers:   Access-Control-Allow-Credentials: true
  CF-Cache-Status: HIT
  CF-RAY: 435ab9d61b61bbcc-LHR
  Cache-Control: public, max-age=14400
  Connection: keep-alive
  Content-Encoding: gzip
  Content-Type: application/json; charset=utf-8
  Date: Thu, 05 Jul 2018 15:04:45 GMT
  Etag: W/"124-yiKdLzqO5gfBrJFrcdJ8Yq0LGnU"
  Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
  Expires: Thu, 05 Jul 2018 19:04:45 GMT
  Pragma: no-cache
  Server: cloudflare
  Set-Cookie: __cfduid=de7c9bc306af9c12658bffab30516cb7d1530803085; expires=Fri, 05-Jul-19 15:04:45 GMT; path=/; domain=.typicode.com; HttpOnly
  Transfer-Encoding: chunked
  Vary: Origin, Accept-Encoding
  Via: 1.1 vegur
  X-Content-Type-Options: nosniff
  X-Powered-By: Express

body (application/json; charset=utf-8):
{'userId': 1,
 'id': 1,
 'title': 'sunt aut facere repellat provident occaecati excepturi optio reprehenderit',
 'body': 'quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto'}

Note on IPython prettification

The way the IPython prettifier works, in order to see the prettified result, it must be returned from the statement, so if one writes;

response = requests.get(...)

there will be nothing printed by IPython. If I want to see what I’ve stuck in a variable I do something like this;

response = requsts.get(...); response

that way IPython will output the contents of response nicely in the shell.

Django models

Inspired by minkr‘s extension I wrote a similar extension to prettify the display of Django Models or QuerySets.

It’s available here.

When the extension is loaded the magic %django_dict_for_model_output can be used to toggle between two different ways of prettifying the output.

The default uses the built-in Django serialiser code. The alternative uses the __dict__ attribute from the model instance for output.

In either case it’s a pretty dumb implementation but I find it quite useful!

It looks like this;

In [1]: from app.models import *
In [2]: SomeModel.objects.all()[0]
Out[2]:
{u'fields': {u'some_id': u'98db4543-70ca-47bd-a5b6-53df176f1245',
  u'created_date': u'2018-03-28T20:40:32.571Z',
  u'last_update': u'2018-06-29T10:12:41.609Z',
  u'status': u'creating',
  u'some_ip': u'8.8.8.8'},
 u'model': u'app.vdc',
 u'pk': 156}

In [3]: %django_dict_for_model_output
Using dict for Django model output.

In [4]: SomeModel.objects.all()[0]
Out[4]:
{'_state': <django.db.models.base.ModelState at 0x7fb9fd156690>,
 'some_id': UUID('98db4543-70ca-47bd-a5b6-53df176f1245'),
 'created_date': datetime.datetime(2018, 3, 28, 20, 40, 32, 571981, tzinfo=<UTC>),
 'id': 156L,
 'last_update': datetime.datetime(2018, 6, 29, 10, 12, 41, 609713, tzinfo=<UTC>),
 'some_ip': u'8.8.8.8',
 'status': u'creating'}

Using extensions

Drop the python files in the ~/.ipython/extensions directory and then load them with the %load_ext magic. For example;

%load_ext ipy_django

Note as I said above storing extensions in the ~/.ipython/extensions path is deprecated but it’s how I’m doing it for now.

Config

I also made a couple of config changes in ipython_config.py file.

Firstly I load the extensions I mentioned above as default so I don’t have to do it manually every time I start;

## A list of dotted module names of IPython extensions to load.
c.InteractiveShellApp.extensions = [
    'ipy_reprrequests',
    'ipy_django',
]

I also switched on the vi keybindings;

## Shortcut style to use at the prompt. 'vi' or 'emacs'.
c.TerminalInteractiveShell.editing_mode = 'vi'

Although I have to confess, despite using vim as my primary editor for years, I’m still struggling to get used to using the vi keybindings in zsh so I’ll see how I get on!

I also added the extra shortcuts that I’m used to to switch to my default external editor when on a line. This can be really useful if one is experimenting defining a new function on the command line, or a long loop block;

## Enable vi (v) or Emacs (C-X C-E) shortcuts to open an external editor. This is
#  in addition to the F2 binding, which is always enabled.
c.TerminalInteractiveShell.extra_open_editor_shortcuts = True

Future

I might add some more stuff in the future like;

• Turning the extensions into proper Python modules.
• Looking what’s already out there in the wild and see if anything else might be useful to me.
• Converting the startup script to a module.
• Make ipy_django check for an existing custom __str__ or __repr__ on models and use that if the users toggles an option.

As a result my repo might change, although the links in this article are to a tagged version so they shouldn’t go dead.

Bonus: pdb++

In a similar vein to IPython, pdb++ (pdb++) module replaces the stock pdb with a much fancier tool.

Once installed with;

pip install -U pdbpp

It’ll automatically get used in the place of the vanilla pdb and adds auto-completion, pretty printing and many other useful features!

Bonus: sh

The sh module is great. It allows you to call any shell command like it was a native python command.

For example;

In [1]: import sh
In [2]: sh.df("/", human_readable=True)
Out[2]:
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       126G   48G   78G  38% /

Great for experimenting or semi-automating some laborious task!

Install it with;

pip install -U sh

Caution!

sh is a great module but I’d be cautious not to overuse it in production code.

It’s very conveniant, but it’s a little bit magic and so I’d advise for production code it’s probably safer to use the standard tools and be more explicit, especially if you want to leave the door open to asyncio in the future.

Even more so now the very conveniant subprocess.run exists.