Mixing Python and C++

2007-02-28 • , • Comments

A while ago, I expressed a desire to work on a mixed language system.

I can’t get over how neat Python is. Sometime soon I’d like to work on a mixed language system: Python for the tricky bits, C++ for the speedy bits.

I haven’t changed my mind. As usual, it’s been an evolution rather than a revolution — step by step, I’ve got closer to realising this ambition, and finally, now, I think I can say I’ve achieved it.

Why Python and C++?

No particular reason. C++ is the systems programming language I know best. When you need to get close to the silicon, it does the job very well. And Python is the high level language I know best. When you want to write concise, flexible software, it does the job very well. Really, though, I’d be just as happy to write the tricky code in any decent high-level language, switching in low-level languages as dictated by performance and platform requirements.

Mixed language development

Perhaps the most obvious way to mix languages is in a single program: you start off by writing the program in Python. If it doesn’t go quickly enough, you profile it and write a C++ library to look after the slow bits. There are disadvantages to this, though:

  • you have to figure out how to interface between the two languages
  • the library and the program can become interdependent and harder to test in isolation
  • your IDE and build system may not like it

As a consequence, this turns out to be a technique I’ve used relatively little. Here are some other techniques I’ve used more frequently.

  • Code Generation. Python is a great language for generating code — either in Python or C++, or indeed in any other language.

  • Multiple Processes. Rather than sharing a single process, the C++ and the Python programs can run as separate executables. C++ does the heavy lifting; Python manages the complex application logic. A nice side-effect of this approach is that the operating system can use separate processors for the separate processes without all the problems associated with multi-threaded code.

  • Prototyping. Sometimes a compiled C++ program is required. I’ve found that sketching out a prototype in Python then simply rewriting it in C++ can work well. The prototype can be used as both a reference implementation and a source of test data.

  • Testing. Python is a great tool for creating test data and for running tests.

  • Refactoring. C++ lacks the refactoring tool support available for (e.g.) Java, but simple Python scripts can do some refactoring jobs well enough.